ACTIVE THREAT TRACKING AND RESPONSE

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for active threat tracking and response. One of the methods includes determining, using sensor data generated by one or more sensors located at one or more properties, that an active threat is in progress at the one or more properties; accessing a virtual model i) of the one or more properties ii) that includes a position of each of the one or more sensors; determining, using the sensor data and the virtual model and for each of two or more areas of the one or more properties, a threat level of the active threat at the respective area; and performing, using the threat level of the active threat at each of the two or more areas of the one or more properties, one or more monitoring system actions.

TECHNICAL FIELD

This disclosure application relates generally to monitoring systems.

BACKGROUND

Many buildings are equipped with property monitoring systems that include sensors and connected system components. Property monitoring systems can receive and analyze data from sensors that are located at a building or at multiple buildings.

SUMMARY

Systems and methods for active threat tracking and response are disclosed. When an active threat such as an active shooter or a fire exists at a building or campus of buildings, it is desirable to quickly locate, track, and respond to the threat. A campus monitoring system can use the disclosed techniques to detect a threat, classify the threat, monitor threat severity and movement, and communicate up-to-date threat status information to users.

The disclosed techniques can be used to detect, monitor, and track active threats such as active shooting incidents, theft, burglary, hostage situations, bomb threats, fire, physical altercations, stampedes, unauthorized access, espionage, property damage, presence of fugitives, etc. Aggregated sensor data generated by sensors located throughout a campus can be used to detect these and other threats. The sensor data can be mapped to a virtual model of a building or campus where the threat is occurring.

Based on the mapped sensor data, the monitoring system can evaluate a threat level at each of multiple different areas of the campus. The evaluated threat level can be communicated to owners, managers, occupants, and emergency responders in a visual format. The visualized threat level information can assist users in escaping from the threat and responding to the threat. The monitoring service can manipulate automated devices located throughout the campus in order to respond to the threat and mitigate the threat. The monitoring system can perform bulk actions, e.g., actions throughout an entire building or campus, or can perform individual actions that are tailored to the specific type of threat and location of the threat.

In general, one innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of determining, using sensor data generated by one or more sensors located at one or more properties, that an active threat is in progress at the one or more properties; accessing a virtual model i) of the one or more properties ii) that includes a position of each of the one or more sensors; determining, using the sensor data and the virtual model and for each of two or more areas of the one or more properties, a threat level of the active threat at the respective area; and performing, using the threat level of the active threat at each of the two or more areas of the one or more properties, one or more monitoring system actions.

In some implementations, the method includes performing the one or more monitoring system actions including sending, to a device at one of the one or more properties, instructions to cause the device to perform a monitoring system action.

In some implementations, the method includes performing the one or more monitoring system actions including sending, to a device at one of the one or more properties, instructions to cause the device to control access to an access point at one of the one or more properties to contain the active threat.

In some implementations, the method includes sending the instructions including sending, to the device, the instructions to cause the device to open the access point.

In some implementations, the method includes sending the instructions including sending, to the device, the instructions to cause the device to close the access point.

In some implementations, the method includes sending the instructions including sending, to the device, the instructions to cause the device to unlock the access point.

In some implementations, the method includes sending the instructions including sending, to the device, the instructions to cause the device to lock the access point.

In some implementations, the method includes: determining, using the sensor data by one or more sensors located at one or more properties, an estimated track of a threat through a property, including sending the instructions to cause the device to perform the one or more monitoring system actions uses the estimated track of the active threat through the property.

In some implementations, the method includes: determining an approximate location of a bystander user device, including sending the instructions to cause the device to perform the one or more monitoring system actions uses the estimated track of a threat through the property and the approximate location of the bystander user device.

In some implementations, the method of includes: determining the threat level including: determining a first threat level for a first area in the two or more areas; and determining a second different threat level for a second different area in the two or more areas; performing the one or more monitoring system actions including: performing, using the first threat level, a first action for the first area; and performing, using the second different threat level, a second different action for the second different area.

In some implementations, the method includes performing the one or more monitoring system actions including sending, to a device, instructions to cause the device to present the threat level of the active threat at each of the two or more areas of the one or more properties.

In some implementations, the method includes performing the one or more monitoring system actions including sending, to a device, instructions to cause the device to present a visualization of the threat level of the active threat at each of the two or more areas of the one or more properties. In some implementations, the method includes: determining a threat level including determining a threat type; and performing the one or more monitoring system actions including performing actions using the threat type.

The foregoing and other implementations can each optionally include one or more of the following features, alone or in combination.

The subject matter described in this specification can be implemented in various implementations and may result in one or more of the following advantages. Information from an emergency situation can provide responders valuable knowledge for planning and response. A map of relevant information presented to a user can provide contextual value to an emergency situation. The ability to control parts of the environment in response to sensed emergencies can provide a means of safety and flexibility to those operating in the environment.

DETAILED DESCRIPTION

FIG.1is a diagram illustrating an example multi-building property monitoring system100for responding to an active threat. For example, the multi-building property may be a campus150. The campus150includes buildings102,104, and106. The campus150can be, for example, a school campus, an office campus, a residential campus, a multiple dwelling unit, an apartment complex, etc. The campus150includes the multiple buildings102,104,106and outdoor spaces between the buildings. The campus150can be owned or managed by a person or by an organization such as an educational or corporate organization.

The monitoring system100can perform threat detection and tracking at the campus150using sensors throughout the campus150. The sensors can include, for example, microphone107at the campus150, camera105at the building104, and camera118at the building106. Other sensors can include fire sensors, smoke sensors, carbon monoxide sensors, motion sensors, lock sensors access control sensors, contact sensors, and other types of sensors.

Threat tracking can include using the sensors to monitor the threats or anomalies that are detected at the campus150. Based on the threats or anomalies detected and tracked by the sensors at the campus150, the monitoring system100can perform one or more actions. The monitoring system100can perform actions to mitigate the threat, to aid emergency responders in responding to the threat, to assist people in evading the threat, or any combination of these.

The buildings102,104,106of the campus150are each opted in to monitoring and tracking by the monitoring system100. For example, the owners or managers of the buildings102,104,106can register each of the buildings102,104,106with a monitoring service. In some examples, the campus150can include a neighborhood of buildings that each may have different residents and owners. Residents and homeowners of the neighborhood can each choose to opt in to or out of monitoring their buildings by the monitoring service. The monitoring service can detect threats on the campus150, track the threats, and take actions to mitigate the threats.

The monitoring system100can include at least one local network. The network can be any communication infrastructure that supports the electronic exchange of data between a control unit112and other components of the monitoring system. For example, the network may include a local area network (LAN). The network may be any one or combination of wireless or wired networks and may include any one or more of Ethernet, Bluetooth, Bluetooth LE, Z-wave, Zigbee, or Wi-Fi technologies.

In some examples, the campus150includes a network, and sensors throughout the campus150communicate with a control unit or multiple control units over the network. In some examples, each property of the campus includes a network, and sensors of the property communicate with a control unit over the network. For example, the building102can include a first network, building104can include a second network, and building106can include a third network. Sensors of the building102, e.g., the microphone107, can communicate with a control unit112of the building102over the first network. Similarly, the camera105can communicate with the control unit114of the building104over the second network, and the camera118can communicate with control unit116over the third network.

The sensors can transmit the sensor data to the control units112,114,116via the network. Example sensor data can include indoor and outdoor motion sensor data, images and video analysis data from security cameras, and door and window position and lock data. The control units112,114,116can collect and assess the data from the sensors to monitor the conditions of the campus150.

The control units112,114,116can each be, for example, a computer system or other electronic device configured to communicate with the sensors. The control units112,114,116can also perform various management tasks and functions for the monitoring system. In some implementations, a resident, a visitor, or another user can communicate with the control units112,114,116(e.g., input data, view settings, or adjust parameters) through a physical connection, such as a touch screen or keypad, through a voice interface, or over a network connection.

In some examples, the control units112,114,116can analyze some or all of the sensor data. For example, the control units112,114,116can analyze motion sensor data, video images, and microphone data to determine the occupancy of the buildings of the campus150. The control units112,114,116can also analyze sensor data to determine locations of the residents and/or other occupants within the campus150.

The monitoring system100includes one or more sensors located at the campus150that collect sensor data related to the campus150. The monitoring system100has the ability to control various sensors and other devices on the campus150through automation controls. The sensors of the monitoring system collect various sensor data from the campus150. Example sensors can include cameras, motion sensors, microphones, thermometers, smoke detectors, and water meters. The sensors can also include position sensors and lock sensors for doors and windows at the campus150.

An example sensor at the campus150is an outdoor security camera105. The outdoor security camera105may be used to monitor people, vehicle, and animals at the campus150. In some implementations, the security camera105may perform video analysis on the images captured by the security camera105. In some implementations, the security camera105may transmit images to a monitoring server130and the monitoring server130may perform video analysis on the images. The security camera105and/or the monitoring server130may perform video analysis on the images to detect and identify objects and/or perform facial recognition within the field of view of the security camera105. For example, the security camera105may detect and identify animals, vehicles, and people.

The cameras105,118can include any type of camera. The cameras can capture images of the interior and exterior areas of the campus150. The images can be generated from any appropriate type of light. For example, the images can be generated from any combination of visible light, IR light, or UV light. The images can also be generated from RADAR, LIDAR, and/or microwave imaging.

The monitoring system100can include one or more drones, e.g., drone144. The drone144can be stored at a location of the campus150. The drone144can include one or more sensors such as a camera. The drone144can be deployed by the monitoring system in response to detecting a threat on the campus150. The drone144can be, for example, an autonomous drone and/or remote controlled drone. The drone144can be an aerial drone or a terrestrial drone. The drone can be capable of mobility, e.g., by flying, rolling, swimming, etc.

The monitoring server130includes a campus model database120. The database120stores a virtual campus model121of the campus150. In some examples, the campus model database120can store virtual models of multiple campuses.

The campus model121can include a two-dimensional (2D) map, a three-dimensional (3D) map, or both, of the campus150. For example, the campus model121can include a map of each of the buildings102,104,106of the campus150. In some examples, the campus model121can include a floor plan of each floor of the buildings102,104,106. In some examples, the campus model121can include a map of outdoor space of the campus150.

The campus model121can include a map of sensors of each of the buildings of the campus150. For example, the campus model121can include a position of the camera105indicated on the map of the building104. The campus model121can include data indicating a sensor area of each sensor. The sensor area of a sensor can include, for example, a maximum range of the sensor, a field of view of the sensor, an area or volume of the property that is within detection range of the sensor, etc. For example, the campus model121can include data indicating a 2D or 3D field of view of the camera105. The field of view of the camera105can be overlaid on the map of the campus150.

The campus model121can include of map of devices at each of the buildings of the campus150. For example, the campus model121can include a position of doors, windows, locks, alarms, lights, speakers, etc. The campus model121can include, e.g., a location of the alarm103at the building102and a location of the speaker142at the building106. The devices can include automated devices, e.g., devices that can be operated by the monitoring server130using automated controls.

The campus model121can be generated at or after a time when the campus150is registered with the monitoring service. For example, when a manager of the campus150registers the campus150with the monitoring service, the manager or another user can provide information to the monitoring server130indicating a layout of the campus, locations of sensors, types of sensors, etc. In some examples, the manager can provide the information to the monitoring server130through a user interface of a computing system. In some examples, the manager can provide the information to the monitoring service by recording or streaming a video walkthrough of the campus150to the monitoring server130.

In some examples, the monitoring server130can update the campus model121over time. For example, the monitoring server130can update the campus model121based on sensor data collected at the campus150over time. In some examples, the monitoring server130can update the campus model121based on user input. As an example, a user may reposition the camera118at the building106. The monitoring server130can update the campus model121based on user input indicating the updated position of the camera118. In some cases, the monitoring server130can detect movement of the camera118, and can prompt the user to input an updated position of the camera118. In some cases, the monitoring server130can detect movement of the camera118, and can automatically update the position of the camera118in the campus model121based on camera images collected from the camera118.

FIG.1illustrates a flow of data, shown as stages (A) to (C), which can represent steps in an example process. Stages (A) to (C) may occur in the illustrated sequence, or in a sequence that is different from the illustrated sequence. For example, some of the stages may occur concurrently.

In the example scenario illustrated inFIG.1, a person110fires gunshots in the building102and carries a firearm111through the campus150past the building104. The person110breaks glass of the door108of the building106and enters the building106.

In stage (A) ofFIG.1, control units112,114,116send monitoring system data115to a monitoring server130. The data115includes audio data from the microphone107at the building102, indicating detected audio of gunshots. In some examples, gunshots can be detected by a dedicated gunshot detection system. The gunshot detection system can include acoustic and/or infrared sensors that are configured to detect gunfire. In some examples, the data115includes an indication that the gunshot detection system has detected gunfire. In some examples, the data115can include a number of gunshots that have been detected by the gunshot detection system. In some examples, the data115can include a confidence value that the gunshot detection system has detected gunfire.

The data115also includes image data from the camera105at the building104, indicating images of a person110walking past the building104. The data115also includes results of image analysis of the images captured by the camera105. The results of the image analysis indicate detection of a firearm111in the images of the person110. The data115also includes audio data from a glass break sensor indicating breaking glass at the building106. The data115also include data indicating that the door108opens and shuts. The data indicating that the door108opens and shuts can include a door position sensor and/or images captured by the camera118showing the door108opening and shutting.

In some examples, cameras at the campus150can analyze captured images, e.g., using video analytics. For example, the camera105, the camera118, or both, can perform video analysis on the images to classify objects within the images. The cameras may identify and classify the person110within the images. The cameras can also perform object tracking of the person110as the person110travels across the campus150.

In some examples, cameras at the campus150can transmit image data to the monitoring server130, and the monitoring server130can perform video analysis on the image data. For example, the camera105can capture an image of the person110with the firearm111and transmit the image to the monitoring server130. The monitoring server130can perform video analysis in order to classify the object in the image as a person, to determine a direction of motion of the person, to perform facial recognition of the person110, to classify the object carried by the person110as a firearm111, etc.

The data115can include timestamps associated with the data. The timestamps can indicate a time that the data was generated by the sensors or a time that the data was sent to the control unit. For example, the microphone data can be associated with a timestamp of 2:00 pm. The image data including the detected firearm111can be associated with a timestamp of 2:24 pm. The audio data from the glass break sensor can be associated with a timestamp of 2:27 pm. The data indicating that the door108opens and shuts can be associated with a timestamp of 2:28 pm.

The data115can include confidence values associated with the data. For example, the microphone data can be associated with a confidence value of seventy percent, indicating that there is a seventy percent chance that the audio data represents gunshots. The image data can be associated with a confidence value of sixty percent confidence that the object in the image is a gun. The data indicating that the door108opens and shuts can be associated with a confidence value of ninety percent confidence that the door108opened and shut. The confidence values can be determined, for example, by the sensor that generated the data or by the control unit.

The monitoring server130may be, for example, one or more computer systems, server systems, or other computing devices that are located remotely from the campus150and that are configured to process information related to the monitoring system at the campus150. In some implementations, the monitoring server130is a cloud computing platform.

The control units112,114,116send the data115to the monitoring server130. The data115includes data collected from sensors at the campus150. In some examples, a central control unit can send the data115to the monitoring server130. For example, the central control unit may be the control unit112. The control units114,116can transmit data to the control unit112, and the control unit112can transmit the data to the monitoring server130. In some examples, the sensors of the buildings102,104,106can transmit data to the central control unit, e.g., control unit112, and the central control unit can transmit the data to the monitoring server130. In some examples, the sensors of the buildings102,104,106can transmit data to the monitoring server130.

The control unit or control units can send the data115to the monitoring server130over a long-range data link. The long-range data link can include any combination of wired and wireless data networks. For example, the control units112,114,116can exchange information with the monitoring server130through a wide-area-network (WAN), a broadband interne connection, a cellular telephony network, a wireless data network, a cable connection, a digital subscriber line (DSL), a satellite connection, or other electronic means for data transmission. In some implementations, the long-range data link between the control units112,114,116and the monitoring server130is a secure data link (e.g., a virtual private network) such that the data exchanged between the control units112,114,116and the monitoring server130is encoded to protect against interception by an adverse third party.

In stage (B) ofFIG.1, the monitoring server130assesses threats at the campus150based on the data115. The monitoring server130can analyze the data115to determine conditions at the campus150.

The monitoring server130includes a threat assessment engine122and a rules engine124. The threat assessment engine122uses the campus model121and the data115to assess a threat level of different areas of the campus150. The threat assessment engine122outputs a threat assessment to the rules engine124. The rules engine124determines one or more actions140to perform based on the threat assessment123.

The threat assessment engine122determines the threat assessment based on the data115. For example, the threat assessment engine122can receive the data115generated from sensors at the campus150. In response to receiving the data115from the sensors at the campus150, the threat assessment engine122can retrieve the campus model121of the campus150from the campus model database120. The threat assessment engine122can map the data115to the campus model121. For example, the threat assessment engine122can map the audio data of detected gunshots to the building102. In some examples, the threat assessment engine122can map the audio data of detected gunshots to a particular area of the building102based on the location of the microphone107. For example, the threat assessment engine122can map the audio data to a particular room, floor, hallway, stairway, etc. of the building102based on the detection range of the microphone107.

In some examples, the monitoring server130can determine confidence values of the data115instead of receiving confidence values determined by the sensors or by the control unit. For example, the monitoring server130may assign the microphone data with a confidence value of sixty percent, indicating that there is a sixty percent chance that the audio data represents gunshots. In some cases, the confidence value for the data115can be based at least in part on coincidence logic. For example, the monitoring server130may assign audio data indicating gunshot noises alone a confidence value of fifty percent. The monitoring server130may assign audio data representing gunshot noises a higher confidence value based on coincidence with additional data. For example, for audio data representing gunshot noises in coincidence with audio data representing screaming, e.g., captured within a time and distance proximity to each other, the monitoring server130may assign a higher confidence value of eighty percent that the audio data represents gunshots.

Based on the data115, the threat assessment engine122can determine that a threat exists at the campus150and can classify the threat. For example, based on the gunshots detected in building102, the threat assessment engine122can determine that an active shooter threat exists at the campus150. In some examples, the threat assessment engine122can detect and classify threats using programmed rules. In some examples, the threat assessment engine122can detect and classify threats using machine learning algorithms. The machine learning algorithms can be trained using supervised or unsupervised methods. The threat assessment engine122can update machine learning parameters over time based on sensor data and threat events detected at the campus150and other campuses.

In some examples, the threat assessment engine122can evaluate a likelihood or confidence that a threat exists at the campus150. The threat assessment engine122can update the confidence of the threat based on additional sensor data. For example, based on the audio data indicating the detected gunshots, the threat assessment engine122may determine a confidence of sixty percent that an active shooter threat exists at the campus150. Based on the image of the person110with the firearm111, the threat assessment engine may determine an updated confidence of eighty percent that an active shooter threat exists at the campus150.

In some examples, the threat assessment engine122can determine the confidence that a threat exists at the campus150based on confidence levels of the sensors data. For example, audio data indicating detected gunshots can have a confidence level of sixty percent for representing an active shooting threat. An image of a firearm in a captured image can have a confidence level of seventy percent for representing an active shooting threat. The threat assessment engine122can determine a confidence that the threat exists at the campus150based on a combination of confidence levels of different sensor data. The combination can include, for example, a weighted sum or a weighted average of confidence levels.

The threat assessment engine122can detect a threat such as an active shooter threat based on any sensor data. The sensor data can include, for example, motion sensor data indicating people escaping from the building102, audio data indicating yelling or screams, doors opening and shutting multiple times, vibration indicating people running, etc.

In some examples, the threat assessment engine122can classify the type of threat. For example, based on the detected gunshots, the threat assessment engine122can determine that a threat exists and can classify the threat as an active shooter threat or as an isolated shooting incident threat. The threat assessment engine122may determine a confidence value of eighty percent that the threat is an active shooter threat and of sixty percent that the threat is an isolated shooting incident.

In some examples, the threat assessment engine122can determine a current threat level at different areas of the campus150. The threat level can be represented by a percentage, a scale, a descriptor, a color, etc. For example, based on the audio data indicating the detected gunshots at the building102, the threat assessment engine122may determine that a high threat level exists at the building102, and that a lower threat level exists at the building104, based on a proximity of the building104to the building102. The threat assessment engine122may determine that an even lower threat level exists at the building106, based on a proximity of the building106to the building102.

The threat assessment engine122can assign a representation to a threat level of different areas of the campus150. A higher threat level can be represented, e.g., by a high percentage such as ninety percent, by a large scale value such as nine out of ten, by a descriptor such as “high” or “very dangerous,” by a color such as red, etc. A medium threat level can be represented, e.g., by a medium percentage such as fifty percent, by a medium scale value such as five out of ten, by a descriptor such as “medium” or “possibly dangerous,” by a color such as yellow, etc. A low threat level can be represented, e.g., by a low percentage such as twenty percent, by a small scale value such as two out of ten, by a descriptor such as “low” or “not dangerous,” by a color such as green, etc.

The threat assessment engine122can detect and track multiple threats at the campus150. In some examples, the threat assessment engine122can detect and track multiple threats at multiple different campuses. Upon detecting a threat, the threat assessment engine122can assign received data115to the threat. For example, the threat assessment engine122can determine that an active shooter threat likely exists at the building102based on the detected gunshots. At or near the same time, the threat assessment engine122can receive a second set of data from a second building on the campus150can determine based on the second set of data that a burglary threat likely exists at the second building.

When the monitoring server130receives the image data from camera105at the building104, the threat assessment engine122can assign the image data to one of the already detected threats on the campus150, or can determine that the image data of the person110corresponds to a new threat on the campus150.

In the example ofFIG.1, the threat assessment engine122can assign the image data to the active shooter threat on the campus150, e.g., based on a proximity of the building104to the building102. For example, the threat assessment engine122can determine that the building104is closer in proximity to the building102than to the second building where the burglary threat was detected.

The threat assessment engine122can assign the image data to the active shooter threat on the campus150, e.g., based on the timestamp associated with the audio data of the detected gunshots at the building102and the timestamp associated with the image data captured by the camera105at the building104. For example, the threat assessment engine122can determine that the timestamp associated with the image data is closer in time to the timestamp associated with the detected gunshots than with the sensor data indicating the burglary threat.

In some examples, the threat assessment engine122can assign data to the active shooter threat on the campus150, e.g., based on tracking movement of the person110. For example, the threat assessment engine122can determine, based on the image data captured by the camera105, that the person110is walking towards the building106. The threat assessment engine122can determine, based on second set of data representing the burglary threat, that the suspected burglar is walking in a direction away from the building106and/or is located far from the building106. Thus, the threat assessment engine122can assign the audio data indicating glass breaking at the building106to the active shooter threat instead of to the burglary threat.

In some examples, the threat assessment engine122can assign data to a detected threat based on proximity or timestamp rules. For example, sensor data captured within a threshold proximity to a detected threat can be assigned to the detected threat. The threshold proximity can be, e.g., 0.1 miles, one hundred feet, two hundred feet, etc. The threshold proximity can also include the sensor data being captured on the same floor of a building where the threat is detected, in a same wing of a building where the threat is detected, etc. For example, the building104is within 0.1 miles of the building102. The threat assessment engine122can therefore associate the image data captured by the camera105with the detected active shooter threat based on the building104being within a threshold distance of 0.1 miles of the building102where the active shooter threat was detected.

The threat assessment engine122outputs the threat assessment123to the rules engine124. The threat assessment123can include a classification of a detected threat, e.g., a classification of the detected threat as an active shooter threat. The threat assessment123can also include a confidence level of the detected threat. The threat assessment123can also include a threat level of different areas of the campus150.

The rules engine124determines monitoring system actions based on the threat assessment123. The rules engine124can determine monitoring system actions based on pre-programmed settings and rules. Rules and settings can be customizable and may be programmed, e.g., by an owner, resident, an installer, an operator, or another user of the monitoring system. For example, a rule may state that the monitoring server130sends a notification to emergency responders when a confidence of an active shooter event exceeds sixty percent. In some examples, a rule may state that the monitoring server130shuts and locks doors at the campus150when an active shooter is determined to be within a particular area of the campus150with a confidence of greater than seventy percent. In some examples, the monitoring server130may be programmed to request permission from a user before adjusting a device at the campus150.

In some examples, the rules engine124can determine bulk actions based on the threat assessment123. The bulk actions can include a pre-determined set of actions to be taken for a given threat. For example, a first set of bulk actions may apply to an active shooter threat anywhere on the campus150. The first set of bulk actions may include notifying emergency responders, activating available sensors within a threshold range of the threat, and broadcasting an emergency alert to mobile devices within a particular geographic range to the threat. A second set of bulk actions may apply to a fire threat anywhere on the campus150. The second set of bulk actions may include notifying emergency responders, activating alarms of buildings within a threshold range of the threat, and activating fire suppression systems at the location of the threat.

In some examples, the rules engine124can determine to change a state of the monitoring system100based on the threat assessment123. For example, the threat assessment123may indicate an active shooter threat with a confidence of greater than sixty-five percent. Based on the threat assessment123, the rules engine124can determine to change the state of the monitoring system100to a higher alert state. In the higher alert state, the monitoring server may collect sensor data from a greater number of sensors at the campus150, may collect sensor data from sensors at the campus150at an increased frequency, etc. For example, based on entering the higher alert state based on sensor data collected at building102, the monitoring server130can obtain sensor data from a greater number of sensors, including sensors at buildings104,106, and/or other buildings at the campus150.

The rules engine124can determine monitoring system actions based on the confidence of the detected threat. The actions can include sending notifications to users. For example, at a threat confidence of fifty percent, the rules engine124can determine to send a notification to an owner or manager of the campus150. At a threat level of seventy percent, the rules engine124can determine to send a notification to emergency responders. At a threat level of seventy-five percent, the rules engine124can determine to send a notification to occupants of the campus and/or to people within a proximity of the campus150.

In an example, the threat assessment123can include a confidence of fifty percent for an active shooter threat. Based on the confidence of fifty percent, the rules engine124can determine an action of notifying the campus manager, e.g., by transmitting a notification to a computing device of the campus manager. The rules engine124may receive an updated threat assessment123indicating a confidence of seventy percent for an active shooter threat. Based on the confidence of seventy percent, the rules engine124can determine an action of notifying emergency responders, e.g., by placing a telephone call to an emergency response center. The rules engine124may receive an updated threat assessment123indicating a confidence of seventy-five percent for an active shooter threat. Based on the confidence of seventy-five percent, the rules engine124can determine an action of notifying occupants of the campus150. For example, the monitoring server130can notify occupants by triggering one or more alarms at the campus150, by broadcasting an audible notification at one or more buildings of the campus150, by broadcasting an emergency alert within a designated radius through programs such as a Wireless Emergency Alert program.

The rules engine124can determine monitoring system actions based on tracking the person110. For example, the threat assessment123can include an estimated path that the person110has taken through the campus150, an estimated current location of the person110, a predicted path of the person110, or any combination of these. The threat assessment123can include an estimated current location of the person110as being in a first room of the building106. Based on the estimated current location of the person110as being in the first room of the building106, the rules engine124can determine monitoring system actions130that prevent the person110from exiting the first room, or from exiting the building106. The monitoring system actions130can include, for example, shutting one or more interior or exterior doors, locking one or more interior or exterior doors, shutting one or more windows, locking one or more windows, deactivating one or more elevators, etc. Thus, the monitoring server130can perform actions in order to isolate or contain the threat.

The rules engine124can determine monitoring system actions based on tracking bystanders and victims at the campus150. A bystander can be, for example, a person who is not considered a threat but who is located near the threat. Bystanders may be able to move about the campus in order to evade the threat. A victim can be, for example, a person who has been harmed by the threat. Victims might be hindered in their movement about the campus, e.g. due to injuries or due to a captive or hostage situation.

The threat assessment engine122can determine bystander and victim locations based on sensor data, and the threat assessment123can include the estimated locations of victims, and locations of bystanders who may be sheltering in the buildings102,104,106. Based on the estimated locations of the victims, the rules engine124can determine monitoring system actions140that assist emergency responders in locating the victims. For example, the monitoring system actions140can include activating lighting to illuminate a path to the victims, broadcasting audible directions through a speaker, etc. Based on the estimated locations of the bystanders, the rules engine124can determine monitoring system actions130that prevent the person110from locating and/or accessing areas of the buildings where bystanders are located. For example, the monitoring system actions140can include locking doors, locking windows, shutting shades or shutters, etc.

In some examples, the rules engine124can determine monitoring system actions130that assist people in escaping from the threat. For example, the monitoring system actions130may contain the threat in the first room of the building106by locking doors and windows of the first room of the building106. The rules engine124can then determine to unlock locked doors and automatically hold open other doors and/or windows of the building106in order to provide an escape path for people in the building106.

The rules engine124can determine escape paths based on the threat assessment123mapped to the campus model121. For example, the threat assessment123may indicate that people are located in a second room at the building106, and that a hallway of the building106leads to an exit without passing the current location of the person110. The rules engine124can determine actions140that permit the people to escape down the hallway. In some examples, the rules engine124can determine actions140that guide the people to follow the escape path, e.g., by broadcasting audio guidance through speakers at the building106.

In some examples, the rules engine124can determine actions140that guide people to follow the escape path by illuminating lights along the escape path. For example, the rules engine124can illuminate overhead lighting along the escape path, and distinguish overhead lighting in unsafe areas. In some examples, the rules engine124can determine actions that use color-coded lights to guide people to follow the escape path. For example, the rules engine124can determine to illuminate safe passages in green light, and to illuminate dangerous passages in red light.

Using the campus model121, the rules engine124can identify isolable areas of the campus. Isolable areas of the campus are areas that are capable of being isolated, e.g., using doors, windows, and/or locks. Isolating an area can include securing all accesses to the area such that a person within the area cannot get out of the isolated area. Isolable areas of the campus can include, for example, a building, a wing of a building, a floor of a building, an elevator, a hallway, a room, a tunnel, an overpass, or another area of the campus.

The rules engine124can monitor each isolable area to determine whether the threat is located within an isolable area. In some examples, the rules engine124can isolate an isolable area based on determining that the threat is located within the isolable area. In some examples, the rules engine124can isolate the isolable area based on determining that the threat is located within the isolable area and that no bystanders are located within the isolable area. Once the isolable area is isolated, the rules engine124can determine actions to impede the threat, e.g., by turning off all lights in the isolable area.

In some examples, the rules engine124may determine that the threat is located within an isolable area and that bystanders are also located within the isolable area. The rules engine124can continue to monitor the isolable area using sensor data to determine when all bystanders have departed from the isolable area. Based on determining that all bystanders have departed from the isolable area, and that the threat is still located within the isolable area, the rules engine124can isolate the isolable area by securing all accesses to the isolable area to contain the threat.

In stage (C) ofFIG.1, the monitoring server130performs system actions140as determined by the rules engine124. For example, based on the determination that an active shooter threat exists at the campus150, and that the person110has a firearm111and has a current location within the building106, the monitoring server130can perform an action140of sending a notification or an alert to an emergency responder132. Emergency responders such as the emergency responder132may be, for example, police personnel, firefighters, security guards, emergency medical personnel, etc. The emergency responder132can receive the alert on a device such as a mobile device136. The monitoring server130can also send alerts and notifications to owners and occupants of the campus150as determined by the rules and settings. The actions140can include providing a visualization of the threat assessment123to the emergency responders. An example visualization of a threat assessment123is described with reference toFIG.2.

The actions140can include activating additional sensors at the campus150. The additional sensors may be installed at the campus150or may be mounted to automated or remotely operated vehicles such as aerial drone144. The monitoring server130transmit an instruction to the drone144that causes the drone to deploy to the estimated location of the threat, e.g., building106. Sensors of the drone144can obtain additional sensor data and provide the additional sensor data to the monitoring server130. Based on the additional sensor data, the threat assessment engine122can update the threat assessment123.

In some examples, autonomous or remotely controlled vehicles such as the drone144can perform additional actions as determined by the rules engine124. For example, the monitoring server130can deploy the drone144to guide people to safety. The drone144can deploy to locations where bystanders are located, and can provide a signal to the bystanders indicating to follow the drone144. The signal can include, e.g., an audio or visual signal. The drone144can then travel along an escape path determined by the monitoring server130using the threat assessment123and the campus model121.

In some examples, the monitoring server130can deploy the drone144to the location of the threat in order to mitigate the threat. For example, the drone144can include lethal and/or non-lethal weapons. The lethal and/or non-lethal weapons can be used to harm, mark, or distract the person110. The lethal and/or non-lethal weapons can include, e.g., tasers, pepper spray, firearms, tear gas, long-range acoustic devices (LRAD), permanent ink, etc. The monitoring server130can deploy the drone144to the estimated location of the person110. When the drone144intercepts the person110, the drone144can attack the person110using the weapons.

In some examples, the monitoring server130can perform system actions140that include adjusting or configuring one or more devices at the campus150. The monitoring server130may send a command to adjust a device at the campus150via the control unit112. For example, the monitoring server130can send a command to the control unit112to shut and lock doors and to shut and lock windows at the campus150. The control unit112can adjust the doors, windows, etc., via automation controls. In some examples, the monitoring server130can trigger an alarm at the campus150, e.g., an audio and/or visual alarm.

In some examples, the monitoring server130can adjust devices at the campus150without any user action. In some examples, the monitoring server130can provide recommended actions to a user, and can perform the actions upon approval by the user. For example, the monitoring server130may provide recommended actions to a user such as the manager or owner of the campus, and request approval of the actions. Upon approval of the actions, the monitoring server can perform the actions140. In some examples, the monitoring server130can provide recommended actions to the user with a time limit. If the user does not respond to approve or deny the actions within the time limit, the monitoring server130may perform the actions140.

In some examples, some of the actions140require approval by a user, while other actions can be performed without user approval. In some examples, user approval may be required for lower confidence threats, while user approval might not be required for higher confidence threats. For example, for an active shooter threat with a confidence value of less than sixty-five percent, the monitoring server130may request approval before notifying emergency responders, in accordance with rules and settings. For an active shooter threat with a confidence value of greater than or equal to sixty-five percent, the monitoring server130may notify emergency responders without requesting approval from a user.

In some examples, the monitoring server130may perform actions140related to increasing security measures at campus buildings. For example, control units of buildings near the campus150may communicate with the same monitoring server130. The monitoring server130can send commands to the buildings to adjust devices and/or equipment at the buildings. For example, the monitoring server130can send commands to buildings to activate external security cameras at the buildings. The security cameras can then send collected images to the monitoring server130.

In some examples, in response to detecting the threat at the campus150, the monitoring server130can send commands to campus buildings to shut and/or lock doors or arm monitoring systems at the buildings. In some examples, the monitoring server130can send data to monitoring systems of campus buildings indicating that the threat occurred. The monitoring systems of the campus buildings can then use automation controls to adjust and configure devices based on rules and settings of the monitoring systems.

The actions140can include send an alert to users that includes the location of the campus150, the time of the threat, and the current location, route, speed of the person110, and/or other information. The alert can also include details about the person110based on the data115, e.g., security camera105images. For example, the alert can include the number of personnel and whether or not the personnel are armed.

The notification can include a message stating that an active shooter situation is in progress. The monitoring server130can send the notification to residents or occupants via, for example, a text message that the occupants can receive on a mobile device. The mobile device can be any type of data carrying computing device. For example, the mobile device can be a laptop computer, a tablet, smart watch, a video game console, or a smart car. The monitoring server130can also send the notification to users via, for example, a telephone call.

In some examples, the monitoring server130can perform system actions140that include adjusting access control devices at the campus150. Access control devices can include, for example, access control readers such as badge readers, ID card readers, key fob readers, mobile credential readers, etc. Access control devices can be installed at any indoor or outdoor access point of the campus150. For example, access control devices can be installed to permit people to enter and/or exit buildings, rooms, hallways, etc. For example, an access control device may be installed on the door108or near the door108, e.g., on a wall next to the door108. When a person scans a badge at the access control device, the access control device can verify the badge and unlock the door108, permitting the person to enter the building106.

Access control devices can include lighting components, audio components, or both. For example, an access control device can include lighting components such as LED lights of various colors. An access control device can also include audio components such as speakers and buzzers. The lighting components and audio components can be used to indicate permission or denial to access a space. For example, when the access control device verifies a scanned credential, the access control device can indicate the verification, e.g., by illuminating a green light. When the access control device fails to verify a scanned credential, the access control device can indicate the failed verification, e.g., by illuminating a red light and activating a buzzer.

The monitoring server130can transmit commands to adjust access control devices, e.g., by sending a command to activate lighting components and/or audio components of access control devices to warn of dangerous conditions. For example, a gunshot detection system may detect gunshots in a particular room of the building102. Based on the detected gunshots in the particular room, the monitoring server130can transmit a command to an access control device installed on a door to the particular room. For example, the command can instruct the access control device to flash LED lights in a pattern, e.g., by alternating between red and orange LED colors to indicate dangerous conditions in the particular room. In some examples, the command can instruct the access control device to broadcast audible beeping or buzzing sounds to indicate dangerous conditions in the particular room.

In some examples, the monitoring server130can transmit commands to activate lighting components and/or audio components of access control devices to indicate safe areas and escape paths. For example, a gunshot detection system may detect gunshots in a particular room of the building102. The monitoring server130can identify safe areas and escape paths that are clear of the particular room, and can transmit commands to access control devices in the safe areas and along the escape paths. For example, the command can instruct the access control devices to flash a green color to indicate safety.

In some examples, the monitoring server130can transmit commands to adjust access control devices, e.g., by sending a command to a control device to lock or unlock a door. For example, the monitoring server130may determine that bystanders are located in a dangerous area of the campus150, and that access through a door to an escape path is controlled by a particular access control device. The monitoring server130can transmit a command to the particular access control device that causes the particular access control device to unlock the door, permitting the bystanders to access the escape path.

In some examples, the monitoring server130may determine that a threat, e.g., person110, is alone in a room, and that an exit door from the room is controlled by a particular access control device. The monitoring server130can transmit a command to the particular access control device that causes the particular access control device to lock the exit door, trapping the person110in the room. The monitoring server130can transmit a command to the particular access control device that revokes access to the room. Therefore, when another person approaches the locked room and scans an authorized access badge, the control device will prevent the person from entering the room where the threat has been trapped.

In some examples, a gunshot detection system can be paired to one or more access control devices. For example, a gunshot detector installed in a room can be paired to access control devices that control access to the room. The gunshot detector can communicate with the access control device, e.g., through a wired or short-range wireless connection. When the gunshot detector detects gunfire in the room, the gunshot detector can transmit a signal to the access control device. The signal can cause the access control device to perform one or more actions. For example, based on receiving the signal from the gunshot detector, the access control device can signal dangerous conditions in the room by flashing lights, by broadcasting audio sounds, etc. In some examples, based on receiving the signal from the gunshot detector, the access control device can unlock in order to permit bystanders to exit from the room.

In some examples, the monitoring server130can perform system actions140that include sending notifications of the threat to residents of campus buildings or nearby properties. For example, properties that are near to the campus150may have monitoring systems that can communicate with the monitoring server130. Residents of the campus buildings may opt-in to receiving alerts and notifications from the monitoring server130based on anomalies detected at the campus150and/or other buildings in the area. In some examples, the monitoring server130may perform system actions140that include adjusting or configuring devices at the campus buildings using automation controls. In some examples, the monitoring server130may perform system actions140that include requesting permission from managers of campus buildings before adjusting devices at the campus buildings in response to the detected threat. The monitoring server130may include preprogrammed rules and settings for each of the campus buildings.

Though described above as being performed by a particular component of monitoring system100(e.g., the control units112,114,116or the monitoring server130), any of the various control, processing, and analysis operations can be performed by either the control units, the monitoring server130, the sensors, or another computer system of the monitoring system100. For example, the control units, the monitoring server130, the sensors, or another computer system can analyze the images and data from the sensors to detect a threat. Similarly, the control units, the monitoring server130, the sensors, or another computer system can control the various sensors, and/or the property automation controls, to collect data or control device operation.

FIG.2is a diagram illustrating an example visualization200of a threat assessment for an emergency responder132. The visualization200can be presented on a display of a computing device, e.g., the mobile device136associated with the emergency responder132. The visualization200can include a 2D or 3D visual representation of the campus model121overlaid with data from the threat assessment123. The visualization200ofFIG.2shows an example threat assessment of the building106. The person110is in a room220of the building106with the firearm111.

Though the visualization200ofFIG.2shows a single floor of a single building, the visualization of the threat assessment123can include additional floors and additional buildings of the campus150. For example, the emergency responder132can view a visualization of multiple floors of the building106and/or a visualization of the buildings102,104, and106. In some examples, the visualization200can include depictions of outdoor spaces, e.g., outdoor spaces of the campus150between the buildings102,104,106. The depictions of outdoor spaces can include a 2D or 3D terrain map, including locations of features such as hills, trees, boulders, hedges, etc.

In some examples, the emergency responder132can access the visualization200using an application of the mobile device136. Emergency response organizations can have registered accounts with the monitoring service that permit the emergency responders to access the visualization200. The application can perform an authentication process to allow the emergency responder132to view the visualization200. In some examples, the authentication process can be performed as part of a dispatch process for the emergency responder. For example, when the emergency responder132is dispatched to respond to the threat, the dispatching officials can send authentication credentials to the mobile device136that permit the emergency responder132to access the visualization.

In some examples, access to the visualization is based in part on a geographic location of the mobile device136, e.g., based on a GPS location of the mobile device136. For example, when the emergency responder132approaches within a particular geographic range to the building106, e.g., a range of a quarter mile or less from the building106, the application of the mobile device136can prompt the emergency responder132to enter credentials to access the visualization200.

The visualization200shows color-coded representations of the threat level of different areas of the building106, according to a legend210. Thus, the visualization of the threat assessment can be represented as a “heat map” of various threat levels at different areas of the campus150. Areas of low threat are represented in a light shade, areas of medium threat are represented in a medium shade, and areas of high threat are represented in a dark shade. In some examples, the color coding can use a first color, e.g., green, for a low threat, a second color, e.g., yellow, for a medium threat, and a third color, e.g., red, for a high threat.

The threat levels shown in the visualization200are based on the threat assessment123. The threat levels can be based on, e.g., a confidence level of the threat, a location of the threat, a speed of movement of the threat, etc. For example, based on analyzing camera image data collected from cameras at the campus150, the threat assessment engine122may determine that there is a high confidence of 90% that the person110is carrying a firearm111. Based on motion sensor data, camera image data, microphone data, and door open/shut data collected from sensors at the campus150, the threat assessment engine122may determine that there is a high confidence of 80% that the person110is in the room220of the building106. Thus, based on the high confidence that the person110is carrying the firearm111and that the person110is in the room220, the threat assessment engine122can determine a high threat level in the room220, and show the room220in a dark shade in the visualization200.

Based on door open/shut data, the threat assessment engine may determine that the doors208,212are shut. The threat assessment engine122can determine a medium threat level for rooms adjacent to the room220based on the location of the person110and based on the doors208,212being shut. Thus, the rooms adjacent to room220are shown as a medium shade in the visualization200. Similarly, rooms that are farther away from the room220can be represented by light shading, representing a low threat level.

The visualization200can be an interactive visualization. The emergency responder132can interact with the visualization200, e.g., through a user interface provided by the mobile device136. For example, the emergency responder132can zoom in and out to view the visualization200in various levels of detail. For example, the emergency responder132can zoom in to view a particular room of the building106, and can zoom out to view the entire building106, or to view the building106and one or more other buildings. In some examples, the visualization200is a 3D visualization, and the emergency responder132can interact with the visualization200by rotating the 3D visualization. By rotating the 3D visualization, the emergency responder132can view various access points of the building106and paths within the building106.

The visualization200shows exterior access points of the building106, e.g., exterior door202, exterior door204, exterior door108, and window206. The visualization200also shows interior access points of the building106, e.g., interior door208, interior door212, and interior door222. In some examples, the visualization200can depict an open/shut status and or a locked/unlocked status of access points of the building106. The open/shut status and the locked/unlocked status of access points can be based on sensor data collected at the building106, e.g., based on door and window sensors, lock sensors, and/or camera image data. For example, the visualization200depicts the doors202,204,208, and212being shut. The visualization200depicts the door222being open. The visualization200depicts the doors204,108being unlocked, and the door202being locked.

The visualization200shows locations of sensors at the building106, e.g., camera118, motion sensor218, camera214, and microphone216. The visualization200can include depictions of fields of view and ranges of the sensors. For example, the visualization200includes a depiction of the field of view215of the camera118.

In some examples, the visualization200can be annotated with indications of sensor data collected by the sensors. For example, an annotation226at the depiction of the microphone216indicates that breaking glass was detected at 2:27 pm. Similarly, the door108includes a door lock sensor, and an annotation224at the depiction of the door108indicates that the door108was unlocked at 2:28 pm. An annotation228at the depiction of the motion sensor218indicates that motion was detected at 2:29 pm. An annotation230at the depiction of the camera214indicates that the person110and the firearm111are currently within the field of view of the camera214.

In some examples, the visualization200can be tailored to the emergency responder. For example, an emergency responder that logs into the application using an account associated with security personnel may be provided with a different view that an emergency responder that logs into the application using an account associated with medical personnel. A view provided to security personnel can display, e.g., the estimated location of the person110, a path through the building106that the person110has already traversed, a predicted path of the person110, and suggested safe routes to approach the person110. A view provided to medical personnel can instead display, e.g., locations of victims and suggested safe routes to approach the victims.

In some examples, the emergency responder132can interact with the visualization200by selecting devices and sensors depicted in the visualization. Selecting a device or sensor of the visualization200can enable the emergency responder132to view additional detail and/or to manipulate the device or sensor. For example, the emergency responder132may select the depiction of the camera214in order to view a current or recent image captured by the camera214. Upon selecting the depiction of the camera214, the emergency responder132may also be able to manipulate the camera214, e.g., by tilting or rotating the camera214using the interface provided on the mobile device136. In another example, the emergency responder132may select the depiction of the door202. Upon selecting the depiction of the door202, the emergency responder132can select an option to unlock the door202in order to permit the emergency responder132to access the building106through the door202.

In some examples, the emergency responder132can interact with the visualization200in order to perform bulk actions to mitigate the threat. For example, the emergency responder132can select an area of the building106and select an option to lock down the area. As an example, the emergency responder132can select the room220on the visualization, and select a “lock down” option displayed through a user interface. Based on the emergency responder132selecting the “lock down” option, the monitoring server130can send a command to doors, windows, and locks of the room220that cause the doors and windows to shut and lock. In the example of a fire threat, the emergency responder132can interact with the visualization200to select options to initiate fire suppression systems in one or more areas of the building106.

FIG.3is a flow diagram illustrating an example process300for active threat tracking and response. Process300can be performed by one or more computer systems, for example, the monitoring server130of monitoring system100. In some implementations, some or all of the process can be performed by a control unit, e.g., control unit112,114,116of the monitoring system100, or by another computer system located at the monitored campus150.

Briefly, process300includes determining, using sensor data generated by one or more sensors located at one or more properties, that an active threat is in progress at the one or more properties (302), accessing a virtual model of the one or more properties, the virtual model including a position of each of the one or more sensors (304), determining, using the sensor data and the virtual model, a threat level of the active threat at each of two or more areas of the one or more properties (306), and based on the threat level of the active threat at each of the two or more areas of the one or more properties, performing one or more monitoring system actions (308). The process300can optionally include providing, to a user, a visualization of the threat level of the active threat at each of the two or more areas of the one or more properties (310) and controlling one or more access points to the one or more properties to contain the active threat.

In more detail, the process300includes determining, using sensor data generated by one or more sensors located at one or more properties, that an active threat is in progress at the one or more properties (302). For example, the threat assessment engine122may receive sensor data115and determine that an active shooter is in progress. The sensor data can include, for example, video camera data, audio data, motion sensor data, and temperature data. The sensor data can also include a status of one or more devices at the property. For example, the sensor data can include a door and window position and lock status of the properties. The sensor data can also include the arming status of the monitoring system at each of the properties, e.g., “armed stay,” “armed away,” or “unarmed.” The active threat can be, for example, an active shooter, a fire, a carbon monoxide leak, a flood, a burglary, a physical altercation, etc. In an example, the sensor data can include camera image data showing smoke and flames and smoke sensor data indicating smoke in the building104. Based on the sensor data, the system can determine that an active threat of a fire is in progress at the building104.

The process300includes accessing a virtual model of the one or more properties, the virtual model including a position of each of the one or more sensors (304). For example, the threat assessment engine122can access a virtual model of the building104. The virtual model can include a position of the camera and of the smoke sensor in the building104. The virtual model may indicate that the camera and the smoke sensor are located on the fourth floor of the building104.

The process300includes determining, using the sensor data and the virtual model and for each of two or more areas of the one or more properties, a threat level of the active threat at the respective area (306). For example, the threat assessment engine122can determine, based on the sensor data115and the virtual model a threat level of the fire at multiple different areas of the building104and surrounding buildings. For example, the system may determine a high threat level for the fourth floor of the building104, a medium threat level for the second and third floor of the building104, and a low threat level for the first floor of the building104. The system may determine a low threat level for adjacent buildings102and106.

The process300includes, using the threat level of the active threat at each of the two or more areas of the one or more properties, performing one or more monitoring system actions (308). For example, based on the high threat level of the fire in the multiple different areas of the building104, the rules engine124may activate a fire alarm at the building104and send a notification to an owner of the building104and to emergency responders. In some examples, the system may activate additional sensors at the property, such as additional cameras, microphones, and/or motion sensors. The monitoring system may also mitigate the risk of fire by adjusting one or more devices at the property, e.g., by locking doors and windows to contain the fire and by activating a sprinkler system.

The process300optionally includes providing, to a user, a visualization of the threat level of the active threat at each of the two or more areas of the one or more properties (310). For example, the monitoring server130may generate a visualization that is sent to the mobile device136of the emergency responder132, where the visualization of the threat level of the fire at the building104can use a red color to indicate a high threat level of the fourth floor, a yellow color to indicate a medium threat level of the third floor and the second floor, and a green color to indicate a low threat level of the first floor. The visualization can include a depiction of a location of the camera and of the smoke sensor, and an indication of an open/shut status of doors and windows and a locked/unlocked status of doors and windows at the building104.

The process300optionally includes the ability to distinguish threats from non-threats in various ways. Processes include, but are not limited to video or image detection, facial recognition, databases using various methods of identification, electronic tagging, or any combination thereof. For example, the system can identify the threat by video detection of a weapon. In other instances, the system can identify threats or non-threats through facial recognition.

The process300optionally includes controlling one or more access points to the one or more properties to contain the active threat (312). For example, the system can control door and window accesses to the building104to contain the fire in the building104, or on the fourth floor of the building104.

The order of steps in the process300described above is illustrative only, and active threat tracking and response can be performed in different orders. For example, the process300can perform step304and then step302or perform these steps substantially concurrently.

In some implementations, the process300can include additional steps, fewer steps, or some of the steps can be divided into multiple steps. For example, the process300can include steps302through308without steps310or312. The process300can include steps302through310without step312. The process300can include steps302through308and step312without step310.

FIG.4is a diagram illustrating an example of a home monitoring system400. The monitoring system400includes a network405, a control unit410, one or more user devices440and450, a monitoring server460, and a central alarm station server470. In some examples, the network405facilitates communications between the control unit410, the one or more user devices440and450, the monitoring server460, and the central alarm station server470.

The network405is configured to enable exchange of electronic communications between devices connected to the network405. For example, the network405may be configured to enable exchange of electronic communications between the control unit410, the one or more user devices440and450, the monitoring server460, and the central alarm station server470. The network405may include, for example, one or more of the Internet, Wide Area Networks (WANs), Local Area Networks (LANs), analog or digital wired and wireless telephone networks (e.g., a public switched telephone network (PSTN), Integrated Services Digital Network (ISDN), a cellular network, and Digital Subscriber Line (DSL)), radio, television, cable, satellite, or any other delivery or tunneling mechanism for carrying data. Network405may include multiple networks or subnetworks, each of which may include, for example, a wired or wireless data pathway. The network405may include a circuit-switched network, a packet-switched data network, or any other network able to carry electronic communications (e.g., data or voice communications). For example, the network405may include networks based on the Internet protocol (IP), asynchronous transfer mode (ATM), the PSTN, packet-switched networks based on IP, X.25, or Frame Relay, or other comparable technologies and may support voice using, for example, VoIP, or other comparable protocols used for voice communications. The network405may include one or more networks that include wireless data channels and wireless voice channels. The network405may be a wireless network, a broadband network, or a combination of networks including a wireless network and a broadband network.

The control unit410includes a controller412and a network module414. The controller412is configured to control a control unit monitoring system (e.g., a control unit system) that includes the control unit410. In some examples, the controller412may include a processor or other control circuitry configured to execute instructions of a program that controls operation of a control unit system. In these examples, the controller412may be configured to receive input from sensors, flow meters, or other devices included in the control unit system and control operations of devices included in the household (e.g., speakers, lights, doors, etc.). For example, the controller412may be configured to control operation of the network module414included in the control unit410.

The network module414is a communication device configured to exchange communications over the network405. The network module414may be a wireless communication module configured to exchange wireless communications over the network405. For example, the network module414may be a wireless communication device configured to exchange communications over a wireless data channel and a wireless voice channel. In this example, the network module414may transmit alarm data over a wireless data channel and establish a two-way voice communication session over a wireless voice channel. The wireless communication device may include one or more of a LTE module, a GSM module, a radio modem, cellular transmission module, or any type of module configured to exchange communications in one of the following formats: LTE, GSM or GPRS, CDMA, EDGE or EGPRS, EV-DO or EVDO, UMTS, or IP.

The network module414also may be a wired communication module configured to exchange communications over the network405using a wired connection. For instance, the network module414may be a modem, a network interface card, or another type of network interface device. The network module414may be an Ethernet network card configured to enable the control unit410to communicate over a local area network and/or the Internet. The network module414also may be a voice band modem configured to enable the alarm panel to communicate over the telephone lines of Plain Old Telephone Systems (POTS).

The control unit system that includes the control unit410includes one or more sensors. For example, the monitoring system may include multiple sensors420. The sensors420may include a lock sensor, a contact sensor, a motion sensor, or any other type of sensor included in a control unit system. The sensors420also may include an environmental sensor, such as a temperature sensor, a water sensor, a rain sensor, a wind sensor, a light sensor, a smoke detector, a carbon monoxide detector, an air quality sensor, etc. The sensors420further may include a health monitoring sensor, such as a prescription bottle sensor that monitors taking of prescriptions, a blood pressure sensor, a blood sugar sensor, a bed mat configured to sense presence of liquid (e.g., bodily fluids) on the bed mat, etc. In some examples, the health-monitoring sensor can be a wearable sensor that attaches to a user in the home. The health-monitoring sensor can collect various health data, including pulse, heart rate, respiration rate, sugar or glucose level, bodily temperature, or motion data.

The sensors420can also include a radio-frequency identification (RFID) sensor that identifies a particular article that includes a pre-assigned RFID tag.

The control unit410communicates with the home automation controls422and a camera430to perform monitoring. The home automation controls422are connected to one or more devices that enable automation of actions in the home. For instance, the home automation controls422may be connected to one or more lighting systems and may be configured to control operation of the one or more lighting systems. In addition, the home automation controls422may be connected to one or more electronic locks at the home and may be configured to control operation of the one or more electronic locks (e.g., control Z-Wave locks using wireless communications in the Z-Wave protocol). Further, the home automation controls422may be connected to one or more appliances at the home and may be configured to control operation of the one or more appliances. The home automation controls422may include multiple modules that are each specific to the type of device being controlled in an automated manner. The home automation controls422may control the one or more devices based on commands received from the control unit410. For instance, the home automation controls422may cause a lighting system to illuminate an area to provide a better image of the area when captured by a camera430.

The camera430may be a video/photographic camera or other type of optical sensing device configured to capture images. For instance, the camera430may be configured to capture images of an area within a building or home monitored by the control unit410. The camera430may be configured to capture single, static images of the area and also video images of the area in which multiple images of the area are captured at a relatively high frequency (e.g., thirty images per second). The camera430may be controlled based on commands received from the control unit410.

The camera430may be triggered by several different types of techniques. For instance, a Passive Infra-Red (PIR) motion sensor may be built into the camera430and used to trigger the camera430to capture one or more images when motion is detected. The camera430also may include a microwave motion sensor built into the camera and used to trigger the camera430to capture one or more images when motion is detected. The camera430may have a “normally open” or “normally closed” digital input that can trigger capture of one or more images when external sensors (e.g., the sensors420, PIR, door/window, etc.) detect motion or other events. In some implementations, the camera430receives a command to capture an image when external devices detect motion or another potential alarm event. The camera430may receive the command from the controller412or directly from one of the sensors420.

In some examples, the camera430triggers integrated or external illuminators (e.g., Infra-Red, Z-wave controlled “white” lights, lights controlled by the home automation controls422, etc.) to improve image quality when the scene is dark. An integrated or separate light sensor may be used to determine if illumination is desired and may result in increased image quality.

The camera430may be programmed with any combination of time/day schedules, system “arming state”, or other variables to determine whether images should be captured or not when triggers occur. The camera430may enter a low-power mode when not capturing images. In this case, the camera430may wake periodically to check for inbound messages from the controller412. The camera430may be powered by internal, replaceable batteries if located remotely from the control unit410. The camera430may employ a small solar cell to recharge the battery when light is available. Alternatively, the camera430may be powered by the controller's412power supply if the camera430is co-located with the controller412.

In some implementations, the camera430communicates directly with the monitoring server460over the Internet. In these implementations, image data captured by the camera430does not pass through the control unit410and the camera430receives commands related to operation from the monitoring server460.

The system400also includes thermostat434to perform dynamic environmental control at the home. The thermostat434is configured to monitor temperature and/or energy consumption of an HVAC system associated with the thermostat434, and is further configured to provide control of environmental (e.g., temperature) settings. In some implementations, the thermostat434can additionally or alternatively receive data relating to activity at a home and/or environmental data at a home, e.g., at various locations indoors and outdoors at the home. The thermostat434can directly measure energy consumption of the HVAC system associated with the thermostat, or can estimate energy consumption of the HVAC system associated with the thermostat434, for example, based on detected usage of one or more components of the HVAC system associated with the thermostat434. The thermostat434can communicate temperature and/or energy monitoring information to or from the control unit410and can control the environmental (e.g., temperature) settings based on commands received from the control unit410.

In some implementations, the thermostat434is a dynamically programmable thermostat and can be integrated with the control unit410. For example, the dynamically programmable thermostat434can include the control unit410, e.g., as an internal component to the dynamically programmable thermostat434. In addition, the control unit410can be a gateway device that communicates with the dynamically programmable thermostat434. In some implementations, the thermostat434is controlled via one or more home automation controls422.

A module437is connected to one or more components of an HVAC system associated with a home, and is configured to control operation of the one or more components of the HVAC system. In some implementations, the module437is also configured to monitor energy consumption of the HVAC system components, for example, by directly measuring the energy consumption of the HVAC system components or by estimating the energy usage of the one or more HVAC system components based on detecting usage of components of the HVAC system. The module437can communicate energy monitoring information and the state of the HVAC system components to the thermostat434and can control the one or more components of the HVAC system based on commands received from the thermostat434.

In some examples, the system400further includes one or more robotic devices490. The robotic devices490may be any type of robots that are capable of moving and taking actions that assist in home monitoring. For example, the robotic devices490may include drones that are capable of moving throughout a home based on automated control technology and/or user input control provided by a user. In this example, the drones may be able to fly, roll, walk, or otherwise move about the home. The drones may include helicopter type devices (e.g., quad copters), rolling helicopter type devices (e.g., roller copter devices that can fly and roll along the ground, walls, or ceiling) and land vehicle type devices (e.g., automated cars that drive around a home). In some cases, the robotic devices490may be devices that are intended for other purposes and merely associated with the system400for use in appropriate circumstances. For instance, a robotic vacuum cleaner device may be associated with the monitoring system400as one of the robotic devices490and may be controlled to take action responsive to monitoring system events.

In some examples, the robotic devices490automatically navigate within a home. In these examples, the robotic devices490include sensors and control processors that guide movement of the robotic devices490within the home. For instance, the robotic devices490may navigate within the home using one or more cameras, one or more proximity sensors, one or more gyroscopes, one or more accelerometers, one or more magnetometers, a global positioning system (GPS) unit, an altimeter, one or more sonar or laser sensors, and/or any other types of sensors that aid in navigation about a space. The robotic devices490may include control processors that process output from the various sensors and control the robotic devices490to move along a path that reaches the desired destination and avoids obstacles. In this regard, the control processors detect walls or other obstacles in the home and guide movement of the robotic devices490in a manner that avoids the walls and other obstacles.

In addition, the robotic devices490may store data that describes attributes of the home. For instance, the robotic devices490may store a floorplan and/or a three-dimensional model of the home that enables the robotic devices490to navigate the home. During initial configuration, the robotic devices490may receive the data describing attributes of the home, determine a frame of reference to the data (e.g., a home or reference location in the home), and navigate the home based on the frame of reference and the data describing attributes of the home. Further, initial configuration of the robotic devices490also may include learning of one or more navigation patterns in which a user provides input to control the robotic devices490to perform a specific navigation action (e.g., fly to an upstairs bedroom and spin around while capturing video and then return to a home charging base). In this regard, the robotic devices490may learn and store the navigation patterns such that the robotic devices490may automatically repeat the specific navigation actions upon a later request.

In some examples, the robotic devices490may include data capture and recording devices. In these examples, the robotic devices490may include one or more cameras, one or more motion sensors, one or more microphones, one or more biometric data collection tools, one or more temperature sensors, one or more humidity sensors, one or more air flow sensors, and/or any other types of sensors that may be useful in capturing monitoring data related to the home and users in the home. The one or more biometric data collection tools may be configured to collect biometric samples of a person in the home with or without contact of the person. For instance, the biometric data collection tools may include a fingerprint scanner, a hair sample collection tool, a skin cell collection tool, and/or any other tool that allows the robotic devices490to take and store a biometric sample that can be used to identify the person (e.g., a biometric sample with DNA that can be used for DNA testing).

In some implementations, the robotic devices490may include output devices. In these implementations, the robotic devices490may include one or more displays, one or more speakers, and/or any type of output devices that allow the robotic devices490to communicate information to a nearby user.

The robotic devices490also may include a communication module that enables the robotic devices490to communicate with the control unit410, each other, and/or other devices. The communication module may be a wireless communication module that allows the robotic devices490to communicate wirelessly. For instance, the communication module may be a Wi-Fi module that enables the robotic devices490to communicate over a local wireless network at the home. The communication module further may be a900MHz wireless communication module that enables the robotic devices490to communicate directly with the control unit410. Other types of short-range wireless communication protocols, such as Bluetooth, Bluetooth LE, Z-wave, Zigbee, etc., may be used to allow the robotic devices490to communicate with other devices in the home. In some implementations, the robotic devices490may communicate with each other or with other devices of the system400through the network405.

The robotic devices490further may include processor and storage capabilities. The robotic devices490may include any suitable processing devices that enable the robotic devices490to operate applications and perform the actions described throughout this disclosure. In addition, the robotic devices490may include solid-state electronic storage that enables the robotic devices490to store applications, configuration data, collected sensor data, and/or any other type of information available to the robotic devices490.

The robotic devices490are associated with one or more charging stations. The charging stations may be located at predefined home base or reference locations in the home. The robotic devices490may be configured to navigate to the charging stations after completion of tasks needed to be performed for the monitoring system400. For instance, after completion of a monitoring operation or upon instruction by the control unit410, the robotic devices490may be configured to automatically fly to and land on one of the charging stations. In this regard, the robotic devices490may automatically maintain a fully charged battery in a state in which the robotic devices490are ready for use by the monitoring system400.

The charging stations may be contact based charging stations and/or wireless charging stations. For contact based charging stations, the robotic devices490may have readily accessible points of contact that the robotic devices490are capable of positioning and mating with a corresponding contact on the charging station. For instance, a helicopter type robotic device may have an electronic contact on a portion of its landing gear that rests on and mates with an electronic pad of a charging station when the helicopter type robotic device lands on the charging station. The electronic contact on the robotic device may include a cover that opens to expose the electronic contact when the robotic device is charging and closes to cover and insulate the electronic contact when the robotic device is in operation.

For wireless charging stations, the robotic devices490may charge through a wireless exchange of power. In these cases, the robotic devices490need only locate themselves closely enough to the wireless charging stations for the wireless exchange of power to occur. In this regard, the positioning needed to land at a predefined home base or reference location in the home may be less precise than with a contact based charging station. Based on the robotic devices490landing at a wireless charging station, the wireless charging station outputs a wireless signal that the robotic devices490receive and convert to a power signal that charges a battery maintained on the robotic devices490.

In some implementations, each of the robotic devices490has a corresponding and assigned charging station such that the number of robotic devices490equals the number of charging stations. In these implementations, the robotic devices490always navigate to the specific charging station assigned to that robotic device. For instance, a first robotic device may always use a first charging station and a second robotic device may always use a second charging station.

In some examples, the robotic devices490may share charging stations. For instance, the robotic devices490may use one or more community charging stations that are capable of charging multiple robotic devices490. The community charging station may be configured to charge multiple robotic devices490in parallel. The community charging station may be configured to charge multiple robotic devices490in serial such that the multiple robotic devices490take turns charging and, when fully charged, return to a predefined home base or reference location in the home that is not associated with a charger. The number of community charging stations may be less than the number of robotic devices490.

In addition, the charging stations may not be assigned to specific robotic devices490and may be capable of charging any of the robotic devices490. In this regard, the robotic devices490may use any suitable, unoccupied charging station when not in use. For instance, when one of the robotic devices490has completed an operation or is in need of battery charge, the control unit410references a stored table of the occupancy status of each charging station and instructs the robotic device to navigate to the nearest charging station that is unoccupied.

The system400further includes one or more integrated security devices480. The one or more integrated security devices may include any type of device used to provide alerts based on received sensor data. For instance, the one or more control units410may provide one or more alerts to the one or more integrated security input/output devices480. Additionally, the one or more control units410may receive one or more sensor data from the sensors420and determine whether to provide an alert to the one or more integrated security input/output devices480.

The sensors420, the home automation controls422, the camera430, the thermostat434, and the integrated security devices480may communicate with the controller412over communication links424,426,428,432,438, and484. The communication links424,426,428,432,438, and484may be a wired or wireless data pathway configured to transmit signals from the sensors420, the home automation controls422, the camera430, the thermostat434, and the integrated security devices480to the controller412. The sensors420, the home automation controls422, the camera430, the thermostat434, and the integrated security devices480may continuously transmit sensed values to the controller412, periodically transmit sensed values to the controller412, or transmit sensed values to the controller412in response to a change in a sensed value.

The communication links424,426,428,432,438, and484may include a local network. The sensors420, the home automation controls422, the camera430, the thermostat434, and the integrated security devices480, and the controller412may exchange data and commands over the local network. The local network may include 802.11 “Wi-Fi” wireless Ethernet (e.g., using low-power Wi-Fi chipsets), Z-Wave, Zigbee, Bluetooth, “Homeplug” or other “Powerline” networks that operate over AC wiring, and a Category 5 (CATS) or Category 6 (CAT6) wired Ethernet network. The local network may be a mesh network constructed based on the devices connected to the mesh network.

The monitoring server460is an electronic device configured to provide monitoring services by exchanging electronic communications with the control unit410, the one or more user devices440and450, and the central alarm station server470over the network405. For example, the monitoring server460may be configured to monitor events generated by the control unit410. In this example, the monitoring server460may exchange electronic communications with the network module414included in the control unit410to receive information regarding events detected by the control unit410. The monitoring server460also may receive information regarding events from the one or more user devices440and450.

In some examples, the monitoring server460may route alert data received from the network module414or the one or more user devices440and450to the central alarm station server470. For example, the monitoring server460may transmit the alert data to the central alarm station server470over the network405.

The monitoring server460may store sensor and image data received from the monitoring system and perform analysis of sensor and image data received from the monitoring system. Based on the analysis, the monitoring server460may communicate with and control aspects of the control unit410or the one or more user devices440and450.

The monitoring server460may provide various monitoring services to the system400. For example, the monitoring server460may analyze the sensor, image, and other data to determine an activity pattern of a resident of the home monitored by the system400. In some implementations, the monitoring server460may analyze the data for alarm conditions or may determine and perform actions at the home by issuing commands to one or more of the controls422, possibly through the control unit410.

The monitoring server460can be configured to provide information (e.g., activity patterns) related to one or more residents of the home monitored by the system400. For example, one or more of the sensors420, the home automation controls422, the camera430, the thermostat434, and the integrated security devices480can collect data related to a resident including location information (e.g., if the resident is home or is not home) and provide location information to the thermostat434.

The central alarm station server470is an electronic device configured to provide alarm monitoring service by exchanging communications with the control unit410, the one or more user devices440and450, and the monitoring server460over the network405. For example, the central alarm station server470may be configured to monitor alerting events generated by the control unit410. In this example, the central alarm station server470may exchange communications with the network module414included in the control unit410to receive information regarding alerting events detected by the control unit410. The central alarm station server470also may receive information regarding alerting events from the one or more user devices440and450and/or the monitoring server460.

The central alarm station server470is connected to multiple terminals472and474. The terminals472and474may be used by operators to process alerting events. For example, the central alarm station server470may route alerting data to the terminals472and474to enable an operator to process the alerting data. The terminals472and474may include general-purpose computers (e.g., desktop personal computers, workstations, or laptop computers) that are configured to receive alerting data from a server in the central alarm station server470and render a display of information based on the alerting data. For instance, the controller412may control the network module414to transmit, to the central alarm station server470, alerting data indicating that a sensor420detected motion from a motion sensor via the sensors420. The central alarm station server470may receive the alerting data and route the alerting data to the terminal472for processing by an operator associated with the terminal472. The terminal472may render a display to the operator that includes information associated with the alerting event (e.g., the lock sensor data, the motion sensor data, the contact sensor data, etc.) and the operator may handle the alerting event based on the displayed information.

In some implementations, the terminals472and474may be mobile devices or devices designed for a specific function. AlthoughFIG.4illustrates two terminals for brevity, actual implementations may include more (and, perhaps, many more) terminals.

The one or more authorized user devices440and450are devices that host and display user interfaces. For instance, the user device440is a mobile device that hosts or runs one or more native applications (e.g., the home monitoring application442). The user device440may be a cellular phone or a non-cellular locally networked device with a display. The user device440may include a cell phone, a smart phone, a tablet PC, a personal digital assistant (“PDA”), or any other portable device configured to communicate over a network and display information. For example, implementations may also include Blackberry-type devices (e.g., as provided by Research in Motion), electronic organizers, iPhone-type devices (e.g., as provided by Apple), iPod devices (e.g., as provided by Apple) or other portable music players, other communication devices, and handheld or portable electronic devices for gaming, communications, and/or data organization. The user device440may perform functions unrelated to the monitoring system, such as placing personal telephone calls, playing music, playing video, displaying pictures, browsing the Internet, maintaining an electronic calendar, etc.

The user device440includes a home monitoring application452. The home monitoring application442refers to a software/firmware program running on the corresponding mobile device that enables the user interface and features described throughout. The user device440may load or install the home monitoring application442based on data received over a network or data received from local media. The home monitoring application442runs on mobile devices platforms, such as iPhone, iPod touch, Blackberry, Google Android, Windows Mobile, etc. The home monitoring application442enables the user device440to receive and process image and sensor data from the monitoring system.

The user device440may be a general-purpose computer (e.g., a desktop personal computer, a workstation, or a laptop computer) that is configured to communicate with the monitoring server460and/or the control unit410over the network405. The user device440may be configured to display a smart home user interface452that is generated by the user device440or generated by the monitoring server460. For example, the user device440may be configured to display a user interface (e.g., a web page) provided by the monitoring server460that enables a user to perceive images captured by the camera430and/or reports related to the monitoring system. AlthoughFIG.4illustrates two user devices for brevity, actual implementations may include more (and, perhaps, many more) or fewer user devices.

In some implementations, the one or more user devices440and450communicate with and receive monitoring system data from the control unit410using the communication link438. For instance, the one or more user devices440and450may communicate with the control unit410using various local wireless protocols such as Wi-Fi, Bluetooth, Z-wave, Zigbee, HomePlug (ethernet over power line), or wired protocols such as Ethernet and USB, to connect the one or more user devices440and450to local security and automation equipment. The one or more user devices440and450may connect locally to the monitoring system and its sensors and other devices. The local connection may improve the speed of status and control communications because communicating through the network405with a remote server (e.g., the monitoring server460) may be significantly slower.

Although the one or more user devices440and450are shown as communicating with the control unit410, the one or more user devices440and450may communicate directly with the sensors and other devices controlled by the control unit410. In some implementations, the one or more user devices440and450replace the control unit410and perform the functions of the control unit410for local monitoring and long range/offsite communication.

In other implementations, the one or more user devices440and450receive monitoring system data captured by the control unit410through the network405. The one or more user devices440,450may receive the data from the control unit410through the network405or the monitoring server460may relay data received from the control unit410to the one or more user devices440and450through the network405. In this regard, the monitoring server460may facilitate communication between the one or more user devices440and450and the monitoring system.

In some implementations, the one or more user devices440and450may be configured to switch whether the one or more user devices440and450communicate with the control unit410directly (e.g., through link438) or through the monitoring server460(e.g., through network405) based on a location of the one or more user devices440and450. For instance, when the one or more user devices440and450are located close to the control unit410and in range to communicate directly with the control unit410, the one or more user devices440and450use direct communication. When the one or more user devices440and450are located far from the control unit410and not in range to communicate directly with the control unit410, the one or more user devices440and450use communication through the monitoring server460.

Although the one or more user devices440and450are shown as being connected to the network405, in some implementations, the one or more user devices440and450are not connected to the network405. In these implementations, the one or more user devices440and450communicate directly with one or more of the monitoring system components and no network (e.g., Internet) connection or reliance on remote servers is needed.

In some implementations, the one or more user devices440and450are used in conjunction with only local sensors and/or local devices in a house. In these implementations, the system400includes the one or more user devices440and450, the sensors420, the home automation controls422, the camera430, and the robotic devices490. The one or more user devices440and450receive data directly from the sensors420, the home automation controls422, the camera430, and the robotic devices490, and sends data directly to the sensors420, the home automation controls422, the camera430, and the robotic devices490. The one or more user devices440,450provide the appropriate interfaces/processing to provide visual surveillance and reporting.

In other implementations, the system400further includes network405and the sensors420, the home automation controls422, the camera430, the thermostat434, and the robotic devices490, and are configured to communicate sensor and image data to the one or more user devices440and450over network405(e.g., the Internet, cellular network, etc.). In yet another implementation, the sensors420, the home automation controls422, the camera430, the thermostat434, and the robotic devices490(or a component, such as a bridge/router) are intelligent enough to change the communication pathway from a direct local pathway when the one or more user devices440and450are in close physical proximity to the sensors420, the home automation controls422, the camera430, the thermostat434, and the robotic devices490to a pathway over network405when the one or more user devices440and450are farther from the sensors420, the home automation controls422, the camera430, the thermostat434, and the robotic devices490.

In some examples, the system leverages GPS information from the one or more user devices440and450to determine whether the one or more user devices440and450are close enough to the sensors420, the home automation controls422, the camera430, the thermostat434, and the robotic devices490to use the direct local pathway or whether the one or more user devices440and450are far enough from the sensors420, the home automation controls422, the camera430, the thermostat434, and the robotic devices490that the pathway over network405is required.

In other examples, the system leverages status communications (e.g., pinging) between the one or more user devices440and450and the sensors420, the home automation controls422, the camera430, the thermostat434, and the robotic devices490to determine whether communication using the direct local pathway is possible. If communication using the direct local pathway is possible, the one or more user devices440and450communicate with the sensors420, the home automation controls422, the camera430, the thermostat434, and the robotic devices490using the direct local pathway. If communication using the direct local pathway is not possible, the one or more user devices440and450communicate with the sensors420, the home automation controls422, the camera430, the thermostat434, and the robotic devices490using the pathway over network405.

In some implementations, the system400provides end users with access to images captured by the camera430to aid in decision making. The system400may transmit the images captured by the camera430over a wireless WAN network to the user devices440and450. Because transmission over a wireless WAN network may be relatively expensive, the system400can use several techniques to reduce costs while providing access to significant levels of useful visual information (e.g., compressing data, down-sampling data, sending data only over inexpensive LAN connections, or other techniques).

In some implementations, a state of the monitoring system and other events sensed by the monitoring system may be used to enable/disable video/image recording devices (e.g., the camera430). In these implementations, the camera430may be set to capture images on a periodic basis when the alarm system is armed in an “away” state, but set not to capture images when the alarm system is armed in a “home” state or disarmed. In addition, the camera430may be triggered to begin capturing images when the alarm system detects an event, such as an alarm event, a door-opening event for a door that leads to an area within a field of view of the camera430, or motion in the area within the field of view of the camera430. In other implementations, the camera430may capture images continuously, but the captured images may be stored or transmitted over a network when needed.

The described systems, methods, and techniques may be implemented in digital electronic circuitry, computer hardware, firmware, software, or in combinations of these elements. Apparatus implementing these techniques may include appropriate input and output devices, a computer processor, and a computer program product tangibly embodied in a machine-readable storage device for execution by a programmable processor. A process implementing these techniques may be performed by a programmable processor executing a program of instructions to perform desired functions by operating on input data and generating appropriate output. The techniques may be implemented in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device.

Each computer program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language if desired; and in any case, the language may be a compiled or interpreted language. Suitable processors include, by way of example, both general and special purpose microprocessors. Generally, a processor will receive instructions and data from a read-only memory and/or a random access memory. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including by way of example semiconductor memory devices, such as Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and Compact Disc Read-Only Memory (CD-ROM). Any of the foregoing may be supplemented by, or incorporated in, specially designed ASICs (application-specific integrated circuits).