Determining localized weather by media classification

Systems and techniques are described for utilizing video classification capabilities for providing accurate local weather. In some implementations, the techniques include the actions of obtaining images from cameras located at a monitored property. An expected weather forecast and an actual weather condition is obtained for the monitored property. A machine-learning model is trained to classify a current weather condition for the monitored property using the images from the cameras, the expected weather forecast, and the actual weather condition. A weather condition is obtained from the trained machine-learning model that indicates a particular weather condition at the monitored property based on one or more images from a camera and the expected local weather forecast at the monitored property.

TECHNICAL FIELD

The subject matter of the present disclosure is generally related to weather recognition, and more particularly, to weather recognition using classification models.

BACKGROUND

Obtaining and distributing accurate weather prediction is often a challenge, as weather prediction is generally aggregated at a level that is either too geographically broad or over too long of a time frame to be entirely reliable. For example, knowing that rain has been identified in a zip code does not mean that it is raining at every point within that zip code. Similarly, a weather station may never receive any rain, but a location nearby might. This lack of reliable knowledge makes it difficult to provide a confident description of the state of a property.

SUMMARY

The subject matter of the present disclosure is related to techniques of utilizing video classification capabilities to provide accurate local weather data to users. Specifically, the video classification capabilities may make of use of recorded media (e.g., video clips and images) from an integrated security environment for security measures. The security measures may include monitoring a property, such as a residential or commercial facility, with a monitor unit for detecting intrusions and other potentially harmful acts.

The monitor unit obtains recorded media from each of the one or more cameras found in the monitored property. The recorded media may be recording of an area surrounding an exterior of the monitored property. The recorded media includes one or more visual frames of data. In addition, the monitor unit obtains non-visual data, such as sensor data and device status data of the monitored property. The monitor unit provides the recorded media and the non-visual data to a data analyzer for processing. The monitor unit may alternatively include the data analyzer for further processing of the recorded media and the non-visual data.

The data analyzer obtains the recorded media and the non-visual data from the monitored property. In addition, the data analyzer obtains expected local forecast weather data to supplement the processing of the recorded media and the non-visual data. The data analytics server trains a neural network model with previously classified footage of labeled weather. Once the data analytics server sufficiently trains the neural network model to classify weather patterns in images, the data collection system may store the trained neural network model in memory and transmit the trained neural network model to cameras located at other monitored properties for weather classification. The cameras located at each monitored property can use the trained neural network model to classify weather patterns in camera footage without the use of the sensor data and weather forecasting data. Additionally, the data analytics server can provide automation instructions to a monitored property based on the weather classification and the sensor data. The automation instructions may provide instructions to the monitored property to close and lock opened doors, close and lock one or more opened windows, and close garage doors in view of a potentially damaging weather classification, such as hail.

In one general aspect, a method is performed by one or more computers that includes: obtain images from cameras located at a monitored property; obtain an expected weather forecast and an actual weather condition for the monitored property; train a machine-learning model to classify a current weather condition for the monitored property using the images from the cameras, the expected weather forecast, and the actual weather condition; and obtain a weather condition from the trained machine-learning model that indicates a particular weather condition at the monitored property based on one or more images from a camera and the expected local weather forecast at the monitored property.

Implementations may include one or more of the following features. For example, in some implementations, the method includes pairing the images from the cameras with data identifying the expected local weather forecasts in proximity to the monitored property; and providing the paired images with the data identifying the expected local weather forecasts and the actual weather condition for training the machine-learning model.

In some implementations, obtaining the expected weather forecast and the actual weather condition for the monitored property further includes: obtaining the expected weather forecast from a third party resource; and obtaining the actual weather condition from water sensors at the monitored property.

In some implementations, the method includes determining a device at the monitored property that exposes a portion of the monitored property to the particular weather condition; and providing an instruction to the device that adjusts a position of the device to reduce an exposure of the portion of the monitored property to the particular weather condition.

In some implementations, providing the instruction to the device further includes providing an instruction to close and lock a front door of the monitored property when the particular weather condition includes rain at the monitored property.

In some implementations, the method includes: providing the current weather condition to a client device owned by a property owner of the monitored property; receiving a correction to the current weather condition from the client device; and training the trained machine-learning model to generate the correction to the current weather condition using the correction to the current weather condition, the one or more images from the camera and the expected local weather forecast used to generate the current weather condition.

In some implementations, the method includes: providing the trained machine-learning model to each of the cameras at the monitored property; and receiving a weather condition from the trained machine-learning at each of the cameras.

DETAILED DESCRIPTION

FIG. 1is a contextual diagram of an exemplary system100that utilizes video analytics classification of media to detect localized weather. Though integrated security environment100is shown and described including a particular set of components including a monitor unit110-1through110-4(collectively monitor unit110), sensors104-1through104-4(collectively sensors104), cameras106-1through106-4(collectively cameras106), home devices108-1through108-4(collectively home device108), data analyzer112, visual data114, and expected local weather forecast116, the present disclosure need not be so limited.

As shown inFIG. 1, a geographic region can include one or more monitored properties. For example, geographic region 1 includes monitored property102-1and monitored property102-2. Additionally, geographic region 2 includes monitored property102-3and monitored property102-4. A geographic region can be defined by a geographic area such as, for example, a state, a city, or a county. The geographic region can include more than two monitored properties. Only two monitored properties are shown in each geographic region inFIG. 1for illustrative purposes.

A monitor unit110-1and other components within a monitored property may monitor the monitored property, such as monitored property102-1. The monitored property may be a residential facility, such as a home, a commercial facility, such as a corporate office building, a storage facility, such as a warehouse, or a transportation facility, such as an airport, to name a few examples. Each monitored property may include one or more sensors, one or more cameras, and one or more home devices. The components within each monitored property102can include one or more sensors104, one or more cameras106, and one or more home devices108. The one or more video cameras106may include video cameras, such as video camera111-1for monitored property102-1that can be located at the exterior of each monitored property102near the front door, located at the exterior of each monitored property102near the backyard, as well as located at the interior of each monitored property102. The cameras111-1can be anchored to the wall and include a unit to swivel over a predetermined direction, such as 180 degrees, for viewing its surrounding area.

The one or more sensors104in each of the monitored properties102can be located throughout each room of the monitored property102as well as at the exterior of the monitored property102. The one or more sensors104can include a contact sensor positioned at the front door to determine if the front door is open or closed. The one or more sensors can also include a pressure sensor that receives button presses at a light switch device, a lock sensor that is positioned at the front door and each window of each monitored property102. Additionally, the one or more sensors104can include a passive infrared sensor (PIR) located at the interior of each monitored property102, as well as thermal sensors, depth sensors, infrared beam trip sensors, weight sensors, seismic sensors, inductive loop sensors such as vehicle detection loops, an RGB-D camera, and one or more light curtain sensors. Each of these sensors may be positioned in and around each monitored property102for the monitor unit110to utilize its output for monitoring purposes.

The contact sensor may sense whether the front door, the windows, or garage door corresponding to each monitored property102is in an open position or a closed position. The lock sensor can sense whether the front door, each window, and each garage door is in an unlocked or locked position. The one or more home devices108can include home appliances such as a washing machine, a dryer, a dishwasher, an oven, a stove, a microwave, and a laptop, to name a few examples. In a commercial facility, the one or more home devices108may include a printer, a copier, a vending machine, and a fax machine to name a few examples. In another example, should the monitored property102be a transportation facility, such as an airport, the one or more home device108may include a ticket booth, a printer, a baggage scanner, and a scale, to name a few examples.

Each monitor unit110communicates over a wired or a wireless network connection with each of the one or more sensors104, the one or more cameras106, and the one or more home device108(e.g., for a residential facility, these home devices can include a washing machine, a dryer, a dishwasher, an oven, a stove, a microwave, a laptop, etc.). In addition, the monitor unit110can receive data from each of the sensors, cameras, and home devices that describe events detected by each of the sensors, the cameras, and the home devices.

In some implementation, each of the connected devices may connect to the monitor unit110using Wi-Fi, Bluetooth, or any other communication protocol utilized to communicate over a network to the monitor unit110. Additionally, the monitor unit110communicates over a long-range wired or wireless connection with the data analyzer112over a network131utilizing one or more communication links.

In some implementations, the data analyzer112is located in a remote position from each of the monitored properties102, and provides the data analytics for media provided by each monitored property102. For example, other monitored properties such as monitored properties102-2,102-3, and102-4. The data analyzer112may also monitor additional other residential properties located in various geographic regions not illustrated inFIG. 1. In other implementations, the data analyzer112may be located locally at each monitored property102. For instance, each data analyzer112may be incorporated in the monitor unit110. In other implementations, the data analyzer112may be located in each client device103,107,113, and117that corresponds to each monitored property102.

In some implementations, the data analyzer112communicates bi-directionally with each monitor unit110. Specifically, the data analyzer112receives media from the monitor unit110from each monitored property102. Additionally, the data analyzer112receives sensor data descriptive of events detected by the sensors104in each monitored property102. The data analyzer112can also receive data from each of the home device108that describe a status for each of the home devices108. For example, a home device108such as a printer may indicate a status of powered on and printing. In another example, a home device108of an oven may provide a status of powered on and operating at 400 degrees F. In response to receiving the media, sensor data, and the status from each of the home devices108, the data analyzer112provides a description of the detected weather from the media and property automation instructions to each of the monitor unit110. In some implementations, the data analyzer112provides the description of the detected weather from the media and the property automation instructions to a client device that corresponds to each of the monitored properties102.

In some implementations, the one or more sensors104can detect a presence or movement of an object, such as an individual, an animal, a vehicle, or an item, to name a few examples, at each of the monitored properties102. For example, the monitored property102may include a weight sensor under a doormat behind the front door to indicate a measurement of force applied downward on the doormat. For instance, the weight sensor may detect a change in weight that has been applied to the doormat when a property owner or a pet has stepped on the doormat. In other instances, the monitored property102may include one or more inductive loop sensors under a garage floor to detect a vehicle or movement of a vehicle entering or leaving a garage of the monitored property102. The inductive loop sensors may detect a vehicle passing over which induces eddy currents in the inductive loop sensors, decreases inductance of the inductive loop sensors, which generates a pulse to provide as a likelihood of vehicle detection/movement. The inductive loop sensors may also be placed underneath driveways and yards to detect weather patterns, such as winds blowing. In another example, the sensors104may include one or more motion sensors that detect movement, such as a PIR motion sensor. The motion sensors provide a likelihood that motion has occurred in the motion sensors field of view, whether the motion comes from a moving person, a moving animal, a falling object, a tree moving from weather activity, or a car moving towards or away from the monitored property. In another example, the sensors104may include one or more seismic sensors to measure seismic vibrations within proximity to the monitored property102. In this instance, the one or more seismic sensors may be placed in the driveway, garage, or even a room of the monitored property102to detect ground vibrations because of movement on top of the ground or beneath the ground.

Each of the one or more sensors104, the one or more home devices108, and the one or more cameras106, can provide a continuous stream of data to the monitor unit110. In other implementations, these devices can provide data to the monitor unit110upon request. In particular, the one or more sensors104can provide a continuous stream of data to the monitor control110upon a detection of a likelihood of movement or some other activity. The likelihood of detection may indicate object identification, object movement, or some other detected activity, depending on the type of sensor that provides the likelihood. For example, a sensor that detects weight may provide object identification when the sensor detects weight exceeding a specified weight threshold. Additionally, a PIR sensor may provide object movement detection when the PIR sensor detects movement in a frame. In some implementations, the one or more cameras106can provide the continuous stream of data to a local video storage device located at the monitor unit110.

In other implementations, the one or more sensors104may provide data to the monitor unit110only when a sensor has tripped (i.e., a sensor tripping indicates a strong likelihood of detection). The tripped sensor that streams data to the monitor unit110may provide data indicating a likelihood of detection in the format of voltage. For example, the data can indicate a voltage high, such as 3.3V, or a voltage low, such as 1.5V, depending on the likelihood of detection. For example, a high voltage value may indicate a likelihood of strong detection and a low voltage value may indicate no detection. In other implementations, the high voltage value may not indicate a likelihood of detection and a low voltage value may indicate a strong likelihood of detection. The sensor may additionally provide a timestamp to the monitor unit110indicating at time at which the detection or non-detection has occurred.

In some implementations, the monitor unit110cross-correlates data provided by the sensors104, the cameras106, and the home devices108. The cross-correlation of data from each of these devices helps improve the detection accuracy of the monitor unit110.

In some implementations, the monitor unit110may receive a continuous stream of media from the one or more cameras106. The received media stream can be real time video or image clips from the cameras106, such as camera111. The monitor unit110may receive the continuous stream of media in real time or after the camera111has recorded the footage. In addition, the one or more cameras106can time-tag the media footage for tracking and reassembling purposes by the monitor unit110. Typically, the one or more cameras106can also insert an ID number into each frame of streamed media to the monitor unit110for reassembly. By time stamping and providing IDs for each frame of media, the monitor unit110can cross correlate the media packets from the one or more cameras106with data provided by the one or more sensors104and data provided by the one or more home devices108.

The monitor unit110can determine a location of the one or more cameras106in the monitored property102based on the identified device ID provided by each camera. For example, the monitor unit110can receive a stream of media from a camera, such as camera111that has a device ID of 011. The monitor unit110can retrieve a device type from a device table stored in the monitor unit110to determine the device type and its corresponding location. For example, the device ID may indicate to the monitor unit110that the device providing the data stream is camera111and is located in the living room of the monitored property102. Upon moving the camera111around the monitored property, a property owner101can specify the newly moved location of the camera in the device table through an application of the client device103.

In some implementations, the one or more cameras106may be located in areas in the monitored property102that focuses on areas also monitored by the one or more sensors104. For example, the camera111-1may located outside of the front door of the monitored property102-1focused on the driveway and the sidewalk. In this instance, a motion sensor may also be placed outside the front door to detect motion of a person or object in close proximity to the camera111-1. In other instances, the camera111-1may be placed inside the monitored property102-1, such as in the living room, the basement, the kitchen, where other sensors from the one or more sensors104-1currently monitor for motion, weight, and presence of light.

In some implementations, the monitor unit110can provide data to the data analyzer112over network131. The data can include media120from monitor unit110-1, media124from monitor unit110-2, media122from monitor unit110-3, and media126from monitor unit110-4, recorded from each respective camera111. Additionally, each monitor unit110-1provides sensor data from the one or more sensors104and home device data from the one or more home devices108from each corresponding monitored property102.

The monitor unit110can provide media120,122,124, and126to the data analyzer112to generate a trained neural network model to detect weather patterns. In some implementations, the monitor unit110can provide the media120,122,124, and126at various times throughout the day. In other implementations, the monitor unit110can provide the media120,122,124, and126to the data analyzer112when the one or more sensors104indicate that a detection has occurred.

The media120,122,124, and126provided by each monitor unit110to the data analyzer112can include data as a packet. In particular, the packet data can include the media format provided by each particular camera from the one or more cameras106, the device ID for each of the one or more cameras106, a timestamp for each frame of the media, and a particular description of the location of the camera111found in the monitored property102. For example, the media120can include a 20 second video clip of the sky from camera111-1pointed outwards in a form such as MPEG, a device ID such as 001, timestamps from 4/13/2001 at 10:00:00 to 4/13/2001 at 10:00:20, and “OUTDOOR’ as a string to describe the location of camera111-1located in the monitored property102.

In some implementations, the monitor unit110also provides the sensor data from the one or more sensors104and the status of the home devices108to the data analyzer112. The monitor unit110can provide the sensor data and the status of the home devices108to the data analyzer112at the same time it provides the media. Alternatively, the monitor unit110can provide the sensor data and the status of the home devices108to the data analyzer112at a different time than the media.

In some implementations, the data analyzer112receives expected local weather116from various sources. The expected local weather116incorporates data from various weather sources such as, for example, weather forecast provided by various websites on the INTERNET, weather devices such as, for example, barometers to measure atmospheric pressure; thermometers to measure out door temperature; hygrometers to measure outdoor temperature and humidity; anemometers to the measure the direction and speed of wind; and, rain gauges to measure the amount of rainfall. The data from each of these devices and sources is provided as expected local weather forecast data to the data analyzer112for further analysis.

The data analyzer112can receive the media120,122,124,126, the expected local weather forecast116, and the sensor data over network131to store in its data in the visual data114. For instance, the visual data114stores a 20 second video clip by recorded by the camera111-1of a flower outside the monitored property102-1, motion data of motion detection of the flower moving, and weather data provided by the various weather websites and various weather devices for the zip code encompassing the monitored property102-1. Additionally, the data analyzer112can also store any other data provided by the sensors104and data provided by the status of the home devices108with weather data and the media data in the visual data114. For example, the stored data in the visual data114can include the 20 second video clip including a timestamp of 9/28/2018 10:00:00 to 9/28/2018 10:00:20, device ID of 001, “OUTSIDE” as a string to represent the location of the camera111-1in the monitored property102-1, weather data that indicates in zip code of 20001 the weather is sunny with a temperature of 75 degrees Fahrenheit. In other implementations, the visual data114stores previously recorded media that is classified by a weather label.

In some implementations, the data stored in the visual data114may be used to train a weather classifier stored in the data analyzer112. The data analyzer112may initiate training a model for the weather classifier using data stored in the visual data114and additional data received from the expected local weather116. For instance, the data analyzer112begins to train the weather classifier when a sufficient amount of data exists in the visual data114. The weather classifier may be trained to detect a weather pattern provided in the media data. For example, the weather classifier can be trained to detect the weather patterns in a frame of media data from a video clip or a frame of image data. The weather pattern can include a detection of snow, rain, hail, strong winds, sunny skies, cloudy skies, and nighttime, to name a few examples.

In some implementations, the output of the weather classifier can be augmented with the data from the expected local weather forecast116to increase the performance of the weather classifier. In other implementations, the weather classifier can receive the data from the expected local weather forecast116to improve the output of the weather classifier. For example, warm weather forecasts provided by the local weather forecast116could help the weather classifier differentiate between snowflakes blowing in the wind versus small flower petals blowing in the wind. With the results of the weather classifier, the data analyzer112can accurately provide, with a known confidence level, awareness of the current weather conditions at the location specified by the zip code.

The weather illustrated over geographic region one is sunny and bright while the weather illustrated over geographic region two is cloudy and rainy. The outdoor camera111-1records media120of various outdoor locations within proximity to the monitored property102-1in order to classify the weather. Additionally, the outdoor camera111-2records media124of outdoor locations within proximity to the monitored property102-2for weather classification. The monitor unit110-1transmits the recorded media120to the data analyzer112and the monitor unit110-2transmits the recorded media124to the data analyzer112for weather classification. The data analyzer112processes the recorded media120along with data from the expected local weather forecasts116of a corresponding zip code to produce a weather classification128. In response, the weather classifier in the data analyzer112determines the weather classification128to be “sunny,” as illustrated inFIG. 1. Additionally, the data analyzer112processes the recorded media124along with data from the expected local weather forecasts116for a corresponding zip code to produce a weather classification132. Like weather classification128, the weather classifier in the data analyzer112determines that the weather classification132is also “sunny.”

As shown in geographic region 2, the outdoor camera111-3records media122of outdoor locations within proximity to the monitored property102-3for weather classification. Additionally, the outdoor camera111-4records media126of outdoor locations within proximity to the monitored property102-4for weather classification. The monitor unit110-3transmits the recorded media122to the data analyzer112and the monitor unit110-4transmits the recorded media126to the data analyzer112for weather classification. The data analyzer112processes the recorded media122and the recorded media126along with data from the expected local weather forecasts116of a corresponding zip code to produce weather classifications130and134, respectively. The weather classifier in the data analyzer112, which processes the recorded media, determines the weather classification130to be “rainy” and the weather classification134to be “rainy.”

In some implementations, the data analyzer112provides the weather classifications128,130,132, and134to each respective monitor unit110. Alternatively, the data analyzer112can provide the weather classifications128,130,132, and134to each client device corresponding to the respective monitored property102.

In some implementations, the data analyzer112can additionally provide safety instructions to each monitor unit110. In other implementations, the data analyzer112can provide instructions to a client device of a property owner corresponding to the monitored property102. In particular, the data analyzer112can determine the safety instructions using the weather classification and data from the one more sensors104and the one or more home devices108. For example, if the data analyzer112determines that the localized weather is rainy and determines from the sensor data of a particular monitored property102that a door is open, the data analyzer112transmits an indication to a corresponding monitored control unit server110or to a client device instructing the property owner to close the door. This safety instruction helps protect the inside of monitored property102from damage due to current weather or potentially future weather.

FIG. 2Aillustrates a contextual diagram of an exemplary system200for training a neural network model using media and localized weather data. The system200illustrates a process for training the neural network model inside the data analyzer112. In particular, the neural network model can be a deep learning model for weather classification. The data analyzer112can train the weather classification model such that the weather classification model detects a type of weather in image data provided by the monitored property without the use of non-visual sensors. In order for the data analyzer112to train and generate the weather classification model, the data analyzer112needs a sufficient amount of training data such that the weather classification model can detect a weather type in visual image frames alone. In particular, the weather classification model may detect the weather type in the visual image frames regardless of the background in the visual image frames. For example, the weather classification model can detect snow in a foreground of a visual image when the background includes trees, houses, roads, and grass. In another example, the weather classification model can detect snow in a background or a mid-ground of a visual image when the foreground includes objects such as trees, humans, flowers, parks, and other objects.

In some implementations, the weather classification model will be able to distinguish between objects moving that look similar in a visual image. For example, the weather classification model will be able to determine a difference between snowflakes, rainfall, and flower petals blowing in the wind. Using expected local weather forecast data and previously labeled weather data, the weather classification model may intelligently distinguish between the objects using data that includes temperature, wind pressure, atmospheric pressure, color of object, and predicted weather.

In some implementations, the system200includes a combining module206. The combining module206receives weather labeled data202that includes previously labeled recorded media with a weather classification from the visual data114, recorded media204from a monitored property102, and data indicative of expected local weather forecast214. The combining module206correlates the recorded media204with the data indicative of the expected local weather forecast214. In particular, the data indicative of the expected local weather forecast214suggests the current weather within proximity to the monitored property102and more particularly, the weather shown in the frame of the recorded media204. For example, if the data indicative of the expected local weather forecast214suggests the weather to be currently raining, then the recorded media204may illustrate rain in its framed image data. The rain may be shown in the recorded media204in a foreground, mid-ground, or a background of the recorded media204. In some implementations, the combining module206can tag the portion of the recorded media that includes the weather. For example, snow shown in the foreground of the recorded media can be tagged with a label of snow with data indicative of the expected local weather116illustrating snow. In another example, rain shown in the background of the recorded media can be tagged with data indicative of the expected local weather116illustrating rain. This data indicative of the expected local weather116can include information from the weather web sites, data from the barometers, thermometers, and rain gauges corresponding to weather within proximity to the monitored property102.

In some implementations, the data analyzer112retrieves weather labeled data202from the visual data114based on data indicative of the expected local weather forecast214. For example, if the data indicative of the expected local weather forecast214illustrates rain, then the data analyzer112retrieves weather labeled data202from the visual data114of rainy weather. In some implementations, a type of weather classification indexes the data stored in the visual data114. For example, one row of data includes an indexed entry for rain as the type of weather from the expected local weather forecast, recorded media from a camera illustrating rain, sensor data corresponding to a time when the recorded media was captured, and data from the home devices corresponding to a time when the recorded media was captured.

In some implementations, the combining module206labels the recorded media204with the data indicative from the expected local weather forecast214. For example, the combining module206labels the recorded media204with the rainy weather forecast from the expected local weather forecast214. In particular, by labeling the recorded media204with a weather type, the model trainer210can instruct the trained visual model212to understand what type of weather is found in the recorded media204and where the particular weather is found in the recorded media204.

In some implementations, the combining module206additionally provides the weather labeled data202along with the labeled recorded media204as additional data used to train the weather classification model. The combining module206outputs video labeled with weather type208to provide to the model trainer210. The video labeled with weather type208includes labeled recorded media204and recorded media from the weather labeled data202. The video labeled with weather type208is provided to the model trainer210.

In some implementations, the model trainer210receives the video labeled with the weather type208that includes an image frame, of a detection of weather type, as designated by the expected local weather forecast214. The model trainer210uses data from the received video labeled with weather type208to train a neural network model or any other type of machine learning model. The neural network model may include an input layer, an output layer, and one or more hidden layers. The model trainer210may use a machine learning technique to continuously train the deep learning model as the model trainer210receives more data when applying the model. A deep learning model may be neural network model with one or more hidden layers between an input layer and an output layer. For example, the model trainer210may train a convolutional neural network model (CNN) or a recurrent neural network model (RNN). The model trainer210seeks to train the deep learning model with the video labeled with weather type208input such that the neural network model can recognize the presence or absence of a weather type in the image frame from the image frame alone. In essence, with enough training data of weather identification from the visual data114, the deep learning model should be able to detect a weather type, such as snow, distinguish the weather types from other weather types, and provide an accurate weather type to a property owner of a residential property, thus obviating the need for measuring tools to determine the weather type in other detection systems.

In some implementations, the model trainer210uses one or more sequential frames in the visual data114to train the deep learning model to detect a particular weather type. For example, by training the deep learning model to detect a particular weather type, such as rain, by analyzing one or more sequential frames in the visual data114, the deep learning model can detect individual rain droplets as they move across the camera's field of view. Additionally, the deep learning model can detect a particular weather type, such as snow, by analyzing the rate at which an object moves in the one or more sequential frames in the visual data114. In particular, by analyzing the rate at which objects move across the camera's field of view, the camera can determine whether the object is precipitation, such as rain or snow, based on the object's rate of speed from frame to another. The speed can be measured by determining the objects change in distance across a first frame and a second frame and dividing the distance by the time difference between the two frames. In particular, by the model trainer210measuring the time difference between a time-tag corresponding to the first frame and a time-tag corresponding to the second frame.

In some implementations, the model trainer210can use object identification in the one or more visual data114to screen out particular objects when identifying a weather classification. In particular, the model trainer210can identify one or more non-weather objects in the visual data114to assist the deep learning model with recognizing a particular weather type in the visual data114. For example, if an image frame in the visual data114includes two humans, two trees, and rain drops, the model trainer210can label the two humans and the two trees as non-weather objects in order to assist the deep learning model with focusing its efforts on detecting the rain drops. In other implementations, the model trainer210can use one or more of the non-weather objects to help in identifying the particular weather type. For example, the model trainer210can use a direction that the leaves of the two trees are moving to determine the direction the raindrops are moving. In addition, the model trainer210can detect to a change in the weather by analyzing the leaf movement on the two trees. For example, if in one frame the leaves of the two trees are moving in the wind, and in the second frame the leaves of the two trees are not moving, the deep learning model can determine the first frame of image data indicates windy weather and the second frame indicates non-windy weather.

In some implementations, the model trainer210may generate a weather classification model, such as a trained visual model212, after sufficiently training the neural network model. In some implementations, the data analyzer112may transmit the trained visual model212to one or more monitored properties102-1through102-4. In particular, the data analyzer112may transmit the trained visual model212to each monitored control unit server110corresponding to each monitored property102. In response to receiving the trained neural network model212, each monitored control unit server110will transmit the trained neural network model212to each camera111in the respective monitored property102. The processing for determining a detected weather type comes from the camera106when the trained visual model212is on the camera106. For instance, the trained visual model212may be stored in the camera111's memory and utilized by the camera111for weather classification without the use of other sensors or home devices.

In some implementations the system100for detecting a particular localized weather may use only a subset of the aforementioned components. In one example, implementations may exist that do not use the home device108. Similarly, other implementations may exist in which the visual data114is stored within the data analyzer112. Yet other alternative exemplary systems also fall within the scope of the present disclosure such as a system100that does not use a monitor unit110. For these reasons, the system100should not be viewed as limiting the present disclosure to any particular set of necessary components.

FIG. 2Billustrates a contextual diagram of an exemplary system201for utilizing a trained neural network model to detect localized weather and providing control instructions for security measures. In some implementations, the system201may be implemented on the data analyzer112. In other implementations, the system201may be implemented on each camera106in a monitored property102. In other implementations, the system201may be implemented in each monitor unit110. In some implementations, a pre-trained weather classifier model218is preloaded onto the data analyzer112(or the camera106). The pre-trained weather classifier model218can be adaptively retrained with incorrect and correct weather classification data from the monitor unit110and/or the visual data114.

In some implementations, a camera, such as camera111-1, may record real time footage of an area within proximity to the monitored property102-1. For example, the camera111-1may record footage216of a view of the backyard of the monitored property102-1. In some implementations, the camera111-1may provide each frame of the recorded footage216to the trained visual model218in memory. In other implementations, the camera111-1may provide each frame of the recorded footage216to the trained visual model218on the data analyzer112over the network131. As illustrated in system201, a frame of the recorded media214is provided to the trained visual model218. In some implementations, data indicative of the expected local weather forecast216is provided to the trained visual model218to supplement the trained visual model218's accuracy of a weather prediction. The expected local weather forecast216is an optional input and not necessary for the trained visual model218to output a weather classification220from the recorded media216.

In some implementations, if the trained visual model218produces an output indicating a weather classification220in response to analyzing a frame from the recorded media214, the camera111may provide an indication of the weather classification220to the monitored control unit server110. Similarly, if the trained visual model218executes on the data analyzer112, the data analyzer112produces a weather classification220and provides the weather classification220to the corresponding monitored control unit server110. In some implementations, the trained visual model218, whether on the data analyzer112or on the camera111, provides the indication of the weather classification220to the client device corresponding to the monitored property102. In other implementations, in response to the monitor unit110receiving the notification of the weather classification220, the monitor unit110may transmit a notification to the client device corresponding to that monitored property102. For example, the notification may include a push notification, a short message service (SMS), or an instant message. In other implementations, the monitor unit110may transmit the live video feed of the camera111-1to the client device103for real time viewing purposes along with a current label indicating the notification of the weather classification220.

In some implementations, the data analyzer112can use the weather classification from each of the monitored properties to generate a weather model of a geographic region. In particular, although geographic one region is clearly indicated to have sunny weather, as illustrated by the sun shown over geographic region one, portions of geographic region one may not be sunny. For example, the data analyzer112may produce a weather classification128that is sunny for the monitored property102-1and a weather classification132that is cloudy for the monitored property102-2. The data analyzer112can aggregate this weather classification information from each monitored property102to create a weather classification model from each area. In particular, the weather classification model can update in real time such that a property owner can see weather patterns according to weather classifications. In other implementations, the weather classification model can update periodically.

In some implementations, a property owner can view the weather classification model overlaid on top of a map. The property owner can view this interface by interacting with a smart home application on his or her client device, such as client device103. The overlay may include labels and pictures of a type of weather classification over particular areas of the map. In addition, the property owner can select on the pictures and labels over the particular areas to view additional information. The additional information can include data provided by the expected local weather forecast116. This includes website and weather device data accompanied by the frame of recorded media.

In some implementations, if the trained visual model218executes on the data analyzer112, the data analyzer112may receive feedback from each of the monitored properties102to tune the trained visual model218. In particular, the feedback data may be similar data utilized to train the neural network model by the model trainer210. In other implementations, the feedback data may include a correction of the labeled data along with the recorded media data to tune the trained visual model218. For example, the data analyzer112provides an indication of a weather classification to the client device103of property owner101. The property owner101reviews on his or her client device103the indication of the weather classification from the data analyzer112that includes the recorded media120and a label of the weather classification corresponding to the output of the trained visual model218.

After reviewing the indication of the weather classification, the property owner101determines that the label of the weather classification is incorrect. The property owner101can interact with the application on the client device103to indicate that the determined weather is not correct. For example, the property owner101can indicate that the labeled weather of “snow” is not correct, and the actual weather in the recorded media is “rain.” In response, the client device103transmits the corrected label and the recorded media to the monitor unit110-1. The monitor unit110-1provides an indication to the data analyzer112that the trained visual model218incorrectly detected weather in the recorded media. As a result, the data analyzer112provides the newly received recorded media216and corrected classification label to the model trainer210in the same manner used to train the neural network model. In response, the data analyzer112may then provide the updated trained visual model212to each of the cameras111for an improved and updated weather classification detection.

In some implementations, the data analyzer112can use the weather classification220, data222from sensors104, and data224from home devices to determine one or more safety instructions to provide to a corresponding monitored property102. In particular, the data analyzer112includes a control module226that processes the weather data, the sensor data, and the home device data, to determine the one or more safety instructions to provide to the monitored property102. For example, the weather classification220indicates to the data analyzer112that the current weather is rain, the sensor data222indicates to the data analyzer112that the front door of the monitored property102-1is open, and the sensors data222indicates that a self-automated vacuum is currently vacuuming behind the front door. The control module226determines that the weather classification220includes an indication of weather that could potentially damage a portion of an interior or exterior of the monitored property102. In addition, should any of monitored property's doors or windows be unlocked or open, the control module226should notify the corresponding property owner. In this instance, since the control module226receives an indication that the front door is open and that the weather currently forecasts rain, the control module226determines that the property owner is to be notified. As such, the control module226provides a safety instruction in the instructions and weather message228to provide to the client device of the corresponding property owner.

In addition, the control module226correlates the location of the self-automated vacuum in the monitored property102to the indication that the front door of the monitored property102is open. As such, the control module226provides an additional notification in the instructions and weather message228to move the self-automated vacuum away from the front door and to close the front door. Additionally, the control module226can determine to provide a message to the property owner to shut and lock one or more opened windows in the monitored property102. In other instances, the control module226can determine to provide a message to property owner102to move car associated with the monitored property102inside the garage when the weather is hail or snow. Other examples are possible for the control module226to protect the valuables of his monitored property102when the trained visual model218has identified a potentially damaging weather classification.

In some implementations, the control module226provides instructions that can automatically interact with items in a monitored property102. In particular, the control module226can provide automation instructions in the instructions and weather message228for the monitor unit110to automatically close, lock, or turn off the front door, windows, or the one or more home devices108. For example, the control module226can provide an automation instruction to the monitor unit110to turn off a grill left on located outside the monitored property102when the weather classifier outputs determines the weather is raining from the recorded media. In another example, the control module226can provide an automation instruction to the monitor unit110close and lock the front door of the monitored property102when the weather classifier outputs determines the weather is raining from the recorded media. In another example, the control module226can provide an automation instruction to the monitor unit110to close the garage door when the weather classifier determines the weather is hailing from the recorded media.

In some implementations, in response to the monitor unit110receiving the automation instruction228from the data analyzer112, the monitor unit110transmits a notification to the corresponding device for action. For example, if the automation instruction228includes an indication to close the front door because the current local weather is raining, the monitor unit110will automatically close the front door. In another example, if the automation instruction includes an indication to turn off the grill because it is raining, the monitor unit110will automatically turn off the grill. In some implementations, the monitor unit110can communicate with the house doors, garage doors, and windows along with the sensors104, camera106, and home devices108.

FIG. 3illustrates an example of a monitoring system300for weather classification. The system300includes a network308, a monitoring system control unit350, one or more user devices320,324, a monitoring application server310, and a central alarm station server306. In some examples, the network308facilitates communications between the monitoring system control unit350, the one or more user devices320,324the monitoring application server310, and the central alarm station server306.

The network308is configured to enable exchange of electronic communications between devices connected to the network308. For example, the network308may be configured to enable exchange of electronic communications between the monitoring system control unit350, the one or more user devices320,324, the monitoring application server310, and the central alarm station server306. The network308may include, for example, one or more of the Internet, Wide Area Networks (WANs), Local Area Networks (LANs), analog or digital wired and wireless telephone networks (e.g., a public switched telephone network (PSTN), Integrated Services Digital Network (ISDN), a cellular network, and Digital Subscriber Line (DSL)), radio, television, cable, satellite, or any other delivery or tunneling mechanism for carrying data. The network308may include multiple networks or subnetworks, each of which may include, for example, a wired or wireless data pathway. The network308may include a circuit-switched network, a packet-switched data network, or any other network able to carry electronic communications (e.g., data or voice communications). For example, the network308may include networks based on the Internet protocol (IP), asynchronous transfer mode (ATM), the PSTN, packet-switched networks based on IP, X.25, or Frame Relay, or other comparable technologies and may support voice using, for example, VoIP, or other comparable protocols used for voice communications. The network308may include one or more networks that include wireless data channels and wireless voice channels. The network308may be a wireless network, a broadband network, or a combination of networks including a wireless network and a broadband network.

The monitoring system control unit350includes a controller358and a network module352. The controller358is configured to control a monitoring system (e.g., a home alarm or security system) that includes the monitoring system control unit350. In some examples, the controller358may include a processor or other control circuitry configured to execute instructions of a program that controls operation of an alarm system. In these examples, the controller358may be configured to receive input from sensors, detectors, or other devices included in the alarm system and control operations of devices included in the alarm system or other household devices (e.g., a thermostat, an appliance, lights, etc.). For example, the controller358may be configured to control operation of the network module352included in the monitoring system control unit350.

The network module352is a communication device configured to exchange communications over the network308. The network module352may be a wireless communication module configured to exchange wireless communications over the network308. For example, the network module352may be a wireless communication device configured to exchange communications over a wireless data channel and a wireless voice channel. In this example, the network module352may transmit alarm data over a wireless data channel and establish a two-way voice communication session over a wireless voice channel. The wireless communication device may include one or more of a GSM module, a radio modem, cellular transmission module, or any type of module configured to exchange communications in one of the following formats: LTE, GSM or GPRS, CDMA, EDGE or EGPRS, EV-DO or EVDO, UMTS, or IP.

The network module352also may be a wired communication module configured to exchange communications over the network308using a wired connection. For instance, the network module352may be a modem, a network interface card, or another type of network interface device. The network module352may be an Ethernet network card configured to enable the monitoring system control unit350to communicate over a local area network and/or the Internet. The network module352also may be a voiceband modem configured to enable the alarm panel to communicate over the telephone lines of Plain Old Telephone Systems (POTS).

The monitoring system that includes the monitoring system control unit350includes one or more sensors or detectors. For example, the monitoring system may include multiple sensors312. The sensors312may include a contact sensor, a motion sensor, a glass break sensor, or any other type of sensor included in an alarm system or security system. The sensors312also may include an environmental sensor, such as a temperature sensor, a water sensor, a rain sensor, a wind sensor, a light sensor, a smoke detector, a carbon monoxide detector, an air quality sensor, etc. The sensors312further may include a health monitoring sensor, such as a prescription bottle sensor that monitors taking of prescriptions, a blood pressure sensor, a blood sugar sensor, a bed mat configured to sense presence of liquid (e.g., bodily fluids) on the bed mat, etc. In some examples, the sensors312may include a radio-frequency identification (RFID) sensor that identifies a particular article that includes a pre-assigned RFID tag.

The monitoring system control unit350communicates with the module314and the camera362to perform surveillance or monitoring. The module314is connected to one or more lighting systems and is configured to control operation of the one or more lighting systems. The module314may control the one or more lighting systems based on commands received from the monitoring system control unit350. For instance, the module314may cause a lighting system to illuminate an area to provide a better image of the area when captured by a camera362.

The camera362may be a video/photographic camera or other type of optical sensing device configured to capture images. For instance, the camera362may be configured to capture images of an area within a building monitored by the monitoring system control unit350. The camera362may be configured to capture single, static images of the area and also video images of the area in which multiple images of the area are captured at a relatively high frequency (e.g., thirty images per second). The camera362may be controlled based on commands received from the monitoring system control unit350.

The camera362may be triggered by several different types of techniques. For instance, a Passive Infra-Red (PIR) motion sensor may be built into the camera362and used to trigger the camera362to capture one or more images when motion is detected. The camera362also may include a microwave motion sensor built into the camera and used to trigger the camera362to capture one or more images when motion is detected. The camera362may have a “normally open” or “normally closed” digital input that can trigger capture of one or more images when external sensors (e.g., the sensors312, PIR, door/window, etc.) detect motion or other events. In some implementations, the camera362receives a command to capture an image when external devices detect motion or another potential alarm event. The camera362may receive the command from the controller358or directly from one of the sensors312.

In some examples, the camera362triggers integrated or external illuminators (e.g., Infra-Red, Z-wave controlled “white” lights, lights controlled by the module314, etc.) to improve image quality when the scene is dark. An integrated or separate light sensor may be used to determine if illumination is desired and may result in increased image quality.

The camera362may be programmed with any combination of time/day schedules, system “arming state”, or other variables to determine whether images should be captured or not when triggers occur. The camera362may enter a low-power mode when not capturing images. In this case, the camera362may wake periodically to check for inbound messages from the controller358. The camera362may be powered by internal, replaceable batteries if located remotely from the monitoring system control unit350. The camera362may employ a small solar cell to recharge the battery when light is available. Alternatively, the camera362may be powered by the controller's358power supply if the camera362is co-located with the controller358.

The sensors312, the module314, the camera362, and the extended camera354communicate with the controller358over communication links316,318,356, and360. The communication links316,318,356, and360may be a wired or wireless data pathway configured to transmit signals from the sensors312, the module314, and the camera362to the controller358. The sensors312, the module314, and the camera362may continuously transmit sensed values to the controller358, periodically transmit sensed values to the controller358, or transmit sensed values to the controller358in response to a change in a sensed value.

The communication link356over which the extended camera354and the controller358communicate may include a local network. The extended camera354and the controller358may exchange images and commands over the local network. The local network may include 802.11 “WiFi” wireless Ethernet (e.g., using low-power WiFi chipsets), Z-Wave, Zigbee, Bluetooth, “Homeplug” or other “Powerline” networks that operate over AC wiring, and a Category 5 (CATS) or Category 6 (CAT6) wired Ethernet network.

The monitoring application server310is an electronic device configured to provide monitoring services by exchanging electronic communications with the monitoring system control unit350, the one or more user devices320,324, and the central alarm station server306over the network308. For example, the monitoring application server310may be configured to monitor events (e.g., alarm events) generated by the monitoring system control unit350. In this example, the monitoring application server310may exchange electronic communications with the network module352included in the monitoring system control unit350to receive information regarding events (e.g., alarm events) detected by the monitoring system control unit350. The monitoring application server310also may receive information regarding events (e.g., alarm events) from the one or more user devices320,324.

In some examples, the monitoring application server310may route alarm data received from the network module352or the one or more user devices320,324to the central alarm station server306. For example, the monitoring application server310may transmit the alarm data to the central alarm station server306over the network308.

The monitoring application server310may store sensor and image data received from the monitoring system and perform analysis of sensor and image data received from the monitoring system. Based on the analysis, the monitoring application server310may communicate with and control aspects of the monitoring system control unit350or the one or more user devices320,324.

The central alarm station server306is an electronic device configured to provide alarm monitoring service by exchanging communications with the monitoring system control unit350, the one or more user devices320,324, and the monitoring application server310over the network308. For example, the central alarm station server306may be configured to monitor alarm events generated by the monitoring system control unit350. In this example, the central alarm station server306may exchange communications with the network module352included in the monitoring system control unit350to receive information regarding alarm events detected by the monitoring system control unit350. The central alarm station server306also may receive information regarding alarm events from the one or more user devices320,324.

The central alarm station server306is connected to multiple terminals302and304. The terminals302and304may be used by operators to process alarm events. For example, the central alarm station server306may route alarm data to the terminals302and304to enable an operator to process the alarm data. The terminals302and304may include general-purpose computers (e.g., desktop personal computers, workstations, or laptop computers) that are configured to receive alarm data from a server in the central alarm station server306and render a display of information based on the alarm data. For instance, the controller358may control the network module352to transmit, to the central alarm station server306, alarm data indicating that a sensor312detected a door opening when the monitoring system was armed. The central alarm station server306may receive the alarm data and route the alarm data to the terminal302for processing by an operator associated with the terminal302. The terminal302may render a display to the operator that includes information associated with the alarm event (e.g., the name of the user of the alarm system, the address of the building the alarm system is monitoring, the type of alarm event, etc.) and the operator may handle the alarm event based on the displayed information.

In some implementations, the terminals302and304may be mobile devices or devices designed for a specific function. AlthoughFIG. 3illustrates two terminals for brevity, actual implementations may include more (and, perhaps, many more) terminals. For instance, the client devices103,107,113, and117may be the illustrative examples of the terminals.

The one or more user devices320,324are devices that host and display user interfaces. For instance, the user device324is a mobile device that hosts one or more native applications (e.g., the native surveillance application346). The user device324may be a cellular phone or a non-cellular locally networked device with a display. The user device324may include a cell phone, a smart phone, a tablet PC, a personal digital assistant (“PDA”), or any other portable device configured to communicate over a network and display information. For example, implementations may also include Blackberry-type devices (e.g., as provided by Research in Motion), electronic organizers, iPhone-type devices (e.g., as provided by Apple), iPod devices (e.g., as provided by Apple) or other portable music players, other communication devices, and handheld or portable electronic devices for gaming, communications, and/or data organization. The user device324may perform functions unrelated to the monitoring system, such as placing personal telephone calls, playing music, playing video, displaying pictures, browsing the Internet, maintaining an electronic calendar, etc.

The user device324includes a native surveillance application346. The native surveillance application346refers to a software/firmware program running on the corresponding mobile device that enables the user interface and features described throughout. The user device324may load or install the native surveillance application346based on data received over a network or data received from local media. The native surveillance application346runs on mobile devices platforms, such as iPhone, iPod touch, Blackberry, Google Android, Windows Mobile, etc. The native surveillance application346enables the user device324to receive and process image and sensor data from the monitoring system

The user device320may be a general-purpose computer (e.g., a desktop personal computer, a workstation, or a laptop computer) that is configured to communicate with the monitoring application server310and/or the monitoring system control unit350over the network308. The user device320may be configured to display a surveillance monitoring user interface322that is generated by the user device320or generated by the monitoring application server310. For example, the user device320may be configured to display a user interface (e.g., a web page) provided by the monitoring application server310that enables a user to perceive images captured by the camera362and/or reports related to the monitoring system. AlthoughFIG. 3illustrates two user devices for brevity, actual implementations may include more (and, perhaps, many more) or fewer user devices.

In some implementations, the one or more user devices320,324communicate with and receive monitoring system data from the monitoring system control unit350using the communication link348. For instance, the one or more user devices320,324may communicate with the monitoring system control unit350using various local wireless protocols such as Wi-Fi, Bluetooth, zwave, zigbee, HomePlug (ethernet over powerline), or wired protocols such as Ethernet and USB, to connect the one or more user devices320,324to local security and automation equipment. The one or more user devices320,324may connect locally to the monitoring system and its sensors and other devices. The local connection may improve the speed of status and control communications because communicating through the network308with a remote server (e.g., the monitoring application server310) may be significantly slower.

Although the one or more user devices320,324are shown as communicating with the monitoring system control unit350, the one or more user devices320,324may communicate directly with the sensors and other devices controlled by the monitoring system control unit350. In some implementations, the one or more user devices320,324replace the monitoring system control unit350and perform the functions of the monitoring system control unit350for local monitoring and long range/offsite communication.

In other implementations, the one or more user devices320,324receive monitoring system data captured by the monitoring system control unit350through the network308. The one or more user devices320,324may receive the data from the monitoring system control unit350through the network308or the monitoring application server310may relay data received from the monitoring system control unit350to the one or more user devices320,324through the network308. In this regard, the monitoring application server310may facilitate communication between the one or more user devices320,324and the monitoring system.

In some implementations, the one or more user devices320,324may be configured to switch whether the one or more user devices320,324communicate with the monitoring system control unit350directly (e.g., through link348) or through the monitoring application server10(e.g., through network308) based on a location of the one or more user devices320,324. For instance, when the one or more user devices320,324are located close to the monitoring system control unit350and in range to communicate directly with the monitoring system control unit350, the one or more user devices320,324use direct communication. When the one or more user devices320,324are located far from the monitoring system control unit350and not in range to communicate directly with the monitoring system control unit350, the one or more user devices320,324use communication through the monitoring application server310.

Although the one or more user devices320,324are shown as being connected to the network308, in some implementations, the one or more user devices320,324are not connected to the network308. In these implementations, the one or more user devices320,324communicate directly with one or more of the monitoring system components and no network (e.g., Internet) connection or reliance on remote servers is needed.

In some implementations, the one or more user devices320,324are used in conjunction with only local sensors and/or local devices in a house. In these implementations, the system300only includes the one or more user devices320,324, the sensors312, the module314, and the camera362. The one or more user devices320,324receive data directly from the sensors312, the module314, and the camera362and sends data directly to the sensors312, the module314, and the camera362. The one or more user devices320,324provide the appropriate interfaces/processing to provide visual surveillance and reporting.

In other implementations, the system300further includes network308and the sensors312, the module314, and the camera362are configured to communicate sensor and image data to the one or more user devices320,324over network308(e.g., the Internet, cellular network, etc.). In yet another implementation, the sensors312, the module314, and the camera362(or a component, such as a bridge/router) are intelligent enough to change the communication pathway from a direct local pathway when the one or more user devices320,324are in close physical proximity to the sensors312, the module314, and the camera362to a pathway over network308when the one or more user devices320,324are farther from the sensors312, the module314, and the camera362. In some examples, the system leverages GPS information from the one or more user devices320,324to determine whether the one or more user devices320,324are close enough to the sensors312, the module314, and the camera362to use the direct local pathway or whether the one or more user devices320,324are far enough from the sensors312, the module314, and the camera362that the pathway over network308is required. In other examples, the system leverages status communications (e.g., pinging) between the one or more user devices320,324and the sensors312, the module314, and the camera362to determine whether communication using the direct local pathway is possible. If communication using the direct local pathway is possible, the one or more user devices320,324communicate with the sensors312, the module314, and the camera362using the direct local pathway. If communication using the direct local pathway is not possible, the one or more user devices320,324communicate with the sensors312, the module314, and the camera362using the pathway over network308.

In some implementations, the system300provides end users with access to images captured by the camera362to aid in decision making. The system300may transmit the images captured by the camera362over a wireless WAN network to the user devices320,324. Because transmission over a wireless WAN network may be relatively expensive, the system300uses several techniques to reduce costs while providing access to significant levels of useful visual information.

In some implementations, a state of the monitoring system and other events sensed by the monitoring system may be used to enable/disable video/image recording devices (e.g., the camera362). In these implementations, the camera362may be set to capture images on a periodic basis when the alarm system is armed in an “Away” state, but set not to capture images when the alarm system is armed in a “Stay” state or disarmed. In addition, the camera362may be triggered to begin capturing images when the alarm system detects an event, such as an alarm event, a door opening event for a door that leads to an area within a field of view of the camera362, or motion in the area within the field of view of the camera362. In other implementations, the camera362may capture images continuously, but the captured images may be stored or transmitted over a network when needed.

In some implementations, all of the processing described throughout this disclosure may be implemented in a monitoring system control panel located inside the property being monitored, as opposed to sending data to an external server for processing. For example, rather than being a separate server located in a remote location, the monitoring application server310may be a logical component inside of the monitoring system control unit350. In this example, the monitoring system control unit350performs the processing of supervising property access without having to send image/video data to a separate server separated by a network.

In other implementations, all of the processing described throughout this disclosure may be performed on a remote server (e.g., monitoring application server310). In these implementations, the monitoring system control panel (or sensors themselves) may send image/video data to the remote server over a network and the remote server may perform all of supervising property access. For instance, the monitoring system control unit350sends all captured image/video data to the monitoring application server310and the monitoring application server310performs the processing of supervising property access.

In still further implementations, the processing described throughout this disclosure may be mixed between a monitoring system control panel and a remote server. In these implementations, the monitoring system control panel and the remote server may share operations needed to analyze the sensor data. For instance, the monitoring system control panel may perform the interpretation of image/video data collected relatively recently (e.g., image/video data collected within the past three months) and the remote server may perform the detection of patterns in image/video data collected over a longer period of time (e.g., image/video data collected over the past several years). Alternatively, the monitoring system control panel may perform pre-processing of the image/video data, including collection and aggregation of the image/video data, and the remote server may perform the detailed analysis of detecting patterns within the image/video data. In the example shown inFIG. 3, the processing described throughout this disclosure may be mixed between the monitoring system control unit350and the monitoring application server310.

FIG. 4is a flowchart of an example process400for utilizing a trained neural network model that detects localized weather and provides automated instructions to devices of a corresponding area. Generally, the process400includes obtaining data from one or more expected local weather devices and a camera from a first area; based on the obtained data from the expected local weather devices and the camera, pairing expected local weather forecasts from the expected local weather devices with corresponding media from the camera; providing the expected local weather forecasts paired with the corresponding media to a weather classifier to determine a current weather prediction in proximity to a geographic area of the first area; providing the current weather prediction paired and obtained sensor data from the first area to a control module to determine property automation instructions to provide to a client device and a home device both corresponding with the first area; and providing the determined property automation instructions and the current weather prediction to the client device and the home device of the first area. Alternatively, the process400can be used for utilizing a trained neural network model for other implementations for one or more cameras, such as identifying weather damage, to name an example. The process400will be described as being performed by a computer system comprising one or more computers, for example, the system100as shown inFIG. 1or the computing system200as shown inFIGS. 2A and 2B.

During402, the system obtains data from one or more expected local weather devices and a camera from a first area. In some implementations, the data analyzer112receives media from each monitor unit110recorded by a corresponding camera111. For instance, the monitor unit110-1of monitored property102-1provides recorded media120, the monitor unit110-2of monitored property102-2provides recorded media124, the monitor unit110-3of monitored property102-3provides recorded media122, and the monitor unit110-1of monitored property102-4provides recorded media126to the data analyzer112. The recorded media can be real time video or image frames from each of the cameras106in a respective monitored property102. In some implementations, the monitor unit110can transmit real-time recorded media from each of the cameras106. In other implementations, the monitor unit110can transmit post-recorded media from each of the cameras106to the data analyzer112.

In some implementations, the data analyzer112receives expected local weather116from various weather sources. For instances, the expected local weather116acquires sensor data of weather forecasting from various weather sources, such as for example, weather forecasts from an expected local weather source, weather data from barometers, thermometers, hygrometers, anemometers, and rain gauges for weather analysis. The expected local weather116retrieves weather data for weather in proximity and encompassing the geographic regions. For instance, the data analyzer112knows geographic region 1 is found within zip code 11111 and geographic region 2 is found within zip code 11112. The data analyzer112uses the location information of each geographic region to determine a corresponding location for the weather data to retrieve from the expected local weather116.

During404, the system, based on the obtained data from the expected local weather devices and the camera, pairs expected local weather forecasts from the expected local weather devices with corresponding media from the camera. In some implementations, the combining module206within the data analyzer112correlates recorded media from each monitor unit corresponding to a monitored property with the data indicative of the expected local weather forecast214. For example, the data from the one or more websites, data from the barometers, thermometers, anemometers, and rain gauges suggests the weather within proximity to the monitored property102, and more particular, to the weather shown in each frame of the recorded media from each monitored property. In order for the weather devices (e.g., barometers, thermometers, anemometers, and rain gauges) to accurately forecast weather within proximity to a particular monitored property102, the weather devices need to also be within proximity to the particular monitored property102.

In some implementations, the weather can be shown in a particular portion of the recorded media. In particular, the weather can be shown in a foreground, in a mid-ground, or a background of the media. For example, the snow, which is shown in the foreground, is front of a tree, which is shown in the background, of the recorded media. During training of the trained neural network model, the combining module206can tag the portion of the recorded media that includes the weather. For example, if the recorded media includes snow in the background, then the combining module206tags the snow in the background with one or more labels. This allows the model trainer210to properly train the neural network model to detect a particular weather in any portion of the recorded media.

During406, the system provides the expected local weather forecasts paired with the corresponding media to a weather classifier to determine a current weather prediction in proximity to a geographic area of the first area. In some implementations, a camera, such as camera111-1, corresponding to monitored property102-1provides recorded media to a weather classifier (e.g., trained visual model) to produce a current weather prediction. In particular, the camera111-1may provide each frame of the recorded media to the trained visual model in the data analyzer112over the network131. In some implementations, data indicative of the expected local weather forecast216is provided to the trained visual model along with the frame of the recorded media to supplement the trained visual model's accuracy of the weather classification. For example, the trained visual model will receive one or more frames of the recorded media illustrating a backyard of the monitored property102-1along with data indicative of the expected local weather forecast216that describe sunny weather within proximity to the monitored property102-1.

During408, the system provides the current weather prediction paired and obtained sensor data from the first area to a control module to determine property automation instructions to provide to a client device and a home device both corresponding with the first area. In some implementations, the trained visual model outputs a weather classification in response to analyzing a frame of the recorded media along with the data indicative of the expected local weather forecast216. In other implementations, the data indicative of the expected local weather forecast216is provided as an optional feature to the trained visual model. In other implementations, the data indicative of the expected local weather forecast216is augmented with the output of the weather classification to check the correctness of the weather classification. For instance, the data indicative of the expected local weather forecast216can provide a strong indication of the weather forecast, especially the data pulled from one or more weather websites. As a result, the website data can be compared to the weather classification to determine the trained visual model's accuracy.

In some implementations, the data analyzer112can use the output weather classification220, sensors data222, and home device data224to determine one or more safety automation instructions to provide to the corresponding monitored property102. For instance, the data analyzer112includes a control module226that processes the output weather classification220, sensor data22, and home device data224to determine the one or more safety automation instructions. For example, the trained visual model produces a weather classification of rain within proximity to the monitored property102-1, the sensor data222provides an indication that the garage door of the monitored property102-1is open, and the home device data224indicates that all devices are powered down. In some implementations, the control module226determines that the weather classification220indicates that the type of weather could potentially damage a monitored property. If the control module226notices any of the monitored property's doors, windows, or garages are unlocked or opened, the control module226will generate an instruction to notify the property owner of the corresponding monitored property.

During410, the system provides the determined property automation instructions and the current weather prediction to the client device and the home device of the first area. In some implementations, the control module226adds the determined property automation instruction to the message228to provide to the monitor unit110. In some implementations, the data analyzer112transmits the message228to the monitor unit110corresponding to the monitored property102. In other implementations, the data analyzer112transmits the message228to the client device corresponding to the property owner of the monitored property102. In response to the monitor unit110receiving the automation instruction228from the data analyzer112, the monitor unit server110transmits a notification to the corresponding device for action. For example, if the automation instruction228includes an indication to the close the front door of the monitored property102-1because the local weather forecasts rain, the monitor unit110automatically closes the front door. In other implementation, when the client device corresponding to the property owner of the monitored property102receives the message228from the data analyzer112, the property owner is prompted with message228in order to perform the requested action determined by the control module226.

FIG. 5is a flowchart of another example process500for utilizing a trained neural network model that detects localized weather and provides automated instructions to devices of a monitored property. Generally, the process500includes obtaining images from cameras located at the monitored property; obtaining an expected weather forecast and an actual weather condition for the monitored property; training a machine learning model to classify a current weather condition for the monitored property using the images from the cameras, the expected weather forecast, and the actual weather condition; and, obtaining a weather condition from the trained machine learning model that indicates a particular weather condition at the monitored property based on one or more images from a camera and the expected local weather forecast at the monitored property. The process500will be described as being performed by a computer system comprising one or more computers, for example, the system100as shown inFIG. 1or the computing system200as shown inFIGS. 2A and 2B.

During502, the system obtains images from cameras located at the monitored property. In some implementations, the data analyzer112receives media from a monitor unit at a monitored property. The data analyzer112can receive media from each of the monitor units from each monitored property. The monitor unit at each monitored property provides media received from its cameras to the data analyzer. For example, as illustrated in system100, the data analyzer112receives media data120from monitor unit110-1, which received the media data120from a camera111-1or multiple cameras106-1. The media can include real time video footage or images from the cameras106at a corresponding monitored property. In some implementations, the data analyzer112can receive post-recorded media from each of the monitor units110.

The media can included recorded footage or images of areas within a monitored property. For example, the areas can include areas surrounding the interior and exterior of the front door of the monitored property; areas surrounding a backyard of the monitored property; areas within the monitored property, such as a living room, basement, and bedroom. The media may illustrate a deck area of the monitored property or areas within the monitored property that include one or more windows with a view to the outside of the monitored property. The media can include recorded images and recorded footage.

During504, the system obtains an expected weather forecast and an actual weather condition for the monitored property. For example, the data analyzer can acquire the expected weather forecast from a third party resource, such as a local weather source or the Internet. Additionally, the data analyzer retrieves an actual weather condition for the monitored property from various weather sources. For example, the data analyzer can retrieve pressure data from barometers, temperature data from thermometers, humidity and water data from hygrometers, wind speed data from anemometers, and rain data from rain gauges. The data analyzer can acquire the actual weather condition data for regions in proximity to a particular monitored property. For example, the data analyzer112determines that geographic region 1 has a zip code of 12345 and geographic region 2 has a zip code of 12346. Then, the data analyzer112retrieves expected local forecast data and actual weather condition data corresponding to the zip codes of those geographic regions.

During506, the system trains a machine-learning model to classify a current weather condition for the monitored property using the images from the cameras, the expected weather forecast, and the actual weather condition. In particular, the data analyzer112includes a model trainer that provides the media data from the cameras, an expected weather forecast, and an actual weather condition to train a neural network model or any other type of machine-learning model. The model trainer trains the machine-learning model with the media data, the data describing the forecasted weather, and the actual weather to recognize the presence or absence of a particular weather condition in an image frame or multiple image frames provided by a camera. With sufficient training data, the machine-learning model can generate a particular weather condition, e.g., snow, rain, or sleet. In response, the data analysis can provide the generated particular weather condition to a property owner of the monitored property, thus obviating the need for weather measuring tools to determine the weather.

In some implementations, the data analysis pairs the images from the cameras with data identifying the expected weather forecasts for the monitored property. For example, the data analyzer112includes a combining module206that receives recorded media from cameras at a monitored property and data indicative of an expected weather forecast. The combining module206correlates and combines the recorded media with the data indicative of the expected weather forecast. The recorded media may illustrate a particular weather condition in a foreground, a mid-ground, or a background of its image or video footage. For example, the combining module206tags or labels the particular weather condition in the recorded media with the expected weather forecast. For example, the combining module206can tag snow shown in the foreground of the recorded media with a label of data indicative of the expected local weather, e.g., snow. In another example, the combining module206can tag rain shown in the background of the recorded media with a label of data indicative of the expected local weather, e.g., rain.

The combining module206then provides the tagged media data to the model trainer for training the machine-learning model. The combining module206can also provide an indication of the actual weather condition for the monitored property to the model trainer. The actual weather condition describes the current weather condition at the monitored property. The actual weather condition can be different from the expected weather forecast. For example, the expected weather forecast may indicate rainy weather at the monitored property, while the actual weather condition is sunny. By providing the actual weather condition with the expected weather forecast, the trained machine-learning model benefits by improving its accuracy of weather conditions found in images where the expected weather forecast is different from the actual weather condition. The trained machine-learning model may produce a result that is different from the input expected weather forecast during its implementation on the camera.

The model trainer trains the machine-learning model using multiple frames of media to detect a particular weather condition. Additionally, the machine-learning model can detect a particular weather type by analyzing a rate at which an object, e.g., snowflake, moves across the frames of images. For example, by training the machine-learning model to detect a particular weather condition using multiple subsequent image frames, the machine-learning model can detect snowflakes and their speed as they move across the frames of images. By analyzing an object's rate of movement or speed in subsequent images, the trained machine-learning model can determine whether the object is precipitation, such as rain or snow, or another object. The combining module measures the speed of the object by determining the object's change in distance across a first frame and a second frame and dividing the distance by the time difference between the two frames. Once the data analyzer sufficiently trains the machine-learning model to produce accurate weather conditions in image data, the data analyzer112can provide the trained machine-learning model to each camera at each monitored property.

During508, the system obtains a weather condition from the trained machine-learning model that indicates a particular weather condition at the monitored property based on one or more images from a camera and the expected local weather forecast at the monitored property. For example, the camera, at a particular monitored property, may record media footage of a view of the front yard in the monitored property. The camera can provide each frame of the recorded footage to its trained visual model. In some implementations, the cameras may provide the recorded footage to the data analyzer for processing through the data analyzer's trained visual model.

The camera also retrieves expected weather forecasts from the data analyzer when processing the recorded footage through the trained visual model. The camera transmits a request to the data analyzer for the expected weather forecasts and in response, receives the expected weather forecast for the particular monitored property where the camera is located from the data analyzer. The camera provides the recorded media and the expected weather forecast to the trained machine-learning model.

In response, the trained machine-learning model produces an output of a weather condition that describes a particular weather condition at the monitored property. In other implementations, if the trained machine-learning model executes on the data analyzer, the trained machine-learning model produces a weather condition for the monitored property and the data analyzer provides the output weather condition to the monitored control unit at the monitored property. In some implementations, the data analyzer can transmit the weather condition to the client device of the owner of the monitored property. In other implementations, in response to the monitor unit receiving the notification of the weather condition from the camera, the monitor unit transmits a notification to the client device of the owner of the monitored property. In other implementations, the camera or the monitor unit transmits a live video feed of the camera to the client device of the property owner for real time viewing purposes along with a current weather label describing the output weather condition.

In some implementations, in response to the trained visual model executing on the data analyzer or the camera, the data analyzer receives feedback from each of the monitored properties to fine tune the trained machine-learning model. For example, the feedback data can include new image data corresponding to a newly installed camera at the monitored property. In another example, the feedback data can include a correction of the labeled data associated with media data. For example, the data analyzer provides an indication of a weather condition to a client device of a property owner for a monitored property. The property owner can review the weather condition on his or her client device that includes the media data and a label of the weather classification corresponding to the output of the trained machine-learning model.

After the property owner reviews the media data and the corresponding weather classification label, the property owner can determine that the weather classification label is incorrect. The property owner can then interact with an application on his or her client device to indicate that the determined weather classification is incorrect, for example. In addition, the property owner can indicate that the weather classification label should read “rain” and not “snow.” In response, the client device can transmit the correction to the monitor unit corresponding to the monitored property or to the data analyzer. If the client device transmits the correction to the monitor unit, the monitor unit relays the correction to the data analyzer to indicate that the trained machine-learning model has incorrectly detected a weather classification. In response, the data analyzer can take steps to retrain the trained machine-learning model. For example, the data analyzer provides the recorded media with the corrected classification label to the model trainer for retraining the machine-learning model. Once the model trainer has retrained the machine-learning model to properly classify the media, the data analyzer provides the updated trained machine-learning model to each of the cameras at the monitored properties for an improved and updated weather classification detection.

In some implementations, the data analyzer can use the weather classification data, data from sensors at the monitored property, and data from the devices to determine one or more safety instructions to provide to devices at the monitored property. For example, the data analyzer can process the weather classification data, the sensor data from a monitored property, and the device data, to determine the safety instructions. In particular, the weather classification data can indicate that the current weather at the monitored property is rain, the sensor data indicates that the front door of the monitored property is open and unlocked, and other sensor data indicates that the back door of the monitored property is open. The data analyzer determines that the weather classification label, which indicates that it is raining at the monitored property, includes a weather condition that can potentially damage a portion of an interior or exterior of the monitored property. In addition, the data analyzer determines that based on the weather classification, each door and window of the monitored property should be closed and the property owner should be notified. In this example, the data analyzer notifies the property owner to close the front door and the back door through his or her client device. In another example, the data analyzer can transmit a notification to a control module located at both the front and back doors to automatically close both doors and lock them upon closing. In another example, a window at the monitored property may be opened, and the data analyzer can transmit a notification to the property owner and a notification to a control module located at the window to close and lock the window to avoid a portion of the monitored property becoming wet due to the rain.