Wireless integration of security camera and door lock

An apparatus includes an audio output device, an audio input device and a processor. The audio output device may be configured to generate an audio message. The audio input device may be configured to receive audio input. The processor may be configured to analyze the audio input to perform a first authentication of a user and determine a command corresponding to the audio input, determine a confidence level of a classification of a user based on the first authentication and authenticate the user if the confidence level is above a pre-determined threshold. The classification may correspond to an approved list of users. The confidence level may be adjusted in response to one or more authentication factors. If the user is authenticated, the processor may be configured to perform the command.

FIELD OF THE INVENTION

The invention relates to security devices generally and, more particularly, to a method and/or apparatus for implementing wireless integration of security camera and door lock.

BACKGROUND

Internet-connected devices are increasing in popularity, particularly for home automation. However, services are segmented. Each internet-connected device provides a separate functionality, but can also include a separate interface/app, a separate user account, and separate security concerns. Many internet-connected devices have poor security or fail to patch security flaws.

Internet-connected devices can combine functionality by physically integrated components. Physically integrating multiple types of functionality on a single device increases costs, increases device complexity and increases the size of the device. Furthermore, physically integrating multiple components still results in many of the same shortcomings as single-function internet-connected devices.

It would be desirable to implement wireless integration of security camera and door lock.

SUMMARY

The invention concerns an apparatus comprising an audio output device, an audio input device and a processor. The audio output device may be configured to generate an audio message. The audio input device may be configured to receive audio input. The processor may be configured to analyze the audio input to perform a first authentication of a user and determine a command corresponding to the audio input, determine a confidence level of a classification of a user based on the first authentication and authenticate the user if the confidence level is above a pre-determined threshold. The classification may correspond to an approved list of users. The confidence level may be adjusted in response to one or more authentication factors. If the user is authenticated, the processor may be configured to perform the command.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention include providing wireless integration of a security camera and door lock that may (i) authenticate users, (ii) implement artificial intelligence for authentication, (iii) control remote devices, (iv) respond to voice commands, (v) implement a wireless hub to communicate with remote devices, (vi) perform multiple authentication steps and/or (vii) be implemented as one or more integrated circuits.

Referring toFIG. 1, a diagram illustrating an example embodiment of the present invention is shown. A system100is shown. The system100may comprise a user50, a network60, a number of blocks (or circuits)102a-102n,a block (or circuit)104and a number of blocks (or circuits)106a-106n.The user50may be a visitor to a premises or a homeowner. The network60may be a local network and/or a wide area network (e.g., the internet). The circuits102a-102nmay implement devices. In one example, the devices may implement one or more sensors. In another example, each of the devices102a-102nmay implement a smart security camera. In yet another example, the devices102a-102nmay implement an outdoor hub for outdoor internet-of-things (IoT) devices (shown in association withFIG. 2). The circuit104may implement a remote device and/or cloud service. The circuits106a-106nmay implement user devices. The system100may comprise other components (not shown). The number, type and/or implementation of the components of the system100may be varied according to the design criteria of a particular implementation.

The devices102a-102nmay implement smart security devices. In the example shown, the device102amay be a smart security light, the device102bmay be a smart security doorbell and the device102nmay be a smart security camera. The smart security devices102a-102nmay comprise various sensors. The devices102a-102nmay be configured to read sensor data from the sensors to make inferences about the environment. In one example, one of the sensors implemented by the devices102a-102nmay be a camera sensor.

Camera sensors implemented by the devices102a-102nmay be configured to capture video data. The devices102a-102nmay be configured to perform video analysis (e.g., object detection, behavior detection, facial recognition, object classification, etc.). For example, each of the devices102a-102nmay implement on-board artificial intelligence configured to interpret the video data, determine characteristics of objects in the video data and communicate the data in a format that may be stored and/or read by the remote device104.

The devices102a-102nmay implement a wireless module. The wireless module may enable the devices102a-102nto communicate wirelessly (e.g., using Wi-Fi, ZigBee, Bluetooth, LTE, etc.) via the internet60and/or a local connection. In the example shown, the device102amay communicate directly with the user device106a(e.g., a device-to-device connection, such as Bluetooth). The wireless communication capability may enable the devices102a-102nto operate as a hub for a variety of network-connected devices. For example, the network-connected devices may communicate directly with the devices102a-102non a local network and the devices102a-102nmay communicate information from the network-connected devices to the remote device104via the interne60.

The devices102a-102nmay be configured to communicate the sensor data and/or the inferences made in response to performing sensor fusion operations on the sensor data to the remote device104and/or the user devices106a-106nvia the network60. The devices102a-102nmay operate independently of the network60(e.g., without instructions from the remote device104and/or the user devices106a-106n). Communication with the remote device104and/or the user devices106a-106nmay enhance the capabilities of the devices102a-102n.In one example, the processing capabilities of the remote device104may enable faster and/or more detailed video analysis of the video data and/or audio data captured by the devices102a-102n.

The remote device104may be configured to provide processing and/or mass storage for the system100. Generally, the remote device104may be located off-site from the devices102a-102n.The remote device104may be configured to store data, retrieve and transmit stored data, process data and/or communicate with other devices. The remote device104may be implemented as part of a cloud computing platform (e.g., distributed computing). In an example, the remote device104may be implemented as a group of cloud-based, scalable server computers. By implementing a number of scalable servers, additional resources (e.g., power, processing capability, memory, etc.) may be available to process and/or store variable amounts of data. For example, the remote device104may be configured to scale (e.g., provision resources) based on demand. The remote device104may implement scalable computing (e.g., cloud computing). The scalable computing may be available as a service to allow access to processing and/or storage resources without having to build infrastructure (e.g., the provider of the system100may not have to build the infrastructure of the remote device104). In the example shown, the remote device104is shown as a single block. However, the remote device104may be implemented as several computing devices (e.g., servers) located in many different locations.

The user devices106a-106nmay enable a user to send and/or receive information to/from the devices102a-102n.The user devices106a-106nmay provide a graphical user interface to enable a homeowner to interact with the devices102a-102n(e.g., an app). In an example, the graphical user interface of the user devices106a-106nmay be used to adjust the settings of the various sensors implemented by the devices102a-102n.

The user devices106a-106nmay be configured to communicate with the remote device104. For example, the user devices106a-106nmay be configured to retrieve video data and/or audio data stored by the remote device104. The user devices106a-106nmay be configured to receive notifications from the devices102a-102nand/or the remote device104. In one example, the notification may be a message that indicates that the sensors of the devices102a-102nhave detected an object (e.g., a person).

The user devices106a-106nmay be implemented as portable devices configured to display graphics (e.g., text, video, etc.), communicate audio (e.g., send and receive audio) and/or communicate with a network (wired or wirelessly). In some embodiments, one or more of the user devices106a-106nmay be implemented as a smartphone, a smart watch, a tablet computing device, a phablet, a desktop computer, a laptop computer, a netbook, etc. The user devices106a-106nmay be configured to execute instructions to provide an app that enables communication between the user devices106a-106nand the devices102a-102n.The type of devices implemented as the user devices106a-106nmay be varied according to the design criteria of a particular implementation.

A speech bubble108is shown. The speech bubble108may be a greeting generated by the devices102a-102n(e.g., an audio message). The devices102a-102nmay comprise an audio output device (e.g., an audio processor and/or a speaker) to generate audio and/or the speech108. In the example shown, the speech108may be a greeting. The devices102a-102nmay be configured to use artificial intelligence to generate context-relevant speech (e.g., a greeting when a visitor is detected, a personalized greeting when a resident is detected, delivery instructions when a package is detected, etc.). The devices102a-102nmay be configured to implement sensor fusion to make inferences based on various sensor readings. The sensor fusion may enable inferences to be made by combining sensor readings from disparate sources that would not be able to be determined from reading one of the sensor readings alone. Implementing the sensor fusion may enable the devices102a-102nto implement a conversational artificial intelligence.

Dotted lines110a-110bare shown. The dotted lines110a-100bmay represent a field of view of the camera implemented by the devices102a-102n.The field of view110a-110bmay capture the user50. The devices102a-102nmay be configured perform video analysis on the user50to determine an identity of the user50. For example, one or more features (e.g., the face, the height, the gait, etc.) of the user50may be detected by the devices102a-102nand compared with stored features. The video analysis may be used to determine one or more authentication factors.

A signal (e.g., AUTH) is shown. The signal AUTH may represent an authentication and/or classification of the user50. The devices102a-102nmay be configured to perform the authentication of the user50. In one example, the identification performed by the audio analysis may implement the authentication. Other sensors implemented by the devices102a-102nmay be configured to provide evidence and/or confidence for the authentication. For example, video analysis, voice detection, RFID, etc. may be implemented to perform the authentication. In some embodiments, the speech108may respond to the authentication (e.g., the authentication may provide the identity of the user50, and the speech108may be configured to greet the user50by name based on the identification).

The authentication may be configured to enable the devices102a-102nto receive commands from the user50. In one example, if the user50has been authenticated, the user50may provide voice-based commands. For example, there may be a list of voice commands that homeowners would like to have while outside the home. To enable voice commands outside the home, authentication may be implemented. With authentication, the voice commands may be configured to provide wireless control of various network-connected devices (e.g., door locks, garage door openers, outdoor lighting, keeping the devices quiet, arm/disarm door alarm systems, etc.). The devices102a-102nmay be configured to receive the commands, and then perform a functionality corresponding to the commands The functionality corresponding to the received commands may be performed by one or more of the network-connected devices. The devices102a-102nmay communicate (e.g., wirelessly) the commands to the network-connected devices.

A robust and easy outdoor authentication process may be valuable to all outdoor IoT devices (e.g., door locks). Robust and easy authentication may rely on two or more data points and/or signals (e.g., sensor fusion). Once visitors are authenticated (e.g., against a white list of approved visitors such as family and friends), other detected visitors may be considered ‘unfamiliar’ and/or ‘unwanted’ visitors. Commands provided by unfamiliar, unknown and/or unwanted visitors may not be performed by the devices102a-102n.For example, in an outdoor environment any person may walk up to the devices102a-102nand attempt to issue a command. The authentication provided by the devices102a-102nmay prevent unwanted commands and/or ensure that only approved users can provide the commands.

The system100may be configured to provide a response to a voice command and perform voice authentication. The devices102a-102nmay implement a hub that provides a connection to the Internet (e.g., the network60) and provides control to other network connected devices (shown in association withFIG. 2). For example, one or more of the devices102a-102nmay perform an authentication and initiate a command in response to a voice command. The command may be initiated by controlling another network connected device such as a door lock.

Generally, one or more of the devices102a-102nand/or one or more of the internet-connected devices may implement a microphone (e.g., for voice authentication and/or to receive the voice command). In some embodiments, voice authentication alone may be inadequate to authenticate a user and the devices102a-102nmay perform other types of authentication (e.g., authentication factors) for security. For example, the devices102a-102nmay receive multiple input signals for authenticating the visitor50(e.g., audio input, video input, motion input, etc.).

Performing a first authentication using voice analysis and using additional authentication factors may provide reliable authentication (e.g., 2-pass authentication, 3-pass authentication, etc.). The first authentication and other authentication factors may adjust a confidence level of the authentication. When the confidence level is determined to be above a threshold (e.g., 99% confidence) then the user may be authenticated. The confidence level may represent a likelihood that the identity of the visitor50determined is correct.

One or more of the devices102a-102nmay receive an audio input (e.g., capture a voice) from the visitor50and perform a voice analysis to authenticate the visitor50. The audio input may comprise a voice command to control the devices102a-102nand/or other network-connected devices. In one example, the visitor50may approach one of the devices102a-102nand speak, “This is Alice, 3467”. The authentication may comprise recognizing the voice of the visitor50(e.g., recognizing the voice as Alice). Other authentication factors may comprise a passphrase (e.g., correlating the spoken name of Alice to the vocal signature of Alice, correlating a code number to a code association with Alice). In the example, providing the code (e.g., 3467) may also provide a known command. For example, the code 3467 may be understood by the devices102a-102nto be a command to open the door.

Generally, the audio input may be captured by the devices102a-102nand/or an internet-connected device with a microphone wirelessly connected to the devices102a-102n.In some embodiments, video analysis may be implemented as an additional authentication factor (e.g., to improve authentication accuracy). Further authentication factors may comprise facial recognition, a barcode, hand gestures, height of visitors, etc.

Referring toFIG. 2, a diagram illustrating an example embodiment performing a command in response to an authenticated input is shown. The system100′ is shown. The system100′ shown may provide additional details of the system100shown in association withFIG. 1. The system100′ may comprise the device102. The device102may be a representative example of the devices102a-102nshown in association withFIG. 1. The system100′ may comprise one or more blocks (or circuits)120a-120n.The circuits120a-120nmay implement network-connected devices (e.g., IP devices, IoT devices, etc.). The number, arrangement and/or types of devices of the system100′ may be varied according to the design criteria of a particular implementation.

The user50is shown. A speech bubble122is shown. The speech bubble122may correspond to speech provided by the user50(e.g., audio input). A block (or circuit)124is shown. The circuit124may be a user device in the possession of the user50. In some embodiments, the circuit124may be one of the user devices106a-106nshown in association withFIG. 1.

A number of signals (AUTH_A-AUTH_N) are shown. The signals AUTH_A-AUTH_N may be authentication signals having a similar implementation as the signal AUTH shown in association withFIG. 1. The device102may be configured to perform a multi-factor authentication. The multi-factor authentication may be implemented to adjust a confidence level for the authentication. For example, each factor of the authentication may be analyzed by the device102to adjust (e.g., increase/decrease/no change) the confidence level. Whether the user50is authenticated may be determined in response to the aggregate confidence level of each authentication factor.

Various factors may be analyzed by the device102to perform the authentication of the user50. In the example shown, the signal AUTH_A may correspond to the speech122. The device102may analyze the speech (e.g., a passphrase) and/or voice of the user50(e.g., vocal patterns, voice biometric markers, etc.) as one factor for authenticating the user50. In the example shown, the signal AUTH_B may correspond to the user50. The device102may analyze the characteristics of the user50based on video analysis (e.g., facial recognition, gait recognition, height detection, etc.). In the example shown, the signal AUTH_N may correspond to the smartphone124. The device102may analyze the smartphone124(e.g., detect a MAC address, provide a notification that the user50can respond to, etc.). The number and/or methods of authentication may be varied according to the design criteria of a particular implementation.

The device102may compare the data received via the signals AUTH_A-AUTH_N may to known values for the corresponding authentication factors. The known values may be information that corresponds to one or more users on a white list (e.g., an approved list of users). In some embodiments, the known values may be stored by the device102and the comparison may be performed by the device102to classify the user50as a person on the approved list of users. In some embodiments, the known values may be stored by the cloud service104and the device102may upload the signals AUTH_A-AUTH_N to the cloud service104, the cloud service104may perform the comparison and provide the results to the device102.

In one example, the signal AUTH_A may be compared to the known factor of stored voices. If the voice122matches one or more of the known stored voices, the device102may increase the confidence level for each user identity that corresponds to the matches voices). In an example, the signal AUTH_A may have a 0% match for stored identity A, a 25% match for stored identity B, a 100% match for stored identity C and a 60% match for stored identity D. Since stored identity A and stored identity B have a low probability match, the device102may lower the confidence level for stored identity A and stored identity B. Since stored identity C and stored identity D have a higher probability match, the device102may increase the confidence level for stored identity C and stored identity D.

Continuing the example, the facial analysis of the signal AUTH_B may be compared to one or more stored faces and the signal AUTH_B may have a 0% match for the stored identity A, a 0% match for the stored identity B, a 90% match for the stored identity C and a 25% match for the stored identity D. Since the stored identity A, the stored identity B and the stored identity C have a low probability match for the signal AUTH_B, the device102may lower the confidence level for stored identity A, the stored identity B and the stored identity D. Since stored identity C has a higher probability match for the signal AUTH_B, the device102may increase the confidence level for the stored identity C.

Continuing the example, the MAC address of the signal AUTH_N may be compared to one or more stored MAC addresses and the signal AUTH_N may have a 0% match for the stored identity A, a 0% match for the stored identity B, a 100% match for the stored identity C and a 0% match for the stored identity D. Since the stored identity A, the stored identity B and the stored identity C have a low probability match for the signal AUTH_N, the device102may lower the confidence level for stored identity A, the stored identity B and the stored identity D. Since stored identity C has a higher probability match for the signal AUTH_N, the device102may increase the confidence level for the stored identity C.

Similarly, the device102may perform analysis, comparisons and/or adjustments to the confidence level for each of the signals AUTH_A-AURH_N. Some factors may have a higher or lower weighting when determining an aggregate confidence level. In one example, if facial analysis is considered unreliable technology, the confidence level determined for the factor AUTH_B may have a lower weight for the aggregate confidence level. In another example, the MAC address may have a higher weight for the aggregate confidence level. The amount of weight for each factor may be varied for each specific scenario (e.g., facial recognition may have a low weight if the face of the user50is obscured, but may have a higher weight if the face of the user50is looking directly at the camera).

If the confidence level for one the stored identities is high enough (e.g., above a threshold value for the confidence level), then the device102may authenticate the user. Continuing the example, the stored identity A, the stored identity B and the stored identity D may have a low aggregate confidence level for matching the user50. The stored identity C may have a high confidence level for matching the user50. If the aggregate confidence level for the stored identity C is above the threshold level, then the device102may authenticate the user50. If none of the stored identities are above the pre-determined confidence level threshold then the user50may not be authenticated.

The device102is shown communicating corresponding signals (e.g., CMD_A-CMD_N) to the IP devices120a-120n.The IP devices120a-120nare each shown communicating a corresponding signal (SEN_A-SEN_N) the device102. The signals CMD_A-CMD_N may represent a communication of a command from the device102to the IP devices120a-120n.The signals SEN_A-SEN_N may represent a communication of sensor readings from the IP devices120a-120nto the device102.

The IP devices120a-120nmay be configured to execute computer readable instructions. Executing the computer readable instructions may enable the IP devices120a-120nto perform one or more types of functionality (e.g., tasks). Each of the IP devices120a-120nmay be configured to perform one or more types of functionality. In one example, the functionality may be to capture video data. In another example, the functionality may be to detect motion. In yet another example, the functionality may be to read soil moisture levels. In still another example, the functionality may be to water a lawn according to a schedule. The number and/or types of functionality implemented by the IP devices120a-120nmay be varied according to the design criteria of a particular implementation.

One or more of the IP devices120a-120nmay be configured as sensors. The sensors implemented by the IP devices120a-120nmay be configured to read sensor data. The sensor data read and/or captured by the IP devices120a-120nmay be communicated to the device102via the signals SEN_A-SEN_N.

The device102may be configured to issue commands to the IP devices120a-120n. The commands CMD_A-CMD_N may provide instructions and/or interrupt requests to the IP devices120a-120n.In one example, the IP devices120a-120nmay be configured to independently perform the functionality (e.g., water a lawn according to a schedule based on the time of day). The IP devices120a-120nmay perform the scheduled tasks without additional instruction (e.g., independently follow the schedule). The commands CMD_A-CMD_N may provide direct control over the IP devices120a-120n.For example, the signals CMD_A-CMD_N may change and/or adjust the functionality of and/or how the functionality is performed by the IP devices120a-120n. In an example, if the IP device120ais configured to initiate lawn irrigation at a particular time of day, the signal CMD_A may provide instructions to immediately turn on the irrigation system (e.g., override the schedule).

In some embodiments, the device102may be configured to generate the signals CMD_A-CMD_N based on the sensor data SEN_A-SEN_N. In one example, if the IP device120ais implemented as a soil moisture sensor, and the signal SEN_A indicates that the moisture level for the soil is low, then the device102may generate the signal CMD_B to instruct the IP device120b(e.g., implementing an irrigation control system) to initiate watering the lawn. Similarly, the device102may be configured to generate one or more of the signals CMD_A-CMD_N in response to sensor data implemented by the device102. For example, the device102may implement a motion sensor, and when motion is detected the device102may communicate the signal CMD_C to communicate to the IP device120c(e.g., implementing a smart light bulb) to turn on a light.

The user50may provide user commands to the device102. In one example, the user50may provide voice commands. In another example, the user50may provide a user command to the device102via an app implemented on the smartphone106. The device102may interpret the user command provided by the user50and generate one or more of the signals CMD_A-CMD_N in response to the interpreted command. In the example shown, speech122may be a voice command. The device102may interpret the voice command and generate one or more of the signals CMD_A-CMD_N in response to the voice command122.

The device102may only generate the commands CMD_A-CMD_N in response to voice commands provided by the user50if the user50has been authenticated. In one example, the device102may ignore the voice command if the user50has not been authenticated. In another example, the device102may provide the communication108(e.g., a recorded message indicating that the user50does not have authorization to provide the command), if the user50has not been authenticated. In one example, the voice command122may provide the voice command and the authentication. For example, the device102may analyze the voice of the voice command122to authenticate the user50. If the user50is not authenticated, then the device102may not perform the voice command122. If the user50is authenticated, then the device102may interpret the voice command122and generate the signals CMD_A-CMD_N.

The device102and/or the IP devices120a-120nmay implement functionality such as lights, camera, door lock control, irrigation control, etc. In an example, the IP devices120a-120nmay function together via the device102to implement a light controlled by a camera, a door lock controlled by a camera, and/or an irrigation system controlled by a camera. The device102may combine various functionality (e.g., a camera implemented by the device102) of the device102with the various functionality of each of the IP devices120a-120nvia wireless signals. The device102may operate as an outdoor hub configured to control the functionality of the IP devices120a-120n. Using the AI-based authentication of users, the device102may operate as the outdoor hub to enable the benefits of voice commands to control the IP devices120a-120njthat would otherwise be unable to respond to voice commands. For example, the IP devices120a-120nmay be simple (e.g., low-cost, low feature) devices (e.g., incapable of interpreting voice commands) that may be controlled by the device102(e.g., a device that is capable of interpreting voice commands).

In one example, for authenticated users (e.g., if the user50is on a whitelist), the user50may issue a command such as “Kung, keep quiet” to prevent notifications for a particular amount of time (e.g., an hour, no alerts, no greetings, no lights, etc.). In another example, the user50may issue a command such as “Kung, open the door” (e.g., one of the IP devices120a-120nmay be configured to lock or unlock a door lock). In yet another example, the user50may issue a command such as “Kuna, turn on the lights” (e.g., one of the IP devices120a-120nmay be configured to control a light bulb). In still another example, the user50may issue a command such as “Kung, open the garage 1 foot” (e.g., one of the IP devices120a-120nmay operate a garage door opener). In another example, the user50may issue a command such as “Kuna, tell Beatrice that I am outside to wash the car” (e.g., the device102may record the audio from the user50to be played back by another authenticated user).

The authentication may be determined based on a 2D facial recognition in combination with a voice (e.g., performing audio analysis to determine a voice based on biometric analysis) and/or stating a name as a soft password). The authentication may be configured to control an alarm control panel and/or a door. The video and/or audio analysis may be combined with other sensor data and/or manual input of a password (e.g., authentication based on multiple factors), such as a numerical password.

In some embodiments, the visitor50may be detected based on the seashell effect of a visitor shape on resonances of the environment (e.g., porch resonance). In an example, an active swept frequency audio stimulus may be presented by the speaker (e.g., a bat ear echo detection). Ultrasonic, near ultrasonic and/or an audible chirp may be emitted and the return signal may be analyzed to determine dimensions of the visitor50and/or if the visitor50is carrying a package.

In some embodiments, WiFi and/or Bluetooth Received Signal Strength Indication (RSSI) levels from multiple AP directions, phones, cars, etc. may be analyzed to determine the attenuation of signals from different directions that occur as a visitor drives up to the home and/or walks up to towards the door, doorbell and/or light fixture antenna. In an example, a resident that parks the same car in the same place in the driveway may cause changes to the reflected and/or attenuated signals from the access points of the resident and/or neighboring home. The RSSI levels may be detected and/or recognized. The RSSI may add to the confidence level of detecting a visitor (e.g., the resident drives the same car daily and parks in the same place, causing similar RSSI levels daily). The additional data point may increase confidence for performing the action (e.g., enabling more security for passive unlocking of the access points).

The system100′ may enable an integration of camera functionality with door lock control. Since the device102may be powered using a wire, the camera functionality may not drain a battery of one of the IP devices120a-120n.The camera and video analysis and/or audio analysis may enable AI detection that would not be otherwise available to the IP devices120a-120n. Furthermore, the device102may have more physical space available for a higher quality camera than would be available for a low cost IP device120a-120n.The system100′ may enable the user50to provide commands using convenient vocal commands as well as through a phone app. The device102may be configured to provide a high confidence authentication with wireless integration between the device102and the IP devices120a-120n.The system100′ may provide convenience, wired-power for advanced authentication methods, reduced cost and reduced power usage for locks to connect to internet.

Similarly, the system100′ may enable an integration of an outdoor authentication with a garage door opener. Since the device102may be an outdoor device and one or more of the IP devices120a-120nmay operate as a garage door opener, the system100′ may provide an alternate to indoor only smart hubs and/or app-controlled devices. The system100′ may enable a detection of a delivery person and open the garage door a small amount (e.g., 1 foot high) for package delivery, then close the garage door after the delivery is made (e.g., secure storage of packages for delivery).

In some embodiments, the system100′ may enable an integration of an outdoor authentication with a smart irrigation system. For example, many homeowners decide to water the lawn when they are outside and see that the lawn needs watering. The outdoor device102may enable the user50to control one of the IP devices120a-120noperating as a wireless irrigation system control (e.g., “Kuna, sprinkler on for 5 minutes”). Similarly, when the homeowner is outside (e.g., hosting a barbeque party) the homeowner may realize that it is getting dark outside. The system100′ may enable the user50to provide a voice command to control the IP devices120a-120nthat may operate as a smart light (e.g., “Kuna, turn on the yard light”, “Kuna, turn off the outdoor lights after 10 pm”, etc.).

In some embodiments, the system100′ may enable an integration with outdoor lighting. For example, the system100′ may not rely on a phone app for control and may work with outdoor lighting instead of indoor only lighting. For example, the device102and/or the IP devices120a-120nmay be weatherproofed and/or meet regulations for outdoor usage. Instead of using an indoor light switch to control outdoor lighting, control of the outdoor lighting may be managed by the commands of the device102. The authentication may ensure that only particular approved people are able to control the lighting. For example, the voice command122may be “Kuna, turn on the yard light” or “Kuna, turn off door light 2”.

Referring toFIG. 3, a diagram illustrating an example embodiment of an outdoor hub is shown. The device102may be a multi-functional device that comprises at least a video/image recording function and/or an audio recording/playback function. The device102may be configured as an outdoor wireless hub. In the example shown, the device102may be a smart security doorbell camera (e.g., configured to generate video recordings of visitors approaching and/or entering a premises). In another example, the device102may be implemented as a smart security light. In some embodiments, the device102may be implemented as an outdoor security device (e.g., a doorbell security camera mounted to the outside of the home). In some embodiments, the device102may be implemented as an indoor security device. The implementation and/or the functionality of the device102may be varied according to the design criteria of a particular implementation.

The device102may comprise a housing154. The housing154may be configured to encase various components of the device102(e.g., electronics configured to implement and/or facilitate the implementation of features). The housing154may be molded together and/or snapped together to form a single enclosure. In some embodiments, the housing154may be non-removable. The housing154may comprise various openings to enable and/or facilitate functionality of the device102(e.g., openings for speakers, microphones, cables, camera lenses, etc.).

In the example shown, the device102may comprise various components. The device102is shown comprising a component150, a component152, a component156, a component158, a component160, a component162, components164a-164band/or a component166. The components150-166may enable the various functionality and/or features for the device102. In the example shown, the component152may implement a lens, the component156may implement status lights and/or a microphone grille, the component158may implement a speaker grille, the component160may implement a button (e.g., a doorbell), the component162may implement a status light, the components164a-164nmay implement lights (e.g., for illumination and/or infrared lighting) and/or the component166may implement a power cable. In one example, the speaker grille158may facilitate an audio playback feature by allowing sound waves generated by a speaker component (e.g., the audio output device) within the housing154to easily pass through the housing154(e.g., provide good sound quality). In another example, the button160may enable a doorbell feature by providing a contact that activates the doorbell. The number, type, design, layout and/or functionality of the components150-166may be varied according to the design criteria of a particular implementation.

The component150may implement a camera lens opening. The component150is shown having an oblong shape. For example, the oblong shape of the component150may have a larger width dimension than height dimension. The lens opening150may be configured to accept light for a capture device (e.g., a camera sensor). The lens opening150may be configured to provide protection for an internal component of the device102(e.g., the capture device). In an example, a lens (e.g., a wide-angle lens) and an image sensor (described in association withFIG. 5) may be located behind the lens opening150.

Dotted lines170are shown extending from the lens opening150. The dotted lines170may represent a capture area (e.g., field of view) for the capture device located behind the lens opening150. In the example shown, the field of view170may be a truncated rectangular pyramid. The shape of the field of view170may generally match the shape of the lens opening150(e.g., the width of the field of view170may be greater than the height of the field of view170). Similarly, dotted lines172a-172bare shown extending from the lens152. The dotted lines172a-172bmay represent a capture area (e.g., field of view) for a capture device located behind the lens152. In the example shown, the lens152may have a circular shape. In the example shown, the device102may implement a dual-camera security device configured to capture two separate capture areas (e.g., provide two different views).

Referring toFIG. 4, a diagram illustrating an alternate view of the outdoor hub is shown. A front view of the device102is shown. The front view of the device102may show the lens opening150, the lens152, the microphone grille156, the speaker grille158, the button160, the status light162and/or the lights164a-164b.

The lens150is shown having the oblong shape. The lens150may be horizontally centered on the device102. In some embodiments, the lens150may be implemented having a tilt. The tilt of the lens150may be configured to accommodate an angle of the image sensor of the capture device within the device102. In one example, the lens150may have a 5 degree tilt (e.g., upwards) to capture a view with a different capture area than the capture area of the lens152(e.g., so that video data captured may provide a view of faces of visitors). For example, the device102may enable capturing a head-to-toe image of a person using a combination of the view captured using the lens opening150and the lens152.

The device102may be implemented having a thin profile. For example, the device102may be relatively tall compared to the width. The thin profile of the device102may impose size and/or space restrictions for the components150-166and/or the lens150. One example of the functionality of the device102may be implemented by the capture device located behind the lens opening150. For example, the capture device may be a wide angle camera. In some embodiments, the device102may be implemented as a video doorbell. Implementing the device102with a thin profile may restrict the amount of room (e.g., the device102may not provide enough space for adding a clear glass cover over the entire device). In an example, the device102may be configured to implement the functionality of an outdoor camera and the lens opening150may be configured to minimize a “greenhouse effect” of clear lens cover with minimal air trapped by the lens cover.

The lens opening150may have a relatively small height compared to width (e.g., an oblong shape). The oblong shape of the lens opening150may reduce a size footprint of the lens opening150on the front face of the device102. A small height for the lens opening150may enable more components (e.g., which allows for greater functionality) to fit on the front face of the device102. If the lens opening150had a larger height, the larger height may occupy more space that could be used for other components.

In the example shown, the shape of the lens opening150may have rounded corners. In some embodiments, the shape of the lens opening150may have edged corners (e.g., rectangular corners that meet at a 90 degree angle). In some embodiments, the shape of the lens opening150may have a combination of rounded and edged corners. The shape of the corners may be cosmetic. The lens opening150may have a width dimension greater than a height dimension.

Referring toFIG. 5, a diagram illustrating an exploded view of the outdoor hub is shown. The exploded view200may illustrate various components internal to the device102. The internal components of the device102may provide the various functionality and/or implement various features of the device102.

An object202is shown. The object202may be a protective cover for the lens opening150. The cover202may have a size and shape that matches the size and shape of the lens opening150(e.g., a width may be greater than the height). The cover202may be implemented as a clear material (e.g., to provide protection while still allowing light to reach the image sensor). In one example, the clear material of the cover202may be implemented as a piece of glass. In another example, the cover202may be implemented as a piece of clear plastic. The cover202may not provide a removable, opaque shutter. The implementation of the cover202may be varied according to the design criteria of a particular implementation.

The housing154is shown. In the example shown, the housing154is hollow. The hollow housing154may be configured to provide a protective enclosure for the components of the device102. The enclosure154is shown separated. An upper faceplate204and a lower faceplate206are shown. The upper faceplate204and the lower faceplate206may provide a front face of the housing154. Together, the upper faceplate204, the lower faceplate206and the housing154may provide the protective and/or decorative enclosure for the internal components of the device102. The upper faceplate204and/or the lower faceplate206may implement a body plate for the device102. The upper faceplate204and/or the lower faceplate206may provide a front cover for the housing154. In the exploded view200, the upper faceplate204, the lower faceplate206and the housing154are shown as separated (e.g., for illustrative purposes). However, the upper faceplate204, the lower faceplate206and the housing154may form a single enclosure for the device102. In some embodiments, the upper faceplate204, the lower faceplate206and the housing154may snap together. In some embodiments, the upper faceplate204, the lower faceplate206and the housing154may be molded together, fused together and/or permanently adhered together. In one example, the upper faceplate204may be non-removable.

The upper faceplate204may comprise the microphone grille156and/or the lens opening150. The lower faceplate206may comprise the speaker grille158, an opening208and an opening210. The microphone grille156and/or the lens opening150may be implemented as an integrated portion of the upper faceplate204. For example, the lens opening150may not be removed from the upper faceplate204. The speaker grille158, the opening208and/or the opening210may be implemented as an integrated portion of the lower faceplate206. Generally, the lens opening150may not be removable from the upper faceplate204and/or the housing154.

The opening208may provide a space for the button160to fit into the opening208. For example, when the button160is pressed, the opening208may allow the button160to move into the housing154to make contact with a component of the device102. The opening210may provide a space for the lights164a-164band/or the lens152.

Various components of the device102are shown in the exploded view200. Circuits212a-212bare shown. The circuits212a-212bmay provide interconnections and/or mounting locations for the various components of the device102. The various components of the device102may fit within a limited amount of vertical space of the housing154. A block (or circuit)214is shown implemented on the circuit212a.The circuit214may implement a processor. For example, the circuit214may be a video processing device configured generate video data from the image sensor. A block (or circuit)216is shown implemented on the circuit212b.The circuit216may implement a wireless communication device. The wireless communication device216may enable the device102to transmit and/or receive data wirelessly (e.g., connect to the IP devices120a-120nand/or the internet60). A lens housing218is shown. A lens232is shown. The circuits234a-234nmay implement image sensors.

The image sensors234a-234bmay implement video capture device. In an example, the image sensors234a-234bmay be rectangular with a 16:9 aspect ratio. The image sensor234a-234bmay receive light within the truncated rectangular pyramid field of view170and/or172a-172b. The image sensors234a-234bmay be configured to convert received light into computer readable data (e.g., pixels that comprise video frames). The image sensors234a-234bmay be implemented to record video data. The lens232may correspond with the image sensor234a.The lens232may be configured to focus incoming light onto the image sensor234a.The lens232may be a wide-angle lens. In an example, the lens232, the image sensor234aand/or other circuits may implement a capture device.

The lens housing218may be a housing for the front wide angle lens232. The cover202may fit over top of the lens housing218and the lens232may be behind the cover202. The lens housing218may isolate the wide angle lens232from other components of the device102.

The circuit212amay further comprise a block (or circuit)220, a block (or circuit)222, a block (or circuit)224and/or a block (or circuit)226. The circuit220may implement a speaker (e.g., the audio output device). The circuit222may implement an audio processor. The circuit224may implement a microphone (e.g., the audio input device). The circuit226may implement a fingerprint scanner. The circuit212bmay further comprise a block (or circuit)228and/or a block (or circuit)230. The circuit228may implement storage (e.g., memory). The circuit230may implement a power storage/power conversion (e.g., a battery, an AC power converter, voltage regulation, etc.). A connector236is shown attached to the housing154. The power cable(s)166may provide power via the connector236. For example, the connector236may connect to the power storage/power conversion device230to provide an external power supply. In some embodiments, the power connectors may be a USB power connection. In the example shown, the connector236may be a T-Terminal (e.g., with the wires238a-238bas a source power supply for the premises) connection. In an example, for a doorbell style security device102, the T-Terminal connection may be implemented (e.g., most doorbells use a T-Terminal connection). The circuits212a-212bmay comprise other components (not shown). The components214-236may be implemented on either of the circuits212a-212b(e.g., based on physical space constraints, heating constraints, operating condition constraints, etc.). The number, type and/or arrangement of the components214-236may be varied according to the design criteria of a particular implementation.

The circuits212a-212band/or the components214-236may be entirely and/or partially enclosed within the housing154(and the upper faceplate204and the lower faceplate206). The components214-236may provide the functionality of the device102. In some embodiments, the components214-236may implement the functionality for a smart security camera. In one example, the speaker220and the audio processing device222may be configured to generate audio for communicating to a visitor and the microphone224may be configured to receive audio (e.g., to enable 2-way communication). In another example, the fingerprint scanner226may be configured to identify a visitor that presses the button160based on a fingerprint. Details of the functionality of the circuits212a-212band/or the various components214-236may be described in U.S. patent application Ser. No. 16/143,853, filed on Sep. 27, 2018, appropriate portion of which are hereby incorporated by reference in their entirety.

The size of the opening150may affect the number of components214-236that may be implemented by the device102. For example, the hollow housing154may have a limited amount of space. To provide functionality, the components214-236may need to fit within the constraints of the hollow housing154. If the size of the opening150takes up too much space, one or more of the components214-236may be unable to fit within the constraints of the hollow housing154.

Referring toFIG. 6, a diagram illustrating a dual view video capture displayed on a user device is shown. An example context300of the invention is shown. In the example context300, the user device106is shown held by a user50(e.g., a homeowner). For example, the user device106may be a portable (e.g., handheld) device (e.g., one of the user devices106a-106nshown in association withFIG. 1). In the example shown, the user device106may be a smartphone.

The smartphone106is shown having a display302, a speaker304and a microphone306. In an example, the display302may be a touchscreen display enabling the user50to view output from the smartphone106and/or provide input (e.g., touch controls) to the smartphone106. The speaker304may playback audio. The microphone306may receive audio. In an example, the speaker304and the microphone306may enable the user50to interact with the device102as a two-way intercom. In one example, the speaker304on the smartphone106may playback audio captured by the microphone component224and wirelessly transmitted by the device102. In another example, the microphone306on the smartphone106may capture audio that may be wirelessly transmitted to the device102and the device102may playback the audio (e.g., using the speaker component220).

The display302is shown displaying a companion application310. The device102may be compatible with the companion application310. For example, the companion application310may be a smartphone app (e.g., iOS app, an Android app). In some embodiments, the companion application310may be a computer program (e.g., for Windows, macOS, Linux, etc.). The companion application310may enable the user50to remotely interact with the features of the device102. For example, the companion application310may be configured to interface with the smartphone microphone306and speaker304to enable the user50to use the device102as an intercom (e.g., audio received by the smartphone106may be played by the speaker220of the device102and audio received by the microphone of the device102may be played back to the user by the smartphone speaker304).

In the example shown, the companion application310shown on the smartphone106may display a video stream312aand a video stream312b.The video stream312amay correspond with the video stream generated by the image sensor234a.The video stream312bmay correspond with the video stream generated by the image sensor234b.For example, the device102may be configured to wirelessly communicate (e.g., a live stream and/or a recorded file for later playback) the video streams to the smartphone106using the communication device216. The companion application310may playback the video streams312a-312b.

In the example shown, the companion application310may further comprise controls314. The controls314may enable access to various features of the companion application310. In one example, the controls314may be a play/pause video progress bar (e.g., used to control what portion of the video streams312a-312bthe user50is viewing). In another example, the controls314may be a live view indicator (e.g., to indicate whether the video streams312a-312bare live video currently captured by the device102or a pre-recorded file). In yet another example, the controls314may be a toggle button to toggle between options (e.g., enable/disable a live view). In another example, the controls314may be a button configured to activate/deactivate audio communication (e.g., one-way audio by the user50to be played back by the device102and/or a two-way intercom to additionally receive audio from the device102). In still another example, the controls314may be a button to access other features of the smartphone106(e.g., an app-drawer). In another example, the controls314may be a button configured to control a light that may be coupled with the device102. The features and/or controls314implemented by the companion application310may be varied according to the design criteria of a particular implementation.

In the example shown, the companion application310operating on the smartphone106may show a dual view displaying both video streams312a-312b.In some embodiments, the companion application310may display one of the video streams312a-312bat a time or neither of the video streams312a-312b.The dual view may comprise the front view video stream312acorresponding to the field of view170captured by the front facing lens232. The dual view may comprise the bottom view video stream312bcorresponding to the field of view172a-172bcaptured by the bottom lens152.

The video streams312a-312bshown may be captured by the image sensors234a-234bfrom the doorbell embodiment of the apparatus100. In the example shown, the video stream312amay comprise a view captured by the front-facing capture device234a.For example, in a doorbell embodiment, the device102may be mounted to a surface at approximately 48 inches from the ground (e.g., a standard doorbell height). The video stream312amay capture a view of a visitor320from the height of a doorbell. In the example shown, the field of view170for the video stream312amay capture a waist, torso, chest and face of the visitor320.

For the doorbell embodiment of the device102, the video stream312bmay comprise a view captured by the bottom capture device234b.The bottom capture device234bmay capture an area below the device102. In the example shown, the video stream312bmay comprise a high angle view of the bottom (e.g., from the waist down) of the visitor320. The video stream312bmay further comprise a view of a package330. For example, the field of view172a-172bfor the bottom view video stream312bmay be oriented to capture packages330left at the premises. A tracking number332is shown on the package330. A portion of a welcome mat334is also visible in the example shown.

The video processor214may be configured to scan the tracking number332. For example, the image quality of the capture device234bmay be high enough to enable the video processor214to read various tracking numbers (e.g., bar codes, numbers, QR codes, etc.). The video processor214may read and/or decipher the tracking number332using optical character recognition and/or object detection. In some embodiments, the wireless communication device216may be configured to send a notification using the companion application310to indicate that the package330with the tracking number332read by the video processor214has been delivered. In some embodiments, the user50may use the companion application310to input and store particular tracking numbers of expected packages, and the video processor214may compare the detected tracking number332against the stored tracking numbers.

In an example, the video streams312a-312bmay be played back simultaneously. For example, the front view video stream312amay show a view of the face of a visitor320and the bottom view312bmay show the waist down view of the visitor320and the package330at the same time. For example, if the visitor320is a burglar stealing the package330, the front view video312amay provide a clear view of the face (e.g., identity) of the visitor320but not the package330and the bottom view video312bmay show the visitor320stealing the package330but not provide a view of the face to identify the thief. Similarly, if the visitor320is attempting to break into the home by opening the door, the front view video312amay not provide the view of the door but the bottom view video312bmay show the visitor320attempting to open the door. The videos312a-312bcaptured by both the front capture device234aand the bottom capture device234bmay be used as evidence for police of the visitor320trying to steal the package330and/or attempting to break into the premises.

In some embodiments, the device102may be configured to send notifications to the companion application310in response to the detection and/or authentication. For example, a text message and/or an email may be transmitted in response to the notification. In another example, a notification may be sent via an API (e.g., push notifications) for a particular operating system (e.g., Android notifications, iOS notifications, Windows notifications, etc.). Generally, the user50may create a user account (e.g., comprising at least an email address and a password as credentials) for the cloud service104(e.g., via an app and/or a web-based interface). The user account may allow the user50to configure preferences. The preferences may comprise the notification settings. The type of notifications sent by the device102may be based on the notification settings. In an example, the notification may provide an indication of when the package330has been delivered.

The video output streams312a-312nmay be configured to capture the head to toe view of the visitor320. For example, the FOV170and the FOV172a-172bof the device102may be configured to provide a view of the full height of a person. Since two FOVs are used from two different cameras, the two streams may not provide a continuous view of the person320, but the view may appear to be somewhat continuous. In some embodiments, video operations may be performed on the video data from the dual camera sensors234a-234bto modify (e.g., warp) the images to provide a continuous view of the person320.

In an example, the top camera sensor234amay provide 1080p HDR video data. The top camera sensor234amay have a 30 ft range with night vision. The bottom camera sensor234bmay be configured to prevent blindspots. For example, the bottom camera sensor234bmay provide coverage for areas that are not covered by the top camera sensor234a(e.g., to provide a full top to bottom view of a visitor). The bottom camera sensor234bmay implement a white LED night vision (e.g., using the lights164a-164b). In a doorbell dualcam embodiment, a 110 dB+ siren may be implemented and a speaker for playing back the greetings and/or other audio.

In the example shown, the app310is used to access the dual video streams312a-312b.The app310may be further configured to provide a premium plan with artificial intelligence (e.g., package monitoring and theft protection).

The video analysis may be configured to detect special events (e.g., interrupt events) such as a package delivery (e.g., at the door). The detection of a package delivery may initiate an alert state to prevent package theft. When the package is detected at the door, the current event may be overridden with the alert state (e.g., the default schedule may be interrupted by the alert state). A special greeting may be enabled during the package alert state.

The device102may be configured to perform automatic package detection using computer vision and/or AI technology. In some embodiments implementing the dualcam doorbell with the bottom camera sensor234b,the package330may be detected when left on the ground at the front door (e.g., the package is in the FOV172a-172bof the bottom camera sensor234b). The camera FOV may be further adjusted by the user50to reduce false detection.

Referring toFIG. 7, a diagram illustrating an example authentication is shown. An example scenario350is shown. The example scenario350may comprise the device102and the user50. The memory228is shown in the device102. A more detailed view of components of the memory228is shown as an expanded view. The FOV170a-170bof the camera sensor234ais shown. The FOV172a-172bof the camera sensor234bis shown. In the example scenario350, the user50may be captured by the FOV170a-170band the FOV172a-172b.A speech bubble352and a speech bubble354aare shown. The speech bubble352may correspond to an audio instruction (e.g., an audio message) generated by the device102. The speech bubble354amay correspond to a vocal reply from the user50.

Authentication characteristics354a-354nare shown. The authentication characteristic354amay correspond to the voice of the user50. The authentication characteristic354bmay correspond to video analysis (e.g., facial recognition) of the user50. The authentication characteristic354cmay correspond to a gesture performed by the user50. In the example shown, the user50may be waving. The authentication characteristic354nmay correspond to audio generated by the user50. In the example shown, the user50may generate audio by a foot tap. The device102may analyze other types of authentication characteristics. For example, the video analysis may be used to determine a height of the user50. The authentication characteristics354a-354nmay be representative examples of some of the characteristics that the device102may detect and/or analyze.

The memory228is shown comprising a block (or circuit, or module)360, a block (or circuit, or module)362and/or blocks (or circuits, or modules)364a-364n.The block360may represent an approved list of users (e.g., a whitelist). The block362may represent a list of undesired users (e.g., a blacklist). The blocks364a-364nmay comprise various known values and/or factors for the users. In an example, each user on the whitelist360and each user on the blacklist362may have one or more associated stored values364a-364n.Each user on the whitelist360or the blacklist362may not have every one of the stored values364a-364n.The number and/or type of stored values364a-364nassociated with the users on the whitelist360or the users on the blacklist362may be varied according to the design criteria of a particular implementation.

The outdoor hub device102may be configured to determine a classification the user50as a person on the whitelist360. Each user on the whitelist360may be approved for particular commands or groups of commands. For example, each user and/or category of user may have permission-based access to the functionality of the device102and/or the IP devices120a-120n.In some embodiments, not every user on the whitelist360may have access to all of the functionality of the device102and/or the IP devices120a-120n.In one example, the homeowner may be a user on the whitelist360that has full access to the functionality of the device102and/or the IP devices120a-120n(e.g., full permissions). In another example, a child of the homeowner may be a user on the whitelist360that has permission to unlock a door but may not have permission to initiate the irrigation system (e.g., partial permissions). Each person on the whitelist360may have a certain level of access to doors, outdoor devices to control (e.g., lights, music, greetings, etc.), phone numbers to call/text, etc.

The outdoor hub device102may be configured to determine a classification the user50as a person on the blacklist list362. Each user on the blacklist362may be denied permission for every command. When a user on the blacklist362is detected, one or more of the commands may be automatically initiated by the device102. For example, users on the blacklist362may be a known criminal, a solicitor, a pest animal, etc. In one example, if a criminal is detected, the device102may automatically initiate a command to have one of the IP devices120a-120nsound an alarm, and the device102may contact the authorities. In another example, if a solicitor is detected the device102may playback an audio message of, “Please do not disturb this household”.

In some embodiments, users on the whitelist360and/or users on the blacklist362may not refer to specific individuals. The users on the whitelist360and/or the users on the blacklist362may comprise groups and/or categories of users. In one example, a group of users on the whitelist360may be each member of the household. In another example, a group of users on the whitelist360may be a person detected as a delivery person (e.g., determined based on a uniform). In yet another example, a group of users on the blacklist362may be people exhibiting the behavior of a burglar (e.g., jiggling doorknobs, checking windows, attempting to enter multiple entrances, etc.). The types of people and/or groups of people on the whitelist360and/or the blacklist362may be varied according to the design criteria of a particular implementation.

The stored values364a-364nmay comprise passive characteristics and/or active characteristics. The passive characteristics may comprise the detection of authentication characteristics that are performed on the visitor50without a specific action by the visitor50. In an example, the video analysis used for facial recognition may be a passive authentication characteristic. The active characteristics may comprise authentication characteristics that are detected in response to an action by the visitor50. In an example, the device102may ask the visitor for a passphrase and the active authentication characteristic may be the audio analysis of the passphrase. The types of passive and/or active detection performed may be varied according to the design criteria of a particular implementation.

The stored values364amay correspond to faces (e.g., 2D face detection and/or 3D face detection detecting based on analyzing video frames of the visitor50approaching the device102). The stored values364bmay be a phone ID (e.g., determined based on using a Bluetooth MAC address from the smartphone124detected by the communication device216). The stored values364cmay correspond to a visitor height (e.g., an estimate of the height may be determined by triangulation using at least 2 cameras). The stored values364dmay correspond to a gait of the visitor50(e.g., performing video analysis on how the visitor50walks when approaching the device102, performing audio analysis on the sound of the footsteps approaching the device102, etc.). The stored values364a-364dmay correspond to passive detection.

The stored values364emay correspond to a voice of the visitor50(e.g., using the microphone224, the device102may use artificial intelligence to start a conversation with the visitor50and device102may analyze the voice format). The stored values364fmay correspond to a rhythm performed by the visitor50(e.g., the visitor50may be requested to tap a rhythm of a pre-selected favorite song near the microphone224). The stored values364e-364fmay correspond to active detection.

The stored values364gmay correspond to biometric information of the visitor50(e.g., measuring a body part such as a palm, or finger size when the visitor50is asked to hold up a hand to the camera, measuring a quick wave while walking in, etc.). The stored values364hmay correspond to a passkey (e.g., show barcode/QR code from a package, a RFID wallet card, a code displayed on a phone app screen, etc.). The stored values364imay correspond to a passphrase (e.g., the visitor50speaks one or more pre-approved key words).

Other types of stored values364nmay be implemented. In one example, the device102and/or the IP devices120a-120nmay detect vehicle sounds (e.g., diesel engine from a UPS/FedEx delivery vehicle, a car door slam, etc.). In another example, the device102may analyze the seashell effect volume estimate. In yet another example, the device102may request an automated challenge response (e.g., spoken letters/numbers/phrases, analyzing a reply to a challenge phrase based on a known code cipher (e.g., respond with the 6th letter of 1st word, 3rd letter of 2nd word, 2nd letter of 3rd word, etc.). In still another example, the device102may make a Bluetooth or WiFi direct connection to the app310and initiate the app310to playback a sound in response to a sound emitted from the speaker220and/or initiate the app310to modulate a brightness from the phone screen302(e.g., or cause the camera flash LED to illuminate, which could work at long distances).

In the example scenario350, the visitor50may approach the device102(e.g., walk up to the front door). In the example shown, the detection of the authentication characteristics354a-354nmay be performed by the device102. However, the detection of the authentication characteristics354a-354nmay be performed by the device102and/or the IP devices120a-120n.As the visitor50approaches, the FOV170a-170band the FOV172a-172nmay capture the visitor50. The gait may be analyzed using video analytics and compared to the stored values364d.The face354bmay be detected using facial recognition analysis and compared to the stored values364a.The FOV170a-170bmay capture a top portion of the visitor50and the FOV172a-172bmay capture a bottom portion of the visitor50. The device102may combine the top portion and bottom portion of the visitor50to generate a head-to-toe video capture of the visitor50and estimate the height in order to compare to the stored values364c.For example, the device102may compensate for the difference (e.g., location, angle, resolution, etc.) between the image sensors234a-234bin order to recreate a full head-to-toe estimate of the visitor50from two separate and partial views of the visitor50.

Continuing the example, when the visitor50arrives within a pre-determined range and/or area of the device102, the device102may provide the instructions352to request information for further authentication. The user50may provide the vocal reply354a.The vocal reply354amay be analyzed to determine a vocal format/pattern and compared to the stored values364e.In another example, the instructions352may request a passphrase and the vocal reply354amay be analyzed for keywords to compare to the passphrase of the stored values364i.The instructions352may further request that the visitor50perform a specific gesture (e.g., a wave). The visitor50may perform the gesture354c.For example, the device102may analyze the gesture354cto compare to the biometric information in the stored values364g(e.g., the wave may provide a view of the palm of the hand to perform a hand measurement, the wave may be analyzed as a specific gesture to use as a form of password, etc.). The instructions352may request that the visitor50perform a specific rhythm (e.g., a beat of a particular song) and the device102may analyze the foot tapping354nto compare to the stored values364f.A comparison of the detected authentication characteristics354a-354nand the stored values364a-364nmay comprise the first authentication and/or other authentication factors that may adjust the confidence level of the identification and/or classification of the user50.

The device102may perform sensor fusion operations to analyze a progressive number of authentication steps and/or factors until a threshold of the confidence level is achieved for authentication (e.g., 99% certainty). The authentication steps may be analyzed independently and/or in aggregate (e.g., sensor fusion). Similarly, the authentication steps and/or factors may be used to determine that the visitor50is not on the whitelist360or the blacklist362. In an example, the sensor fusion may be used to implement two-factor authentication and/or multi-factor authentication. The device102may stop performing the authentication steps when a high enough confidence level is achieved to determine that the visitor50is a match or is not a match to the whitelist360or the blacklist362.

The instructions352may combine a greeting with a request for information. For example, the instructions352may be, “Good morning, may I have your name? And how can I help you?” The reply354amay be analyzed for a specific name as the passphrase364i.In another example, the instructions352may combine other audio such as a camera shutter sound with the audio recording, “my boss would like to call you back, what is your phone number?” The camera shutter sound may be used to deter a potential criminal as well as the request for personal contact information. In yet another example, the instructions352may inform the visitor50that personal information has been received such as, “Good evening, I see your cell phone MAC address. How are you?” The declaration that information has been collected may further deter a potential criminal. In still another example, the instructions352may provide context-aware intelligent conversation such as “It is 1 am and 40 degrees cold here, how can I help you?” The conversational AI may prompt the visitor50to respond in order to perform voice analysis and/or receive commands.

In some embodiments, the device102may automatically perform a command based on the detected behavior of the visitor50. If the visitor50is not detected as a person on the whitelist360or the blacklist362(e.g., an unknown visitor), the behavior may be analyzed for particular patterns. In one example, after a pre-determined amount of time of detecting the same person the device102may conclude that the visitor50is loitering and play a sound such as an alarm. In another example, if the visitor50is loitering at a suspicious time of the day then a camera sound may be played, the police may be contacted and/or a police siren may be played back to scare away the loiterer/potential criminal.

The device102may be used to authenticate people on the whitelist360such as domestic helpers. For example, the visitor50may be a domestic helper that has to use 2-part authentication for access to the premises. The device102may perform the facial analysis354band/or the vocal analysis354a.Then the domestic helper may need to use the keypad of a door lock to unlock the door. The lock may be wirelessly controlled by the device102but connected to the Internet60. The two factor authentication may provide extra security by preventing the domestic helper from allowing others to gain access by simply giving out the keypad code. Furthermore, the domestic helper knows there is video recording for additional security. The domestic helper may be an example of a category of a person on the whitelist360that has partial permissions (e.g., the domestic helper360may have permission to unlock the door, but may not have permission to enter the garage).

Referring toFIG. 8, a diagram illustrating an example of video analysis detecting a package delivery in a video frame is shown. The example video frame450may correspond to one of the video frames captured by the front capture device234a(e.g., via the front lens232).

Generally, the video frames captured by the front-facing capture device234amay be used by the device102to determine whether the package330is being delivered. For example, the field of view170may be directed to a location that would capture visitors approaching the premises. The bottom capture device234bmay also be used to determine whether the package330is being delivered. For example, the bottom capture device234bmay be used to detect the package330being placed in a particular location by a delivery person. Both sets of video frames may be used to determine whether the package330is being delivered. However, the video data captured by the front-facing capture device234amay be more likely to detect events corresponding to the package delivery before the bottom-facing capture device234b.

The example video frame450may comprise a delivery truck452and a delivery person454. In the example video frame450, the delivery person454is shown carrying the package330. For example, the front-facing capture device234amay capture images of the delivery person454approaching the premises. For example, the video frames may capture a sequence of events corresponding to the delivery truck452approaching and parking near the premises402a,the delivery person454getting out of the truck452and retrieving the package330from the truck452and then carrying the package330up to the access point404a(e.g., the front door). The video processor214may be configured to intelligently analyze the video frames to determine the behavior of the visitor (e.g., the delivery person454) and come to the conclusion that the behavior is consistent with a package delivery behavior. Sensor fusion may be implemented for further authentication (e.g., detecting a diesel sound of the truck452, audio of the delivery person454announcing themselves, etc.).

A logo460ais shown on a side of the truck452. A logo460bis shown on the hat worn by the delivery person454. A logo460cis shown on the uniform (e.g., on the chest) of the delivery person454. The video processor214may be configured to perform video analysis to determine the characteristics of the delivery truck452and/or the delivery person454. For example, the detected logos460a-460cmay be used by the video processor214to determine that the visitor454is a delivery person. Additionally, the detected logos460a-460cmay be used by the video processor214to determine which package is being delivered. For example, multiple packages may be delivered by different parcel services (e.g., different services may have different logos).

An object462may be detected as the package330. For example, the video processor214may be configured to distinguish between various types of objects. For example, the object462may be determined to have a rectangular shape (e.g., a common shape for a package), and a brown color (e.g., a common color of boxes that are shipped). The characteristics of the object462(e.g., size, shape, color, markings, etc.) detected by the video processor214may be used to determine that the object462is the package330.

An object464may be detected as the tracking ID (or shipping label)332. The tracking ID332may be used by the video processor214to correlate the package330with a particular expected delivery. In some embodiments, the companion application310may provide an API to enable the homeowner to input expected package deliveries to monitor. In some embodiments, the companion application310may provide an API to enable automatically scraping data from various delivery services to determine expected package deliveries. For example, the tracking ID332may be used to distinguish between multiple deliveries. In another example, markings on the package330may be used to distinguish between multiple deliveries. For example, particular merchants have stylized boxes for deliveries (e.g., an Amazon shipment has the Amazon logo on the box), which may be recognized by the video processor214to help distinguish between deliveries.

The video processor214may be configured to detect the delivery of the package330as early as possible (e.g., as the delivery person454is approaching the premises). Using artificial intelligence and/or machine learning to detect objects, the delivery person454may be detected by uniform, by having a package in hand, etc. Accuracy may be improved by knowing the expected time of delivery. For example, an expected time of delivery may be input by the homeowner and/or may be provided by shipment tracking provided by various parcel carriers and/or provided by various merchants (e.g., Amazon, Jet.com, Costco.com, etc.). For example, the companion application310and/or the cloud services410may be configured to read information from the various shipment tracking sources and determine an expected time of delivery. For example, the device102may be awoken from a sleep state several minutes before the package330is scheduled for delivery. In another example, the delivery person454may scan the next package to deliver and send the signal to the cloud services410.

In some embodiments, the device102may perform sensor fusion operations to determine whether the package330is being delivered. For example, the sensor fusion operations may combine various data inputs (e.g., from the signals SEN_A-SEN_N) in order to make inferences about what is happening that would not be able to be determined from one data source alone. For example, the sensor fusion operations may combine the data generated in response to the video analysis by the video processor214(e.g., the facial recognition, detecting the package330, detecting the logos460a-460c,etc.). In another example, the sensor fusion operations may combine the data about the tracking information received via the companion application310(e.g., the expected time of delivery, the parcel service, the weight/size of the package330, etc.). Other information may be combined such as voice recognition implemented by the audio processing component222. In some embodiments, the sensor fusion operations may combine fingerprint information generated by the fingerprint scanner226. For example, the pre-recorded audio instructions generated by the device102may instruct the delivery person454to touch the button136so that the fingerprint may be scanned. The combination of computer vision data, fingerprint data, tracking information data, behavior analysis, voice recognition and/or information from other sensors may increase a confidence level of the determination that the package330is being delivered.

When the video processor214determines that the behavior and/or characteristics of the visitor454indicates that a package is being delivered, the device102may generate one or more control signals to activate various responses. The control signals may be configured to control the various components of the device102(e.g., the speaker220, the microphone224, the lights140a-140b,etc.) and/or the IP devices120a-120n.The various components of the device102and/or the IP devices120a-120nmay perform the response(s) based on the control signals (e.g., CMD_A-CMD_N) received.

In one example, one of the responses may be to use the speaker220to emit a pre-recorded message to ask the delivery person454to place the package330in a particular (e.g., hidden) location. For example, the pre-recorded message may be “Hello. Thank you for delivering the package. Please bring the package to the side door”. In another example, the message may be, “The garage door is slightly opened, please slide the package under the door.” Other messages may provide instructions such as to place the package in a bush that hides the package from a view from the street.

In another example, the pre-recorded message may provide instructions to place the package330in a particular area. When the package is placed in the field of view of the bottom capture device234b,the device102may be able to monitor the package330. In an example, the tracking ID332may be scanned using the bottom capture device234b(e.g., read the barcode) and the response may be to send a delivery confirmation to the user devices106a-106nvia the companion application310.

When the device102detects someone with a package (e.g., the delivery person454carrying the detected package462), the device102may playback the instructions352. For example, the instructions may state, “Please face the barcode of the package at the bottom camera to scan”. If the delivery person454faces the tracking ID barcode332in the FOV172a-172bof the camera152, then the device102may perform the detection464. The device102a-102nmay generate one of the command signals CMD_A-CMD_N to one of the IP devices120a-120nconfigured to activate the garage door. For example, the command may instruct the garage door IP device120a-120nto open about 1 foot high. After issuing the command to open the garage door, the device102may provide another instruction352. For example, the device102may playback audio that states, “Please put the package inside the garage”. After the device102detects that the package330has been placed in the garage and the delivery person454is outside of the garage, the device102may issue another one of the commands CMD_A-CMD_N to the IP devices120a-120nto close the garage door.

Referring toFIG. 9, a diagram illustrating the outdoor hub controlling multiple remote devices is shown. An example delivery scenario500is shown. The device102is shown wall mounted next to a door502(e.g., a front door) and a garage door504. The device102may operate as the outdoor hub and/or a doorbell. In the example delivery scenario500, the garage door504is shown partially open with a gap506between the garage door504and the ground.

IP devices120a-120dare shown. In the example delivery scenario500, the IP device120amay implement a keypad door lock, the IP device120bmay implement a smart light, the IP device120cmay implement a garage door opener and the IP device120dmay implement a pressure sensor. Each of the IP devices120a-120dmay be connected wirelessly to the outdoor hub device102.

The package330is shown located on the ground on top of the pressure sensor120d. The package330is shown in the field of view172a-172bof the bottom camera152of the outdoor hub device102. In an example, analysis of the video frames captured by the bottom camera152may detect the presence of the package330in the FOV172a-172b.In another example, the pressure sensor120dmay provide the signal SEND indicating that weight has been added to the pressure sensor120d.The processor214may be configured to perform sensor fusion operations to combine the information from the signal SEN_D (e.g., the presence of weight) and the video analysis detection of the package330to make an inference that a package is present. In some embodiments, the video analysis detection of the package330may be sufficient for the processor214to determine that a package is present.

In response to the package detection, the outdoor hub device102may initiate one or more commands. The speech bubble352is shown. The instructions of the speech bubble352may state, “Please slide the package under the garage door”. The outdoor hub device102may further generate the command signal CMD_C to the garage door opener IP device120c.The command signal CMD_C may provide instructions to the IP device120cto open the garage door504a small amount to create the gap506. In one example, the video analysis may determine a height of the package330and the instructions in the command CMD_C may indicate how high to make the gap506(e.g., large enough for the package330, but keep the gap506as small as needed for the package330to create difficulty for a potential intruder. The video analysis may be performed to detect whether the delivery person has placed the package330into the garage door through the gap506. After the package330has been placed in the garage (and the delivery person is no longer close to the garage door), the outdoor hub device102may provide another command signal CMD_C to the garage door IP device120cwith instructions to close the garage door504so that the gap506is no longer available.

In another example, in response to the package detection, the outdoor hub device102may send the command signal CMD_A to the door lock IP device120a.For example, the wireless control of the door lock IP device120aby the outdoor hub device102may enable the door502to be unlocked in response to the detection of the package330by the camera152. The instructions352may playback audio stating, “Please open the door and place the package inside”. After the video analysis performed by the outdoor hub device102determines that the package330has been delivered inside the door502and the delivery person is outside again, the outdoor hub device102may send another command signal CMD_A to instruct the door lock IP device120ato lock the door502.

In yet another example, in response to the package detection, the outdoor hub device102may send the command signal CMD_B to the smart light IP device102b.For example, the wireless control of the smart light IP device120bby the outdoor hub102may enable the area near the package330to be illuminated. For example, if the door lock IP device120aand the garage door IP device120care unavailable, then the outdoor hub device102may decide that the next best option is to provide illumination for the package330(e.g., since thieves may avoid well-lit areas and/or to provide better lighting for the cameras234a-234bto capture video of package thieves).

The outdoor hub device102may be configured to intelligently determine which commands to automatically initiate in response to the detected scenario. Which commands to select may be determined based on the detected scenario, available functionality (e.g., which functionality is available from the IP devices120a-120n) and/or available battery life for the IP devices120a-120n.For example, the signals SEN_A-SEN_N may provide a battery life measurement to the outdoor hub device102.

For the example of the package delivery scenario500, the device102may decide that the preferred option is to use the garage door IP device120cto open the garage door504for the package330(e.g., does not provide full access to the home, the small gap506may prevent unauthorized access to the home, etc.). If the garage door IP device120cis unavailable (e.g., not installed, unable to communicate, low battery, etc.), then the outdoor hub device102may decide that the next preferred option is to open the door502using the door lock IP device120a(e.g., provides greater access to the home than the garage door, but still secures the package330). If both the garage door IP device120cand the door lock IP device120aare unavailable, then the outdoor hub device102may decide that the next preferred option is to use the smart light IP device120bto illuminate the area near the package330(e.g., package is easily accessed by potential thieves, but the light may deter theft attempts). The user50(if authorized to have permission based on the authentication and the whitelist360) may override the automatic command selection by the outdoor hub device102using the voice command122.

The automatic selection of the commands performed by the outdoor hub device102may change based on the scenario. For example, for a package delivery, the command CMD_C for the garage door IP device120cmay be preferred over the command CMD_A for the door lock IP device120aand/or the command CMD_B for the smart light IP device120b.However, if the video analysis determines that the visitor50carrying the package330is not a delivery person, but is instead the homeowner on the whitelist360bringing the package home then the outdoor hub device102may adjust the preference for the commands. For example, for the homeowner carrying the package330, the outdoor hub device102may issue the command CMD_A to instruct the door lock IP device120ato open the door502(e.g., for easier access to the home while carrying the package330), instead of the command CMD_C for the garage door IP device120b.

In some embodiments, the outdoor hub device102may provide communication with a central monitoring service (e.g., a third-party security alarm monitoring service, police, etc.). The verbal authentication performed by the outdoor hub device102may be configured to prevent unintentional calls to the central monitoring service. For example, if the central monitoring service feature is in an armed state (e.g., configured to contact the central monitoring service), then the outdoor hub device102may provide a number of escalating queries and/or voice challenges. In one example, the outdoor hub device102may greet the visitor50and state that the police will be notified unless the system is disarmed and prompt the visitor50to provide a passcode and/or perform facial recognition analysis. One or more pre-recorded messages may be played back to the visitor50for authentication to enable automated authentication before an employee of the central monitoring service is contacted (e.g., which may result in fees).

In some embodiments, a unique identifier associated with a wireless device (e.g., a wireless device carried by the visitor50and/or in the possession of the visitor50) may be one of the authentication characteristics354a-354n.In the example scenario500, the smartphone124and a key fob510are shown. The smartphone124and/or the key fob510are shown comprising a block (or circuit, or module)512. The block512may be the unique identifier.

The unique identifier512of the wireless device may be used as the first authentication and/or one of the authentication factors. In one example, the wireless device providing one of the unique identifier512as one of the authentication characteristics354a-354nmay be the smartphone124(or one of the user devices106a-106n) shown in association withFIG. 2. For example, the unique identifier512associated with the smartphone124(e.g., a MAC address, an electronic certificate, an automatically expiring electronic token, a randomly generated access code, a Bluetooth signal, etc.) may be one of the authentication characteristics354a-354n.In another example, the wireless device providing the unique identifier512as one of the authentication characteristics354a-354nmay be the fob device510(e.g., a key fob). For example, the unique identifier512associated with the fob device510(e.g., a MAC address, an electronic certificate, an automatically expiring electronic token, a randomly generated access code, a Bluetooth signal, etc.) may be one of the authentication characteristics354a-354n.

The door lock IP device120amay be configured to lock/unlock and/or control the door502. The door lock IP device120amay be a battery-powered device. The battery-powered door lock IP device120amay not be able to reliably perform authentication of the user50without a wired power source (e.g., the artificial intelligence-based authentication steps performed by the outdoor hub device102may be power consuming operations that may quickly drain batteries). Furthermore, a battery-powered device may not be always on and/or may introduce latency.

The outdoor wireless hub device102may be configured to perform the authentication of the user50and generate the command signal CMD_A to the door lock IP device120a.For example, the outdoor wireless hub device102may have the wired power supply to analyze the authentication factors354a-354nto perform the authentication of the user50. For example, the outdoor wireless hub device102may receive the unique identifier512from the smartphone124and/or the fob device as one of the authentication factors354a-354nand perform the comparison to the stored values364a-364n(e.g., the phone ID stored value364b) to classify the visitor50(e.g., determine whether the visitor50is on the approved list of users). If the confidence level of the classification of the user is high enough, then the outdoor wireless hub device102may generate the signal CMD_A to the door lock IP device120a.The door lock IP device120amay unlock and/or open the door502in response to the signal CMD_A.

In some embodiments, the unique phone/fob identifier512may be the first authentication and/or may provide a sufficient confidence level to classify the visitor50as an authenticated user. For example, the unique identifier512may adjust the confidence level above the pre-determined threshold without performing the analysis of the audio input (e.g., the unique identifier512may further comprise a pre-set command readable by the outdoor hub device102for opening the door502using the door lock IP device120a). In some embodiments, the voice authentication may be the first authentication and/or may provide a sufficient confidence level to classify the visitor50as an authentication user. In some embodiments, the outdoor wireless hub device102may use a combination of the unique phone/fob identifier512, the voice authentication and/or other authentication factors to achieve a confidence level above the pre-determined threshold for classifying the visitor50as an authenticated user. The number and/or type of authentication factors analyzed may be varied according to the design criteria of a particular implementation.

Referring toFIG. 10, a method (or process)550is shown. The method550may authenticate a visitor. The method550generally comprises a step (or state)552, a step (or state)554, a step (or state)556, a step (or state)558, a step (or state)560, a step (or state)562, a step (or state)564, a decision step (or state)566, a step (or state)568, a decision step (or state)570, a step (or state)572, and a step (or state)574.

The step552may start the method550. In the step554, the device102may receive the audio input. In one example, the microphone224may capture the audio input. In another example, one of the IP devices120a-120nmay implement a microphone functionality and send the captured audio input as one of the signals SEN_A-SEN_N to the outdoor hub device102. Next, in the step556, the outdoor hub device102may analyze the audio input. For example, the processor214and/or the audio processor222may perform the analysis (e.g., voice extraction, voice frequency analysis, speech to text analysis, etc.). In the step558, the processor214and/or the audio processor222may determine the command from the audio input. Next, the method550may move to the step560.

In the step560, the outdoor hub device102may perform the first authentication of the user50based on the analysis of the audio input. For example, the first authentication may be recognizing the voice of the user50. Next, in the step562, the outdoor hub device102may perform a next authentication of the user50. For example, the next authentication may be based on the authentication factors AUTH_A-AUTH_N shown in association withFIG. 2. The next authentication may be one of the other authentication factors. In one example, the other authentication factors may be based on video analysis by the processor214. In the step564, the processor214may adjust the confidence level of the authentication (e.g., based on the first authentication and/or a combination of one or more of the other authentication factors). Next, the method550may move to the decision step566.

In the decision step566, the processor214may determine whether the confidence level is above the pre-determined threshold. If the confidence level is above the threshold, the method550may move to the step568. In the step568, the outdoor hub device102may initiate the command (e.g., determined in the step558). The command may be performed by the outdoor hub device102and/or the signals CMD_A-CMD_N may be generated to cause one or more of the IP devices120a-120nto perform the command. Next, the method550may move to the step574. In the decision step566, if the confidence level is not above the threshold, the method550may move to the decision step570.

In the decision step570, the processor214may determine whether other authentication factors are available. If there are more authentication factors available, the method550may return to the step562. If there are not more authentication factors available, the method550may move to the step572. In the step572, the outdoor hub device102may not initiate or perform the command. Next, the method550may move to the step574. The step574may end the method550.

Referring toFIG. 11, a method (or process)600is shown. The method600may determine permissions for a received command The method600generally comprises a step (or state)602, a step (or state)604, a decision step (or state)606, a step (or state)608, a decision step (or state)610, a step (or state)612, a step (or state)614, a decision step (or state)616, a step (or state)618, a step (or state)620, and a step (or state)622.

The step602may start the method600. In the step604, the processor214may compare the authentication factors (e.g.,354a-354b) to the stored values364a-364nto identify and/or authenticate the visitor50. Next, the method600may move to the decision step606. In the decision step606, the processor214may determine whether the visitor50is on the blacklist362. If the user50is on the blacklist362then the method600may move to the step608. In the step608, the processor214may initiate deterrence commands. For example, the outdoor hub device102and/or the IP devices120a-120nmay be configured to sound an alarm, make camera shutter noises, contact the authorities, contact a central monitoring service employee, etc. Next, the method600may move to the step622. In the decision step606, if the user50is not on the blacklist362, then the method600may move to the decision step610.

In the decision step610, the processor214may determine whether the user50is on the whitelist360. For example, the authentication factors may be compared to the stored values364a-364nand then correlated to a user on the whitelist360. If the user50is not on the whitelist360, the method600may move to the step612. In the step612, the outdoor hub device102may perform other analysis (e.g., determine whether the visitor is an unknown delivery person, determine whether the visitor is loitering and/or performing suspicious behavior, etc.) and/or select an audio message based on video analysis and/or analysis of audio input from the visitor50. Next, the method600may move to the step622. In the decision step610, if the user50is on the whitelist362, the method600may move to the step614. In the step614, the processor214may determine the permissions of the user50. For example, each user on the whitelist362may have an associated set of permissions for available commands. Next, the method600may move to the decision step616.

In the decision step616, the processor214may determine whether the user50has permissions for the command provided (e.g., the command122). If the user50does not have permission, the method600may move to the step618. In the step618, the processor214may not perform the command. Next, the method600may move to the step622. In the decision step616, if the user50does have permission for the command, the method600may move to the step620. In the step620, the processor214may initiate the command. Next, the method600may move to the step622. The step622may end the method600.

Referring toFIG. 12, a method (or process)650is shown. The method650may determine responses using sensor fusion. The method650generally comprises a step (or state)652, a decision step (or state)654, a step (or state)656, a step (or state)658, a step (or state)660, a step (or state)662, a step (or state)664, a step (or state)666, and a step (or state)668.

The step652may start the method650. Next, the method650may move to the decision step654. In the decision step654, the processor214may determine whether the detected visitor50is on the whitelist360or the blacklist362. If the user50is on either the whitelist360or the blacklist362, the method650may move to the step656. In the step656, the processor214may manage commands and/or responses based on the known user (e.g., greet user, allow entrance, deter user, sound alarm, etc.). Next, the method650may move to the step668. In the decision step654, if the user50is not on the whitelist360or the blacklist362, then the method650may move to the step658.

In the step658, the outdoor hub device102may greet the visitor50and/or provide the instructions352(e.g., request that the visitor50do something for authentication). Next, in the step660, the outdoor hub device102may receive sensor data from the IP devices120a-120nand/or the processor214may analyze sensor data of the outdoor hub device102. In the step662, the processor214may perform sensor fusion operations on the received sensor data. For example, the sensor fusion operations may enable the processor214to make inferences based on multiple sources of data that may not be determined from a single source of data alone. Next, in the step664, the processor214may determine the response to perform based on the sensor fusion operations. In the step666, the wireless communication device216may wirelessly communicate one or more of the command signals CMD_A-CMD_N to the IP devices120a-120nbased on the selected response. The IP devices120a-120nmay perform the commands. Next, the method650may move to the step668. The step668may end the method650.

Referring toFIG. 13, a method (or process)700is shown. The method700may enable a keypad entry in response to a first authentication. The method700generally comprises a step (or state)702, a step (or state)704, a step (or state)706, a step (or state)708, a decision step (or state)710, a step (or state)712, a step (or state)714, a step (or state)716, a decision step (or state)718, a step (or state)720, and a step (or state)722.

The step702may start the method700. In the step704, the wireless communication device216may wirelessly connect to one of the IP devices120a-120nthat implements smart door lock control (e.g., the IP device120i). Next, in the step706, the outdoor hub device102may detect a visitor (e.g., based on audio input, video input, information in the signals SEN_A-SEN_N, etc.). In the step708, the processor214may perform a first authentication on the visitor50. Next, the method700may move to the decision step710.

In the decision step710, the processor214may deter mine whether the user50passes the first authentication. If the user50does not pass the first authentication, the method700may move to the step712. In the step712, the outdoor hub device102and/or the IP devices120a-120nmay monitor the unknown visitor50for suspicious behavior. Next, the method700may move to the step722. In the decision step710, if the user does pass the first authentication, the method700may move to the step714. In the step714, the processor214may send the command signal CMD_I to the IP device door lock120ito enable the keypad entry feature. Next, in the step716, the outdoor hub device102may wait for a response (e.g., the signal SEN_I) from the IP device door lock120i. For example, the IP device door lock120imay be configured to receive the keypad input from the user50and determine if the entered code matches the stored entry code. The result may be communicated back to the outdoor hub device102. Next, the method700may move to the decision step718.

In the decision step718, the processor214may analyze the signal SEN_I to determine whether the visitor50has entered the correct code into the IP device keypad120i.If the user50has not entered the correct code, the method700may move to the step712. If the user50has entered the correct code, the method700may move to the step720. In the step720, the outdoor hub device102may welcome the guest (e.g., provide a greeting, grant access, play welcoming music, etc.). Next, the method700may move to the step722. The step722may end the method700.