Computer vision to enable services

This application describes techniques for providing computer vision for manual services. In some instances, a remote system may determine that a current time is within a threshold period of time of a scheduled service and, based on the determination, send a first message requesting image data to a camera apparatus located within an environment. After sending the first message, the remote system may receive image data associated with the environment from the camera apparatus and use the image to detect an object within the environment. The remote system can then send a second message to a device of a guest and a third message to a device of a user, where each of the second message and the third message indicates that the object is within the environment. In some instances, the object can include an animal while in some instances, the object can include a person.

BACKGROUND

Homes are connected. They now include computing devices, such as desktops, tablets, entertainment systems, portable communication devices, speech-enabled devices, camera-based devices, security devices, and the like. Users interact with the computing devices in many ways, such as through mechanical devices (e.g., keyboards, mice, etc.), touch screens, motion, and gesture. In some instances, users may desire to use these devices in new ways. Provided herein are technical solutions to enable these and other types of devices to be used in new ways.

DETAILED DESCRIPTION

Techniques for providing computer-vision-based messages for manual services are described herein. Homes, offices, and other environments can include computing devices, such as those that have cameras, such as security cameras, thermal imaging cameras, and the like. The camera apparatuses described herein may include one or more network interfaces for sending image data, such as still image data and/or video data of an environment, to a remote system. For instance, a camera apparatus may receive a message from the remote system instructing the remote system to send image data to the remote system. Based on receiving the message, the camera apparatus may capture image data of the environment in which the camera apparatus is located, and send the image data to the remote system.

The environment may further include one or more security locks configured to secure one or more entrances of the environment. For instance, the environment can include a locking apparatus on a main entrance (e.g., door, window, or other type of entrance) of the environment. In some instances, the locking apparatus can include a network interface for receiving messages from other electronic devices, such as the remote system, the camera apparatus, a device of a user, a device of a guest, or the like. The message can instruct the locking apparatus to move a locking mechanism from a locked position to an unlocked position and/or from an unlocked position to a locked position. In a locked position, the locking apparatus may cause the main entrance to be secure and inaccessible, such that a guest (e.g., a delivery person) cannot enter the environment. In an unlocked position, the locking apparatus may cause the main entrance to be unsecure and inaccessible, such that the guest can enter the environment.

In some instances, a guest may attempt to perform a manual service, such as an in-home service, within the environment. An in-home service can include any type of activity that can be performed by a guest within the environment. For example, the guest can be a delivery person and the in-home service can include an in-home delivery of a physical package. As another example, the guest can include a chef and the in-home service can include a catered meal cooked by the chef. As yet another example, the guest can include a housekeeper and the in-home service can include cleaning one or more areas (e.g., rooms) within the environment.

In some instances, the guest may include a robot that performs the in-home service within the environment. For instance, the guest can include any type of mechanical or virtual artificial agent, such as an electromechanical machine, that is guided by a computer program and/or electronic circuitry to perform the in-home service. In some instances, the robot can include an autonomous robot while in some instances, the robot can include a semi-autonomous robot.

Upon arrival of the guest at the environment, the remote system can receive a message indicating that the guest is at the environment. In some instances, the remote system may receive the message from a camera apparatus that detects the guest at the environment (e.g., a camera apparatus that is located outside of the environment). Additionally or alternatively, in some instances, the remote system may receive the message from a device associated with the guest. Based on receiving the message, the remote system then provides the guest with access to the environment.

In some instances, to provide the guest with access to the environment, the remote system may send a message to the camera apparatus that instruct the camera apparatus to provide the guest with access to the environment. In such instances, the camera apparatus can receive the message, and then send an additional message (and/or relay the message) to the locking apparatus that causes the locking apparatus to unlock the entrance of the environment. In some instances, the remote system may send the message to the locking apparatus that causes the locking apparatus to unlock the entrance of the environment. Still, in some instances, the remote system may send the message to a different electronic device, which may or may not be located within the environment. The different electronic device can receive the message, and then send an additional message (and/or relay the message) to the locking apparatus that causes the locking apparatus to unlock the entrance of the environment.

After unlocking the entrance for the guest, the guest can enter the environment through the entrance and perform the service within the environment. In some instances, one or more camera apparatuses within the environment capture image data associated with the environment while the guest is within the environment. The one or more camera apparatuses then send the image data to the remote system for monitoring the environment. When the guest is finished performing the in-home service and exits the environment, the locking apparatus can return to a locked state in order to secure the environment.

In some instances, before the locking apparatus provides access to the guest, a remote system can send the guest messages indicating what is inside of the environment. For instance, the remote system can a message to the guest that alerts the guest if there is something within the environment (e.g. objects, an unfavorable conditions, etc.). Objects within the environment can include people, animals (dogs, cats, etc.), items (e.g., jewelry, etc.), or the like. Unfavorable conditions within the environment can include anything that might cause the guest to not want to enter the environment. For instance, message can notify the guest that the environment is dark, that there is a fire within the environment, or the like. In addition to providing the messages, in some instances, the remote system can further cause the environment to remain secure by not unlocking the entrance for the guest, if the environment is currently unsecure, or the remote system can cause the environment to be secure by locking the entrance of the environment, if the environment is current unsecure.

To send the messages, the remote system can store data associated with one or more scheduled services at a location corresponding to the environment. Data associated with a scheduled service can include, for example, the geographical location of the environment, a time for the scheduled service, an identity of a user, information on how to perform the service, an identity of the guest, or the like. In some instances, the remote system can store data associated with a scheduled service in association with a user profile of the user. The user profile can include data indicating a number of occupants within the environment, a number of animals within the environment, designated animal zones (e.g., animal safe zones within the environment, animal notification zones within the environment, etc.), speaker identification data of each occupant or other related people (e.g., data that can be used to identify a person based on audio data representing the person's voice), among other things.

In some instances, the remote system can further store data associated with one or more classifiers that the remote system uses to monitor the environment. An object classifier can include data indicating one or more characteristics (e.g., features) of an object, data indicating when to monitor the environment for the object, among other things. In some instances, characteristics of an object can include one or more of a size of the object, a shape of the object, a color of the object, or the like.

In some instances, the remote systems may train the one or more object classifiers using image data that the remote system receives from one or more environments. For instance, the remote system can receive, from one or more camera apparatuses, positive image data corresponding to images that include an object and negative image data corresponding to images that do not include the object. The remote system can then use the positive image data and the negative image data to train an object classifier for the object using one or more machine learning algorithms.

In some instances, the remote system can use the data associated with the scheduled services and the data associated with the object classifiers to monitor the environment. For instance, the remote system can use data associated a scheduled service to determine that a service is scheduled at the environment at a specific time. Based on the time, the remote system can determine a starting time in which to start monitoring the environment for various objects and unfavorable conditions. For instance, the remote system can use a user profile to determine that an animal, such as a dog, may be within a portion of the environment. The remote system can then use an object classifier associated with dogs to determine that the environment should be monitored for a specified time before the guest arrives at the environment. For instance, if the time of the service is scheduled for 4:00 p.m., and the monitoring time indicates that the remote system should monitor the environment for three hours before the service time, then the remote system can determine that a starting time for monitoring the environment for the dog is around 1:00 p.m.

In some instances, the remote system may use a similar process for determining when to monitor for other objects within the environment. For instance, the remote system can determine that a person, such as the user, may be home at the scheduled service time. As such, the remote system may use an object classifier for people in order to determine a starting time for monitoring the environment for the user. For instance, and using the example above, if the object classifier for people indicates that the remote system should monitor the environment for thirty minutes, the remote system can determine that a starting time for monitoring the environment for the user is around 3:30 p.m.

The remote system can use various factors when determining the threshold period of time for an object classifier. In some instances, the remote system can determine the threshold period of time based on a type of object that the object classifier represents. For example, the remote system can determine that a shorter threshold period of time should be used for people than for animals since people have a greater tendency to enter and leave the environment. As such, a detected person within the environment is more likely to leave the environment before the guest arrives than a detected animal.

In some instances, the remote system can determine the scheduled time for the scheduled service using the data associated with the scheduled service. For instance, the data can indicate a time (e.g., 4:00 p.m.) for the scheduled service. Additionally or alternatively, in some instances, the remote system can determine the scheduled time using other factors. For instance, the remote system can determine an initial time using the data associated with the scheduled service, and then update the initial time based on various factors. Factor can include among other things, weather, traffic conditions, a current location between the guest performing the scheduled service and the location of the scheduled service, average service times associated with the scheduled service (e.g., average delivery times), other scheduled services (e.g., other scheduled deliveries), or the like.

Additionally or alternatively, in some instances, the remote system can determine the scheduled time for the scheduled service using data associated with other scheduled services. For instance, the remote system can use data indicating a number of services that are going to be performed during a given period of time (e.g., a day) and an average time for each service to determine a scheduled time for each scheduled service during the given time period. Using the scheduled time for each service, the remote system can then determine the scheduled time for the scheduled service at the environment. Additionally, in some instances, the remote system can then update the scheduled time based on weather, traffic conditions, a current location between the guest performing the scheduled service and the location of the scheduled service, or the like.

In some instances, the remote system uses one or more determined starting times to monitor the environment before the guest arrives at the environment. For instance, the remote system can begin receiving image data from the camera apparatus based on the earliest starting time (e.g., 1:00 p.m. in the example above). To begin receiving the image data, the remote system can send a message to the camera apparatus at the earliest starting time that requests the image data from the camera apparatus. For instance, the camera apparatus can receive the message from the remote system. In some instances, if the camera apparatus is not already capturing image data, the camera apparatus can begin capturing image data of the environment and send the image data to the remote system. In some instances, if the camera apparatus is already capturing image data, the camera apparatus can begin sending the image data to the remote system.

In some instances, in addition to or alternatively from determining the starting times, the remote system can receive a message from a device of the guest. The message can indicate that the guest is at the environment. Based on receiving the message, the remote system can then send the message to the camera apparatus that request the image data.

The remote system can receive the image data from the camera apparatus and then use the image data to monitor the environment for objects and unfavorable conditions. In some examples, the remote system may continually monitor the environment using image data that the remote system receives from the camera apparatus. For instance, the remote system can receive a constant feed of video data from the camera apparatus and continually monitor the video data for objects. Additionally or alternatively, in some instances, the remote system may monitor the environment using image data at specified time intervals. For instance, the remote system can analyze image data of the environment every five seconds, thirty seconds, minute, or the like to monitor for objects and unfavorable conditions within the environment. In some instances, the specified time intervals may change over the course of monitoring the environment.

To monitor the image data, the remote system can analyze the image data using one or more object classifiers in order to detect objects (and similarly unfavorable conditions described below) within the environment. To analyze the image data, in some instances, the remote system can use initial image data received from the camera apparatus and generate a background image for the environment. The remote system can then process image data that the remote system later receives from the camera apparatus using the background image. For instance, the remote system can subtract the background image from images associated with image data in order to detect an object and/or unfavorable condition within the environment. Based on detecting an object and/or unfavorable condition, the remote system can then analyze the object and/or unfavorable condition using the one or more object classifiers in order to determine a type of object (e.g., a person, animal, item, etc.) and/or type of unfavorable condition (e.g., fire, low light, etc.) within the environment.

In some instances, analyzing the object using the one or more classifiers can include comparing characteristics of the object to characteristics of different objects associated with the one or more object classifiers. For instance, the remote system can identify one or more characteristics of the object within the environment. The remote system can then compare the one or more characteristics to characteristics of objects stored in each of the one or more object classifiers. Based on matching characteristics of the object to characteristics of a specified object within an object classifier, the remote system can determine which type of object is within the environment.

After identifying an object within the environment, the remote system can determine whether the object is within the environment between the starting time for the respective object and the scheduled time of the service. For instance, the remote system can determine a time associated with image data that includes the object. The remote system can then determine whether the time associated with the image data is between the starting time and the scheduled time for the service (e.g., within the threshold period of time). Based on the object being within the environment between the starting time and the scheduled time for the service, the remote system can perform different processes based on the type of object that is within the environment. For instance, the remote system can perform a first process based on the object including an animal, and perform a second process based on the object including a user.

For instance, based on the object including an animal, the remote system can send message to a device of the guest (and/or a device associated with scheduled service). The message can include one or more of an indication that the animal is within the environment, a type of animal that is within the environment, an image of the environment that includes an indication of the position of the animal. In some instances, when the service includes an in-home delivery, the message can further include an indication of a position for performing the in-home delivery (e.g., a drop-off zone (e.g., room, position, etc.) for a package), an indication of how long it will take for the object to reach the position of the in-home delivery and/or the delivery door, information specifying how to proceed with the delivery, or the like. In some instances, the guest can then decide how to proceed with the delivery based on the message.

For instance, the information can include text, an image, audio, or other type of data that the device can display to the guest. In some examples, the information can specify that the guest is not to enter the environment. In some examples, when the service includes an in-home service, the information can specify that the guest is to leave the package outside of the environment (e.g., perform an outside delivery). Additionally or alternatively, in some examples, the information can request an indication from the guest of whether the guest still wants to proceed with performing the in-home service. Based on the information, the guest can then send the remote system a message indicating whether to proceed with the in-home service.

In some instances, the device of the guest can send content to the guest, such as image or video of the environment that indicates the current position of the animal, the type of animal, the time that it will take for the animal to reach the entrance, and/or the like. Based on the content, the guest may determine that it is still safe to proceed with performing the in-home service. As such, the device of the guest can receive input to proceed with the in-home service. Based on the input, the device of the guest can send a message to the remote system to unlock the environment if the guest has indicated that he or she is comfortable continuing with the in-home service.

In some instances, based on receiving the message, the remote system can send a message to the locking apparatus that causes the locking apparatus to unlock the entrance for the guest. In some instances, the remote system can send the message to the camera apparatus and the camera apparatus can cause the locking apparatus to unlock the entrance. For instance, the camera apparatus can receive the message from the remote system. Based on the message, the camera apparatus can then send an additional message to the locking apparatus that causes the locking apparatus to unlock the entrance for the guest.

After unlocking the locking apparatus, the remote system can determine when to send an additional message to camera apparatus and/or the locking apparatus that causes the locking apparatus to once again secure (e.g., lock) the entrance of the environment. In some instances, the remote system sends the additional message based on a threshold period time passing after sending the unlocking message. For instance, the remote system can send the additional message two minutes after sending the unlocking message. In some instances, the remote system can send the additional message based on receiving a first message from the device of the guest. For instance, the device of the guest can receive input that causes the device to send the first message to the remote system. Additionally or alternatively, the device of the guest can send the first message based on the detecting that the device is outside of a threshold location (e.g., a geolocation) of the environment.

In some instances, based on detecting the animal within the environment, the remote system can further send a message to a device of the user. The message can include an indication that the animal is within the environment, an indication of how the guest is proceed with the in-home service, a request to re-schedule the service, or the like. In some instances, the user can use the user device to send a message back to the remote system indicating that the animal is secure within the environment and/or that it is safe for the guest to perform the in-home service. Based in part on receiving the message, the remote system can cause the locking mechanism to unlock the entrance using a similar process as described above.

It should be noted that, in some instances, the remote system can receive, from the device of the user, a message that includes information designating locations within the environment as designated animal zones (e.g., animal safe zones, animal notification zones, etc.). For instance, the information can designate a first area within the environment as an animal safe zone and a second area within the environment as an animal notification zone. An animal safe zone can include an area within the environment in which the animal is secure. For instance, the animal safe zone can correspond to a closed room, kennel, and/or other secure area within the environment in which the animal does not have access to guest while the guest performs the in-home service. An animal notification zone can include an area within the environment in which the animal is unsecure. For instance, the animal notification zone can correspond an area within the environment in which the animal does have access to the guest while the guest performs the in-home service.

For instance, for an in-home delivery, the animal safe zone can include an area within the environment in which animal does not have access to a zone (e.g., area within the environment) in which the delivery of the physical package is to occur. Additionally, the animal notification zone can include an area within the environment in which the animal does have access to the zone in which the delivery of the physical package is to occur.

When the user sends the information to the remote system, the remote system can use the information to determine whether to notify the guest and/or the user about the animal. For instance, the remote system may refrain from sending the messages to the guest and/or the user based on the animal being detected within the animal safe zone. Additionally, based on detecting the animal is the animal safe zone, the remote system may cause the locking apparatus to unlock the entrance for the guest when the guest arrives at the location to perform the in-home service.

Additionally, in some instances, the remote system can determine whether the animal is within an animal safe zone or an animal notification zone. For instance, the remote system can analyze image data from the environment to determine whether the animal is within an area (closed room, kennel, or other secure area) of the environment that does not have access to the guest while the guest performs the in-home service. Based on determining that the animal is within an animal safe zone of the environment, the remote system may refrain from sending the messages to the guest and/or the user. Additionally, based on detecting the animal is the animal safe zone, the remote system may cause the locking apparatus to unlock the entrance for the guest when the guest arrives at the environment to perform the in-home service.

Besides detecting animals within the environment, the remote system can further detect people (such as the user) within the environment. Based on detecting a person within the environment, the remote system can determine that the person is within the environment between the starting time associated with people and the scheduled delivery time. In some instances, the remote system can then send a message to the device of the guest that includes an indication that a person is within the environment, information specifying how to proceed with the service, an image of the environment that includes an indication of the position of the person, and/or the like. In some instances, the remote system can further cause the locking apparatus to secure the entrance if the locking apparatus is currently in an unlocked position.

In some instances, the remote system can additionally send a message to a device of the user. The message can indicate that the remote system detected a person within the environment, that the guest is near to the location, the scheduled time for the service, that the guest will not automatically enter the environment to perform the in-home service, and/or the like. In some instances, the user can then use the user device to send a message to the remote system to unlock to locking apparatus for the guest. Additionally or alternatively, in some instances, the user can move to the entrance to manually unlock the locking apparatus for the guest.

In some instances, the user can send a message to the remote system indicating how the remote system is to proceed when detecting a person within the environment. For example, the message can indicate that the remote system is to allow the guest to proceed with the in-home service when the person includes the user, but that the remote system is not to allow the guest to proceed with the in-home service when the person includes someone other than the user (and/or visa-versa). For another example, the message can indicate that the remote system is to notify the user of the person only when the person is someone other than the user.

In some instances, the remote system can further send messages to one or more additional electronic device within the environment. For instance, the remote system can determine that the user includes a wireless electronic device within the environment. In some instances, the wireless electronic device outputs audio within the environment via a speaker. The remote system can send an audio signal to the wireless electronic device that causes the wireless electronic device to output audio. In some instances, the output audio can include a notification that the guest is near to the location of the environment.

As discussed above, in addition to detecting objects within the environment, the remote system can further detect unfavorable conditions within the environment. For instance, at a time that the guest arrives at the environment and/or a threshold period of time before the guest arrives at the environment, the remote system can analyze the image data to determine whether any unfavorable conditions exist within the environment (using a similar process as above for objects). Unfavorable conditions within the environment can include anything that might cause the guest to not want to enter the environment. Based on detecting an unfavorable condition within the environment, the remote system can send a message to the device of the guest warning the guest of the unfavorable condition. Additionally, the remote system can send a message to the device of the user that warns the user of the unfavorable condition. In some instances, additionally to sending the messages, the remote system can send message to the locking apparatus that causes the locking to remain in a locked state.

Additionally, in some instances, the remote system may be configured to detect specific objects and/or unfavorable conditions at a time that the guest arrives at the environment. For instance, when the guest arrives at the environment, the remote system may analyze the image data in order to determine whether any valuable objects (e.g., electronics, jewelry, money, etc.) are located within an area that the guest is to perform the in-home service. Based on detecting a valuable object, the remote system may send a message to the device of the user indicating that the valuable object is within the area. The remote system may further send a message to the device of the guest indicating that the guest will not be provided access to the environment. In some instances, the user may wish to continue with the in-home service and send a message to the remote system. The remote system can receive the message from the user and provide the guest access to the environment.

It should be noted that, in some instances, after detecting an object and/or unfavorable condition within the environment, the remote system may monitor the object and/or unfavorable condition until the scheduled time of the service and/or until the guest arrives at the environment. For example, the remote system may detect an animal at a position within the environment that is not secure within the threshold period of time before the scheduled time of the service. Based on detecting the animal, the remote system may monitor (e.g., track) the animal until the guest arrives at the environment to perform the in-home service. While tracking the animal, the remote system may detect that the animal moves from the position that is not secure to a position within environment that is secure. Based on detecting that the animal moved the position that is secure, the remote system may determine not to send the messages to the device of the guest and/or the device of the user. The remote system may further send a message to the camera apparatus and/or the locking apparatus to unlock the entrance to the environment when the guest arrives to perform the in-home service.

For another example, the remote system may detect a person in the environment within the threshold period of time before the scheduled time for the service. Based on detecting the person, the remote system may monitor (e.g., track) the person until the guest arrives at the environment to perform the in-home service. While tracking the person, the remote system may detect that the person leaves the environment. Based on detecting that the person leaves the environment, the remote system may determine not to send the messages to the device of the guest and/or the device of the user. The remote system may further send a message to the camera apparatus and/or the locking apparatus to unlock the entrance to the environment when the guest arrives to perform the in-home service.

Additionally, in some examples, in addition to or alternatively from the remote system monitoring the environment, the camera apparatus may monitor the environment for objects and/or unfavorable conditions. For instance, the camera apparatus may receive the data associated with a scheduled service and the one or more object classifiers. In such an instance, the camera apparatus can determine a time period for monitoring the environment using the data, and monitor the environment during the time period using a similar process as described above for the remote system. Additionally, based on detecting an object and/or unfavorable condition within the environment, the camera apparatus can send messages to the device of the guest and/or the devices of the user. The camera apparatus can further send a message to the locking apparatus to lock and/or unlock the entrance of the environment.

As described herein, messages can include any type of electronic communication that electronic devices can send and receive with other electronic devices. For instance, a message can include an email message, a short message service (SMS), multimedia messages (MMS), a voicemail message, an audio signal, or any other type of electronic communication that an electronic device can send to another electronic device. In some instances, an electronic device may use messages to provide indications, notifications, alerts, and/or requests to another electronic device.

Additionally, as discussed above, image data can include still image data and video data. Still image data can represent images captured within an environment that the remote system receives from a camera apparatus at given time intervals. Video data can represent a video captured within the environment that the remote system receives from a camera apparatus. For instance, formats for image data can include Joint Photographic Expert Group (JPEG), Exchangeable Image File Format (Exif), Tagged Image File Format (TIFF), Graphics Interchange Format (GIF), Portable Network Graphics (PNG), Better Portable Graphics (BPG) or any other type of image data format that electronic devices can send and receive with one another.

As a preliminary note,FIGS. 1 and 2are described with respect to a manual service including an in-home delivery for clarity reasons. For instance,FIGS. 1 and 2illustrate a driver (e.g., a guest) arriving at the environment to perform an in-home delivery (e.g., an in-home service) of a physical package, and receiving a message using a driver device (e.g., a user device). However, in other examples, the manual service can include any type of activity that can be performed by a guest within the environment and/or outside of the environment.

FIG. 1is a schematic diagram of a first example of performing a delivery at an environment102. In the example ofFIG. 1, the environment102that includes one or more camera apparatuses104(1)-(4) that each include a camera106(althoughFIG. 1only illustrates details of camera apparatus104(1) for clarity purposes) configured to acquire image data (e.g., still image data, video data, etc.) of different respective areas108(1)-(3) (e.g., rooms, zones, etc.) within the environment102and outside of the environment102. For instance, the camera apparatus104(1) is configured to acquire image data from the first area108(1) within the environment102, the camera apparatus104(2) is configured to acquire image data from the second area108(2) within the environment102, the camera apparatus104(3) is configured to acquire image data from the third area108(3) within the environment102, and the camera apparatus104(4) is configured to acquire image data from outside of the environment.

Environment102further includes a locking apparatus110that includes a locking mechanism112configured to secure the main entrance114of the environment102. In addition, the locking apparatus110further includes physical controls116that are configured to cause the locking apparatus110to move the locking mechanism from a locked position to an unlocked position and/or from an unlocked position to a locked position. In a locked position, the locking apparatus110causes the main entrance to be secure and inaccessible, such that a driver118cannot enter the environment to perform an in-home delivery. In an unlocked position, the locking apparatus110causes the main entrance to be accessible, such that the driver118can enter the environment102to perform in-home deliveries.

In some instances, a remote system120associated with network accessible resource(s)122is configured to send messages to the camera apparatuses104(1)-(4) that request the camera apparatuses104(1)-(4) to send acquired image data to the remote system120via a network connection124. For instance, the remote system120can determine that the driver118is scheduled to make a delivery at the environment102. In some instances, the remote system120makes the determination using stored data associated with the scheduled delivery. The remote system120can then determine starting time for monitoring the environment102using one or more object classifiers. In some instances, the starting time can include a threshold period of time before the scheduled delivery in which the remote system is to monitor the environment. Based on the starting time, the remote system120can send a message to one or more of the camera apparatuses104(1)-(4) that requests the image data.

In some instances, in addition to or alternatively from determining the starting time, the remote system120can receive a message indicating the driver118is at the environment. For instance, the driver device132can send the message to the remote system120when the driver118arrives at the environment. Additionally, in some instance, the camera apparatus104(4) can send the message to the remote system120after detecting the driver118outside of the environment102. Moreover, in some instance, the remote system120can determine that the driver118is at the environment based on geographical data associated with the driver device132. For instance, the remote system120can receive global positioning data (GPS) data associated with the driver device132and use the GPS data to determine that the driver118is at the environment102. Based on determining that the driver118is at the environment102, the remote system120can send the message to one or more of the camera apparatuses104(1)-(3) that requests the image data.

The camera apparatuses104(1)-(3) can receive the message from the remote system120and based on the message, send the image data to the remote system120. Using the image data, the remote system120can determine whether an object (e.g., animal126) or other unfavorable condition of potential concern is within a area108(1)-(3) of the environment102during the time window. In some instances, a user128associated with the scheduled delivery can designate one or more of the areas (e.g., areas108(1)-(3)) within the environment102as animal safe zones and one or more of the areas (e.g., areas108(1)-(3)) within the environment102as animal notification zones. An animal safe zone can include an area within the environment102in which the animal126is secure. For instance, the animal safe zone can correspond to a closed room, kennel, and/or other secure location within the environment102in which the animal126does not have access to the main entrance114and/or a delivery zone for a physical package. An animal notification zone can include an area within the environment102in which the animal126is unsecure. For instance, the animal notification zone can correspond an area within the environment102that includes access to the main entrance114and/or a delivery zone for a physical package.

For instance, in the example ofFIG. 1, the user128associated with the scheduled delivery can designate the first area108(1) within the environment102as an animal notification zone, and the second area108(2) and the third area108(3) within the environment102as an animal safe zone. Using the image data from the camera apparatuses104(1)-(3), the remote system120can determine that the animal126is within the first area108(1) of the environment102. Since the animal126is within an animal notification zone of the environment102, the remote system120can send a first message130to a driver device132of the driver118and a second message134to a user device136of the user128.

The first message130can include one or more of an indication that the animal126is within the environment102(e.g., within a notification zone), a type of animal126that is within the environment102, information on how to proceed with the delivery, an image of the environment102that includes an indication of the position of the animal126and/or an indication of a position for delivering the package (e.g., a delivery zone), or the like. For instance, in some instances, the information can specify that the driver118is to stop the delivery. In some instances, the information can specify that the driver118is to leave the package outside of the environment102(e.g., perform an outside delivery). Still, in some instances, the information can result in the delivery device outputting a query regarding whether the driver118still wants to proceed with performing the in-home delivery.

For instance, the driver device132can display content associated with the environment102. The content can include an image of the environment102that indicates a current position of the animal126, the indication of the type of animal, an indication of the time that it will take for the animal126to reach the main entrance114, and/or the like. Based on the image, the driver118may determine that it is still safe to proceed with performing the in-home delivery. As such, the driver device132can receive input from the driver118indicating that it is still safe and based on the input, send a message to the remote system120indicating that the driver118still wants to proceed with the in-home delivery. The remote system120can receive the message from the driver device132and provide the driver118with access to the environment102.

In some instances, to provide access to the driver, the remote system120, sends a message to the locking apparatus110that causes the locking apparatus110to unlock the main entrance114for the driver118. Additionally or alternatively, in some instances, the remote system120sends a message to one or more of the camera apparatuses104(1)-(4) that instruct the one or more camera apparatuses104(1)-(4) to provide the driver118with access to the environment102. For instance, the camera apparatus104(4) can receive a message from the remote system120to unlock the locking apparatus110. The camera apparatus104(4) can then send a message to the locking apparatus110that causes the locking apparatus to unlock the main entrance114. In some instances, the camera apparatus104(4) sends the message to the locking apparatus110using short-range communication, such as IEEE 802.15.4 (ZIGBEE), IEEE 802.15.1 (BLUETOOTH), IEEE 802.11 (WiFi), or any other PAN communication protocol.

Additionally, in some instance, the first message130can include contact information associated with the user128. The driver118can then use the contact information to communicate with the user128in order to determine whether the driver118should still access the environment. For instance, the driver118can use the driver device132to send and receive message with the user driver136in order to inquire whether the driver118should still perform the in-home delivery even though the animal126was detected.

The second message134can include an indication that the animal126is within the environment102(e.g., within a notification zone), a message that the driver118is not going to perform the in-home delivery, a request to re-schedule the delivery, or the like. In some instances, the user128can use the user device136to send a message back to the remote system120indicating that the animal126is secure within the environment102and/or that the driver118should still perform the in-home delivery. Based on receiving the message the remote system can cause the locking apparatus110to unlock the main entrance114using as similar process as described above for the driver118.

As illustrated inFIG. 1, the camera apparatus104(1) (and similarly the camera apparatuses104(2)-(4)) further include network interface(s)138, processor(s)140, and memory142, which stores a controller144. The controller144is configured to cause the camera apparatus104(1) to start receiving image data (e.g., activate) and/or stop receiving image data (e.g., deactivate). Additionally, the controller144is further configured to cause the camera apparatus104(1) to send image data to the remote system120. For instance, the controller144can cause the camera apparatus104(1) to both activate and send image data to the remote system120based on receiving messages from the remote system120.

The locking apparatus110further includes network interface(s)146, processor(s)148, and memory150, which stores controller152. The controller152is configured to cause the locking mechanism112to move from a locked position to an unlocked position and/or from the unlocked position to the locked position. For instance, the controller152can cause the locking mechanism112to move from a locked position to an unlocked position based on the locking apparatus110receiving a message from an electronic device, such as one or more of the camera apparatus104(1)-(4), the remote system120, the driver device132, the user device136, or the like. The controller can further cause the locking mechanism112to move from the unlocked position to the locked position based on receiving an additional message from the electronic device.

Also illustrated inFIG. 1, the driver device132further includes processor(s)154, network interface(s)156, and memory158, which stores application160. User device136further includes processor(s)162, network interface(s)164, and memory166, which stores application168. In some instances, the driver device132can use the application160to send and receive notifications with the remote system120and/or the user device136. Additionally, the user device136can use the application168to send and receive messages with the remote system120and/or the driver device132.

As used herein, a processor (e.g., processor(s)140, processor(s)148, processor(s)154, and processor(s)162) may include multiple processors and/or a processor having multiple cores. Further, the processors may comprise one or more cores of different types. For example, the processors may include application processor units, graphic processing units, and so forth. In one implementation, the processor may comprise a microcontroller.

Further, functional modules are shown stored in the respective memories, the same functionality may alternatively be implemented in hardware, firmware, application specific integrated circuits, field programmable gate arrays, or as a system on a chip (SoC). In addition, while not illustrated, each respective memory discussed herein may include at least one operating system (OS) module that is configured to manage hardware resource devices such as the network interface(s), the I/O devices of the respective apparatuses, and so forth, and provide various services to applications or modules executing on the processors. Such OS module may implement a variant of the FreeBSD™ operating system as promulgated by the FreeBSD Project; other UNIX or UNIX-like variants; a variation of the Linux operating system as promulgated by Linus Torvalds; the Windows® operating system from Microsoft Corporation of Redmond, Wash., USA; LynxOS as promulgated by Lynx Software Technologies, Inc. of San Jose, Calif.; Operating System Embedded (Enea OSE) as promulgated by ENEA AB of Sweden; and so forth.

Each of the network interface(s)138, network interface(s)146, network interface(s)156, and network interface(s)164may enable communications between camera apparatus104(1), locking apparatus110, delivery device132, user device136, respectively and other networked devices, such as remote system120. Such network interface(s) can include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications over a network.

For instances, each of the network interface(s)138, network interface(s)146, network interface(s)156, and network interface(s)164may include a personal area network (PAN) component to enable communications over one or more short-range wireless communication channels. For instance, the PAN component may enable communications compliant with at least one of the following standards IEEE 802.15.4 (ZIGBEE), IEEE 802.15.1 (BLUETOOTH), IEEE 802.11 (WiFi), or any other PAN communication protocol. Furthermore, each of the network interface(s)138, network interface(s)146, network interface(s)156, and network interface(s)164may include a wide area network (WAN) component to enable communication over a wide area network.

The network124may represent an array or wired networks, wireless networks (e.g., WiFi), or combinations thereof. The remote system120may generally refer to a network-accessible platform—or “cloud-based service”—implemented as a computing infrastructure of processors, storage, software, data access, and so forth that is maintained and accessible via the network124, such as the Internet. As such, the remote system may comprise one or more devices, which collectively may comprise a remote device. Cloud-based services may not require end-user knowledge of the physical location and configuration of the system that delivers the services. Common expressions associated with cloud-based services, such as the remote system120, include “on-demand computing”, “software as a service (SaaS)”, “platform computing”, “network accessible platform”, and so forth.

In some instances, the remote system120may be local to the environment102. For instance, the remote system120can be located within the environment102. In some instances, some or all of the functionality of the remote system120may be performed by one of the camera apparatuses104(1)-(3).

FIG. 2is a schematic diagram of a second example of performing a delivery at the environment102. In the example ofFIG. 2, the environment102that includes the camera apparatus104(1) and the locking apparatus110. In the example ofFIG. 2, the remote system120is configured to analyze the image data of the environment102during a specified time interval before a scheduled delivery in order to detect whether the user128(and/or any other person) is within the environment102. Based on detecting the user128, the remote system120is configured to send a message130to the driver device132and send a second message134to the user device136(not shown for illustrative purposes). The remote system120is further configured to cause the locking apparatus110to move to a locked position.

For instance, similar to the example ofFIG. 1above, the remote system120can determine a starting time for monitoring the environment102using a delivery time for a scheduled delivery at the environment102. In some instances, the remote system120determine the starting time using a threshold period of time from the delivery time of the scheduled delivery. The remote system120can then send a message to the camera apparatuses104(1)-(3) at the location that causes the camera apparatuses104(1)-(3) to start sending image data to the remote system. In some instances, the camera apparatuses104(1)-(3) send a constant stream of image data (e.g., a video data) to the remote system120while in some instances, the camera apparatuses104(1)-(3) send image data to the remote system120at given time intervals. In some instance, the camera apparatuses104(1)-(3) can send image data to the remote system120based on determining that the driver118is near to the environment102.

The remote system120analyzes the image data from the camera apparatuses104(1)-(3) using a similar process as described above. While analyzing the image data, the remote system120can detect that the user128is within the environment102. The remote system120can further determine that a time at which the user128is detected within the environment102is within a threshold period of time of the delivery time for the scheduled delivery. In some examples, the remote system determines the threshold period of time using an object classifier associated with people.

Based on detecting the user128, the remote system120can cause the locking apparatus110to move to a locked position if the locking apparatus110is currently in an unlocked position. For instance, the remote system can send an message to the locking apparatus110requesting the locking apparatus110to move the locking mechanism112from the locked position to the unlocked position. The remote system120can further send a first message130to the driver device132. The first message130can include one or more of an indication that the user128is within the environment102, information on how to proceed with the delivery, or the like. For instance, the information can specify that the driver118is to knock on the main entrance114of the environment102when the driver118arrives at the environment102.

In some instances, the remote system further sends a second message134to the user device136. The second message134can include an alert that the driver118is near to the environment102and a notification on how the driver118is going to proceed with the delivery. In some instances, the user128can receive the second message134and physically unlock the locking apparatus110for the driver118. Additionally or alternatively, in some instances, the user128can use the user device136to send a message back to the remote system120that indicates that the driver118can still perform the in-home delivery. Based on receiving the message, the remote system can cause the locking apparatus110to unlock the main entrance114using a similar process as described above.

Also illustrated inFIG. 2, the environment includes an electronic device202. In some instances, the remote system120can send a message204to the electronic device202based on detecting the user128within the environment. In some instances, the remote system120only sends the message204to the electronic device202based on determining that the user128is near the electronic device202(e.g., within the area108(3) of the environment102.

The message204can include an alert that the driver118is within proximity to the environment102. For instance, the message204can indicate an estimated time that the driver118is going to arrive at the environment102. In some instances, the message204can include an audio signal that the electronic device202outputs via speaker(s)206. Additionally or alternatively, in some instances, the message204can include a message that the electronic device202displays to the user128via a display device.

In some instances, when the environment102includes the electronic device,202, the remote system120can use an identifier associated with the electronic device202to determine a location of the user128within the environment102. For instance, the remote system120can use the identifier to determine a position within the environment102of the electronic device202(e.g., the third area108(3)). The remote system120can then determine that the user128is near to the position of the electronic device202based on detecting a voice of the user128received by the electronic device202. In some instances, the remote system120can then use the determined position of the user128when monitoring the environment102.

In the example ofFIG. 2, the electronic device202further includes processor(s)208, network interface(s)210, and memory212, which stores application214. The processor(s)208, network interface(s)210, and memory212can respectively be similar to the processor(s), network interface(s), and memory described above. Additionally, the application214can configure the electronic device202to send and receive messages (e.g., notifications, or the like) with other electronic devices, such as the remote system120.

It should be noted that, in some instances, the remote system120can detect people other than the user128within the environment102. For instance, the remote system120can detect a family member and/or a guest of the user128within the environment102. In some examples, when detecting a person other than the user128within the environment102, the remote system120can perform a similar process as described above for the user128. Additionally or alternatively, in some instances, when detecting a person other than the user128within the environment102, the remote system120may send different notifications to the driver118and/or the user128.

For instance, the remote system120may send a message to the user device134indicating that a person other than the user is within the environment102. The message can further ask the user128whether he/she would like to continue with the scheduled delivery. In some instances, the user128can use the user device134to send a message back to the remote system120instructing the remote system120to cancel the scheduled delivery. Based on receiving the message, the remote system120can send a message to the driver device132that notifies the driver118that a person other than the user128is within the environment102. The message can further include information for the driver118to cancel the delivery at the environment102.

It should further be noted that, in some instances, each of driver device132and user device134can represent any type of electronic device having one or more processors and one or more memories. Such electronic devices can include a variety of categories or classes of devices, such as traditional client-type devices, desktop computer-type devices, mobile devices, special purpose-type devices, embedded-type devices, and/or wearable-type devices.

Finally,FIGS. 1 and 2describe an example environment102that includes a structure, such as a house, business, or some other type of structure. In some instances, the environment can include areas beyond the structure. For instance, in some examples, the environment can include the structure (e.g., the house) along with a yard surrounding the structure. In such examples, the environment may be secure by a fence and/or gate that surrounds at least a portion of the environment. The fence or gate can correspond to the entrance of the environment fromFIGS. 1 and 2. For instance, to provide access to the environment, the remote system may cause a locking apparatus on the fence and/or gate to unlock such that the delivery driver can enter.

Additionally or alternatively, in some instances, the environment may not include a structure, such as a home. For example, the environment can include an area of land that is at least partially secure by a gate and/or fence (e.g., the entrance fromFIGS. 1 and 2). In such examples, the remote system may provide the delivery driver access to the environment (e.g., the area of land) by unlocking a locking apparatus associated with the fence and/or gate.

FIG. 3is a block diagram illustrating relevant functional components of the network accessible resource(s)122of the remote system120. In some instances, the network accessible resource(s)122may include other components not shown inFIG. 3.

Logical functionality of the remote system120may be implemented by one or more processor(s)302, associated memory304, and software that is stored in the memory304and executed by the processor(s)302. As discussed above, the processor(s)302may include multiple processors and/or a processor having multiple cores. Further, the processor(s)302may comprise one or more cores of different types. For example, the processor(s)302may include application processor units, graphic processing units, and so forth. In one implementation, the processor(s)302may comprise a microcontroller.

Further, functional modules are shown stored in the memory304, the same functionality may alternatively be implemented in hardware, firmware, application specific integrated circuits, field programmable gate arrays, or as a system on a chip (SoC). In addition, while not illustrated, each respective memory discussed herein may include at least one operating system (OS) module that is configured to manage hardware resource devices such as the network interface(s), the I/O devices of the respective apparatuses, and so forth, and provide various services to applications or modules executing on the processor(s)302. Such OS module may implement a variant of the FreeBSD™ operating system as promulgated by the FreeBSD Project; other UNIX™ or UNIX-like variants; a variation of the Linux™ operating system as promulgated by Linus Torvalds; the Windows® operating system from Microsoft Corporation of Redmond, Wash., USA; LynxOS as promulgated by Lynx Software Technologies, Inc. of San Jose, Calif.; Operating System Embedded (Enea OSE) as promulgated by ENEA AB of Sweden; and so forth.

The network interface(s)306may include can include one or more network interface controllers (NICs) or other types of transceiver devices to send and receive communications over a network. For instance, network interface(s)306may include a personal area network (PAN) component to enable communications over one or more short-range wireless communication channels. For instance, the PAN component may enable communications compliant with at least one of the following standards IEEE 802.15.4 (ZIGBEE), IEEE 802.15.1 (BLUETOOTH), IEEE 802.11 (WiFi), or any other PAN communication protocol. Furthermore, the network interface(s)306may include a wide area network (WAN) component to enable communication over a wide area network.

The remote system120can include a training component308for training object classifiers stored in the classifier database310. In some instances, the training component308trains the object classifiers using image data312. For instance, to train an object classifier for an object (e.g., an animal, person, item, etc.), the training component308can receive image data (e.g., positive image data314) corresponding to images that include the object and image data (e.g., negative image data316) corresponding to images that do not include the object. The training component308can then use the positive image data314and the negative image data316to train the object classifier using one or more machine learning algorithms.

In some instances, the training component308creates one or more object classifiers that are specific to an environment, such as environment102. For instance, the training component308can receive positive image data314and negative image data316from various locations within the environment. The training component308can then use the positive image data314and the negative image data316to train the one or more object classifiers that are specific to the environment. For instance, the training component308can create an object classifier for each type of object (e.g., a person, dog, cat, etc.) that is within the environment.

In some instances, the user provides the remote system120with a list of objects within the environment. The training component308can then use the list of objects in order to create an object classifier for each object. Additionally or alternatively, in some instances, the training component308can identify different objects within the environment using the positive image data314and the negative image data316. The training component308can then create an object classifier for each identified object.

In some instances, the training component308creates object classifiers that the remote system120uses for multiple environments. For instance, the training component308can receive positive image data314and negative image data316from multiple environments. The training component can then use the position image data314and the negative image data316to train object classifiers for each of the objects detected within the environments.

In some instances, the remote system120further determines a threshold period of time for monitoring an object associated with a created object classifier, and then associate the threshold period of time with the created object classifier. In some instances, the remote system120can determine the threshold period of time based on a type of object. For instance, the remote system120can use a greater threshold period time for animals than for people. In some instances, the remote system120can receive the threshold period of time from another source. For instance, a user that occupies an environment can provide the remote system120with a threshold period of time for each object within the environment.

In some instances, the remote system120uses a similar process as described above for object classifiers to create one or more classifiers for unfavorable conditions within the environment. For instance, the remote system120can create classifiers for fires, flooding, or the like.

The remote system120can further include a scheduling component318for creating records (e.g., data) for scheduled services (e.g., scheduled deliveries) stored in the schedules database320. For instance, the remote system120can receive information associated with a scheduled service. The information can include a service location, a scheduled service time, an identity of the user, contact information for the user (e.g., phone number, email, etc.), an identity of a guest performing the service, contact information for the guest (e.g., phone number, email, etc.), information identifying the device of the guest (e.g., MAC address, etc.), or the like. The scheduling component318can then use the information associated with the scheduled service to create a record for the scheduled service. After creating the record, the scheduling component318can store the record for the scheduled service in the schedules database320.

The remote system120can further include a profile component322for creating user profiles stored in the profiles database324. For instance, the remote system120can receive information associated with a user. The information can include an identity of the user, contact information for the user (email address, phone number for a user device, etc.), a location of an environment associated with the user, identities of people that occupy the environment, a list of animals that occupy the environment, designations associated with different locations within the environment (e.g., animal safe zone or animal notification zone), content/watch history, order/purchase history for online shopping, location/GPS history, speaker ID profile, or the like. The profile component322can then use the information associated with the user to create a user profile for the user. After creating the user profile, the profile component322can store the user profile in the profiles database324.

In some instances, the remote system120can use the profile component322to create an account that includes multiple user profiles. For instance, the remote system120can use the profile component322can create an account for an environment using data associated with the environment. The data can include a location of the environment (e.g., home address), contact information (e.g., home phone number) for the environment, payment information (e.g., credit card number), or the like. The remote system120can then use the profile component322to create user profiles for one or more occupants (e.g., users) within the environment.

A user profile for an occupant can include information that is associated with the occupant. For instance, the user profile can include information such as an identity of the occupant, contact information for the occupant (email address, phone number for a user device, etc.), content/watch history, order/purchase history for online shopping, location/GPS history, speaker ID profile, or the like. The remote system120can then store the user profiles for the one or more occupants within the account for the environment.

The remote system120can further include a monitoring component326for monitoring environments. For instance, the remote system120can use the monitoring component326to determine when to monitor an environment for objects. To determine when to monitor an environment, the monitoring component326can analyze a record of a scheduled service from the schedules database320to determine a scheduled time for the service and scheduled location for the service. Additionally, in some instances, the monitoring component326can analyze a user profile associated with the scheduled service (e.g., user profile of the user) in order to identify objects that may be within the environment. Using the scheduled time for the service and the user profile, the monitoring component326can determine a starting time for monitoring the environment.

For instance, the monitoring component326can determine that one or more animals occupy the environment. For each animal, and using an object classifier for the respective animal, the monitoring component326can determine a starting time for monitoring the environment. Based on the starting times for the animals, the monitoring component326can cause the remote system120to send a request for monitored image data328to one or more camera apparatus within the environment. The monitored image data328can include images and/or a video feed of the environment associated with the scheduled delivery.

In some instances, the monitoring component326monitors the monitored image data328between the starting time and the time of the scheduled service (and/or when the guest arrives at the environment). To monitor the environment, the monitoring component326can analyze the monitored image data328to detect objects within the environment. For instance, the monitoring component326can use initial image data from the monitored image data328to create a background image of the environment. The monitoring component326can then process the monitored image data328using the background image. For instance, the monitoring component326can subtract the background image from images associated with the monitored image data328in order to detect an object within the environment. The monitoring component326can then analyze the object using the one or more object classifiers in order to determine a type of object (e.g., a person or animal, such as a dog, cat, etc.) within the environment.

It should be noted that, in some instances, the monitoring component326can use other image processing techniques for detecting objects within the environment. For instance, the monitoring component326can use edge detection, image analysis, noise filters, math processing, convolution, trend removal, enhancement, or the like to detect objects within the monitored image data326. The monitoring component326can then use the object classifiers stored in the classifier database310to determine the type of the object within the environment.

In some instances, after detecting an object within the environment, the monitoring component326can further track the object. For instance, the monitoring component326can track the object from a time of the detection to the scheduled time for the service (and/or when the guest arrives at the environment). In some instances, when the object includes an animal, the monitoring component326tracks the object to determine whether the object moves from an animal notification zone to a safe zone and/or from a safe zone to an animal notification zone within the environment. In some instances, when the object includes an animal or a person, the monitoring component326tracks the object to determine whether the object exits the environment before the scheduled delivery.

In some instances, the monitoring component326can further detect unfavorable conditions within the environment. For instances, the monitoring component326can use the classifiers for the unfavorable conditions to detect the unfavorable conditions using a similar process as objects described above. Additionally, in some instances, the monitoring component326can detect unfavorable conditions using other techniques, such as one or more sensors within the environment. For instance, the monitoring component326may receive a message indicating a unfavorable condition from a camera apparatus (or other electronic device) within the environment that detects unfavorable conditions using sensors. For instance, the camera apparatus (or other electronic device) may include sensors that detect fires, low light situations, gas, water, or the like. The camera apparatus (or other electronic device) can then send the message to the remote system120based on detecting a unfavorable condition using the sensors.

The remote system120can further include a messaging component330for sending messages to the guest and the user. For instance, after the monitoring component326detects an object within the environment, the messaging component330can generate and send messages to various electronic devices. For instance, the messaging component330can generate the first message130for the guest and the second message134for the user. The messaging component330can then cause the remote system120to send the first message to the guest and send the second message to the user.

FIG. 4is a schematic diagram of an example of a remote system detecting objects within an environment402. For instance, in the example ofFIG. 4, the environment402includes three separate areas406(1)-(3) (e.g., areas, such as rooms). In some instances, each of the areas406(1)-(3) may be monitored by one or more camera apparatus that capture image data of the respective location. As illustrated, the remote system detects a first object404(1) in the first area406(1) and a second object404(2) in the second area406(2).

In some instances, the remote system can determine whether to notify a guest of the objects404(1)-(2) within the environment402based on the positions of the objects404(1)-(2). For instance, the remote system may determine that each of the objects404(1)-(2) includes an animal. The remote system can further determine that the first animal (e.g., object404(1)) is within an animal notification zone of the environment402since the first animal has access to the entrance408, and that the second animal (e.g., object404(2)) is within an animal safe zone of the environment402since the second animal does not have access to the entrance. Based on the determinations, the remote system may send a message to the guest indicating that the first animal is not secure. In some instances, the message may further specify the type of animal.

In some instances, the remote system may detect objects402(1)-(2) other than animals. For instance, the remote system may determine that each of the objects402(1)-(2) includes an item, such a jewelry. The remote system may then determine that the first piece of jewelry (e.g., object404(1)) is not secure since the guest can see the first piece of jewelry from the entrance, and that the second piece of jewelry (e.g., object404(2)) is secure since the second piece of jewelry is locked in the second location404(2). Based on the determinations, the remote system may send a message to the device of the guest indicating that the remote system will not provide access to the environment402.

FIG. 5is a schematic diagram of an example user interface502of a user device that a user may use to inform a delivery driver with a position (e.g., zone) for performing an in-home delivery. For instance, in the example ofFIG. 5, the remote system may receive image data from a camera apparatus located at an entrance504of an environment506. The remote system may then send the image data to a device associated with a user. Based on receiving the image data, the device of the user can use the image data to generate the user interface502. For instance, inFIG. 5, the user interface502includes an image of the entrance504of the environment506.

In some instances, the user can then use the user interface502to specify a position within the environment in which a delivery driver is to perform an in-home delivery (e.g., drop packages off). In some instances, the device receives input from the user corresponding to the position within the environment506for performing the in-home delivery. For instance, in the example ofFIG. 5, the user draws a zone508around the position in which the user wants the delivery driver to leave the package.

FIG. 6is a schematic diagram of an example user interface602that a remote system may present to a delivery driver performing an in-home delivery. For instance, in the example ofFIG. 6, the remote system may receive image data from a camera apparatus located at an entrance604of an environment606at a time the delivery driver arrives at the environment606. The remote system may then send the image data to a device associated with delivery driver. Based on receiving the image data, the device of the delivery driver can use the image data to generate the user interface602. For instance, inFIG. 6, the user interface602includes an image of the entrance604of the environment606.

In some instances, the user interface602further indicates the position608(e.g., zone) for performing the in-home delivery, which may represent the position508indicated by the user inFIG. 5. Additionally, the user interface602illustrates a position of an object610within the environment606. In some instances, the delivery driver can use the user interface602to determine whether to proceed with the in-home delivery, or whether to perform an outdoor delivery (e.g., leave the package outside of the environment606).

FIG. 7illustrates a flow diagram of an example process700for providing computer vision-based alerts for an in-home delivery. This process (as well as each process described herein) is illustrated as a logical flow graph, each operation of which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more non-transitory computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement the process. In some instances, the operations performed in the processes700-1200are performed by the remote system120. In some instances, the operations performed by the processes700-1200can be performed by another electronic device, such as the camera apparatus104(1).

At702, the remote system120receives data associated with a scheduled service. In some instances, the remote system120can retrieve the data associated with the scheduled service from a schedules database. In some instances, the remote system120can receive the data associated with the scheduled service from an electronic device, such as an online marketplace. The data associated with the scheduled service can indicate, among other things, a location of the service and a scheduled time for the service.

At704, the remote system120determines a starting time for monitoring an environment associated with the scheduled service. For instance, the remote system120can determine a starting time for analyzing image data associated with the environment. In some instances, the remote system120can use the scheduled time for the service to determine the starting time for monitoring the image data. For instance, the remote system120can determine the starting time based on a threshold period of time before the scheduled time for the service. In some instances, the remote system120determines the threshold period of time using one or more object classifiers for objects that may occupy the environment. Additionally or alternatively, in some instances, the remote system120can determine the starting time based on detecting that a guest associated with the scheduled service is near the environment.

At706, the remote system120sends a message to a camera apparatus requesting image data. For instance, the remote system120can send the message to the camera apparatus at the starting time for monitoring the environment. The message can cause the camera apparatus to start capturing image data within the environment, and then send the image data to the remote system120.

At708, the remote system120receives the image data from the camera apparatus. In some instances, the image data can include video data that the remote system120receives from the camera apparatus. For instance, the remote system120can receive the video image data from the camera apparatus from the starting time for monitoring the environment until the scheduled time of the service. Additionally or alternatively, in some instances, the image data can include still image data that the remote system120receives from the camera apparatus. For instance, the remote system120can receive still image data corresponding to an image within the environment at different time intervals.

At710, the remote system120analyzes the image data using one or more classifiers. For instance, the remote system120can store object classifiers for different objects (and/or classifiers for unfavorable conditions) in a classifier database. In some instances, a classifier can be associated with a specific object, such as a person or type of animal, and include characteristics representing the specific object. For instance, characteristics can include one or more of a size of the object, a shape of the object, a color of the object, or the like. In some instances, to analyze the image data, the remote system120uses one or more image processing techniques to identify objects within the image data. The remote system120then compares the objects within the image data to one or more classifiers from the classifier database.

At712, the remote system120detects an object within the environment. For instance, based on analyzing the image data, the remote system120can determine that an object is within the environment. In some instances, the object can include an animal within the environment. In some instances, the object can include a person within the environment. Additionally or alternatively, in some instance, the object can include an item within the environment. The remote system120continue to track the object using the image data after detecting the object. For instance, the remote system120can track the object using the image data until the guest associated with the scheduled service arrives at the environment.

At714, the remote system120sends a first message to a first electronic device. For instance, the remote system120can send a message to a device of a guest associated with the scheduled service (e.g., device associated with the scheduled service) that indicates that the object is within the environment. Based on the scheduled service including an in-home delivery, the message can further include information on how to proceed with the scheduled delivery. For instance, the message can indicate that the delivery driver should leave a package outside of the environment.

At716, the remote system sends a second message to a second electronic device. For instance, the remote system120can send a message to a device of a user (e.g., a customer) that is associated with the scheduled service and/or the environment. In some instances, the message can indicate that the object is within the environment. Additionally, the message can indicate that the guest is within proximity of the environment for performing the in-home service.

In some instances, in addition to or alternatively from detecting the object, the remote system120can detect an unfavorable condition within the environment. For instance, the remote system120can detect a fire, floor, low lighting, or the like within the environment. In some instances, the remote system120detects the unfavorable condition using a classifier. In some instances, the remote system120detects the unfavorable condition using one or more sensors within the environment. The remote system120can then send notifications to the device of the guest and the device of the user based on detecting the unfavorable condition.

FIG. 8illustrates a flow diagram of an example process800for providing computer vision-based notification based on detecting an object within an environment. In some instances, operations802-806from example process800can represent operations712-714from example process700. For instance, the operations802-814from example process800can occur after operations702-710from example process700.

At802, the remote system120detects an object within an environment using image data. As discussed above, in some examples, the remote system120can use one or more image processing techniques to detect objects within the environment. Objects can include people, animals, items, or the like. For instance, in some examples, the object can include an animal. Based on the object including an animal, the remote system120may determine that the animal is within a designated animal notification zone of the environment.

At804, the remote system120sends a first message to a first electronic device. For instance, the remote system120can send a message to a device of a guest associated with a scheduled service at the environment. The message can indicate that an object, such as an animal, person, item or the like, is within the environment. In some instances, the message can further include an image captured within the environment that indicates a position of the object within the environment.

For instance, if the object is an animal and the in-home service includes an in-home delivery, the remote system120can send a message to a device of a delivery driver performing the in-home delivery. The message can indicate that the animal is within the environment. In some instances, the message can further include an image of the environment that indicates a position of the animal within the environment.

At806, the remote system120sends a second message to a second electronic device. For instance, the remote system120can send a message to a device of the user that indicates that the object is within the environment. In some instances, the message can further include an image captured within the environment that indicates a position of the object within the environment. For instance, if the object includes an animal, the remote system120can send the device of the user a message indicating that the animal is unsecure within the environment. The message can further include an image within the environment indicating a position of the animal.

At808, the remote system120receives, from the first electronic device, a third message to unlock the environment. For instance, the remote system120can receive a message from the device of the guest that indicates that the guest wants to proceed with the in-home service. In some instances, the guest can determine to proceed with the in-home service based on the message received from the remote system120.

For instance, if the object is an animal and the in-home service includes an in-home delivery, the delivery driver may determine that he or she still wants to perform the in-home delivery. In some instances, the delivery driver may make the determination based on the type of animal, a position of the animal within the environment (e.g., within a locked location, a far distance from the delivery entrance), or the like. Based on the determination, the remote system120can receive a message from the device of delivery driver that indicates that the delivery driver is going to proceed with the in-home delivery.

At810, the remote system120unlocks the environment. For instance, based on receiving the message from the first electronic device, the remote system120can send a message instructing the locking apparatus to unlock the entrance of the environment. In some instances, the remote system120sends the message to the camera apparatus, which in turn sends the message and/or an additional message to the locking apparatus that causes the locking apparatus to unlock the entrance. In some instances, the remote system120sends the message to the locking apparatus.

At812, the remote system120receives a fourth message to secure the environment. For instance, the remote system120can receive a message that includes a request to lock the environment. In some instances, the remote system120can receive the message from the device of the guest when the guest is finished performing the in-home service. In some instances, the remote system120can receive the message from the camera apparatus when the guest is finished with performing the in-home service.

At814, the remote system120causes the environment to be secure. For instance, based on receiving the message to secure the environment, the remote system120can send a message instructing the locking apparatus to lock the entrance of the environment. In some instances, the remote system120sends the message to the camera apparatus, which in turn sends an additional message to the locking apparatus to lock the entrance. In some instances, the remote system120sends the message to the locking apparatus.

FIG. 9illustrates a flow diagram of an example process900for providing computer vision-based messages based on detection of a person. In some instances, operations902-906from example process900can represent operations712-714from example process700. For instance, the operations902-912from example process900can occur after operations702-710from example process700.

At902, the remote system120detects a person within an environment using image data. As discussed above, in some examples, the remote system120can use one or more image processing techniques to detect the person within the environment. In some examples, the remote system further determines that a time associated with the detection is within a threshold period of time of a scheduled service (e.g., a scheduled delivery).

At904, the remote system sends a first message to a first electronic device. For instance, the remote system120can send a message to a device of a guest associated with a scheduled service at the environment. The message can indicate that a person is within the environment. In some instances, when the scheduled service includes scheduled delivery, the message can further include information to perform an outside delivery (e.g., not an in-home delivery).

At906, the remote system120sends a second message to a second electronic device. For instance, the remote system120can send a message to a device of the user that indicates that the person was detected within the environment. In some instances, the message can include an identity of the person within the environment.

At908, the remote system120causes the environment to be secure. For instance, based on detecting the person within the environment, the remote system120can send a message instructing the locking apparatus to lock an entrance of the environment. In some instances, the remote system120sends the message to the camera apparatus, which in turn sends an additional message to the locking apparatus to lock the entrance. In some instances, the remote system120sends the message to the locking apparatus.

Optionally, at910, the remote system120receives, from the second electronic device, a third message that includes a request to provide access to the environment. For instance, the user may decide that he/she would like the guest to still perform the in-home service. As such, the remote system120can receive a message from the device of the user indicating that the user still wants the in-home service. In some instances, based on receiving the message, the remote system120sends a message to the device of the guest that instructs the guest to continue with the in-home delivery.

For instance, if the scheduled service includes a scheduled delivery, the user can determine that he/she want the delivery driver to perform the in-home delivery. As such, the remote system120can receive a message to perform the in-home delivery from the device of the user. In some instances, based on receiving the message, the remote system120can send the device of the delivery driver a message that instructs the delivery driver to continue with the in-home delivery.

Additionally, at912, the remote system120provides access to the environment. For instance, based on receiving the message from the second electronic device, the remote system120can send a message instructing the locking apparatus to unlock the entrance of the environment. In some instances, the remote system120sends the message to the camera apparatus, which in turn sends an additional message to the locking apparatus that causes the locking apparatus to unlock the entrance. In some instances, the remote system120sends the message to the locking apparatus.

FIG. 10illustrates a flow diagram of an example process1000for designating areas within an environment as animal safe zones and animal notification zones.

At1002, the remote system120receives data associated with a scheduled service at a location. In some instances, the remote system120can retrieve the data associated with the scheduled service from a schedules database. In some instances, the remote system120can receive the data associated with the scheduled service from an electronic device, such as an online marketplace. The data associated with the scheduled service can indicate, among other things, a location for the service, a scheduled time for the service. For instance, if the scheduled service includes a scheduled delivery, the data can indicate a delivery location and a delivery time.

At1004, the remote system120receives information designating an area of an environment associated with the location as an animal notification zone. For instance, the remote system120can receive a message from a device associated with a user of the scheduled service that designates areas (e.g., rooms, zones, etc.) within the environment as animal safe zones and animal notification zones. As discussed above, an animal safe zone can include an area within the environment in which the animal is secure. For instance, the animal safe zone can correspond to a secured room, kennel, and/or other secure area within the environment in which the animal does not have access to a zone (e.g., area, room, etc.) in the environment where an in-home service is being performed. An animal notification zone can include an area within the environment in which the animal is unsecure. For instance, the animal notification zone can correspond an area within the environment in which the animal does have access to the zone in the environment where the in-home service is being performed.

At1006, the remote system120receives image data from a camera apparatus within the environment. For instance, the remote system120can send a message to the camera apparatus requesting the image data. In some instances, the image data can include a video data that the remote system120continuously receives from the camera apparatus. Additionally or alternatively, in some instances, the image data can include still image data that the remote system120from the camera apparatus.

At1008, the remote system120detects an animal within the environment using the image data. For instance, the remote system120can analyze the image data using one or more image processing techniques in order to detect the animal within the environment. In some instances, after detecting the animal within the environment, the remote system120can track the animal within the environment using the image data.

At1010, the remote system120determines that the scheduled service should not include an in-home service based at least in part on the information. For instance, the remote system120can determine that the animal is within the area of the environment that the user designated as the animal notification zone. Based on that determination, the remote system120can determine that guest performing the scheduled service should not enter the environment.

For instance, if the scheduled service includes a scheduled delivery, the remote system120can determine that the delivery driver should not perform an in-home delivery. Additionally, the remote system120can update the scheduled delivery to include an outdoor delivery. In some instances, an outdoor delivery includes a delivery in which the delivery driver does not enter the environment. For instance, the delivery driver may leave a package for the user outside of the environment.

At1012, the remote system120sends a message associated with the scheduled service to an electronic device. For instance, based on determining that the scheduled service should not include an in-home service, the remote system120can send a message to a device of the guest that is performing the scheduled service. In some instances, the message can indicate that the animal is within an animal notification zone of the environment. In some instances, the message can further include an image of the environment that indicates a position of the animal.

For instance, if the scheduled service includes a scheduled delivery, the remote system120can send a message to the device of the delivery driver that indicates that the animal is within an animal notification zone of the environment. In some instances, the message can further instruct the delivery driver to perform the outdoor delivery.

FIG. 11illustrates a flow diagram of an example process1100for automatically determining whether an animal is within an animal safe zone or an animal notification zone.

At1102, the remote system120receives data associated with a scheduled service at a location. In some instances, the remote system120can retrieve the data associated with the scheduled service from a schedules database. In some instances, the remote system120can receive the data associated with the scheduled service from an electronic device, such as an online marketplace. The data associated with the scheduled service can indicate, among other things, a location for the service and a scheduled time for the service. For instance, if the scheduled service includes a scheduled delivery, the data can indicate a delivery location and a delivery time.

At1104, the remote system120receives image data from one or more camera apparatuses within an environment associated with the location. For instance, the remote system120can receive image data associated with the environment from multiple camera apparatuses within an environment. In some instance, each camera apparatus may capture image data associated with a given area (e.g., room, zone, etc.) within the environment. For instance, each camera apparatus may capture image data associated with a room within the environment.

At1106, the remote system120detects an animal within the environment using the image data. For instance, the remote system120can analyze the image data using one or more image processing techniques in order to detect the animal within the environment. In some instances, after detecting the animal within the environment, the remote system120can track the animal within the environment using the image data.

At1108, the remote system120determines that the animal has access to an entrance of the environment. For instance, the remote system120can use the image data of the environment to determine a position of the animal within the environment. The remote system120can then determine whether the position corresponds to an animal safe zone or an animal notification zone within the environment. For instance, the remote system120can determine that the animal is located within an animal notification zone based on the animal having access to the entrance of the environment. In some instance, the entrance of the environment is the location wherein the in-home service occurs.

At1110, the remote system120determines that the scheduled service should not include an in-home service. For instance, based at least in part on detecting the animal in an animal notification zone, the remote system120can determine that the guest should not enter the environment to perform the in-home service. Additionally, when the scheduled service includes a scheduled delivery, the remote system120can update the scheduled delivery to include an outdoor delivery. In some instances, an outdoor delivery includes a delivery in which the delivery driver does not enter the environment.

At1112, the remote system120sends a message associated with the scheduled service to an electronic device. For instance, based on determining that the scheduled service should not include an in-home service, the remote system120can send a message to a device of the guest that is performing the scheduled service. In some instances, the message can indicate that the animal is within an animal notification zone of the environment. In some instances, the message can further include an image of the environment that indicates a location of the animal.

For instance, if the scheduled service includes a scheduled delivery, the remote system120can send a message to the device of the delivery driver that indicates that the animal is within an animal notification zone of the environment. In some instances, the notification can further instruct the delivery driver to perform the outdoor delivery.

FIG. 12illustrates a flow diagram of an example process for training one or more classifiers for an environment. At1202, the remote system receives image data from an environment. For instance, the remote system1202can receive image data from one or more camera apparatus located within the environment. In some instances, the image data can include positive image data corresponding to images that include an object and negative image data corresponding to images that do not include the object.

At1204, the remote system120identifies an object within the environment. For instance, the remote system120can identify a person, animal, item, or the like within the environment. In some instances, the remote system120identifies the object by receiving a message from an electronic device that indicates that the object is within the environment. For instance, the remote system120can receive a message from a device of a user that indicates that the object occupies the environment. In some instances, the remote systems120identifies the object by detecting the object within the environment using the image data.

At1206, the remote system creates a classifier for the object using the image data. For instance, the remote system120can use the positive image data and the negative image data train an object classifier for the object using one or more machine learning algorithms. As discussed above, the remote system120can then use the object classifier to later detect the environment within the environment.

It should be noted that, in some instances, the remote system120can create multiple object classifiers for multiple objects within the environment using a similar process. For instance, the remote system120may receive a message from the device of the user that indicates that more than one object is within the environment. In some instances, for each object, the remote system120can receive positive image data corresponding to images that include the respective object and negative image data corresponding to images that do not include the respective object. The remote system can then use the positive image data and the negative image data train an object classifier for the respective object using one or more machine learning algorithms.