Generating detection parameters for a rental property monitoring solution using computer vision and audio analytics from a rental agreement

A system comprising a server and a camera. The server may be configured to receive a rental offer for a location from a first user, receive a rental request for the location from a second user, generate a rental agreement comprising conditions in response to the rental offer and the rental request and generate a feature set based on the conditions. The camera may be configured to receive the feature set from the server, convert the feature set to parameters for the camera, generate a plurality of video frames of the location, perform video operations to detect objects in the video frames, extract data about the objects using the video operations, compare the data to the parameters and generate a notification in response to the data matching the parameters. The video frames may be discarded after performing the video operations. The video operations may be performed locally by the camera.

FIELD OF THE INVENTION

The invention relates to security cameras generally and, more particularly, to a method and/or apparatus for generating detection parameters for a rental property monitoring solution using computer vision and audio analytics from a rental agreement.

BACKGROUND

With the popularity of services like AirBNB, renting property is becoming increasingly popular and has less oversight. Rentals can be set up without the property owner and the renter(s) ever meeting in person. Property owners face the problem of how to protect themselves against renters holding unauthorized parties at rental properties. In addition to online rental services, property owners in areas that have colleges or universities have long had problems with renters involving large parties, loud noises, drugs and alcohol (i.e., keg parties). Additionally, some property owners want to protect themselves against renters bringing pets to the property. Large parties and renters not using a rental property as intended can result in damage to the property, liability issues for the property owner and/or late night angry calls from neighbors. Property owners cannot always be physically present to watch the rental property.

The use of existing video cameras to monitor properties to check on the compliance of renters is problematic because of privacy issues. Renters do not want video of themselves being viewed remotely, voice conversations being listened to, or any recordings, video or audio, being made.

Not all property owners have the time or technical knowledge necessary to configure the sensing camera to make sure that the renters are abiding by the terms of the rental agreement (i.e., number of people, pets, music/noise level etc.). Configuration settings for computer vision and audio analysis can provide an overwhelming number of options. Property owners and renters do not have a way to easily create a human-legible rental agreement that can be applied to a sensing camera to ensure that the rental agreement is properly being followed.

It would be desirable to implement generating detection parameters for a rental property monitoring solution using computer vision and audio analytics from a rental agreement.

SUMMARY

The invention concerns a system comprising a server computer and a camera system. The server computer may comprise a memory configured to store computer readable instructions and a processor configured to execute the computer readable instructions. The computer readable instructions may be configured to receive a rental offer for a location from a first user, receive a rental request for the location from a second user, generate a rental agreement comprising conditions in response to the rental offer and the rental request and generate a feature set based on the conditions, if the first user and the second user agree to the rental agreement. The camera system may be configured to receive the feature set from the server computer, convert the feature set to detection parameters for the camera system, generate a plurality of video frames of the location, perform video operations to detect objects in the video frames, extract data about the objects based on characteristics of the objects determined using the video operations, compare the data to the detection parameters and generate a notification in response to the data matching the detection parameters. The camera system may be implemented at the location. The video frames may be discarded after performing the video operations. The video operations may be performed locally by the camera system.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments of the present invention include generating detection parameters for a rental property monitoring solution using computer vision and audio analytics from a rental agreement that may (i) perform video analytics locally, (ii) perform audio analytics locally, (iii) discard data after processing, (iv) perform video analytics without storing video data, (v) detect restricted activities, (vi) count a number of people at a location, (vii) detect an audio level at a location, (viii) provide a notification when a restricted activity is detected, (ix) provide a web interface for users to enter a rental agreement, (x) provide video and/or audio detection parameters to a camera system based on the rental agreement, (xi) preserve privacy of people at a rental property and/or (xii) be implemented as one or more integrated circuits.

Embodiments of the present invention may be configured to monitor a location (e.g., a property) and notify a property owner in the event of a restricted activity being detected. In an example, the restricted activity may be a party being held and/or the presence of a pet. Using camera technology and/or computer vision, data (e.g., parameters and statistics) may be extracted from captured images and/or sound. The data may be used to make decisions. The decisions may be determined based on the video data and/or the audio data. However, the video data and/or the audio data may not need to be stored and/or communicated. For example, after the decision has been determined, the video data and/or the audio data may be discarded. Discarding the video data and/or the audio data may preserve privacy.

The detection of restricted activities may be determined based on various search parameters performed using computer vision and/or audio analytics. In one example, whether a party is detected may be determined based on using computer vision to detect people and counting the number of people present at the location. In another example, analytics may be implemented to detect a sound level at the location (e.g., detect whether loud music is being played). The computer vision operations may be configured to detect and/or classify objects. In an example, the computer vision operations may be configured to detect animals.

Embodiments of the present invention may be configured to provide a notification to the property owner. The notification may comprise a warning notice. The warning notice may indicate that a restricted activity has been detected. In an example, the notification may be presented from a cloud service to a user device such as a smart phone. The cloud service may be configured to receive the decisions made in response to analysis of video data and/or audio data, but may not receive the video data and/or the audio data.

Embodiments of the present invention may enable use of a web-based application to facilitate generating an agreement to rent a property. The property owner may enter details about the rules for renting the property (e.g., number of people that will be present, whether loud music/movies are allowed, whether there will be a pet allowed, etc.). Based on the information submitted by the property owner, a web-based application may list the property as available for rent. Similarly, a prospective renter may enter details about a desired rental property into the web-based application.

The web-based application may automatically check the requirements of the renter against the rules defined by the property owner. For example, the owner may specify the maximum number of people allowed, whether a pet is allowed, and whether loud music is allowed to be played. If the requirements of the renter fall within the rules of the owner, then a rental booking may be made. The sensing camera may be automatically configured to monitor the property on the specific days rented according to the rules of the rental agreement. For example, the sensing camera may check for the number of people, pets, music etc. as defined by the on-line rental application contract completed by the renter and the property owner. The rental agreement may be used as a basis for setting various search and/or detection parameters for the computer vision and/or audio analytics performed by the sensing camera system.

Referring toFIG.1, a diagram illustrating an example embodiment of the present invention is shown. A system100is shown. The system100may comprise one or more server computers102a-102nand/or one or more camera systems104a-104n.A number of user devices52a-52nmay be connected to the system100. The camera systems104a-104nare shown at a location50. In an example, the location50may be a rental property. The server computers102a-102nmay comprise a number of computing devices configured to operate as a scalable cloud service. For example, the server computers102a-102nmay provide processing and/or storage that may be configured to scale based on demand. The server computers may be referred to generally as the server102, but maybe implemented using a number of computing devices102a-102nconfigured to operate together. The system100may comprise other components and/or interconnections between the components. The arrangement of the system100may be varied according to the design criteria of a particular implementation.

Generally, the server computer(s)102a-102nand the camera systems104a-104may be implemented at different locations. For example, the server computers102a-102nmay be implemented at a centralized location, and the camera systems104a-104nmay be implemented at the rental property50. While multiple camera systems104a-104nare shown at the location50, in some embodiments one of the camera systems104a-104nmay be implemented at the location50. While multiple camera systems104a-104nare shown at the location50, in some embodiments the server computers102a-102nmaybe configured to communicate with multiple camera systems104a-104nthat may be located at the same and/or different properties. For example, the system100may comprise the server computers102a-102nand a plurality of camera systems104a-104nthat may be implemented at one or multiple locations.

In the example shown, the user devices52a-52nmay be desktop computers. However, the user devices52a-52nmaybe implemented as any type of computing device configured to connect to the internet, receive user input and/or provide a graphical display (e.g., a desktop computer, a laptop computer, a netbook, a tablet computing device, a smartphone, a phablet, a wearable headset, a patient monitor, etc.). Generally, the remote devices52a-52nmay enable communication to/from the servers102a-102nand/or the camera systems104a-104n.

Each of the server computers102a-102nmay comprise a respective processor110a-110nand/or a respective memory112a-112n.In the example shown, the processor110aand the memory112aof the server computer102aare shown as a representative example. The processors110a-110nmay be configured to execute computer readable instructions. The memory112a-112nmay be configured to store the computer readable instructions.

The processors110a-110nmay be configured to receive input from the user devices52a-52n,operate on the input according to the computer readable instructions and generate output for the user devices52a-52n.The processors110a-110nmay be configured to receive input from the camera systems104a-104n,operate on the input according to the computer readable instructions and generate output for the camera systems104a-104n.The processors110a-110nmay operate in parallel with each other to collectively perform computational tasks.

The memory112a-112nmay comprise a block120, a block122, a block124and/or a block126. The block120may comprise storage of rental listings. The block122may comprise storage of rental agreements. The block124may comprise a detection engine. The block126may comprise data for a web interface. The memory112a-112nmay comprise other components (not shown). The memory112a-112nmay comprise hard disk drives, cache memory, RAM and/or flash-based memory. The type of storage, the type of information stored and/or the arrangement of the data stored by the memory112a-112nmay be varied according to the design criteria of a particular implementation.

The server computers102a-102nmaybe configured to provide the web interface126to the computing devices52a-52b.The web interface126may be generated by the processors110a-110nin response to the computer readable instructions. For example, the computer readable instructions may comprise web design languages such as HTML, PHP, Javascript, CSS, etc. The data used to generate the web interface126may be generated from the listings120and/or the detection engine124. The web interface126may be configured to receive input from users. For example, the web interface126may enable users to list properties for rent, request a rental property, select terms and/or conditions for a rental agreement, agree to a rental agreement, modify terms of a rental agreement, etc. The listings and rental agreements generated from the web interface126may be listings120and the rental agreements122stored in the memory112a-112n.

In the example shown, one of the user devices52a-52b(e.g., the user device52a) may be used by a rental property owner and the other of the user devices52a-52b(e.g., the user device52b) may be used by a prospective renter. The property owner user device52ais shown receiving a signal (e.g., WEB), generating a signal (e.g., RENTOFR), receiving a signal (e.g., RAGREE) and generating a signal (e.g., CONFO). The signal RENTOFR and the signal CONFO may be communicated to the server computers102a-102nby the property owner user device52a.The renter user device52bis shown receiving the signal WEB, generating a signal (e.g., RENTREQ), receiving the signal RAGREE and generating a signal (e.g., CONFR). The signal RENTREQ and the signal CONFR may be communicated to the server computers102a-102nby the renter user device52b.The signal WEB and the signal RAGREE may be communicated by the server computers102a-102nto the user devices52a-52b.

The signal WEB may represent the communication of the web interface126to the user devices52a-52b.The rental property owner may use the web interface126to provide a rental offer. For example, the signal WEB may be configured to provide a listing form interface for the web application126. The rental offer may be communicated to the server computers102a-102nas the signal RENTOFR. The rental offer may list terms and/or conditions that the property owner has for the rental property50. The server computers102a-102nmay store the rental offers as the listings120. For example, the listings120may be generated based on the rental offers provided by various property owners. Details of the rental listing web interface126may be described in association withFIG.8.

The signal WEB may be provided to the renter user device52bto enable browsing the listings120. The property renter may use the web interface126to request a rental property. For example, the signal WEB may be configured to provide a rental request form interface for the web application126. A rental request may be communicated to the server computers102a-102nas the signal RENTREQ. The rental request signal RENTREQ may provide a list attributes that the renter is seeking in a rental property. Details of the rental request web interface126may be described in association withFIG.9.

The server computers102a-102nmay be configured to list rental properties based on the rental offers. For example, the servers102a-102nmay be configured to receive rental property offers from one or more property owners to provide the rental listings120that prospective renters may use to search using the signal RENTREQ. The server computers102a-102nmay provide the rental listings120(e.g., as the signal WEB) that prospective renters may browse through and/or may use the input signal RENTREQ from the rental request to find properties that closest match what the prospective renter is searching for according to the parameters entered into the web interface126. For example, the server computers102a-102nmay be configured to filter the available listings120based on the data provided in the signal RENTREQ (e.g., at a specific location, available at particular times, allows a particular number of guests, allows pets, etc.). For example, the server computers102a-102nmay provide a match-making service to enable property owners to find suitable renters and to enable prospective renters to find a suitable rental property.

Based on the rental offer signal RENTOFR and/or the rental request signal RENTREQ, the server computers102a-102nmay generate the rental agreement122. The rental agreement122may be communicated to both the property owner and the prospective renter as the signal RAGREE. The rental agreement122presented as the signal RAGREE may be a human-readable (e.g., text-based) contract that the property owner and renter may review and/or agree to. The rental agreement122may be generated from the selections made using the web interface126(e.g., based on the input from both the property owner and the prospective renter). Generally, the terms of the rental agreement122may be determined from the input of the property owner (e.g., using the signal RENTOFR). In some embodiments, the prospective renter may negotiate changes to the terms provided in the rental offer using the signal RENTREQ. Details of the rental agreement web interface126may be described in association withFIG.10.

The property owner and the prospective renter may agree to the terms of the rental agreement122. The web interface126may enable the property renter to agree to the rental agreement122and the property owner user device52amay communicate the signal CONFO to confirm and/or accept the rental agreement122. The web interface126may enable the prospective renter to agree to the rental agreement122and the renter user device52bmay communicate the signal CONFR to confirm and/or accept the rental agreement122. If the server computers102a-102nreceive the signal CONFO and the signal CONFR (e.g., both parties agree to the rental agreement122), then the rental agreement122may be stored in the memory112a-112nand the rental agreement122may be parsed by the detection engine124. If the server computers102a-102ndo not receive both the signal CONFO and the signal CONFR (e.g., at least one party does not confirm the rental agreement), then the rental agreement122may be discarded by the server computers102a-102n.

When the rental agreement122is agreed to, the server computers102a-102nmay be configured to generate a feature set for the camera systems104a-104n.The detection engine124may be configured to parse the conditions of the rental agreement122. Based on the information from the rental agreement122, the detection engine124may generate a feature set for the particular camera(s)104a-104nat the rental property50covered by the rental agreement122. The feature set may comprise the text of the rental agreement122, and/or information that the camera systems104a-104nmay use to generate parameters for performing object detection according to the rental agreement122. The feature set may be generated based on the conditions of the rental agreement122. In an example, the feature set may be a machine readable version (e.g., computer readable data) of the rental agreement122that may enable the camera systems104a-104nto detect breaches of the rental agreement122using computer vision analysis and/or audio analysis.

The server computers102a-102nmay generate a signal (e.g., FEAT_SET). The signal FEAT_SET may provide the feature set to the camera systems104a-104n.The feature set may be generated by the detection engine124in response to parsing the rental agreement122. The camera systems104a-104nmay operate (e.g., perform computer vision operations and/or audio analysis) based on the data in the signal FEAT_SET. For example, the signal FEAT_SET may provide parameters such as a number of people to detect, types of pets to detect, a maximum audio level, a time to search for particular features (e.g., additional guests may be allowed at particular times), a time range that the rental agreement122applies to, which features to ignore (e.g., do not detect faces to ensure privacy protection), etc. The type of information in the signal FEAT_SET may be varied according to the design criteria of a particular implementation.

The camera systems104a-104nare shown comprising a block (or circuit)130and/or a block (or circuit)132. The circuit130may implement a processor. The circuit132may implement a memory. Each of the camera systems104a-104nmay implement the processor130and the memory132. The camera systems104a-104nmay each comprise other components (not shown).

Details of the components of the camera systems104a-104nmay be described in association with

The processor may receive the feature set from the server computers102a-102n.The processor130may convert the feature set into detection parameters that may be used to perform the video and/or audio analysis. The detection parameters may be stored by the memory132. The detection parameters may provide criteria about what the camera systems104a-104nshould look for when analyzing video frames (e.g., number of people detected, detecting pets, detecting a noise level, etc.).

The camera systems104a-104may be configured to perform video surveillance during a normal (or default) mode of operation. In some embodiments, the camera systems104a-104nmay be configured to record (or store) video data and/or communicate video data when in the default mode of operation. For example, when the camera systems104a-104nare not operating according to the feature set provided by the servers102a-102n(e.g., the rental agreement122is not in effect), then the camera systems104a-104nmay operate in the default mode of operation.

When the camera systems104a-104nare operating according to the feature set (e.g., during a time range that is covered by the rental agreement122), the camera systems104a-104nmay not record the video/audio data capture and may not communicate the video/audio data. The video data and/or audio data may not leave (e.g., be communicated by) the camera systems104a-104nto ensure privacy of the renter. The camera systems104a-104nmay perform the computer vision operations to extract data about the video frames (e.g., how many people are detected in a video frame, the type of pet detected, a current audio level, etc.). Once the data is extracted, the video frames and/or audio data may be discarded (or deleted, or overwritten, etc.).

The processor130may analyze the extracted data. The extracted data may be compared to the detection parameters from the feature set for the active rental agreement122. If the processor130detects a match of the data to the detection parameters (e.g., detects a scenario that is a breach of the rental agreement122), the camera systems104a-104nmay generate a notification. A signal (e.g., NTF) is shown. The signal NTF may represent the notification generated by the camera systems104a-104n.The notification NTF may be sent to the property owner and/or the renter. The notification NTF may indicate that the rental agreement122has been breached and/or the type of breach that has been detected. The notification may not provide the video frames and/or audio data that corresponds to the breach of the rental agreement122because the video frames and/or audio data may already be unavailable (e.g., deleted) to protect the privacy of the renter(s).

In the example shown, one feature set signal FEAT_SET is provided to each of the camera systems104a-104n.However, in some embodiments, a different feature set may be generated for each camera system104a-104n.For example, the feature set may be generated according to the specifications and/or detection capabilities of each camera system104a-104n(e.g., different makes/models of cameras may be required to have a feature set in a different format). In another example, the feature set may be different based on particular rooms at the location50(e.g., the main living room may be allowed to have 10 guests and the camera system104ain the main living room may receive a feature set to detect up to 10 guests, while a bedroom may be off limits to everyone and the camera system104bin the bedroom may receive a feature set to detect a breach when any guest is detected). In yet another example, the feature set may be different for each of the camera systems104a-104nbased on the position of the camera systems104a-104nin a room and/or a size of the room. For example, feature sets may be generated based on training data for small rooms and/or large rooms, depending on the height of the camera in the room, etc. In some embodiments, different pre-defined sets of data on the server computers102a-102nmay be downloaded into the camera systems104a-104nbased on the specifications of the camera systems104a-104nand/or the characteristics of the room and/or mounting location of the camera systems104a-104n.The variety of feature sets that may be provided to the camera systems104a-104nmay be defined by the rental agreement122.

The server computers102a-102nmay be configured to generate and/or receive a signal (e.g., QUERY). The signal QUERY maybe communicated to/from the camera systems102a-102n.The signal QUERY may be communicated to enable the servers102a-102nto determine the capabilities, features and/or limitations of the camera systems104a-104n.The signal QUERY may be communicated to enable the servers102a-102nto determine the characteristics of the room and/or environment that the camera systems104a-104nare located in. In an example, the capabilities, features and/or limitations of the camera systems104a-104nmay comprise the type, number, and/or details of the object detection of the camera systems104a-104n(e.g., how many objects may be detected, the types of objects that may be detected, the level of detail of the detection, whether a microphone is connected to receive audio, a height and/or distance from the objects to be detected, etc.). The capabilities of the camera systems104a-104nmay be used to determine what type of entries (e.g., terms and conditions) for the rental agreement122may be detectable by the camera systems104a-104nat the location50and/or the types of feature sets to provide to the camera systems104a-104n.

The signal QUERY may be analyzed by the detection engine124. For example, the signal QUERY may comprise technical specifications, a make/model, etc. of the camera systems104a-104n.The detection engine124may be configured to parse the technical specifications to determine what the camera systems104a-104nmay be capable of detecting. In some embodiments, the detection engine124may be configured to generate input fields for the web interface126in response to the signal QUERY. For example, if the camera systems104a-104nare capable of detecting a type of pet using computer vision operations, then the detection engine124may generate an input field for the web interface126that enables input of a type of pet. Similarly, if the camera systems104a-104nare not capable of detecting a type of pet using computer vision operations, then the input field for a type of pet may not be available on the web interface126.

Referring toFIG.2, a diagram illustrating an example embodiment of the present invention is shown. An example rental scenario150is shown. The example rental scenario150may be at the location50. In an example, the location50may be a rental property in the listings120. In the example shown, the location50may be an interior location. In another example, the location50may be an outdoor location. In yet another example, the location50may be a combination of an indoor and outdoor location comprising multiple rooms of a building and/or multiple floors of a building. The type of location50may be varied according to the design criteria of a particular implementation.

In the example shown, the location50may comprise an interior wall62. An opening64is shown in the wall62(e.g., a doorway to another room). A number of people70a-70eare shown. The person70ais shown partially obscured by the wall62. The people70b-70emay be visible in the opening64.

The camera system104is shown as a representative example of the camera systems104a-104n.The camera system104may be configured to capture video data, capture audio data, perform computer vision operations and/or perform audio analytics. In an example, the camera system104may be configured to implement a rental property monitoring system using computer vision and audio analytics to detect parties and pets while preserving renter privacy.

Lines152a-152bare shown. The lines152a-152bmay represent a field of view of the camera system104. The camera system104may capture video frames of the area within the field of view152a-152b.In the example shown, the lines152a-152bmay provide an illustrative example of the field of view of the camera system104. In some embodiments, the camera system104may comprise one single 360-degree camera (e.g., capturing a 360-degree field of view). In some embodiments, the camera system104may comprise two back-to-back 180-degree cameras for capturing two 180-degree fields of view (e.g., in front and behind the camera system104). In some embodiments, the camera system104may implement a fisheye lens providing a wide-angle field of view. The types of lenses used and/or the field of view captured by the camera system104may be varied according to the design criteria of a particular implementation.

In the example location50, the people70a-70emay be within the field of view152a-152bof the camera system104. In the example shown, the people70a-70emay be visible in video frames captured by the camera system104. Dotted boxes160a-160eare shown. The dotted boxes160a-160emay represent objects detected by the camera system104. The dotted boxes160a-160eare shown around the people70a-70e.In the example shown, the camera system104may be configured to detect the people70a-70eas the objects160a-160e.While the people70a-70eare shown as the detected objects160a-160ein the example shown in association withFIG.2, the detected objects160a-160emay be animals and/or inanimate objects (e.g., furniture, electronics, structures, etc.). While five people70a-70eare shown as the detected objects160a-160e,the camera system104may be configured to detect any number of objects (e.g., detect the people70a-70nas the objects160a-160n). The number and/or types of objects160a-160ndetected by the camera system104may be varied according to the design criteria of a particular implementation.

A dotted box162is shown around the head of the person70c.The dotted box162may represent the camera system104detecting characteristics of the object160c.While the characteristics162are shown corresponding to the object160c,the camera system104may be configured to detect the characteristics of each of the objects160a-160n.The camera system104may be configured to analyze the characteristics162of the objects160a-160nto determine what the objects160a-160nare (e.g., classification), determine what the objects160a-160nare doing (e.g., behavior analysis) and/or to distinguish one object from another object. The types of characteristics162detected and/or analyzed and/or the inferences made based on detecting the characteristics162may be varied according to the design criteria of a particular implementation.

The characteristics162may comprise descriptive and/or identifying attributes of the objects160a-160n.In one example, the characteristics162may comprise clothing worn (e.g., style of clothing, a color of the clothing, the color of the pants, whether pants or shorts are worn, brand of pants worn, etc.). The characteristics162may comprise physical characteristics about the detected objects160a-160n.In an example, the characteristics162may comprise a height, estimated weight, body type, hair color, skin color, gait, etc. The types of characteristics162detected may be used to distinguish one of the people70a-70nfrom another of the people70a-70n(e.g., to ensure that the number of people are counted without double-counting a person). The types of the characteristics162detected may be varied according to the design criteria of a particular implementation.

Waves164a-164bare shown. The waves164a-164bmay represent audio detected by the camera system104. In the example shown, the waves164a-164bmay represent people talking. For example, the waves164amay be an illustrative representation of the person70dtalking and the waves164bmay be an illustrative representation of the person70etalking. The camera system104may be configured to determine an audio level (e.g., amplitude) of the audio164a-164b.In the example shown, the audio164a-164bmay be generated from the people70a-70e.The camera system104may be configured to determine an audio level of the location50. The audio level may comprise audio from the people70a-70e(e.g., talking), televisions, music devices, objects breaking, etc. The type of audio detected may be varied according to the design criteria of a particular implementation.

The computer vision operations performed by the camera system104may be configured to extract data from the video frames and/or the audio captured. The extracted data may comprise parameters and/or statistics about the content of the video frames and/or audio captured. Extracted data170a-170eis shown. In the example shown, the extracted data170a-170emay represent a person count corresponding to the people70a-70ein the location50. The extracted data170a-170emay be used by the camera system104to make decisions. In one example, the decisions made by the camera system104based on the extracted data170a-170emay comprise determining whether terms of the rental agreement122have been breached. After the extracted data170a-170ehas been generated from the video frames and/or the audio captured, the camera system104may discard the video frames and/or the audio captured (e.g., additional information from the video frames and/or the audio captured may be needed to make decisions).

In some embodiments, the computer vision operations performed by the camera system104may be configured to count a number of people at the location50. In the example shown, numbers 1-5 (e.g., the extracted data170a-170e) are shown below a corresponding one of the detected objects160a-160e.The camera system104may be configured to distinguish between the detected objects160a-160nbased on the characteristics162of each of the detected objects160a-160n.For example, the camera system104may be configured to determine that the person70bis one person and the person70cis another person. In another example, the characteristics162may be used to prevent the people70a-70efrom being counted multiple times (e.g., distinguishing a reflection of a person from the actual person).

In the example shown, the person70ais shown partially obscured behind the wall62. The camera system104may be configured to determine that the characteristics162of the visible portion of the person70acorresponds to one person. For example, the camera system104may be configured to infer that a person has been detected based on a partial view. In the example shown, the hand of the person70dis shown in front of the arm of the person70c.The camera system104may be configured to determine that more than one person is present when one person is partially in front of another person.

In some embodiments, the camera system104may be configured to determine a behavior of the objects160a-160n.Inferences may be made about the behavior of the objects160a-160nbased on the characteristics162detected. In an example, a person that is standing still and using arm gestures may be determined to be talking. In another example, regular or rhythmic body movement may be determined to be dancing. The body movement may be compared to the audio data (e.g., music) to determine whether the behavior indicates dancing. In one example, the camera system104may make an inference that there is a party at the location50based on the people70a-70ndancing. In yet another example, the characteristics162may indicate that the people70a-70nare holding drinks, which may indicate alcohol is being consumed at the location50. For example, optical character recognition (OCR) may be implemented to read labels (e.g., to detect beer, liquor, wine, etc.).

In the example shown, the characteristics162may correspond to a face of the person70c(e.g., the detected object160c). The characteristics162may be determined for each of the detected objects160a-160e(e.g., the people70a-70e,items held by the people70a-70e,other items in the location50, etc.). The characteristics162may comprise a color of the detected objects160a-160e(e.g., color of clothing worn). The characteristics162may comprise the size of objects (e.g., a height of a person). The characteristics162may comprise a classification of the detected objects160a-160e(e.g., recognizing the people70a-70eas distinct people, identifying an item as a television, recognizing an animal, etc.). In some embodiments, the characteristics162may be used by the camera system104to distinguish between the detected objects160a-160e.

The extracted data170a-170eabout the number of guests may be compared to the detection parameters converted by the processor130from the signal FEAT_SET. For example, the rental agreement122may indicate a limitation on the number of guests allowed at the rental property50. The detection engine124may convert the machine readable version of the rental agreement122into detection parameters that may be usable by the camera system104at the rental property50shown. The detection parameters may provide computer readable instructions about what types of objects and/or scenarios that the camera system104should detect at the rental property50.

For example, the rental agreement122may indicate a maximum of six guests, the detection engine124may query the camera system104to determine a format of the feature set for the camera system104, the detection engine124may convert the guest limitation from the rental agreement122into the feature set, and the processor130of the camera system104may convert the feature set into detection parameters used to perform the computer vision operations. If six guests is one of the detection parameters, then the camera system104may analyze video frames generated to count the number of the guests70a-70nto determine if the number of guests is less than, equal to or greater than six guests. In the example shown, the extracted data170a-170eabout the number of the guests70a-70emay indicate five guests are at the rental property50. Since the number of guests is less than the amount in the detection parameters, then the camera system104may not indicate a breach has been detected. If more than six guests were detected, then the camera system104may generate the notification signal NTF to indicate that a breach of the rental agreement122has been detected.

Generally, the feature set and/or conditions generated from the rental agreement122may comprise activities of the renter of the property50. In some embodiments, the detection parameters may comprise duties and/or requirements of the property owner. For example, when the property owner is preparing the property50for the renter, the camera system104may provide a check that the property has provided all the amenities agreed to in the rental agreement122(e.g., left out clean towels and clean bedsheets, left out toiletries, etc.). In some embodiments, the camera system104may be configured to detect particular events that the property owner has agreed to respond to in the rental agreement122(e.g., fixing a water leak, replacing a broken appliance, etc.).

In some embodiments, the computer vision operations may detect the type of activities being performed. For example, the rental agreement122may provide restriction on how the property50is allowed to be used. For example, if the property50has a pool and the rental agreement122does not permit usage of the pool, the camera system104maybe configured to detect if the renter(s) are using the pool in breach of the rental agreement122.

Referring toFIG.3, a diagram illustrating monitoring rental properties using computer vision and/or audio analytics is shown. An example scenario180is shown. The example scenario180may comprise a number of locations50a-50n,the network/cloud service102and/or the remote device52i.In some embodiments, the locations50a-50nmay be remote locations (e.g., different geographic locations such as different cities, different countries, different areas within the same city, etc.). In some embodiments, the locations50a-50nmay be different locations within the same building (e.g., different rooms of a rental property, an interior and an exterior of a rental property, etc.). Each of the locations50a-50nare shown implementing a respective camera system104(e.g., the camera systems104a-104n). For example, each of the camera systems104a-104nmay be configured to operate independently of each other (e.g., video data and/or audio data may not be shared). However, the parameters and statistics generated by the camera systems104a-104nmay be shared. Generally, each of the camera systems104a-104nmay be configured to monitor the respective locations50a-50nsimilar to the camera system104monitoring the location50described in association withFIG.2.

The network102may be implemented as part of a cloud computing platform (e.g., distributed computing). In an example, the network102may be implemented as a group of cloud-based, scalable server computers. By implementing a number of scalable servers, additional resources (e.g., power, processing capability, memory, etc.) may be available to process and/or store variable amounts of data. For example, the network102may be configured to scale (e.g., provision resources) based on demand. The network102may implement scalable computing (e.g., cloud computing). The scalable computing may be available as a service to allow access to processing and/or storage resources without having to build infrastructure. In some embodiments, the network102may be configured to provide resources such as training data and/or a database of feature maps (e.g., feature maps of recognized objects to perform object recognition and/or classification). For example, the training data and/or feature maps may be communicated to the camera systems104a-104nby the network102based on the contents of the rental agreement122.

In the example shown, the remote device52imay be implemented as a smartphone. In the example shown, one remote device52iis shown. However, the example scenario180may be implemented having multiple remote devices (e.g., remote devices52a-52n,not shown). In an example, the smartphone52amay be used by the landlord and the smartphone52bmay be used by the renter.

The remote device52imay be configured to receive notifications from the camera systems104a-104n.The camera systems104a-104nmay not communicate the video data and/or the audio data. For example, the camera systems104a-104nmay discard video data and/or audio data after performing the analysis. Discarding the video data and/or audio data may ensure the privacy of the people70a-70n.The notification may provide information that corresponds to the determinations made by the camera systems104a-104nin response to performing the analysis of the video data and/or audio data (e.g., based on the statistic and parameters).

In the example shown, the remote device52iis shown at a location other than the locations50a-50n.For example, the notification may provide the rental property owner information about the locations50a-50nfrom a remote geographic location. In some embodiments, the remote device52imay be located at or near one of the locations50a-50n.For example, to further protect privacy of the people70a-70n,the information about the analysis performed by the camera systems104a-104nmay be stored on-site at the locations50a-50nand the rental property owner may access the information on-site.

The locations50a-50nmay each comprise a respective one of the camera systems104a-104n.Each of the camera systems104a-104nare shown having the field of view152a-152b.In the example shown, the locations50a-50nmay be the subject of the monitoring.

The rental property owner may provide the people70a-70nwith the rental agreement122. The rental agreement122may comprise a list of restrictions. The restrictions may comprise various entries that may comprise a number of people, disallowed animals, noise levels and/or behaviors. The camera systems104a-104nmay be configured to perform the analysis of the video data and/or the audio data to determine whether the data detected matches any entries on the list of restrictions. For example, the list of restrictions may be converted to parameters that may be used by the computer vision operations and/or the audio analytics to perform the detection. If the data detected by the camera systems104a-104nmatches any of the entries on the list of restrictions, the camera system104may generate a notification. The notification may be a warning to the people70a-70nto cure the cause of the warning. The notification may be provided to the rental property owner.

Status signals (e.g., STAT_A-STAT_N) are shown presented to the network102. The status signals STAT_A-STAT_N may be generated by the camera systems104a-104n.The status signals STAT_A-STAT_N may represent the notifications generated by the camera systems104a-104n.In some embodiments, the status signals STAT_A-STAT_N may provide continual updates (e.g., provided even if the camera system104a-104ndoes not detect any of the entries of the list of restrictions). For example, the signals STAT_A-STAT_N may provide continual updates about the extracted data170a-170e(e.g., the number of the people70a-70nat the rental property) generated by the camera systems104a-104n.

In the example shown, the signals STAT_A-STAT_N are shown communicated from the camera systems104a-104nto the network102. In some embodiments, the signals STAT_A-STAT_N may be shared between the camera systems104a-104n.For example, the breach of the rental agreement122may be based on a total number of people detected. If the locations50a-50nare multiple rooms in a single rental property, one of the camera systems104a-104nalone may not be capable of counting all the people at once. By sharing the parameters and statistics in the signals STAT_A-STAT_N, the cameras104a-104nmay determine the total number of people.

The signal NTF is shown. The signal NTF may be presented to the remote device52iin response to one or more of the status signals STAT_A-STAT_N. The signal NTF may be provided in a format readable by the remote device52i.For example, an application may be provided for the rental property owner to use on the remote device52iand the application may be compatible with an API of the network102and/or the camera systems104a-104n.

In the example shown, the notification signal NTF is shown provided by the network102to the remote device52i.In some embodiments, the camera systems104a-104nmay be configured to generate the notification locally. For example, the camera systems104a-104nmay comprise a speaker configured to provide an audio warning to the renters when a breach has been detected (e.g., “The terms of the rental agreement122have been breached. Ten people have been detected and only 4 are allowed. Please ask 6 people to leave.”).

In the location50a,three of the people70a-70care shown. The camera system104amay be configured to count the number of people at the location50ausing the computer vision operations. The computer vision operations performed by the camera system104amay detect three people. The example shown in association with the location50amay represent detecting more people than are allowed by the list of restrictions. For example, the rental agreement122may provide a restriction of two people. In an example, when three people are detected by the camera system104a,the camera system104amay generate the signal STAT_A. In another example, the camera system104amay provide regular updates about the number of people using the signal STAT_A (e.g., the camera system104amay indicate that two people are at the location50a,then update at regular intervals and when the third person arrives the number of people will be updated). The signal NTF may be presented by the network102to the remote device52iin response to the signal STAT_A. In an example, the notification may indicate that the entry on the list of restrictions for number of people has been violated.

In the location50b,the person70dand a speaker74are shown. The camera system104bmay be configured to count the number of people at the location50ausing the computer vision operations. In an example, the computer vision operations performed by the camera system104bmay detect one person, which may be in compliance with the list of restrictions. The example shown in association with the location50bmay represent detecting a higher audio level than is allowed by the list of restrictions. For example, the rental agreement122may provide a restriction on noise level. In the example shown, the speaker74may be set to eleven (e.g., too loud compared to the noise level threshold from the list of restrictions). The camera system104bmay perform the analysis of the audio from the speaker74and determine whether the amplitude of the audio detected matches (or is greater than) the audio amplitude limitation level entry on the list of restrictions. In one example, when the camera system104bdetects that the audio level is greater than allowed by the list of restrictions, the camera system104bmay generate the signal STAT_B. In another example, the camera system104bmay provide regular updates about the detected audio level using the signal STAT_B (e.g., the camera system104bmay indicate the audio level in decibels and then update at regular intervals). The signal NTF may be presented by the network102to the remote device52iin response to the signal STAT_B. In an example, the notification may indicate that the entry on the list of restrictions for noise level has been violated.

In the location50n,the person70nand a bed76are shown. The camera system104nmay be configured to count the number of people at the location50nusing the computer vision operations. In an example, the computer vision operations performed by the camera system104nmay detect one person, which may be in compliance with the list of restrictions. The camera system104nmay further perform the audio analysis to detect the noise level. In the example shown, the person70nmay be sleeping and not making noise. Since the person70nmay be in compliance with the list of restrictions, the camera system104nmay not send the signal STAT_N. In some embodiments, the camera system104nmay send the signal STAT_N that shows the number of people and the audio level is in compliance with the list of restrictions.

In some embodiments, the signals STAT_A-STAT_N may not provide details of the behavior of the people70a-70nto preserve privacy. For example, the signal STAT_N may not indicate that the person70nis sleeping in the bed76. The signals STAT_A-STAT_N may merely provide sufficient information to indicate whether or not the renters (e.g., the people70a-70n) are in compliance with the list of restrictions. For example, the signal STAT_N may provide information that the number of people detected and the noise level was less than the threshold indicated by the list of restrictions in the rental agreement122.

In some embodiments, the camera systems104a-104nmay be configured to share data. The data shared between the camera systems104a-104nmay enable additional information to be aggregated for inferences to be made by the camera systems104a-104n.For example, if the camera systems104a-104nprovide video data and audio data from multiple rooms of the same rental property, the camera systems104a-104nmay share the number of people (e.g., the extracted data170a-170e) counted in each room to determine a total number of people at the rental property. To ensure privacy, the camera systems104a-104nmay not share the video data and/or the audio data. To ensure privacy, the camera systems104a-104nmay share the results determined by the computer vision operations and/or the audio analysis. For example, each camera system104a-104nmay perform the computer vision operations to determine the number count170a-170eof people and share the number count170a-170eto determine a total number of occupants at the rental property. The results shared may be limited to ensure privacy while providing enough information to make a determination about whether the terms of the rental agreement122have been breached. For example, the number count170a-170emay be shared if the number of people is an entry of the rental agreement122, but may not be shared if the number of people is not an entry of the rental agreement122.

The camera systems104a-104nmay be configured to apply various types of audio analysis (e.g., sound levels, frequency analysis, artificial intelligence for detecting words and/or phrases, etc.). In an example, the camera systems104a-104nmay be configured to determine a sound level of the audio and/or detect the type of audio (e.g., determine whether the detected audio is loud music, a loud animal, loud voices, etc.). In some embodiments, the camera systems104a-104nmay be configured to combine the audio analysis with video analysis (e.g., perform the computer vision analysis to determine the body movements to determine that the people70a-70nare dancing and perform the audio analysis to determine that there is loud music). The type of analysis performed may be based on the list of restrictions. For example, merely playing music too loudly may be an issue and the notification may be sent based on the sound level without performing the behavioral analysis to determine that the people70a-70nare dancing (e.g., the sound level may be an issue regardless of what the people70a-70nare doing).

In some embodiments, the camera systems104a-104nmay be implemented as small, discreet cameras that may be hidden from view. Hiding the camera systems104a-104nmay prevent renters from attempting to obscure the lens. In some embodiments, the camera systems104a-104nmay be implemented in plain view and/or highlighted to bring attention to the camera systems104a-104n.The camera systems104a-104nmaybe configured to detect when the field of view152a-152nhas been obscured (e.g., detect all black video frames, detect if more than a threshold percentage of the video frame is dark, etc.). For an example of a rental property, the camera systems104a-104nmay be explicitly described in the rental agreement122, along with the locations of the camera systems104a-104nand the list of restrictions. Implementing the camera systems104a-104nsuch that the camera systems104a-104nare visible may act as a deterrent to those seeking to rent accommodation and hold parties. In an example, when the renter fills out the rental application form there may be a request to acknowledge that the camera systems104a-104nare installed and state how many people will be present and whether there will be any pets.

The camera systems104a-104nmay maintain the privacy of the renter. No video data and/or audio data may be streamed or recorded. The computer vision operations may be configured to detect a crowd and/or count people. The camera systems104a-104nmay be configured to detect if the field of view152a-152nhas been obscured in any way. In some embodiments, the camera systems104a-104nmay not make determinations based on sex, race, other physical features, etc. In one example, the camera systems104a-104nmay be configured to detect when each of the people70a-70nfirst arrive and then compare the people count170a-170eto a threshold (e.g., based on the entry in the list of restrictions). For example, the camera systems104a-104nmay determine whether a party is being held at the rental property based on various parameters (e.g., people count, loud noises, music, etc.).

The camera systems104a-104nmay be configured to detect loud sounds (e.g., identifying music, breaking glass, smoke alarms, etc.). The camera systems104a-104nmay be configured to detect pets (e.g., cats, dogs, birds, ferrets, snakes, gerbils, etc.). The camera systems104a-104nmay be configured to count people to determine if more people are detected than the threshold number of people that may be set by the landlord. The list of restrictions may be provided to the renters in the rental agreement122. The list of restrictions may be converted to threshold parameters that may be readable by the camera systems104a-104n.The types of conditions that the camera systems104a-104nmay search for using the computer vision operations and/or the audio analytics may be determined based on the threshold parameters. In an example, if the landlord does not list pets as an entry on the list of restrictions, the computer vision operations may not search for pets. The method of converting the list of restrictions to the threshold parameters may be varied according to the design criteria of a particular implementation.

In some embodiments, the camera systems104a-104nmay be configured to implement computer vision acceleration hardware to perform the computer vision operations. In some embodiments, the camera systems104a-104nmay be configured to perform cropping and/or zooming techniques as part of the computer vision operations to assist in the person and pet detection.

In some embodiments, the camera systems104a-104nmay be configured to implement facial recognition. Facial recognition may represent a trade-off between available types of detections that may be made for the list of restrictions and privacy concerns. In an example, the camera systems104a-104nmay be configured to perform the facial recognition locally (e.g., comparing feature maps from the faces of previously detected people to the features maps currently detected on the people70a-70nin the location50). However, a database of feature maps to compare against may comprise a large amount of data (e.g., typically stored using services such as the cloud services102). Sending the currently detected feature maps for facial recognition may be a potential privacy concern since the data detected by the camera systems104a-104nwould not all be kept locally and then discarded. In some embodiments, depending on the storage capacity available to the camera systems104a-104na number of feature maps for faces may be stored locally. In an example, parents may want a limited number of people restricted (e.g., detect a boyfriend/girlfriend when the parents are not home). In some embodiments, particular characteristics162may be detected (e.g., hair color, color of clothing, type of clothing worn, etc.). In some embodiments, the processor130of the camera systems104a-104nmay be capable of performing facial recognition, but may not be configured to perform the facial recognition. In an example, in the default mode of operation (e.g., when the rental agreement122is not active), the facial recognition may be performed, and facial recognition may be deactivated when the rental agreement122is active.

The signal NTF may be customizable by the landlord. In an example, the landlord may receive a notification on the smartphone52iwhen music is detected above the threshold level set by the list of restrictions. In some embodiments, the renter may also receive the notification. For example, if the renter also receives the notification, the renter may have an opportunity to correct the issue (e.g., turn down the music) before the landlord has to intervene. In some embodiments, the notification may provide details of the entry on the list of restrictions that has been violated (e.g., “five people have been detected but only two people are allowed on the property”). Providing a detailed notification may enable the renter to take action to ensure that the property is protected according to the list of restrictions. In some embodiments, the notification may be configured to protect privacy by not indicating the particular entry on the list of restrictions that has been violated. For example, the notification may provide the list of restrictions and indicate that a violation has been detected. In some embodiments, the renter may be able to respond to the notification. The response to the notification may be sent to the landlord (e.g., to acknowledge the notification and confirm they have taken action to correct the violation).

The camera systems104a-104nmay be configured to discard the video data after performing the computer vision operations. The video data may not be stored long term. The video data may not be streamed to a remote location. In an example, to perform the computer vision operations, the camera systems104a-104nmay perform the video analysis on a single video frame. Some additional information from data in a buffer maybe used. The camera systems104a-104nmay generate the data (e.g., conclusions, inferences, the number of people70a-70e,etc.), and the video data may be discarded. The data may be updated as new data is extracted from incoming video frames. However, previous video data is unavailable after being discarded. The previously determined data may be used and/or stored but the video data and/or audio data that the data was extracted from may be discarded. In an example, a history of the data may be stored. For example, the noise level may be recorded along with a time stamp of when the noise level was determined. The history of the data may be compared with complaints. For example, if a neighbor calls in a noise complaint with the police, the camera systems104a-104nmay provide the timestamped noise level to determine whether the neighbor has provided a legitimate complaint.

Referring toFIG.4, a block diagram illustrating an example embodiment of the camera system is shown. A block diagram of the camera system104iis shown. The camera system104imay be a representative example of the camera system104a-104nshown in association withFIGS.1-3. The camera system104igenerally comprises the processor130, the memory132, blocks (or circuits)200a-200n,blocks (or circuits)202a-202n,a block (or circuit)204, blocks (or circuits)206a-206nand/or blocks (or circuits)208a-208n.The blocks200a-200nmay implement lenses. The circuits202a-202nmay implement capture devices. The circuit204may implement a communication device. The circuits206a-206nmay implement microphones (e.g., audio capture devices). The circuits208a-208nmay implement audio output devices (e.g., speakers). The camera system104imay comprise other components (not shown). In the example shown, some of the components200-208are shown external to the camera system104i.However, the components200-208may be implemented within and/or attached to the camera system104i(e.g., the speakers208a-208nmay provide better functionality if not located inside a housing of the camera system104i). The number, type and/or arrangement of the components of the camera system104imay be varied according to the design criteria of a particular implementation.

In an example implementation, the circuit130may be implemented as a video processor. The processor130may comprise inputs220a-220nand/or other inputs. The processor130may comprise an input/output222. The processor130may comprise an output224aand an input224b.The processor130may comprise an input226. The processor130may comprise an output228and/or other outputs. The number of inputs, outputs and/or bi-directional ports implemented by the processor130may be varied according to the design criteria of a particular implementation.

In the embodiment shown, the capture devices202a-202nmay be components of the camera system104i.In some embodiments, the capture devices202a-202nmay be separate devices (e.g., remotely connected to the camera system104i,such as a drone, a robot and/or a system of security cameras configured capture video data) configured to send data to the camera system104i.In one example, the capture devices202a-202nmay be implemented as part of an autonomous robot configured to patrol particular paths such as hallways. Similarly, in the example shown, the wireless communication device204, the microphones206a-206nand/or the speakers208a-208nare shown external to the camera system104ibut in some embodiments may be a component of (e.g., within) the camera system104i.

The camera system104imay receive one or more signals (e.g., IMF_A-IMF_N), the signal FEAT_SET and/or one or more signals (e.g., DIR_AUD). The camera system104imay present the signal STAT (e.g., one of the signals STAT_A-STAT_N shown in association withFIG.3) and/or a signal (e.g., DIR_AOUT). The capture devices202a-202nmay receive the signals IMF_A-IMF_N from the corresponding lenses200a-200n.The processor130may receive the signal DIR_AUD from the microphones206a-206n.The processor130may present the signal STAT to the communication device204and receive the signal FEAT_SET from the communication device204. For example, the wireless communication device204may be a radio-frequency (RF) transmitter. In another example, the communication device204may be a Wi-Fi module. In another example, the communication device204may be a device capable of implementing RF transmission, Wi-Fi, Bluetooth and/or other wireless communication protocols. The processor130may present the signal DIR_AOUT to the speakers208a-208n.

The lenses200a-200nmay capture signals (e.g., IM_A-IM_N). The signals IM_A-IM_N may be an image (e.g., an analog image) of the environment near the camera system104ipresented by the lenses200a-200nto the capture devices202a-202nas the signals IMF_A-IMF_N. The lenses200a-200nmay be implemented as an optical lens. The lenses200a-200nmay provide a zooming feature and/or a focusing feature. The capture devices202a-202nand/or the lenses200a-200nmay be implemented, in one example, as a single lens assembly. In another example, the lenses200a-200nmay be a separate implementation from the capture devices202a-202n.The capture devices202a-202nare shown within the circuit104i.In an example implementation, the capture devices202a-202nmaybe implemented outside of the circuit104i(e.g., along with the lenses200a-200nas part of a lens/capture device assembly).

The capture devices202a-202nmay be configured to capture image data for video (e.g., the signals IMF_A-IMF_N from the lenses200a-200n). In some embodiments, the capture devices202a-202nmaybe video capturing devices such as cameras. The capture devices202a-202nmay capture data received through the lenses200a-200nto generate raw pixel data. In some embodiments, the capture devices202a-202nmay capture data received through the lenses200a-200nto generate bitstreams (e.g., generate video frames). For example, the capture devices202a-202nmay receive focused light from the lenses200a-200n.The lenses200a-200nmay be directed, tilted, panned, zoomed and/or rotated to provide a targeted view from the camera system104i(e.g., to provide coverage for a panoramic field of view such as the field of view152a-152b). The capture devices202a-202nmay generate signals (e.g., PIXELD_A-PIXELD_N). The signals PIXELD_A-PIXELD_N may be pixel data (e.g., a sequence of pixels that may be used to generate video frames). In some embodiments, the signals PIXELD_A-PIXELD_N may be video data (e.g., a sequence of video frames). The signals PIXELD_A-PIXELD_N may be presented to the inputs220a-220nof the processor130.

The capture devices202a-202nmay transform the received focused light signals IMF_A-IMF_N into digital data (e.g., bitstreams). In some embodiments, the capture devices202a-202nmay perform an analog to digital conversion. For example, the capture devices202a-202nmay perform a photoelectric conversion of the focused light received by the lenses200a-200n.The capture devices202a-202nmay transform the bitstreams into pixel data, images and/or video frames. In some embodiments, the pixel data generated by the capture devices202a-202nmay be uncompressed and/or raw data generated in response to the focused light from the lenses200a-200n.In some embodiments, the output of the capture devices202a-202nmay be digital video signals.

The communication device204may send and/or receive data to/from the camera system104i.In some embodiments, the communication device204may be implemented as a wireless communications module. In some embodiments, the communication device204may be implemented as a satellite connection to a proprietary system. In one example, the communication device204may be a hard-wired data port (e.g., a USB port, a mini-USB port, a USB-C connector, HDMI port, an Ethernet port, a DisplayPort interface, a Lightning port, etc.). In another example, the communication device204may be a wireless data interface (e.g., Wi-Fi, Bluetooth, ZigBee, cellular, etc.).

The communication device204may be configured to receive the signal FEAT_SET from the network102. The signal FEAT_SET may comprise a feature set that corresponds to the rental agreement122. The feature set information may comprise instructions for the processor130for determining a breach of the rental agreement122. Details of the feature set information may be described in association withFIG.5.

The processor130may receive the signals PIXELD_A-PIXELD_N from the capture devices202a-202nat the inputs220a-220n.The processor130may send/receive a signal (e.g., DATA) to/from the memory132at the input/output222. The processor130may send the signal STAT to the communication device204via the output port224a.In some embodiments, the port224amay be an input/output port and the processor130may receive one of the signals STAT_A-STAT_N from the other camera systems104a-104n.The processor130may receive the signal FEAT_SET from the communication device204via the input port224b.The processor130may receive the signal DIR_AUD from the microphones206a-206nat the port226. The processor130may send the signal DIR_AOUT to the speakers208a-208nvia the port228. In an example, the processor130may be connected through a bi-directional interface (or connection) to the capture devices202a-202n,the communication device204, the memory132, the microphones206a-206nand/or the speakers208a-208n.The processor130may store and/or retrieve data from the memory132. The memory132may be configured to store computer readable/executable instructions (or firmware). The instructions, when executed by the processor130, may perform a number of steps.

The signal PIXELD_A-PIXELD_N may comprise raw pixel data providing a field of view captured by the lenses200a-200n.The processor130may be configured to generate video frames from the pixel data PIXELD_A-PIXELD_N. The video frames generated by the processor130may be used internal to the processor130. In some embodiments, the video frames may be communicated to the memory132for temporary storage. Generally, the video frames generated by the processor130may not leave the processor130. The processor130may be configured to discard the video frames generated.

The processor130may be configured to make decisions based on analysis of the video frames generated from the signals PIXELD_A-PIXELD_N. The processor130may generate the signal STAT, the signal DATA, the signal DIR_AOUT and/or other signals (not shown). The signal STAT, the signal DATA and/or the signal DIR_AOUT may each be generated (in part) based on one or more decisions made and/or functions performed by the processor130. The decisions made and/or functions performed by the processor130may be determined based on data received by the processor130at the inputs220a-220n(e.g., the signals PIXELD_A-PIXELD_N), the input222, the input224b,the input226and/or other inputs.

The inputs220a-220n,the input/output222, the output224a,the input224b,the input226, the output220and/or other inputs/outputs may implement an interface. The interface may be implemented to transfer data to/from the processor130, the communication device204, the capture devices202a-202n,the memory132, the microphones206a-206n,the speakers208a-208nand/or other components of the camera system104i.In one example, the interface may be configured to receive (e.g., via the inputs220a-220n) the pixel data signals PIXELD_A-PIXELD_N each from a respective one of the capture devices202a-202n.In another example, the interface may be configured to receive (e.g., via the input226) the directional audio DIR_AUD. In yet another example, the interface may be configured to transmit parameters and/or statistics about the video frames (e.g., the signal STAT) and/or the converted data determined based on the computer vision operations to the communication device204. In still another example, the interface may be configured to receive the feature set information FEAT_SET (e.g., via the input port224b) from the communication device204. In another example, the interface may be configured to transmit directional audio output (e.g., the signal DIR_AOUT) to each of the speakers208a-208n.The interface may be configured to enable transfer of data and/or translate data from one format to another format to ensure that the data transferred is readable by the intended destination component. In an example, the interface may comprise a data bus, traces, connectors, wires and/or pins. The implementation of the interface may be varied according to the design criteria of a particular implementation.

The signal STAT may be presented to the communication device204. In some embodiments, the signal STAT may comprise parameters and/or statistics determined by the processor130about the video frames. The signal STAT may be generated in response to the computer vision operations performed. The video frames may be encoded, cropped, stitched and/or enhanced versions of the pixel data received from the signals PIXELD_A-PIXELD_N. In an example, the video frames maybe a high resolution, digital, encoded, de-warped, stabilized, cropped, blended, stitched and/or rolling shutter effect corrected version of the signals PIXELD_A-PIXELD_N.

In some embodiments, the signal STAT may be a text message (e.g., a string of human readable characters). In some embodiments, the signal STAT may be a symbol that indicates an event or status (e.g., sound symbol indicating loud noise has been detected, an animal symbol indicating a pet has been detected, a symbol of a group of people to indicate that too many people have been detected at the location50, etc.). The signal STAT may be generated based on video analytics (e.g., computer vision operations) performed by the processor130on the video frames generated from the pixel data PIXELD_A-PIXELD_N. The processor130may be configured to perform the computer vision operations to detect objects and/or events in the video frames and then convert the detected objects and/or events into statistics and/or parameters. The data determined by the computer vision operations maybe converted to the human-readable format by the processor130. The data from the computer vision operations that has been converted to the human-readable format may be communicated as the signal STAT.

In some embodiments, the signal STAT may be data generated by the processor130(e.g., video analysis results, audio/speech analysis results, etc.) that may be communicated to a cloud computing service in order to aggregate information and/or provide training data for machine learning (e.g., to improve object detection, to improve audio detection, etc.). The type of information communicated by the signal STAT may be varied according to the design criteria of a particular implementation.

The circuit104imay implement a camera system. In some embodiments, the camera system104imay be implemented as a drop-in solution (e.g., installed as one component). In an example, the camera system104imay be a device that may be installed as an after-market product (e.g., a retro-fit for a drone, a retro-fit for a security system, etc.). In some embodiments, the camera system104imay be a component of a security system. The number and/or types of signals and/or components implemented by the camera system104imay be varied according to the design criteria of a particular implementation.

The video data of the targeted view captured in the field of view152a-152bmay be generated from the signals/bitstreams/data PIXELD_A-PIXELD_N. The capture devices202a-202nmay present the signals PIXELD_A-PIXELD_N to the inputs220a-220nof the processor130. The signals PIXELD_A-PIXELD_N may be used by the processor130to generate the video frames/video data. In some embodiments, the signals PIXELD_A-PIXELD_N maybe video streams captured by the capture devices202a-202n.In some embodiments, the capture devices202a-202nmay be implemented in the camera system104i.In some embodiments, the capture devices202a-202nmay be configured to add to existing functionality to the camera system104i.

Each of the capture devices202a-202nmay comprise a block (or circuit)230, a block (or circuit)232, and/or a block (or circuit)234. The circuit230may implement a camera sensor (e.g., a complementary metal-oxide-semiconductor (CMOS) sensor). The circuit232may implement a camera processor/logic. The circuit234may implement a memory buffer. As a representative example, the capture device202ais shown comprising the sensor230a,the logic block232aand the buffer234a.Similarly, the capture devices202b-202nmay comprise the camera sensors230b-230n,the logic blocks232b-232nand the buffers234b-234n.The sensors230a-230nmay each be configured to receive light from the corresponding one of the lenses200a-200nand transform the light into digital data (e.g., the bitstreams).

In one example, the sensor230aof the capture device202amay receive light from the lens200a.The camera sensor230aof the capture device202amay perform a photoelectric conversion of the light from the lens200a.In some embodiments, the sensor230amay be an oversampled binary image sensor. The logic232amay transform the bitstream into a human-legible content (e.g., pixel data and/or video data). For example, the logic232amay receive pure (e.g., raw) data from the camera sensor230aand generate pixel data based on the raw data (e.g., the bitstream). The memory buffer234amay store the raw data and/or the processed bitstream. For example, the frame memory and/or buffer234amay store (e.g., provide temporary storage and/or cache) the pixel data and/or one or more of the video frames (e.g., the video signal).

The microphones206a-206nmay be configured to capture incoming audio and/or provide directional information about the incoming audio. Each of the microphones206a-206nmay receive a respective signal (e.g., AIN_A-AIN_N). The signals AIN_A-AIN_N may be audio signals from the environment50near the camera system104i.For example, the signals AIN_A-AIN_N may be ambient noise in the environment50and/or the audio164a-164nfrom the subjects70a-70n.The microphones206a-206nmay be configured to generate the signal DIR_AUD in response to the signals AIN_A-AIN_N. The signal DIR_AUD may be a signal that comprises the audio data from the signals AIN_A-AIN_N. The signal DIR_AUD may be a signal generated in a format that provides directional information about the signals AIN_A-AIN_N.

The microphones206a-206nmay provide the signal DIR_AUD to the interface226. The camera system104imay comprise the interface226configured to receive data (e.g., the signal DIR_AUD) from one or more of the microphones206a-206n.In one example, data from the signal DIR_AUD presented to the interface226may be used by the processor130to determine the location of the source of the audio164a-164n.In another example, the microphones206a-206nmay be configured to determine the location of the audio164a-164nand present the location to the interface226as the signal DIR_AUD.

The number of microphones206a-206nmay be varied according to the design criteria of a particular implementation. The number of microphones206a-206nmay be selected to provide sufficient directional information about the incoming audio (e.g., the number of microphones206a-206nimplemented may be varied based on the accuracy and/or resolution of directional information acquired). In an example, 2 to 6 of the microphones206a-206nmay be implemented. In some embodiments, an audio processing component may be implemented with the microphones206a-206nto process and/or encode the incoming audio signals AIN_A-AIN_N. In some embodiments, the processor130may be configured with on-chip audio processing to encode the incoming audio signals AIN_A-AIN_N. The microphones206a-206nmay capture audio of the environment50. The camera system104imaybe configured to synchronize the audio captured with the images captured by the capture devices202a-202n.

The processor130may be configured to execute computer readable code and/or process information. The processor130may be configured to receive input and/or present output to the memory132. The processor130may be configured to present and/or receive other signals (not shown). The number and/or types of inputs and/or outputs of the processor130may be varied according to the design criteria of a particular implementation.

The processor130may receive the signals PIXELD_A-PIXELD_N, the signal DIR_AUDIO and/or the signal DATA. The processor130may make a decision based on data received at the inputs220a-220n,the input222, the input224b,the input226and/or other input. For example, other inputs may comprise external signals generated in response to user input, external signals generated by the microphones206a-206nand/or internally generated signals such as signals generated by the processor130in response to analysis of the video frames and/or objects detected in the video frames. The processor130may adjust the video data (e.g., crop, digitally move, physically move the camera sensors230a-230n,etc.) of the video frames. The processor130may generate the signal STAT and/or the signal DIR_AOUT in response to data received by the inputs220a-220n,the input222, the input224b,the input226and/or the decisions made in response to the data received by the inputs220a-220n,the input222, the input224band/or the input226.

The signal STAT and/or the signal DIR_AOUT may be generated to provide an output in response to the captured video frames and the video analytics performed by the processor130. For example, the video analytics may be performed by the processor130in real-time and/or near real-time (e.g., with minimal delay).

The cropping, downscaling, blending, stabilization, packetization, encoding, compression and/or conversion performed by the processor130may be varied according to the design criteria of a particular implementation. For example, the video frames generated by the processor130may be a processed version of the signals PIXELD_A-PIXELD_N configured to enable detection of the objects160a-160nand/or determination of the characteristics162. In some embodiments, the video data may be encoded at a high bitrate. For example, the signal may be generated using a lossless compression and/or with a low amount of lossiness.

In some embodiments, the video frames may be some view (or derivative of some view) captured by the capture devices202a-202n.For example, the video frames may comprise a portion of the panoramic video captured by the capture devices202a-202n.In another example, the video frames may comprise a region of interest selected and/or cropped from the panoramic video frame by the processor130(e.g., upscaled, oversampled and/or digitally zoomed) to enable a high precision of object detection. In some embodiments, the video frames may provide a series of cropped and/or enhanced panoramic video frames that improve upon the view from the perspective of the camera system104i(e.g., provides night vision, provides High Dynamic Range (HDR) imaging, provides more viewing area, highlights detected objects, provides additional data such as a numerical distance to detected objects, etc.) to enable the processor130to see the location50better than a person would be capable of with human vision.

The memory132may store data. The memory132may be implemented as a cache, flash memory, DRAM memory, etc. The type and/or size of the memory132may be varied according to the design criteria of a particular implementation. The data stored in the memory132may correspond to a video file, user profiles, user permissions, the rental agreement122, the terms and/or entries of the rental agreement122, contact information for the renter/landlord, etc.

The lenses200a-200n(e.g., camera lenses) may be directed to provide a panoramic view from the camera system104i.The lenses200a-200nmay be aimed to capture environmental data (e.g., light). The lens200a-200nmay be configured to capture and/or focus the light for the capture devices202a-202n.Generally, the camera sensors230a-230nmay be located behind each of the respective lenses200a-200n.Based on the captured light from the lenses200a-200n,the capture devices202a-202nmay generate a bitstream and/or raw pixel data.

Embodiments of the processor130may perform video stitching operations on the signals PIXELD_A-PIXELD_N. In one example, each of the pixel data signals PIXELD_A-PIXELD_N may provide a portion of a panoramic view and the processor130may crop, blend, synchronize and/or align the pixel data from the signals PIXELD_A-PIXELD_N to generate the panoramic video frames. In some embodiments, the processor130may be configured to perform electronic image stabilization (EIS). The processor130may perform de-warping on the video frames. The processor130may perform intelligent video analytics on the de-warped video frames. The processor130discard the video frames after the video analytics and/or computer vision has been performed.

The encoded video frames may be processed locally and discarded. In one example, the encoded, panoramic video may be stored locally by the memory132to enable the processor130to facilitate the computer vision analysis and then discarded. The processor130may discard the video frames as soon as possible after the video frames are no longer needed. Generally, after the processor130determines the parameters and/or statistics170a-170n,the video frames are no longer needed.

The processor130may receive an input to generate the video frames (e.g., the signals PIXELD_A-PIXELD_N) from the CMOS sensor(s)230a-230n.The pixel data signals PIXELD_A-PIXELD_N may be enhanced by the processor130(e.g., color conversion, noise filtering, auto exposure, auto white balance, auto focus, etc.). Generally, the panoramic video may comprise a large field of view generated by one or more lenses/camera sensors. One example of a panoramic video may be an equirectangular 360 video. Equirectangular 360 video may also be called spherical panoramas. Panoramic video may be a video that provides a field of view that is larger than the field of view that may be displayed on a device used to playback the video. For example, the field of view152a-152bcaptured by the camera system104imay be used to generate panoramic video such as a spherical video, a hemispherical video, a 360 degree video, a wide angle video, a video having less than a 360 field of view, etc.

Panoramic videos may comprise a view of the environment near the camera system104i.In one example, the entire field of view152a-152bof the panoramic video may be captured at generally the same time (e.g., each portion of the panoramic video represents the view from the camera system104iat one particular moment in time). In some embodiments (e.g., when the camera system104iimplements a rolling shutter sensor), a small amount of time difference may be present between some portions of the panoramic video. Generally, each video frame of the panoramic video comprises one exposure of the sensor (or the multiple sensors230a-230n) capturing the environment near the camera system104i.

In some embodiments, the field of view152a-152bmay provide coverage for a full 360 degree field of view. In some embodiments, less than a 360 degree view may be captured by the camera system104i(e.g., a 270 degree field of view, a 180 degree field of view, etc.). In some embodiments, the panoramic video may comprise a spherical field of view (e.g., capture video above and below the camera system104i). For example, the camera system104imay be mounted on a ceiling and capture a spherical field of view of the area below the camera system104i.In some embodiments, the panoramic video may comprise a field of view that is less than a spherical field of view (e.g., the camera system104imay be configured to capture the ground below and the areas to the sides of the camera system104ibut nothing directly above). The implementation of the camera system104iand/or the captured field of view152a-152bmay be varied according to the design criteria of a particular implementation.

In embodiments implementing multiple lenses, each of the lenses200a-200nmay be directed towards one particular direction to provide coverage for a full 360 degree field of view. In embodiments implementing a single wide angle lens (e.g., the lens200a), the lens200amay be located to provide coverage for the full 360 degree field of view (e.g., on the bottom of the camera system104iin a ceiling mounted embodiment, on the bottom of a drone camera, etc.). In some embodiments, less than a 360 degree view may be captured by the lenses200a-200n(e.g., a 270 degree field of view, a 180 degree field of view, etc.). In some embodiments, the lenses200a-200nmay move (e.g., the direction of the capture devices may be controllable). In some embodiments, one or more of the lenses200a-200nmay be configured to implement an optical zoom (e.g., the lenses200a-200nmay zoom in/out independent of each other).

In some embodiments, the camera system104imay be implemented as a system on chip (SoC). For example, the camera system104imay be implemented as a printed circuit board comprising one or more components (e.g., the capture devices202a-202n,the processor130, the communication device204, the memory132, etc.). The camera system104imay be configured to perform intelligent video analysis on the video frames of the de-warped, panoramic video. The camera system104imay be configured to crop and/or enhance the panoramic video.

In some embodiments, the processor130may be configured to perform sensor fusion operations. The sensor fusion operations performed by the processor130may be configured to analyze information from multiple sources (e.g., the capture devices202a-202nand the microphones206a-206n). By analyzing various data from disparate sources, the sensor fusion operations may be capable of making inferences about the data that may not be possible from one of the data sources alone. For example, the sensor fusion operations implemented by the processor130may analyze video data (e.g., mouth movements of the subjects70a-70n) as well as the speech patterns from the directional audio DIR_AUD. The disparate sources may be used to develop a model of a scenario to support decision making For example, the processor130may be configured to compare the synchronization of the detected speech patterns with the mouth movements in the video frames to determine which person in a video frame is speaking The sensor fusion operations may also provide time correlation, spatial correlation and/or reliability among the data being received.

In some embodiments, the processor130may implement convolutional neural network capabilities. The convolutional neural network capabilities may implement computer vision using deep learning techniques. The convolutional neural network capabilities may be configured to implement pattern and/or image recognition using a training process through multiple layers of feature-detection.

The signal DIR_AOUT maybe an audio output. For example, the processor130may generate output audio based on information extracted from the video frames PIXELD_A-PIXELD_N. The signal DIR_AOUT may be determined based on an event and/or objects determined using the computer vision operations. In one example, the signal DIR_AOUT may comprise an audio message informing the people70a-70nthat the rental agreement122has been breached. In some embodiments, the signal DIR_AOUT may not be generated until an event has been detected by the processor130using the computer vision operations.

The signal DIR_AOUT may comprise directional and/or positional audio output information for the speakers208a-208n.The speakers208a-208nmay receive the signal DIR_AOUT, process the directional and/or positional information and determine which speakers and/or which channels will play back particular audio portions of the signal DIR_AOUT. The speakers208a-208nmay generate the signals AOUT_A-AOUT_N in response to the signal DIR_AOUT. The signals AOUT_A-AOUT_N may be the audio message played to the people70a-70n.For example, the speakers208a-208nmay emit a pre-recorded message in response to a detected event. The signal DIR_AOUT may be a signal generated in a format that provides directional information for the signals AOUT_A-AOUT_N.

The number of speakers208a-208nmay be varied according to the design criteria of a particular implementation. The number of speakers208a-208nmay be selected to provide sufficient directional channels for the outgoing audio (e.g., the number of speakers208a-208nimplemented may be varied based on the accuracy and/or resolution of directional audio output). In an example, 1 to 6 of the speakers208a-208nmay be implemented. In some embodiments, an audio processing component may be implemented by the speakers208a-208nto process and/or decode the output audio signals DIR_AOUT. In some embodiments, the processor130may be configured with on-chip audio processing. In some embodiments, the signal DIR_AOUT may playback audio received from the remote devices52a-52nin order to implement a 2-way real-time audio communication.

The video pipeline of the processor130may be configured to perform de-warping, cropping, enhancements, rolling shutter corrections, stabilizing, downscaling, packetizing, compression, conversion, blending, synchronizing and/or other video operations. The architecture of the video pipeline of the processor130may enable the video operations to be performed on high resolution video and/or high bitrate video data in real-time and/or near real-time. The video pipeline of the processor130may enable computer vision processing on 4K resolution video data, stereo vision processing, object detection, 3D noise reduction, fisheye lens correction (e.g., real time 360-degree dewarping and lens distortion correction), oversampling and/or high dynamic range processing. In one example, the architecture of the video pipeline may enable 4K ultra high resolution with H.264 encoding at double real time speed (e.g., 60 fps), 4K ultra high resolution with H.265/HEVC at 30 fps and/or 4K AVC encoding. Since the video frames generated by the processor130may be discarded after the data is extracted, encoding operations of the processor130may not be necessary. For example, while the processor130may be capable of performing the encoding operations, in the implementation of the camera systems104a-104n,the encoding may not need to be performed. The type of video operations and/or the type of video data operated on by the processor130may be varied according to the design criteria of a particular implementation.

The sensors230a-230nmay each implement a high-resolution sensor. Using the high resolution sensors230a-230n,the processor130may combine over-sampling of the image sensors230a-230nwith digital zooming within a cropped area. The over-sampling and digital zooming may each be one of the video operations performed by the processor130. The over-sampling and digital zooming may be implemented to deliver higher resolution images within the total size constraints of a cropped area.

In some embodiments, one or more of the lenses200a-200nmay implement a fisheye lens. One of the video operations implemented by the processor130may be a dewarping operation. The processor130may be configured to dewarp the video frames generated. The dewarping may be configured to reduce and/or remove acute distortion caused by the fisheye lens and/or other lens characteristics. For example, the dewarping may reduce and/or eliminate a bulging effect to provide a rectilinear image.

The processor130may be configured to crop (e.g., trim to) a region of interest from a full video frame (e.g., generate the region of interest video frames). The processor130may generate the video frames and select an area. In an example, cropping the region of interest may generate a second image. The cropped image (e.g., the region of interest video frame) may be smaller than the original video frame (e.g., the cropped image may be a portion of the captured video).

The area of interest may be dynamically adjusted based on the location of an audio source. For example, the detected audio source may be moving, and the location of the detected audio source may move as the video frames are captured. The processor130may update the selected region of interest coordinates and dynamically update the cropped section (e.g., the directional microphones106a-106nmay dynamically update the location based on the directional audio captured). The cropped section may correspond to the area of interest selected. As the area of interest changes, the cropped portion may change. For example, the selected coordinates for the area of interest may change from frame to frame, and the processor130may be configured to crop the selected region in each frame.

The processor130may be configured to over-sample the image sensors230a-230n.The over-sampling of the image sensors230a-230nmay result in a higher resolution image. The processor130may be configured to digitally zoom into an area of a video frame. For example, the processor130may digitally zoom into the cropped area of interest. For example, the processor130may establish the area of interest based on the directional audio, crop the area of interest, and then digitally zoom into the cropped region of interest video frame.

The dewarping operations performed by the processor130may adjust the visual content of the video data. The adjustments performed by the processor130may cause the visual content to appear natural (e.g., appear as seen by a person viewing the location corresponding to the field of view of the capture devices202a-202n). In an example, the dewarping may alter the video data to generate a rectilinear video frame (e.g., correct artifacts caused by the lens characteristics of the lenses200a-200n). The dewarping operations may be implemented to correct the distortion caused by the lenses200a-200n.The adjusted visual content may be generated to enable more accurate and/or reliable object detection.

Various features (e.g., dewarping, digitally zooming, cropping, etc.) may be implemented in the processor130as hardware modules. Implementing hardware modules may increase the video processing speed of the processor130(e.g., faster than a software implementation). The hardware implementation may enable the video to be processed while reducing an amount of delay. The hardware components used maybe varied according to the design criteria of a particular implementation.

The processor130is shown comprising a number of blocks (or circuits)240a-240n.The blocks240a-240nmay implement various hardware modules implemented by the processor130. The hardware modules240a-240nmay be configured to provide various hardware components to implement a video processing pipeline. The circuits240a-240nmay be configured to receive the pixel data PIXELD_A-PIXELD_N, generate the video frames from the pixel data, perform various operations on the video frames (e.g., de-warping, rolling shutter correction, cropping, upscaling, image stabilization, etc.), prepare the video frames for communication to external hardware (e.g., encoding, packetizing, color correcting, etc.), parse feature sets, implement various operations for computer vision, etc. Various implementations of the processor130may not necessarily utilize all the features of the hardware modules240a-240n.The features and/or functionality of the hardware modules240a-240nmay be varied according to the design criteria of a particular implementation. Details of the hardware modules240a-240nmay be described in association with U.S. patent application Ser. No. 16/831,549, filed on Apr. 16, 2020, U.S. patent application Ser. No. 16/288,922, filed on Feb. 28, 2019 and U.S. patent application Ser. No. 15/593,493 (now U.S. Pat. No. 10,437,600), filed on May 12, 2017, appropriate portions of which are hereby incorporated by reference in their entirety.

The hardware modules240a-240nmay be implemented as dedicated hardware modules. Implementing various functionality of the processor130using the dedicated hardware modules240a-240nmay enable the processor130to be highly optimized and/or customized to limit power consumption, reduce heat generation and/or increase processing speed compared to software implementations. The hardware modules240a-240nmay be customizable and/or programmable to implement multiple types of operations. Implementing the dedicated hardware modules240a-240nmay enable the hardware used to perform each type of calculation to be optimized for speed and/or efficiency. For example, the hardware modules240a-240nmay implement a number of relatively simple operations that are used frequently in computer vision operations that, together, may enable the computer vision algorithm to be performed in real-time. The video pipeline may be configured to recognize objects. Objects may be recognized by interpreting numerical and/or symbolic information to determine that the visual data represents a particular type of object and/or feature. For example, the number of pixels and/or the colors of the pixels of the video data may be used to recognize portions of the video data as objects.

One of the hardware modules240a-240n(e.g.,240a) may implement a scheduler circuit. The scheduler circuit240amay be configured to store a directed acyclic graph (DAG). In an example, the scheduler circuit240amay be configured to generate and store the directed acyclic graph in response to the feature set information received in the signal FEAT_SET. The directed acyclic graph may define the video operations to perform for extracting the data170a-170nfrom the video frames. For example, the directed acyclic graph may define various mathematical weighting to apply when performing computer vision operations to classify various groups of pixels as particular objects.

The scheduler circuit240amay be configured to parse the acyclic graph to generate various operators. The operators may be scheduled by the scheduler circuit240ain one or more of the other hardware modules240a-240n.For example, one or more of the hardware modules240a-240nmay implement hardware engines configured to perform specific tasks (e.g., hardware engines designed to perform particular mathematical operations that are repeatedly used to perform computer vision operations). The scheduler circuit240amay schedule the operators based on when the operators may be ready to be processed by the hardware engines240a-240n.

The scheduler circuit240amay time multiplex the tasks to the hardware modules240a-240nbased on the availability of the hardware modules240a-240nto perform the work. The scheduler circuit240amay parse the directed acyclic graph into one or more data flows. Each data flow may include one or more operators. Once the directed acyclic graph is parsed, the scheduler circuit240amay allocate the data flows/operators to the hardware engines240a-240nand send the relevant operator configuration information to start the operators.

Each directed acyclic graph binary representation may be an ordered traversal of a directed acyclic graph with descriptors and operators interleaved based on data dependencies. The descriptors generally provide registers that link data buffers to specific operands in dependent operators. In various embodiments, an operator may not appear in the directed acyclic graph representation until all dependent descriptors are declared for the operands.

One of the hardware modules240a-240n(e.g.,240b) may implement a convolutional neural network (CNN) module. The CNN module240bmay be configured to perform the computer vision operations on the video frames. The CNN module240bmay be configured to implement recognition of the objects160a-160nthrough multiple layers of feature detection. The CNN module240bmay be configured to calculate descriptors based on the feature detection performed. The descriptors may enable the processor130to determine a likelihood that pixels of the video frames correspond to particular objects (e.g., the people70a-70n,pets, items, etc.).

The CNN module240bmay be configured to implement convolutional neural network capabilities. The CNN module240bmay be configured to implement computer vision using deep learning techniques. The CNN module240bmay be configured to implement pattern and/or image recognition using a training process through multiple layers of feature-detection. The CNN module240bmay be configured to conduct inferences against a machine learning model.

The CNN module240bmay be configured to perform feature extraction and/or matching solely in hardware. Feature points typically represent interesting areas in the video frames (e.g., corners, edges, etc.). By tracking the feature points temporally, an estimate of ego-motion of the capturing platform or a motion model of observed objects in the scene may be generated. In order to track the feature points, a matching algorithm is generally incorporated by hardware in the CNN module240bto find the most probable correspondences between feature points in a reference video frame and a target video frame. In a process to match pairs of reference and target feature points, each feature point may be represented by a descriptor (e.g., image patch, SIFT, BRIEF, ORB, FREAK, etc.). Implementing the CNN module240busing dedicated hardware circuitry may enable calculating descriptor matching distances in real time.

The CNN module240bmay be a dedicated hardware module configured to perform feature detection of the video frames. The features detected by the CNN module240bmay be used to calculate descriptors. The CNN module240bmay determine a likelihood that pixels in the video frames belong to a particular object and/or objects in response to the descriptors. For example, using the descriptors, the CNN module240bmay determine a likelihood that pixels correspond to a particular object (e.g., a person, an item of furniture, a picture of a person, a pet, etc.) and/or characteristics of the object (e.g., a mouth of a person, a hand of a person, a screen of a television set, an armrest of a couch, a clock, etc.). Implementing the CNN module240bas a dedicated hardware module of the processor130may enable the camera system104ito perform the computer vision operations locally (e.g., on-chip) without relying on processing capabilities of a remote device (e.g., communicating data to a cloud computing service).

The computer vision operations performed by the CNN module240bmay be configured to perform the feature detection on the video frames in order to generate the descriptors. The CNN module240bmay perform the object detection to determine regions of the video frame that have a high likelihood of matching the particular object. In one example, the types of object to match against (e.g., reference objects) may be customized using an open operand stack (enabling programmability of the processor130to implement various directed acyclic graphs each providing instructions for performing various types of object detection). The CNN module240bmay be configured to perform local masking to the region with the high likelihood of matching the particular object(s) to detect the object.

In some embodiments, the CNN module240bmay determine the position (e.g., 3D coordinates and/or location coordinates) of various features (e.g., the characteristics162) of the detected objects160a-160n.In one example, the location of the arms, legs, chest and/or eyes may be determined using 3D coordinates. One location coordinate on a first axis for a vertical location of the body part in 3D space and another coordinate on a second axis for a horizontal location of the body part in 3D space may be stored. In some embodiments, the distance from the lenses200a-200nmay represent one coordinate (e.g., a location coordinate on a third axis) for a depth location of the body part in 3D space. Using the location of various body parts in 3D space, the processor130may determine body position, and/or body characteristics of the people70a-70n.

The CNN module240bmay be pre-trained (e.g., configured to perform computer vision to detect objects based on the training data received to train the CNN module240b). For example, the results of training data (e.g., a machine learning model) maybe pre-programmed and/or loaded into the processor130. The CNN module240bmay conduct inferences against the machine learning model (e.g., to perform object detection). The training may comprise determining weight values for each of the layers. For example, weight values may be determined for each of the layers for feature extraction (e.g., a convolutional layer) and/or for classification (e.g., a fully connected layer). The weight values learned by the CNN module240bmay be varied according to the design criteria of a particular implementation.

The convolution operation may comprise sliding a feature detection window along the layers while performing calculations (e.g., matrix operations). The feature detection window may apply a filter to pixels and/or extract features associated with each layer. The feature detection window may be applied to a pixel and a number of surrounding pixels. In an example, the layers may be represented as a matrix of values representing pixels and/or features of one of the layers and the filter applied by the feature detection window may be represented as a matrix. The convolution operation may apply a matrix multiplication between the region of the current layer covered by the feature detection window. The convolution operation may slide the feature detection window along regions of the layers to generate a result representing each region. The size of the region, the type of operations applied by the filters and/or the number of layers may be varied according to the design criteria of a particular implementation.

Using the convolution operations, the CNN module240bmay compute multiple features for pixels of an input image in each extraction step. For example, each of the layers may receive inputs from a set of features located in a small neighborhood (e.g., region) of the previous layer (e.g., a local receptive field). The convolution operations may extract elementary visual features (e.g., such as oriented edges, end-points, corners, etc.), which are then combined by higher layers. Since the feature extraction window operates on a pixel and nearby pixels, the results of the operation may have location invariance. The layers may comprise convolution layers, pooling layers, non-linear layers and/or fully connected layers. In an example, the convolution operations may learn to detect edges from raw pixels (e.g., a first layer), then use the feature from the previous layer (e.g., the detected edges) to detect shapes in a next layer and then use the shapes to detect higher-level features (e.g., facial features, pets, furniture, etc.) in higher layers and the last layer may be a classifier that uses the higher level features.

The CNN module240bmay execute a data flow directed to feature extraction and matching, including two-stage detection, a warping operator, component operators that manipulate lists of components (e.g., components may be regions of a vector that share a common attribute and may be grouped together with a bounding box), a matrix inversion operator, a dot product operator, a convolution operator, conditional operators (e.g., multiplex and demultiplex), a remapping operator, a minimum-maximum-reduction operator, a pooling operator, a non-minimum, non-maximum suppression operator, a scanning-window based non-maximum suppression operator, a gather operator, a scatter operator, a statistics operator, a classifier operator, an integral image operator, comparison operators, indexing operators, a pattern matching operator, a feature extraction operator, a feature detection operator, a two-stage object detection operator, a score generating operator, a block reduction operator, and an upsample operator. The types of operations performed by the CNN module240bto extract features from the training data may be varied according to the design criteria of a particular implementation.

Each of the hardware modules240a-240nmay implement a processing resource (or hardware resource or hardware engine). The hardware engines240a-240nmay be operational to perform specific processing tasks. In some configurations, the hardware engines240a-240nmay operate in parallel and independent of each other. In other configurations, the hardware engines240a-240nmay operate collectively among each other to perform allocated tasks. One or more of the hardware engines240a-240nmay be homogenous processing resources (all circuits240a-240nmay have the same capabilities) or heterogeneous processing resources (two or more circuits240a-240nmay have different capabilities).

Referring toFIG.5, a diagram illustrating detecting a breach of a rental property agreement is shown. An example scenario250is shown. The example scenario250may comprise the server102, remote devices52b-52nand/or the camera system104. The capture device202aand the microphone206aare shown on the camera system104. The processor130is shown within the camera system104. The example scenario250may further comprise a visualization of the memory132, the processor130and/or a rental agreement122.

The server102may be configured to generate the rental agreement122. The rental agreement122may comprise a text (e.g., human-readable) version252and a machine readable version254. The text version252of the rental agreement122may be used to enable both the property owner and the renter to view and understand the terms and conditions of the rental agreement122. The machine readable version254may comprise a neural network and/or computer readable instructions that define the terms and conditions of the rental agreement122that may be readable by the processor130of the camera system104in order to generate the detection parameters). For example, the machine readable version254may be generated according to an API (e.g., a format) compatible with the camera system104. The machine readable version254of the rental agreement122may comprise one or more neural networks (e.g, each neural network may correspond to various types of objects to detect based on the terms and conditions of the rental agreement122). The text version252and/or the machine readable version254of the rental agreement may be provided to the camera system104in the signal FEAT_SET.

The text version252and/or the machine readable version254of the rental agreement122may define what may be considered a breach of the rental agreement122at the location50. The renter and the landlord may refer to the text version252of the rental agreement122to determine what is acceptable usage of the location50and what may not be acceptable usage of the location50. The camera system104may analyze the video frames based on the machine readable version254. The machine readable version254may correspond to the text version252. In an example, if one of the terms of the rental agreement is that only four people are allowed at the location50, then the text version252may comprise an entry (e.g., “Number of guests allowed: 4”) and the machine readable version254may comprise instructions for the processor130to search the video frames to determine the number of people70a-70ndetected and determine whether more than four guests have been detected. The type of data and/or the format of the text version252and/or the machine readable version254may be varied according to the design criteria of a particular implementation.

The camera system104is shown receiving the signal FEAT_SET based on the rental agreement122from the server102. The camera system104is shown generating the signals NTF_A-NTF_N. In the example shown, the signal NTF_A may be communicated to the server102and the signals NTF_B-NTF_N may be communicated to the remote devices52b-52n.In some embodiments, the camera system104may communicate the signal STAT (as shown in association withFIG.4) to the server102and the server102may communicate the signal(s) NTF_A-NTF_N to the remote devices52a-52n.In an example, the server102may securely store the contact information for the owners of the smartphones52a-52n(e.g., the renters and/or the landlords).

The camera system104may receive the signal FEAT_SET from the server102. The processor130may convert the feature set information in the signal FEAT_SET to detection parameters. The camera system104may capture pixel data of the rental property location50and generate the video frames from the pixel data PIXELD_A-PIXELD_N. The camera system104is shown generating a signal (e.g., VIDEO_DATA). The signal VIDEO_DATA may comprise the video frames generated by the processor130. The signal VIDEO_DATA may comprise data used internally by the processor130. In an example, the signal VIDEO_DATA may never be communicated from the camera system104.

In the example scenario250, the video frames VIDEO_DATA are shown being used by the processor130. In an example, the video frames may be operated on using a video processing pipeline implemented by processor130of the camera system104. The processor130may comprise a block (or circuit)260and/or a block (or circuit)262. The block260may implement computer vision modules. The block262may represent a discard location of the processor130. The computer vision modules260and/or the discard location262may comprise components of the video processing pipeline of the processor130. The processor130may comprise other components (not shown). The number, type and/or arrangement of the components of the processor130may be varied according to the design criteria of a particular implementation.

Video frames270a-270nare shown being operated on by the computer vision modules260. The video frames270a-270nmay be the data in the signal VIDEO_FRAMES. The video frames270a-270nmay be generated by the processor130in response to the pixel data PIXELD_A-PIXELD_N received from the capture devices202a-202n.In the example shown, the video frame270amay generally correspond to the location50shown in association withFIG.2. The computer vision modules260of the processor130may perform the video operations (e.g., the analytics using computer vision) locally (e.g., the video frames270a-270nmay not leave the apparatus104).

The feature set information in the signal FEAT_SET may comprise instructions that may be compatible with the computer vision modules260of the processor130. In an example, the signal FEAT_SET may be configured to provide the machine readable version254of the rental agreement122in a format that may be compatible with the camera system104. The feature set information may provide the parameters that the computer vision modules260may use to analyze the video frames270a-270n.The parameters may define the criteria that the computer vision modules260use to determine whether there has been a breach of the rental agreement122. The signal FEAT_SET may comprise parameters for video (e.g., pet detection, types of pets allowed, behavior allowed, number of guests allowed, locations of particular items, etc.). The signal FEAT_SET may further comprise parameters for audio (e.g., a maximum audio level, frequencies allowed, times of day that particular audio levels are allowed, etc.). Generally, the computer vision modules260may compare the characteristics162detected in the video frames270a-270nwith the rental agreement122by using the parameters defined by the feature set information. The information provided in the feature set information may be varied according to the design criteria of a particular implementation.

Part of the video processing pipeline of the processor130may comprise the computer vision modules260. The computer vision modules260may be configured to perform object detection, classify objects, and/or extract data from the video frames270a-270n.The computer vision modules260may be configured to detect the objects160a-160nin the video frames270a-270nand/or generate the extracted data170a-170nabout the detected objects160a-160nbased on the characteristics162of the objects160a-160n.In the example diagram shown, the video frames270a-270n(e.g., the signal VIDEO_DATA) may be presented to computer vision modules260of the video processing pipeline. The processor130may compare the data extracted with the detection parameters (e.g., the feature set information) of the signal FEAT_SET according to the machine readable version254of the rental agreement122to determine whether or not there has been a breach of the conditions of the rental agreement122.

After the video frames270a-270nhave been analyzed by the computer vision operations performed by the processor130, the video frames270a-270nmay be discarded. In the example shown, the garbage can262may represent the processor130implementing a discarding method for the camera system104. The video frames270a-270nmay be passed from the video processing pipeline to the deletion function262to make the video frames270a-270nunavailable (e.g., flushed from cache, flagged to be overwritten, redirected to /dev/null, etc.). To ensure that privacy of the renter (or other people) is protected, no images or audio may ever leave the camera system104. The video data and/or audio data may be eliminated after processing and may be unavailable for communication beyond the camera system104(e.g., not be stored to long-term memory, not communicated to another device, etc.).

In the example shown, the video processing pipeline of the processor130may detect a breach in the terms of the rental agreement122(e.g., too many people have been detected at the rental property50). For example, the feature set may provide instructions for counting the number of people in the video frames270a-270n,and the computer vision modules260may detect a greater number of visitors (e.g., 5) than the maximum allowable number of visitors in the rental agreement122(e.g., 3). The computer vision modules260may extract the data170that indicates the number of people in the video frames270a-270n(and additional data according to the other detection parameters). In the example shown, the extracted data170may indicate a breach of the rental agreement122.

Data that does not correspond to the detection parameters may be ignored. In one example, if pets are not part of the rental agreement122(e.g., the rental agreement does not restrict having pets at the location50), then the computer vision modules260may not search for pets in video frames270a-270n.In another example, if pets are not part of the rental agreement122, the computer vision modules260may still perform a search for pets as part of the general computer vision operations performed, but the information about the pets may not be included as part of the extracted data170(e.g., any information not defined by the rental agreement may be discarded along with the video frames270a-270n).

The extracted data170may be stored by the camera system104, but the video frames and/or the audio that corresponds to the data extracted may be deleted. The extracted data170is shown stored as the parameters and statistics in the memory132. The extracted data170(e.g., the parameters and statistics) may be used by the processor130to determine whether there has been a breach of the rental agreement122. When the computer vision modules260detect that the extracted data170matches the detection parameters of the machine readable version254, the breach of the rental agreement122may be detected. In an example, the feature set signal FEAT_SET may provide instructions to detect whether there are more than three visitors at the rental property50and the camera system104may use the detection parameters to monitor whether more than three visitors have been detected in the video frames270a-270n.In some embodiments, the computer vision modules260and/or audio analytics modules may determine when the breach of the rental agreement122has been detected. In some embodiments, the processor130may analyze the parameters and statistics170stored in the memory132after the video frames270a-270nhave been discarded to determine whether a breach of the rental agreement122has been detected.

In response to the breach of the rental agreement122, the camera system104may generate a notification. In the example shown, multiple notifications NTF_A-NTF_N may be generated. In some embodiments, the camera system104may not receive contact information about the renter or the property owner. The camera system104may provide a notification signal (e.g., NTF_A) to the server102, and then the server102may contact the property owner and/or the renter (e.g., the server102may store the contact information of the users). In some embodiments, the signal FEAT_SET may comprise the contact information of the users and the camera system104may use the communications device204to contact the property owner and/or the renter. For example, a signal (e.g., NTF_B) may be communicated to the smartphone (or other computing device)52bof the property owner and a signal (e.g., NTF_N) may be communicated to the smartphones (or other computer devices)52nof the renter. The camera system104may be configured to provide the notification to other parties as defined in the rental agreement122. The notification may indicate that there has been a breach of the terms of the rental agreement122. The notification may not comprise the video and/or audio associated with the breach of the rental agreement (e.g., the video and/or audio may no longer be available).

In some embodiments, the camera system104may provide the signal STAT to the server102. The signal STAT may comprise the extracted data170. The server102may use the extracted data170(e.g., the parameters and statistics) to determine whether there has been a breach of the rental agreement122and/or generate the notification signals NTF_A-NTF_N.

Referring toFIG.6, a diagram illustrating an example notification for a breach of a rental agreement is shown. An example scenario400is shown. The example scenario400may comprise a hand70iholding the smartphone52i.The hand70imay be a hand of the renter or a hand of the property owner. The smartphone52iis shown displaying the companion app302. The companion app302may be configured to display the notification of a breach of the terms of the rental agreement122.

The notification shown by the companion app302may be generated in response to one of the notification signals NTF_A-NTF_N. In the example shown, the notification may be displayed by the companion app302. However, the notification may be communicated by other means. In one example, the notification may be provided by a text message (e.g., SMS). In another example, the notification may be provided by a phone call (e.g., an automated voice message). In yet another example, the notification may be provided via email. The format of the notification may be varied according to the design criteria of a particular implementation.

An example notification is shown displayed on the smartphone52i.The notification may comprise a message310. The message310may indicate that a breach of the rental agreement122has been detected. Additional information about the breach of the rental agreement122may be displayed. A property address312is shown as part of the notification. The property address312may indicate the address of the rental property that corresponds to the detected breach. In an example, the property owner may rent out multiple properties and the property address312may provide clarity about which property has the detected breach of the rental property agreement122.

In the example shown, the breach to the rental agreement122may be the number of guests detected by the camera system104. The notification may provide details314-318about the breach of the rental agreement122. The details314-318may comprise the detection314, the condition316and/or a timestamp318of the breach detected based on the statistics and parameters170a-170ndetected by the processor130.

For the example of a breach detected for the maximum number of allowable guests, the detection314may indicate the number of guests detected by the camera system104. In the example shown (as shown in association withFIG.2), the number of detected guests314may be five. The corresponding condition316may be the maximum number of allowed guests according to the rental agreement122. In the example shown, the maximum number of guests316according to the rental agreement122may be three. The timestamp318may comprise the time that the breach was detected. In the example shown, the detection314of five guests may have been detected by the camera system104at three in the morning. Other information may be provided based on the extracted data170a-170nthat corresponds to the breach of the rental agreement122. For example, if a different breach of the agreement122was detected (e.g., audio level), the details314-318may comprise alternate data types (e.g., maximum sound level of the rental agreement122, the sound level detected, etc.).

A button320, a button322and a button324are shown as part of the example notification. The button320may enable the end user to access the text version252of the rental agreement122. For example, the end user may want to review the terms of the rental agreement122.

The button322may enable the end user to contact the other party in the rental agreement122. The contact other party button322may enable the property owner and the renter to communicate. For example, if the notification is provided to the property owner, then the property owner may contact the renter to inform the renter that the extra guests should leave and if they do not leave then the authorities will be contacted. In another example, if the notification is provided to the renter, then the renter may contact the property owner to inform them that the extra guests have left. The contact other party button322may be implemented to enable the parties to remedy the detected breach.

The button324may enable the users to view a data transparency log. The data transparency log may provide access to all the data stored by the servers102a-102nand/or the camera systems104a-104nabout the users. The users may access the data transparency log to ensure the accuracy of any data collected about the users. In one example, the data transparency log may record the number of people that were detected at the location50and at what time. The data transparency log may be used to help resolve a dispute between the landlord and the renter. In an example, the data transparency log may be referenced as feedback (e.g., whether the detections by the camera systems104a-104nare detecting the terms of the rental agreement122accurately) and/or as proof of a violation. In another example, the users may access the data transparency log to verify what the camera systems104a-104nhave extracted (e.g., view the extracted data170a-170n) to see data corresponding to the entries370a-370nmay be stored (e.g., the number of guests was recorded), but there were no identifiers of a particular person and no video/audio stored.

The notification may provide a privacy statement330. The privacy statement330may inform the recipient of the notification that the video (e.g., the video frames270a-270n) and/or audio (e.g., the signals AIN_A-AIN_N) is not available because the video and/or audio has never been stored. Since the video frames270a-270nand/or audio captured may be discarded after the processor130analyzes the video and/or audio, the video frames270a-270nand/or the captured audio may be unavailable for communication. The privacy statement330may further indicate that the determination of the breach of the rental agreement122may be based on the extracted statistics and parameters170a-170n.

Referring toFIG.7, a diagram illustrating an example of comparing statistics and parameters extracted from video and/or audio data to entries of a rental agreement is shown. An example comparison350of the rental agreement122and the detected parameters and statistics170are shown.

Examples of the parameters and statistics170are shown. The parameters and statistics170may comprise a timestamp352. The timestamp352may indicate the time that the video frame270a-270ncorresponding to the extracted data170was recorded. In an example, the timestamp352maybe used to generate the time of breach notification318shown in association withFIG.6. In some embodiments, some of the entries of the rental agreement122may be time sensitive (e.g., before 10 pm ten guests may be allowed, but after 10 pm only 3 guests may be allowed). The timestamp352may provide a temporal reference for when the parameters170were extracted.

The parameters170may comprise detections354a-354n.The detections354a-354nmay be the information that may be determined based on the parameters and statistics170. Generally, the detections354a-354nmay correspond to the entries of the rental agreement122. For example, in order to protect the privacy of the renters, the amount of data collected may be limited. For example, if the rental agreement122does not provide a limitation on the number of guests, then the detections354a-354nmay not comprise information about the number of guests.

The detection354amay be the number of people detected. In the example shown, the number of people detected may be five. The detection354bmay be the number of pets detected. In the example shown, no pets may be detected. If a pet was detected, further detections354a-354nmay be extracted (e.g., the type of pet, where the pet was kept on the property, etc.). The detection354cmay be the audio level detected. In the example shown, the detected audio level may be 80 dB. The detection354dmay be an item condition. In the example shown, the condition of the item (e.g., a TV) may be good (e.g., undamaged, not moved, not used, etc.). The detections354a-354nmay comprise data extracted from both the video frames270a-270nand the captured audio AIN_A-AIN_N. The detections354a-354nmade by the camera system104may be performed in response to the machine readable version254of the rental agreement122. The types of detections extracted in the parameters and statistics170may be varied according to the design criteria of a particular implementation.

The rental agreement122is shown. For illustrative purposes, the text version252of the rental agreement122is shown. The rental agreement122may comprise a property address360. In an example, the property address360may correspond to the property address312shown in association with the notification shown inFIG.6.

The address360may define the location50. The rental agreement122may comprise a time period362a-362b.The time period362a-362bmay comprise a rental start time362aand a rental end time362b.The rental time period362a-362bmay define the time that the rental agreement122is active. The rental time period362a-362bmay indicate at which times the camera system104may use the feature set defined in the machine readable version254of the rental agreement122. For example, when the current time is outside of the time period362a-362b,then the camera system104may not use the machine readable version254of the rental agreement122to extract the data from the video frames270a-270n.

The rental agreement122may further define a number of cameras364and/or a location of cameras366a-366c.The number of cameras364may indicate how many of the camera systems104a-104nare implemented at the rental property50. The location of cameras366a-366cmay define where the camera systems104a-104nare located. Defining the number of cameras364and/or the location of cameras366a-366cmay provide transparency for the renters (e.g., to inform the renters that cameras are detecting behavior and/or watching for breaches to the rental agreement122). Defining the number of cameras364and/or the location of cameras366a-366cmay further ensure that the renters know where the camera systems104a-104nare located to prevent accidental damage and/or accidentally obscuring the camera systems104a-104n.

Entries370a-370eare shown. The entries370a-370emay define the terms of the rental agreement122. The entries370a-370emay define the machine readable instructions254to be used by the camera systems104a-104n.In some embodiments, the entries370a-370emay be pre-defined types of detections that may be performed by the camera systems104a-104n.The renter and the property owner may agree on the criteria for the pre-defined types of detections. While five entries370a-370eare shown in the example350. Any number of entries370a-370nmay be defined in the rental agreement122. In some embodiments, the entries370a-370navailable and/or criteria limitations for the entries370a-370nmay be limited by the detection capabilities of the camera systems104a-104n.The number and/or types of the entries370a-370nmay be varied according to the design criteria of a particular implementation.

The entry370amay be the number of guests allowed on the rental property50. In the example shown, the number of guests allowed370amay be four. The entry370bmay be the number of visitors allowed. In the example shown, the number of guests allowed370bmay be three. For example, the number of guests allowed370amay define how many people may be allowed at any time, while the number of visitors370bmay define how many additional people may be allowed at particular times (e.g., visitors may be allowed before midnight).

The entry370cmay be whether pets are allowed. In the example shown, the pets entry370cmay be that pets are allowed. The entry370dmay be a maximum audio level. For example, the maximum audio level may be defined as described in association with U.S. patent application Ser. No. 16/858,230, filed on Apr. 24, 2020, appropriate portions of which are hereby incorporated by reference. In the example shown, the maximum audio level370dmaybe 75 dB. The entry370emay be whether pool access is allowed. In some embodiments, the property owner may not want renters using certain areas of the rental property (e.g., not allowed to use the pool, not allowed in a particular bedroom, etc.). In the example shown, the pool access entry370emay be that pool access is not allowed.

The processor130and/or the server102may compare the entries370a-370nof the rental agreement122to the detections354a-354nextracted from the video data and/or audio data by the camera system104. The comparison may determine whether there has been a breach of the terms of the rental agreement122.

In the example shown, the people detected354amay be five. Comparing to the number of guests entry370a(e.g., four), more than four guests have been detected. However, the timestamp352maybe before midnight. Since the visitor entry370bprovides for three guests before midnight, then the people detected354amay not breach the rental agreement122.

In the example shown, the pets detected354bmay be zero. Since the pets entry370cof the rental agreement122allows for pets, the pets detected354bmay not breach the rental agreement. The detected audio level354cmaybe 80 dB. Since the maximum audio level entry370dmay be 75 dB, the audio level detected354cmay breach the terms of the rental agreement. The server102and/or the camera system104may generate the signal NTF to provide a notification to the renter and/or the property owner that the maximum audio level entry370dof the rental agreement122has been breached.

Referring toFIG.8, a diagram illustrating an example interface for a rental offer is shown. An example scenario400is shown. The example scenario400may comprise a view of a web browser80. The web browser80may comprise various interface features (e.g., tabs, address bar, navigation buttons, etc.). A tab82of the web browser80is shown. The tab82may comprise a website title indicating an Owner Listing. A URL84is shown in the web browser80. In an example, the URL84may be a web address that points to the servers102a-102n.The browser80is shown displaying a web page. The web page displayed by the browser80may be the web interface126. In an example, the property owner user device52amay access the servers102a-102nusing the web browser80. The servers102a-102nmay generate the signal WEB to present the web interface126to the property owner user device52a.

The web interface126generated by the servers102a-102nshown may be a property listing form interface402. For example, the data from the web interface126stored by the servers102a-102nmay be used to provide data that the browser80may use to output the property listing form interface402. The property listing form interface402may be an example website shown to the property owner. The property listing form interface402may enable the property owner to list the property50for rent.

The property listing form interface402may provide various fields to accept input from the property owner. The fields may correspond to terms and/or conditions that may be used to create the entries370a-370nof the rental agreement122. The fields may be generated in response to the signal QUERY. For example, the servers102a-102nmay communicate with the camera systems104a-104nto determine the capabilities of the computer vision operations (e.g., determine what types of objects160a-160nand/or characteristics162may be detected). The detection engine124may convert the type of objects160a-160nand/or characteristics162that may be detected by the camera systems104a-104ninto fields that may be stored as part of the data for the web interface126. When the property owner wants to list the property50, the servers102a-102nmay generate fields for the property listing form interface402that may be specific to the capabilities of the camera systems104a-104nused by the property owner and/or the characteristics of the room/mounting location of the camera systems104a-104n.

The property listing form interface402may comprise a heading404, date fields406a-406b,a location field408, a price field410, a number of restriction fields412a-412n,a button420and/or a button422. The heading404may indicate that the property owner may list the property50for rent using the property listing form interface402. The various input fields406a-412nmay accept input from the user that may be converted to both the text version252and/or the machine readable version254of the rental agreement122. The machine readable version254may be communicated to the camera systems104a-104nas the signal FEAT_SET. The property listing form interface402shown may be a representative example and the style, layout and/or available input fields may be varied according to the design criteria of a particular implementation.

The date input fields406a-406bmay comprise a start and end time for the rental of the property50(e.g., a time range for the rental offer). For example, the date input fields406a-406bmay be used to enter the time period362a-362bof the rental agreement122shown in association withFIG.7. The location input field408may accept a location (e.g., an address) of the property50. For example, the location input field408may be used to enter the address360of the rental agreement122shown in association withFIG.7. The price input field410may accept a price for renting the location50. In the example shown, the price input field410may be on a per day basis. Other price options may be available (e.g., flat rate, per week, per month, etc.).

In some embodiments, the location input field408may also comprise input fields for the property owner to input data (e.g., technical specifications, room/mounting location characteristics and/or an identification number) for the camera systems104a-104n.In some embodiments, the property listing form interface402may be modified based on the capabilities of the camera systems104a-104nused by the property owner. As advancements are made to the computer vision detection capabilities of the camera systems104a-104n,the property owner may have more options available (e.g., more input fields412a-412nmay be accessible). In an example, when the user inputs an address into the location field408, the servers102a-102nmay generate the signal QUERY to the camera systems104a-104nat the location50to determine the capabilities of the camera systems104a-104nand then update the property listing form interface402(e.g., based on the available parameters, features and/or types of detection that may be performed by the camera systems104a-104n). For example, the signal QUERY may be used to determine the number of cameras364and/or the camera locations366a-366cof the rental agreement122shown in association withFIG.7.

The restriction input fields412a-412nmay enable the property owner to define the terms of the rental agreement122. In an example, the restriction input fields412a-412nmay be used as data for the entries370a-370nof the rental agreement122shown in association withFIG.7. In some embodiments, the number and/or types of the restriction input fields412a-412nmay be determined based on data generated by the detection engine124in response to the signal QUERY and/or the neural networks available in the detection engine124. The number, type and/or format of the input used for the restriction input fields412a-412nmay be varied according to the design criteria of a particular implementation.

In the example shown, the restriction input field412amay comprise a maximum number of guests allowed. The property owner may restrict the number of people allowed on the property50, then signal FEAT_SET may provide the camera systems104a-104nthe maximum number of people that may be detected before a breach is determined to occur. For example, the restriction input field412amay correspond to the entry370ashown in association withFIG.7. In response to the data input in the restriction input field412a,the computer vision operations may be performed by the camera systems104a-104nwhen the rental agreement122is active to detect if more than the maximum number of people have been detected.

In the example shown, the restriction input field412bmay be whether pets are allowed on the property50. In the example shown, the restriction input field412bmay be a binary choice (e.g., to allow pets or not). In some embodiments, the restriction input field412bmay enable the property owner to define how many pets are allowed and/or the type(s) of pets allowed. For example, a property owner that is allergic to cats may allow dogs in the rental agreement122but not cats. In an example, the restriction input field412bmay correspond to the entry370cshown in association withFIG.7.

In the example shown, the restriction input field412cmay comprise whether loud noises (e.g., music, movies, live performances, party guests, etc.) are allowed at the property50. In the example shown, the restriction input field412cmay comprise radio buttons for selecting particular times when loud noises may be allowed. For example, loud noises may be permitted before particular times to avoid noise complaints and/or avoid disturbing neighbors. However, any type of time input may be implemented by the rental listing web interface402. In an example, the restriction input field412cmay correspond to the maximum audio level entry370dshown in association withFIG.7. For example, the maximum audio level may not be applied before midnight based on the selection chosen for the restriction input field412c.In some embodiments, an audio maximum audio level may be defined to indicate what audio level may be considered a loud noise.

In the example shown, the restriction input field412dmay comprise whether social gatherings are permitted. For example, the renter may be permitted to allow guests (e.g., have a party) but only during particular times periods. For example, the maximum number of guests may limit how many people are allowed overnight or past a particular time, but during other times more people may be allowed on the property50. In an example, the restriction input field412dmay correspond to the number of visitors allowed entry370bshown in association withFIG.7. The social gatherings may be defined as the number of extra guests that may be allowed at particular times.

In the example shown, the restriction input field412nmay comprise a free-form text field. The restriction input field412nmay enable the property owner to enter various other terms and/or conditions for renting the property50. In some embodiments, the restriction input field412nmay be further populated in response to the signal QUERY.

The types of restriction input fields412a-412navailable may be varied according to the design criteria of a particular implementation. For example, the property owner may provide restrictions on whether alcohol is allowed on the property50. In another example, the property owner may restrict access to particular locations on the property50(e.g., bedrooms, pool access, etc.). In yet another example, the property owner may define whether fireworks are allowed on the property50, whether only men or only women are allowed on the property50, the types of events allowed on the property50, etc.

The button420may provide a cancel function (e.g., to delete the data entered in the fields406a-412nand/or leave the property listing form interface402). The button422may provide a submit function to enable the property owner to use the data entered on the property listing form interface402. For example, when the property owner interacts with the list rental button422(e.g., clicks, taps, etc.) the user device52amay communicate the signal RENTOFR to the servers102a-102n.The processors110a-110nof the servers102a-102nmay generate the listings120based on the data provided in the input fields406a-412n.

Referring toFIG.9, a diagram illustrating an example interface for a rental request is shown. An example scenario450is shown. The example scenario450may comprise a view of the web browser80. Similar to the example scenario400shown in association withFIG.8, the tab82and the URL84are shown. In the example scenario450, the tab82may comprise a website title indicating Renter Details. The web page displayed by the browser80may be the web interface126. In an example, the prospective renter user device52bmay access the servers102a-102nusing the web browser80. The servers102a-102nmay generate the signal WEB to present the web interface126to the prospective renter user device52b.

The web interface126generated by the servers102a-102nshown may be a rental request form interface452. For example, the data from the web interface126stored by the servers102a-102nmay be used to provide data that the browser80may use to output the rental request form interface452. The rental request form interface452may be an example website shown to the prospective renter. The rental request form interface452may enable the prospective renter to search the listings120to find a suitable rental.

The rental request form interface452may provide various fields to accept input from the prospective renter. The fields may correspond to terms and/or conditions that may be used to create the rental agreement122. In some embodiments, the fields may be generated in response to the listings120based on the data provided by the property owner in the signal RENTOFR. For example, the rental request form interface452may be available after the property owner has provided the signal RENTOFR to communicate information about the property50to be stored as the listings120. When the prospective renter wants to search for a rental property, the servers102a-102nmay generate fields to enable a search of the property listing122. The servers102a-102nmay search the listings120based on the information received based on the input of the rental request form interface452.

The rental request form interface452may comprise a heading454, date fields456a-456b,a location field458, a price field460, a number of feature fields462a-462n,a button470and/or a button472. The heading454may indicate that the prospective renter may search the listings120based on the wants/needs of the renter using the rental request form interface452. In one example, the various input fields456a-462nmay accept input from the user that may be converted to both the text version252and/or the machine readable version254of the rental agreement122. In another example, the various input fields456a-462nmay accept input from the user that may be used by the servers102a-102nto compare against the listings to return a closest match (or a number of the listings120that may be generally close to the input by the prospective renter). The rental request form interface452shown may be a representative example and the style, layout and/or available input fields may be varied according to the design criteria of a particular implementation.

The rental request form interface452may enable the prospective renter to enter criteria for a property to rent that may be listed in the listings120. The web interface may provide the various fields456a-462nto accept input from the user device52b.The fields456a-462nmay accept input from the user that may be converted to search criteria that may be used to filter the available listings120. The content available on the rental request form interface452for finding a rental property may be similar to the input fields406a-412nused for the property listing form interface402shown in association withFIG.8to enable a comparison to the data of the listings120.

The date input fields456a-456bmay comprise a desired start and end time for a desired rental (e.g., a time range for the rental request). For example, the date input fields456a-456bmay be used to compare to the listings120against the data entered in the date input fields406a-406b.The location input field458may accept a location (e.g., an address, a general region, a city, etc.) for a desired rental. For example, the location input field458may be used to compare to the listings120against the data entered in the location input field408. The price input field460may accept a desired price point for a desired rental. For example, the price input field460may be used to compare to the listings120against the data entered in the price input field410.

The feature input fields462a-462nmay enable the prospective renter to define the terms of the rental agreement122. In an example, the feature input fields462a-462nmay be used as data for the entries370a-370nof the rental agreement122shown in association withFIG.7. In another example, the feature input fields462a-462nmay be used to search the listings120by comparing against the similar data entered for the restriction input fields412a-412n.For example, the prospective renter may want to search for a rental property in a particular location that allows loud music late in order to throw a party. By comparing against the data entered with the restriction fields412a-412n,the server computers102a-102nmay filter out available properties that are outside the desired region and/or available properties that do not allow loud music.

The prospective renters use the rental request form web interface452to search for venues at particular locations, for particular dates, in a particular price range that may suit the desires of the renter. For example, if the prospective renter is planning a wedding for two hundred guests, by providing a guest number to the appropriate one of the feature input fields462a-462n,the servers102a-102nmay filter out rental listings120that do not meet the criteria of the prospective renter. The number, type and/or format of the input used for the feature input fields462a-462nmay be varied according to the design criteria of a particular implementation.

In the example shown, the feature input field462amay comprise a maximum number of guests requested. The requested number of guests may be compared against the listings120(e.g., the maximum number of guests defined from the restriction field412a). In the example shown, the restriction input field462bmay be whether a property that allows pets is requested. The request for a pet may be compared against the listings120(e.g., the number and/or types of pets defined from the restriction field412b). Similar to the restriction field412b,the pet feature input462bmay provide options for number and/or types of pets. In some embodiments, the pet feature input462bmay request properties that have never had pets (e.g., for severe allergies).

In the example shown, the feature input field462cmay comprise whether social gatherings are requested. In the example shown, a binary choice is shown. In some embodiments, the feature input field462cmay provide further input details such as number of extra guests, the time of day that the extra guests may be present, etc. The requested number of guests and/or times for the extra guests may be compared against the listings120(e.g., against the number and/or times for social gatherings defined in the restriction input412d). In the example shown, the feature input field462dmay comprise whether loud noises (e.g., music, movies, live performances, party guests, etc.) are requested. In the example shown, the feature input field462dmay comprise radio buttons for selecting particular times when loud noises may be requested. Similar to the noise restriction input field412c,the loud noise feature input462dmay provide further options such as maximum noise level requested. The requested loud music/movies may be compared against the listings120(e.g., against the times and/or maximum noise level defined in the restriction input412c).

In the example shown, the feature input field462nmay comprise a free-form text field. The feature input field462nmay enable the prospective renter to enter various other features requested, terms and/or conditions for renting a property from the listings120. In some embodiments, the prospective renter may not define the terms and/or conditions of the rental agreement122. In some embodiments, the prospective renter may be provided an additional web interface to negotiate one or more terms of the rental agreement122.

The button470may provide a cancel function (e.g., to delete the data entered in the fields456a-462nand/or to leave the rental request interface452). The button472may provide a find a match button to enable the prospective renter to submit the data entered in the rental request interface452to the servers102a-102n.For example, when the prospective renter interacts with the find a match button472(e.g., clicks, taps, etc.) the user device52bmay communicate the signal RENTREQ to the servers102a-102n.The processors110a-110nof the servers102a-102nmay search the listings120based on the data provided in the input fields456a-462n.In some embodiments, the servers102a-102nmay provide a filtered list of suitable listings in response to the signal RENTREQ. In some embodiments, the servers102a-102nmay provide the rental agreement122in response to the signal RENTREQ (e.g., the property owner may list a property and automatically accept offers that fit the criteria of the listing120). The processors110a-110nmay be configured to compare the data from the signal RENTOFR (e.g., stored in the listings120) to the data from the signal RENTOFR to find data that matches and/or data that provides some degree of commonality. In some embodiments, the signal RENTOFR may comprise negotiation terms sent by the prospective renter (e.g., the prospective renter may find a listing120that is close to the desired property but may not allow enough people, the renter may negotiate by asking if more people may be allowed).

Referring toFIG.10, a diagram illustrating an example interface for a rental agreement is shown. An example scenario500is shown. The example scenario500may comprise a view of the web browser80. Similar to the example scenario400shown in association withFIG.8, the tab82and the URL84are shown. In the example scenario500, the tab82may comprise a website title indicating Rental Agreement. The web page displayed by the browser80may be the web interface126. In an example, the user devices52a-52bmay access the servers102a-102nusing the web browser80. The servers102a-102nmay generate the signal RAGREE to present the web interface126to the user devices52a-52b.

The web interface126generated by the servers102a-102nshown may be a rental agreement form interface502. For example, the data from the web interface126, the listings120and/or the rental agreement122stored by the servers102a-102nmay be used to provide data that the browser80may use to output the rental agreement form interface502. The rental agreement form interface502may be an example website shown to both the property owner and the prospective renter. In one example, the rental agreement form interface502may be presented to the prospective renter in response to the signal RENTREQ. The servers102a-102nmay present the rental agreement form interface502when the rental request from the prospective renter from the signal RENTREQ matches (or has commonalities with) the one or more of the listings120. The rental agreement form interface502may enable the property owner and/or the prospective renter to agree to the rental agreement122. The rental agreement form interface502shown may be a representative example and the style, layout and/or available input fields may be varied according to the design criteria of a particular implementation.

In the example shown, the rental agreement form interface502may be an example of a matched listing provided in response to the rental request RENTREQ. An image504is shown. The image504may be an image of the property50that corresponds to the matched listing. In an example, the image504may be submitted by the property owner with the signal RENTOFR. In some embodiments, the image504may be an image captured by one or more of the camera systems104a-104n.In some embodiments, the image504may be an image selected by the property owner. The rental agreement form interface502may comprise a property description506.

The property description506may comprise a list price508. The list price508may be determined based on the data from the listings120(e.g., as defined by the price input410). The property description506may comprise statistics510. The statistics510may provide descriptive data about the location50(e.g., size, number of rooms, number of beds, amenities, etc.). The property description506may comprise a button514and/or additional details516. The additional details516may provide further information about the location50(e.g., directions, nearby restaurants, how old the property is, etc.).

The button514may be a messaging button. In the example shown, the messaging button514may enable the prospective renter to directly contact the property owner. Similarly, the property owner may use the messaging button514to directly contact the prospective renter. The messaging button514may provide an interface for direct contact (e.g., email, text message, video link, etc.). The messaging button514may enable the parties to negotiate terms of the rental agreement122.

In an example, one of the listings120may have a restriction of a maximum of five guests. The renter may use the messaging button514to send a request to the property owner to ask if the rental agreement122could be modified to allow seven guests. The property owner may have the option to agree to the modification, deny the modification and/or provide a counter offer. In an example, the property owner may counter by allowing up to six guests. In another example, the property owner may deny the modification and provide a reason (e.g., allowing more guests may be against the law). The messaging button514may enable negotiation and/or modification of the restrictions provided by the property owner in the signal RENTOFR.

The rental agreement form interface502may have a contract agreement section520. In the example shown, the contract agreement section520may indicate that a match was found between the requests of the prospective renter and the listings120. In the example, one match is shown. However, when the prospective renter provides the signal RENTREQ, the servers102a-102nmay return multiple properties from the listings120that generally fit the criteria of the signal RENTREQ. The contract agreement section520may comprise the text version252of the rental agreement122, a checkbox522and a button524. The checkbox522may be an agreement checkbox. The button522may be a confirmation button.

The rental agreement122displayed on the contract agreement section520may provide the text-based (e.g., human readable) version252listing of the terms and conditions370a-370nof the rental agreement122. In the example shown, the text version252of the rental agreement122may indicate the entry370afor the maximum number of guests allowed on the rental property50, the entry370bfor the times that loud noises are permitted, the entry370cfor the number and/or type of pets allowed, etc. Other terms and conditions370a-370nmay be listed (e.g., how loud the noise may be, the types of activities permitted, the type of noise permitted, etc.).

In some embodiments, the rental agreement122may comprise the terms and conditions370a-370nas input by the property owner using the property listing form interface402. For example, the servers102a-102nmay convert the data input in the restriction input fields412a-412nreceived in the signal RENTOFR into the entries370a-370nof the rental agreement122. In the example shown, the rental agreement web interface502may be provided to the prospective renter. The renter may use the messaging button514to negotiate the terms370a-370n.Similarly, the rental agreement interface502may be provided to the property owner to enable the property owner to agree to the terms370a-370nof the rental agreement122.

In the example shown, the terms370a-370nmay comprise restrictions that may be imposed on the renter if the rental agreement122is agreed to. In some embodiments, the rental agreement122may indicate various requirements and/or guidelines for the property owner. In an example, the terms370a-370nmay outline the types of amenities that the property owner may make available. For example, the rental agreement122may indicate how many beds are to be made available. In another example, the rental agreement122may indicate whether the property owner is charging for usage of utilities. In yet another example, the rental agreement122may indicate whether various conveniences may be provided (e.g., towels, silverware, etc.). The property owner may use the messaging button514to negotiate the amenities provided. For example, if the prospective renter asks for additional guests to be allowed, the property owner may counter-offer with a higher price.

The checkbox522may be configured to enable the users to agree to the rental agreement122. Checking the checkbox522may enable the users to confirm and agree to the rental agreement122. The checkbox522may be used as an indication that each user has read the text version252, understands and agrees to the terms of the rental agreement122. When the checkbox522is checked by the user, the rent it button524may become accessible. Interacting with the rent it button524may submit a confirmation of the rental agreement122to the servers122a-122n.In an example, when the property owner uses the user device52ato check the checkbox522and clicks on the rent it button524, the user device52amay communicate the confirmation signal CONFO to the servers102a-102n.The signal CONFO may provide an indication that the property owner has agreed to the rental agreement122. In another example, when the renter uses the user device52bto check the checkbox522and clicks on the rent it button524, the user device52bmay communicate the confirmation signal CONFR to the servers102a-102n.The signal CONFR may provide an indication that the renter has agreed to the rental agreement122.

When the servers102a-102nreceive both the confirmation signal CONFO and the confirmation signal CONFR, the rental agreement122may be considered agreed to. In response to receiving the confirmation signal CONFO and the confirmation signal CONFR, the memory112a-112nmay store the rental agreement122. The rental agreement122may be parsed by the detection engine124. The detection engine124may determine the feature set based on the entries370a-370nof the rental agreement122. The detection engine124may generate the machine readable version254of the rental agreement122.

The machine readable version254may be generated based on data about the camera systems104a-104nat the location50in the rental agreement122determined from the signal QUERY. The machine readable version254of the rental agreement122may be communicated to the camera systems104a-104nas the signal FEAT_SET. The camera systems104a-104nmay generate the detection parameters of the rental agreement122based on the machine readable version254. The camera systems104a-104nmay perform the computer vision and/or audio analysis of the location50when the rental time range362a-362bis active.

In some embodiments, the restrictions provided by the rental offer interface402, the terms and/or conditions that may be negotiated, and/or the entries370a-370nthat may be available for the rental agreement122may be limited. For example, the entries370a-370nthat may be generated by the servers102a-102nmay generally correspond to the types of detection that the camera systems104a-104nmay perform. In some embodiments, the entries370a-370nmay be pre-defined parameters. For example, an overly complicated system of negotiating the terms370a-370nmay be not desired by end users. For example, the entries370a-370nmay be entered using the restriction input fields412a-412m.Additional terms and conditions may be entered in the restriction free-form input field412nbut the free-form data may not be used as the basis for generating the feature set for the camera systems104a-104n(e.g., other portions of the rental agreement122that may not be used by the camera systems104a-104nmay be generated by the free-form data).

In some embodiments, the detection engine124may convert the entries370a-370nto the feature set used for the computer vision by the camera systems104a-104n.The entries370a-370nmay be used as a basis for comparison with the output from the object/person/pet detection (and recognition) performed by the camera systems104a-104nusing the neural network(s). In one example, the detection parameters used by the camera systems104a-104nmay comprise a single network capable of recognizing various objects, people and/or pets. In another example, the detection parameters used by the camera systems104a-104nmay comprise multiple neural networks running in parallel. In a scenario that there are multiple neural networks, some of the neural networks might not be loaded if not applicable to the entries370a-370n.For example, a pet recognition neural network may not be loaded if the entries370a-370ndo not provide a policy regarding pets.

The detection parameters may comprise the criteria for determining a breach of the rental agreement122. In an example, if one rental agreement122provides a limitation of five guests and another rental agreement122provides a limitation of six guests, the neural networks used for detecting people may be the same in each scenario. The criteria used on the output of the neural networks (e.g., the extracted data170a-170n) may be different. For example, the neural network may detect six people and the output (e.g., the extracted data170a-107nthat indicates six people were detected) may be compared to the detection parameters that indicate a criteria of five people, which may indicate a breach of the rental agreement122. While the type of detection (e.g., people detection and counting) may be the same, the criteria used to determine a breach of the rental agreement122may be different. The entries370a-370ngenerated by agreeing to the rental agreement122using the rental agreement interface502may be used by the detection engine124to determine the feature set to provide to the camera systems104a-104n.The feature set may comprise the neural networks to detect the objects and/or the criteria for comparing against the extracted data170a-170n.

Referring toFIG.11, a method (or process)550is shown. The method550may generate a rental agreement from user input. The method550generally comprises a step (or state)552, a step (or state)554, a step (or state)556, a step (or state)558, a step (or state)560, a step (or state)562, a step (or state)564, a step (or state)566, a step (or state)568, a decision step (or state)570, a step (or state)572, a step (or state)574, a step (or state)576and a step (or state)578.

The step552may start the method550. In the step554, the servers102a-102nmay enable the property listing form interface402to be generated for listing the location50for rent. In an example, data for the property listing form interface402may be stored in the web interface data126. The servers102a-102nmay communicate the data to the user device52avia the signal WEB. The browser80of the user device52amay interpret the data to generate the property listing form interface402. Next, in the step556, the servers102a-102nmay receive the rental offer information. For example, the property owner may use the property listing form interface402to input data into the fields406a-412n.The property owner may submit the data to the servers102a-102nas the signal RENTOFR. In the step558, the processors110a-110nmay receive the information in the signal RENTOFR and generate listing information. The listing information may be stored in the memory112a-112nas the listings120. Next, the method550may move to the step560.

In the step560, the servers102a-102nmay enable the rental request interface452to be generated for viewing the stored rental listings120. In an example, data for the rental request interface452may be stored in the web interface data126. The servers102a-102nmay communicate the data to the user device52bvia the signal WEB. The browser80of the user device52bmay interpret the data to generate the rental request interface452. Next, in the step562, the servers102a-102nmay receive the rental request information. For example, the renter may use the rental request interface452to input data into the fields456a-462n.The renter may submit the data to the servers102a-102nas the signal RENTREQ. Next, the method550may move to the step564.

In the step564, the servers102a-102nmay match the data from the signal RENTREQ to the stored listings120. In one example, the servers102a-102nmay receive the signal RENTREQ, compare the rental request information to the listings120, and find an exact match. In another example, the servers102a-102nmay perform the comparison and return a number of the listings120that may generally correspond to the data in the signal RENTREQ (e.g., based on proximity to the location of the property and the request, based on the price range, based on the requested features and requested restrictions, etc.). The prospective renter may receive one match and/or select a desired choice from a subset of the listings120selected by the servers102a-102nin response to the signal RENTREQ. Next, the method550may move to the step566.

In the step566, the servers102a-102nmay communicate the signal RAGREE to the user devices52a-52b(e.g., to both the renter and the property owner). In an example, the users may agree to the rental agreement122. Next, in the step568, the servers102a-102nmay enable communication between the users (e.g., the renter and the property owner). In one example, the servers102a-102nmay enable the users to communicate to negotiate the entries370a-370nof the rental agreement122. In another example, the servers102a-102nmay not allow the renter to negotiate the terms370a-370n(e.g., the property owner may indicate that the terms are non-negotiable). Generally, the rental agreement interface502may enable the users to negotiate and/or agree to the terms of the rental agreement122. Next, the method550may move to the decision step570.

In the decision step570, the servers102a-102nmay determine whether the users have agreed to the rental agreement122. For example, the property owner may submit an acceptance using the rental agreement interface502and the user device52amay communicate the signal CONFO. Similarly, the renter may submit an acceptance using the rental agreement interface502and the user device52bmay communicate the signal CONFR. The rental agreement122may be agreed to when the servers102a-102nreceive both the signal CONFO and the signal CONFR. If the rental agreement has not been agreed to, the method550may move to the step578. For example, the rental agreement122may not be formed. If the rental agreement122is agreed to, the method550may move to the step572.

In the step572, the processors110a-110nmay parse the terms and conditions of the rental agreement122submitted and/or agreed to in the signal RENTREQ and the signal RENTOFR to generate and store the rental agreement122. Next, in the step574, the detection engine124may generate the machine readable version254of the rental agreement122. The machine readable version254may be generated based on the entries370a-370nof the rental agreement122. The machine readable version254may be the feature set for the camera systems104a-104n.In the step576, the servers102a-102nmay communicate the feature set for the conditions generated by the detection engine124to the camera systems104a-104nof the rental property50. In an example, the feature set may be communicated as the signal FEAT_SET. Next, the method550may move to the step578. The step578may end the method550.

Referring toFIG.12, a method (or process)600is shown. The method600may load a feature set to the camera system when the rental agreement is active. The method600generally comprises a step (or state)602, a step (or state)604, a decision step (or state)606, a step (or state)608, a step (or state)610, a step (or state)612, a step (or state)614, a decision step (or state)616, a decision step (or state)618, and a step (or state)620.

The step602may start the method600. In the step604, the camera systems104a-104nmay receive the feature set from the servers102a-102n.In an example, the camera systems104a-104nmay receive the signal FEAT_SET comprising the machine readable version254of the rental agreement122. Next, the method600may move to the decision step606.

In the decision step606, the processor130may determine whether the current time corresponds to the rental agreement122. In an example, the processor130may compare the current time to rental time range362a-362bto determine whether the rental agreement122is currently in force. If the current time does not correspond to the rental time range362a-362b,then the method600may move to the step608. In the step608, the camera systems104a-104nmay perform default camera functionality. For example, when the rental agreement122is not in force, the camera systems104a-104nmay operate according to various parameters to perform general functionality (e.g., general surveillance, video recording, video streaming, video encoding, etc.). Next, the method600may return to the decision step606. In the decision step606, if the current time does correspond to the rental time range362a-362n,then the method600may move to the step610.

In the step610, the processor130may read the feature set of the machine readable version254of the rental agreement122. Next, in the step612, the processor130may convert the feature set to detection parameters. In an example, the dedicated hardware modules240a-240nmay generate and/or store the directed acyclic graph, and/or configure the various hardware modules240a-240nto efficiently perform the computer vision operations specific to the machine readable version254of the rental agreement122. In the step614, the camera systems104a-104nmay perform the computer vision operations on the video frames270a-270nbased on the detection parameters. Next, the method600may move to the decision step616.

In the decision step616, the processor130may determine whether the rental agreement122has ended. For example, the processor130may compare to the current time to the rental end time value362b.If the rental agreement122has not ended, then the method600may return to the step614. If the rental agreement has ended, then the method600may move to the decision step618.

In the decision step618, the camera systems104a-104nmay determine whether a new rental agreement is available. For example, a new rental agreement122may be communicated by the servers102a-102nand/or the camera systems104a-104nmay store multiple rental agreements that may be active at different times. If no new rental agreement122is available, then the method600may move to the step620. In the step620, the camera systems104a-104nmay perform the default camera functionality. Next, the method600may return to the decision step618. For example, the camera systems104a-104nmay operate according to the default camera functionality until a new rental agreement122becomes available. In the decision step618, if a new rental agreement122is available, then the method600may return to the step604to receive the new feature set (or return to the decision step606if the new rental agreement122is already stored by the camera system104a-104n).

Referring toFIG.13, a method (or process)650is shown. The method650may perform the computer vision analysis of the video frames to detect a breach of the rental agreement. The method650generally comprises a step (or state)652, a step (or state)654, a step (or state)656, a step (or state)658, a step (or state)660, a step (or state)662, a step (or state)664, a step (or state)666, a decision step (or state)668, a step (or state)670, and a step (or state)672.

The step652may start the method650. In the step654, the processor130may load the detection parameters based on the feature set in the machine readable version254of the rental agreement122. For example, the processor130may configure the dedicated hardware modules240a-240nbased on the signal FEAT_SET. Next, in the step656, the capture devices202a-202nmay capture pixel data of the rental property50. For example, the image sensors230a-230nmay generate the signals PIXELD_A-PIXELD_N in response to the light received by the lenses200a-200n.In the step658, the processor130may generate the video frames270a-270nin response to the pixel data PIXELD_A-PIXELD_N. Next, the method650may move to the step660.

In the step660, the processor130may perform the computer vision operations on the video frames270a-270n.For example, the computer vision modules260may perform the computer vision operations on the video frames270a-270nusing the dedicated hardware modules240a-240n.Next, in the step662, the computer vision modules260may generate the extracted data170about the characteristics162of the objects160a-160ndetected in the video frames270a-270n.Next, in the step664, after all the extracted data170(e.g., video and audio data) has been extracted from one of the video frames270a-270n,the video and audio corresponding to the video frame may be discarded. For example, the video frames270a-270nmay be analyzed for data extraction and then, after the data extraction has been completed, sent to the discard function262to discard the data. Discarding the video frames270a-270nand the audio data corresponding to the video frames270a-270nafter the data has been extracted may preserve the privacy of the people70a-70n,while still enabling the processor130to have sufficient data to detect a breach of the rental agreement122. In the step666, the processor130may compare the extracted data170to the detection parameters of the machine readable version254of the rental agreement122. Next, the method650may move to the decision step668.

In the decision step668, the processor130may determine whether the rental agreement122has been breached. For example, the processor130may compare the extracted data170(e.g., the detection354a-354dshown in association withFIG.7) to the detection parameters derived from the entries370a-370n.If the processor130determines that the rental agreement122has not been breached, then the method650may return to the step656. If the processor130determines that the rental agreement has been breached, then the method650may move to the step670.

In the step670, the processor130may generate text from the extracted data170to describe the detected breach of the rental agreement122. In one example, if the breach corresponds to detecting music above the maximum audio level, the processor130may generate the message310(e.g., “a breach has been detected, please lower the volume”), with the detection314(e.g., “75 dB audio detected”), the condition316(e.g., “the maximum audio level is 70 dB”) and the timestamp318(e.g., “time of detection 3 AM”). Next, in the step672, the communication device204may communicate the notification (e.g., the signal NTF) to the users. For example, the notification signal NTF may be communicated to the property owner user device52a,the renter user device52band/or the user devices52a-52nof other parties of interest to the rental agreement122. Next, the method650may return to the step656.

Referring toFIG.14, a method (or process)700is shown. The method700may generate a web-interface based on available features of the camera system. The method700generally comprises a step (or state)702, a step (or state)704, a step (or state)706, a step (or state)708, a step (or state)710, a step (or state)712, a step (or state)714, a step (or state)716, a decision step (or state)718, and a step (or state)720.

The step702may start the method700. In the step704, the servers102a-102nmay receive information about the camera systems104a-104n.For example, the servers102a-102nmay communicate the signal QUERY to the camera systems104a-104nat the location50owned by the property owner. The signal QUERY may request specifications about the capabilities of the camera systems104a-104n.The camera systems104a-104nmay communicate the signal QUERY in return to provide the requested specifications and/or capabilities. Next, in the step706, the detection engine124may determine the available conditions (e.g., available parameters) for a feature set based on the information about the specifications/capabilities of the camera systems104a-104n.For example, the detection engine124may determine what the camera systems104a-104nare capable of detecting (e.g., one camera system may be capable of only detecting people but not pets, another camera may be configured to count people but not capable of detecting various characteristics about the people detected, etc.). Next, the method700may move to the step708.

In the step708, the processors110a-110nmay parse the available conditions. The processors110a-110nmay parse the available conditions based on the type of objects and/or characteristics that may be detected by the camera systems104a-104n(e.g., convert the limitations and/or features of the camera systems104a-104ninto conditions that may be used as the entries370a-370nof the rental agreement122). Next, in the step710, the processors110a-110nmay generate the pre-defined restriction input fields412a-412nthat correspond to the conditions available based on the capabilities/features of the camera systems104a-104n.Next, the method700may move to the step712.

In the step712, the processors110a-110nmay generate the web-interface126with the pre-defined restriction input fields412a-412nfor the rental offer. For example, the servers102a-102nmay communicate the signal WEB to the property owner user device52ato provide the rental offer interface402. Next, in the step714, the servers102a-102nmay receive the rental offer from the user device52a(e.g., the signal RENTOFR). In the step716, the servers102a-102nmay store the rental offer as one of the listings120and provide the listings120to prospective renters. Next, the method700may move to the decision step718.

In the decision step718, the servers102a-102nmay determine whether the camera systems104a-104nat the location50have changed. In one example, the servers102a-102nmay periodically query the camera systems104a-104nto determine whether the capabilities/features have changed. In another example, when a new one of the camera systems104a-104nis installed at the location50, the camera systems104a-104nmay update the capabilities/features of the camera systems104a-104nat the location50. If one or more of the camera systems104a-104nhave changed, then the method700may return to the step704. If one or more of the camera systems104a-104nhave not changed, then the method700may move to the step720. The step720may end the method700.

Referring toFIG.15, a method (or process)750is shown. The method750may communicate pre-defined sets of neural networks to the camera systems based on the position of the camera systems and/or the terms of the rental agreement. The method750generally comprises a step (or state)752, a step (or state)754, a step (or state)756, a step (or state)758, a decision step (or state)760, a step (or state)762, a step (or state)764, and a step (or state)766.

The step752may start the method750. In the step754, the detection engine124my parse the rental agreement122. Next, in the step756, the detection engine124may query the characteristics of the camera systems104a-104n.In an example, the detection engine124may communicate the signal QUERY to the camera systems104a-104nthat correspond to the rental agreement122. The camera systems104a-104nmay return the signal QUERY to provide the characteristics of the camera systems104a-104n.In the step758, the detection engine124may determine the objects to detect from the rental agreement122(e.g., detecting pets, people, types of items, etc.). Next, the method750may move to the decision step760.

In the decision step760, the detection engine124may determine whether there are more neural networks to load for the feature set for the camera systems104a-104n.The neural networks to be loaded may be determined based on the entries370a-370nof the rental agreement122. More than one neural network may be loaded (e.g., one neural network for detecting people, one neural network for detecting pets, one neural network for detecting inanimate objects, etc.). If there are more neural networks to load, then the method750may move to the step762. In the step762, the detection engine124may compare the objects to detect from the rental agreement122to the camera characteristics determined from the signal QUERY. Next, in the step764, the detection engine124may communicate an appropriate one of the neural networks for the objects of the rental agreement122and the room characteristics of the camera systems104a-104n.For example, if the rental agreement122comprises entries370a-370nfor detecting people and the room characteristics for one of the camera systems104a-104nis that the camera is mounted from above, a neural network that comprises feature set libraries for detecting people from above may be selected. In another example, if the rental agreement122comprises entries370a-370nfor detecting people and the room characteristics for one of the camera systems104a-104nis that the camera is mounted at a level that captures a side-view of people (e.g., as shown in association withFIG.2), a neural network that comprises feature set libraries for detecting people from body height level may be selected. Next, the method750may return to the decision step760.

In the decision step760, if there are no more neural networks to load, then the method750may move to the step766. In an example, there may be no additional neural networks to load when the camera systems104a-104nhave received neural networks that correspond to detecting the criteria of each of the entries370a-370nin the rental agreement122. The step766may end the method750.

The characteristics of the camera systems104a-104nmay comprise technical specifications about the camera systems104a-104n(e.g., resolution, shutter speed, focal length, zoom settings, frame rate, etc.). The characteristics of the camera systems104a-104nmay further comprise information about the environment that the camera systems104a-104nare located (e.g., room characteristics). For example, the characteristics of the camera systems104a-104nmay comprise information about where the camera systems104a-104nare mounted (e.g., how high up, how large the room is, the lighting in the room, expected distance from objects, etc.). For example, the detection parameters for detecting objects from different distances and/or in different lighting environments may be different.

The objects to detect from the rental agreement122may be determined based on the entries370a-370n.For example, the detection parameters may be different for detecting different types of objects. In one example, the entries370a-370nmay comprise multiple entries for detecting people (e.g., number of people, number of visitors, activities of the visitors, etc.). To detect people, one neural network may be communicated to the camera systems104a-104n.In another example, the entries370a-370nmay comprise entries for detecting pets. To detect pets, another neural network may be communicated to the camera systems104a-104n.In some embodiments, different types of pets may each be detected based on separate neural networks (e.g., one neural network for detecting cats, another neural network for detecting dogs, another neural network for detecting lizards, etc.). In yet another example, the entries370a-370nmay comprise entries for detecting items and/or the conditions of items. Additional neural networks may be communicated to the camera systems104a-104nfor detecting items (e.g., detecting TVs, detecting types of damage to TVs, detecting a pool, detecting the usage of a pool, etc.).

The types of objects detected maybe combined with the location characteristics (e.g., room characteristics) of the camera systems104a-104n.For example, one neural network may be implemented for detecting people from above and another neural network may be implemented for detecting people from ground level. In some embodiments, the detection engine124may not communicate neural networks that do not apply to the entries370a-370nof the rental agreement122. For example, if none of the entries370a-370nof the rental agreement122applies to pets (e.g., all pets are allowed), then the detection engine124may not communicate the neural networks that correspond to detecting animals to the camera systems104a-104n.The number and/or types of neural networks that may be communicated as the feature set may be varied according to the design criteria of a particular implementation.

Since the camera systems104a-104nmay not store the video frames270a-270nor communicate the video frames270a-270n,the video frames270a-270nmay not be sent back to the servers102a-102nfor ongoing training of neural networks. The memory112a-112nmay store pre-defined sets of data (e.g., neural networks) that may be downloaded into the camera systems104a-104n.The neural networks stored by the servers102a-102nmay be trained using training data acquired from sources other than the camera systems104a-104nwhile the rental agreements122are active. The neural networks stored by the servers102a-102nselected for the camera systems104a-104nmay be selected based on the position of the camera systems104a-104nin the room and/or the size of the room at the location50. For example, some of the neural networks may be trained for small rooms and other neural networks may be trained for large rooms In another example, some of the neural networks may be trained for various heights of the camera systems104a-104nin the rooms at the location50and/or the lighting in the room at the location50. For example, during installation and/or a set up time of the camera systems104a-104n,the camera systems104a-104nmay detect the physical parameters of the room and communicate the physical parameters (e.g., room characteristics) back to the servers102a-102n.The servers102a-102nmay select the neural network parameters optimized for a room according to the room characteristics.