System and method for gesture recognition

Internet of Things (IoT) systems and related methods are disclosed. A method comprises determining whether a first condition holds, the first condition being a condition of the IoT system, performing a function of the IoT system in response to a trigger, wherein the trigger is a determination that the first condition holds, recognizing a gesture based on image data received from an image sensor, and reconfiguring the IoT system for a future performance of the function based on the user feedback data.

BACKGROUND

Aspects of this disclosure relate generally to telecommunications, and more particularly to interactions between user equipment and other devices in a shared wireless access environment.

The Internet is a global system of interconnected computers and computer networks that use a standard Internet protocol suite (e.g., the Transmission Control Protocol (TCP) and Internet Protocol (IP)) to communicate with each other. The Internet of Things (IoT), sometimes referred to as the Internet of Everything (IoE), is based on the idea that everyday objects, not just computers and computer networks, can be readable, recognizable, locatable, addressable, and controllable via an IoT communications network (e.g., an ad-hoc system or the Internet).

A number of market trends are driving development of IoT devices. For example, increasing energy costs are driving governments' strategic investments in smart grids and support for future consumption, such as for electric vehicles and public charging stations. Increasing health care costs and aging populations are driving development for remote/connected health care and fitness services, wherein doctors can, for example, remotely monitor patients' health while people can track the progress of fitness routines. A technological revolution in the home is driving development for new “smart” services, wherein smart homes and buildings can have centralized control over virtually any device or system in the home or office, from appliances to plug-in electric vehicle (PEV) security systems. Buildings are getting smarter and more convenient as a means to reduce operational costs for enterprise facilities. In the field of asset tracking, enterprises, hospitals, factories, and other large organizations can accurately track the locations of high-value equipment, patients, vehicles, and so on.

As such, in the near future, increasing development in IoT systems will lead to numerous IoT devices surrounding a user at home, in vehicles, at work, and many other locations. Accordingly, a need exists for an IoT management device that leverages large amounts of disorganized data in useful ways.

SUMMARY

The following summary is an overview provided solely to aid in the description of various aspects of the disclosure and is provided solely for illustration of the aspects and not limitation thereof.

In one example, a method performed in an Internet of Things (IoT) system is disclosed. The method may include, for example, determining whether a first condition holds, the first condition being a condition of the IoT system, performing a function of the IoT system in response to a trigger, wherein the trigger is a determination that the first condition holds, recognizing a gesture based on image data received from an image sensor, interpreting the recognized gesture to generate user feedback data, and reconfiguring the IoT system for a future performance of the function based on the user feedback data.

In another example, an IoT system is disclosed. The IoT system may include, for example, a memory system configured to store data and/or instructions, and a processing system coupled to the memory system. The processing system may be configured to determine whether a first condition holds, the first condition being a condition of the IoT system, perform a function of the IoT system in response to a trigger, wherein the trigger is a determination that the first condition holds, recognize a gesture based on image data received from an image sensor, interpret the recognized gesture to generate user feedback data, and reconfigure the IoT system for a future performance of the function based on the user feedback data.

In yet another example, another IoT system is disclosed. The IoT system may include, for example, means for determining whether a first condition holds, the first condition being a condition of the IoT system, means for performing a function of the IoT system in response to a trigger, wherein the trigger is a determination that the first condition holds, means for recognizing a gesture based on image data received from an image sensor, means for interpreting the recognized gesture to generate user feedback data, and means for reconfiguring the IoT system for a future performance of the function based on the user feedback data.

In yet another example, a non-transitory computer-readable medium comprising code is disclosed. When executed by a processor, the code may cause the processor to perform operations in an Internet of Things (IoT) system. The non-transitory computer-readable medium may comprise, for example, code for determining whether a first condition holds, the first condition being a condition of the IoT system, code for performing a function of the IoT system in response to a trigger, wherein the trigger is a determination that the first condition holds, code for recognizing a gesture based on image data received from an image sensor, code for interpreting the recognized gesture to generate user feedback data, and code for reconfiguring the IoT system for a future performance of the function based on the user feedback data.

DETAILED DESCRIPTION

An IoT system within, for example, a smart home, may be pre-programmed to perform one or more IoT functions using one or more IoT devices. Each performance of an IoT function may be triggered, as per the programming, by one or more contextual conditions identified by the IoT system. The contextual conditions may be identified using data downloaded from a network and/or data sensed directly using an IoT sensor associated with the IoT system. The IoT system may misbehave by, for example, performing a function that a user of the IoT system disapproves of, performing the function at a wrong time, performing the function in response to the wrong set of contextual conditions, etc.

It may be useful to incorporate into the IoT system some algorithm for training and/or reconfiguring the IoT system. The IoT system may learn, based on feedback from a user, whether the user approves or disapproves of the particular function performed by the IoT system. Accordingly, it may be useful to provide many different mechanisms for providing feedback.

FIGS. 1-3relate to IoT systems generally. In accordance with aspects of the disclosure, the IoT system inFIGS. 1-3may be equipped with a gesture recognition algorithm that analyzes received image data and translates the image data into user feedback data.FIGS. 4A-4Bdepict scenarios in which the gesture recognition algorithm may be utilized for training purposes. The user feedback data generated using the gesture recognition algorithm may subsequently be used to reconfigure the IoT system so as to increase user satisfaction (for example, by maximizing the amount of positive user feedback). The reconfiguring may include confidence adjustments, monitoring of new contextual conditions, adjustment of trigger conditions, or any other suitable reconfiguring, as described in greater detail in the description relating toFIGS. 5-9.

FIG. 1illustrates a high-level system architecture of an IoT system100in accordance with an aspect of the disclosure. The IoT system100contains a plurality of IoT devices, which include an image sensor110, a solar panel111, an HVAC unit112(where HVAC stands for “heating, ventilation, and air conditioning”), a lamp113, a thermostat114, a refrigerator116, and a washer and dryer118.

As used herein, the term “Internet of Things device” (or “IoT device”) may refer to any object (e.g., an appliance, a sensor, etc.) that has an addressable interface (e.g., an Internet protocol (IP) address, a Bluetooth identifier (ID), a near-field communication (NFC) ID, etc.) and can transmit information to one or more other IoT devices over a wired or wireless connection. An IoT system may comprise any combination of IoT devices.

Referring toFIG. 1, IoT devices110-118are configured to communicate with an access network (e.g., an access point120) over a physical communications interface or layer. As shown inFIG. 1, the communication interface may be an air interface122and/or a direct wired connection124. The air interface122may comply with a wireless Internet protocol (IP), such as IEEE 802.11. AlthoughFIG. 1illustrates IoT devices110-118communicating over the air interface122and washer and dryer118communicating over the direct wired connection124, it will be understood that each IoT device may communicate over a wired or wireless connection, or both.

The Internet130includes a number of routing agents and processing agents (not shown inFIG. 1for the sake of convenience). The Internet130is a global system of interconnected computers and computer networks that uses a standard Internet protocol suite (e.g., the Transmission Control Protocol (TCP) and IP) to communicate among disparate devices/networks. TCP/IP provides end-to-end connectivity specifying how data should be formatted, addressed, transmitted, routed and received at the destination.

InFIG. 1, a computer140, such as a desktop or personal computer (PC), is shown as connecting to the Internet130directly (e.g., over an Ethernet connection or Wi-Fi or 802.11-based network). The computer140may have a wired connection to the Internet130, such as a direct connection to a modem or router, which, in an example, can correspond to the access point120itself (e.g., for a Wi-Fi router with both wired and wireless connectivity). Alternatively, rather than being connected to the access point120and the Internet130over a wired connection, the computer140may be connected to the access point120over air interface122or another wireless interface, and access the Internet130over the air interface122. Although illustrated as a desktop computer, computer140may be a laptop computer, a tablet computer, a PDA, a smart phone, or the like. The computer140may be an IoT device and/or contain functionality to manage an IoT network/group, such as the network/group of IoT devices110-118.

The access point120may be connected to the Internet130via, for example, an optical communication system, such as FiOS, a cable modem, a digital subscriber line (DSL) modem, or the like. The access point120may communicate with IoT devices110-118and the Internet130using the standard Internet protocols (e.g., TCP/IP).

Referring toFIG. 1, an IoT server150is shown as connected to the Internet130. The IoT server150can be implemented as a plurality of structurally separate servers, or alternately may correspond to a single server. In an aspect, the IoT server150is optional, and the group of IoT devices110-118may be a peer-to-peer (P2P) network. In such a case, the IoT devices110-118can communicate with each other directly over the air interface122and/or the direct wired connection124. Alternatively, or additionally, some or all of IoT devices110-118may be configured with a communication interface independent of air interface122and direct wired connection124. For example, if the air interface122corresponds to a Wi-Fi interface, one or more of the IoT devices110-118may have Bluetooth or NFC interfaces for communicating directly with each other or other Bluetooth or NFC-enabled devices. In a peer-to-peer network, service discovery schemes can multicast the presence of nodes, their capabilities, and group membership. The peer-to-peer devices can establish associations and subsequent interactions based on this information.

The IoT system100may optionally include a supervisor device160. In one aspect of the disclosure, the supervisor device160may generally observe, monitor, control, or otherwise manage the various other components in the IoT system100. For example, the supervisor device160may communicate with an access network (e.g., access point120) over air interface122and/or a direct wired connection124to monitor or manage attributes, activities, or other states associated with the various IoT devices110-118in the IoT system100. The supervisor device160may have a wired or wireless connection to the Internet130and optionally to the IoT server150. The supervisor device160may obtain information from the Internet130and/or the IoT server150that can be used to further monitor or manage attributes, activities, or other states associated with the various IoT devices110-118. The supervisor device160may be a standalone device (as shown), but it will be understood that the supervisor device may include or be included in one of the IoT devices110-118, the access point120, the computer140, or any other electronic device (smartphone, tablet, etc.). For example, the supervisor device160may be a panel on an IoT air conditioner, or an application on a smartphone or tablet. The supervisor device160may be a physical device or a software application running on a physical device. The supervisor device160may include a user interface that can output information relating to the monitored attributes, activities, or other states associated with the IoT devices110-118and receive input information to control or otherwise manage the attributes, activities, or other states associated therewith. Accordingly, the supervisor device160may generally include various components and support various wired and wireless communication interfaces to observe, monitor, control, or otherwise manage the various components in the IoT system100.

In addition to the various IoT devices110-118, the IoT system100shown inFIG. 1may further include one or more passive IoT devices (in contrast to the active IoT devices110-118) that can be coupled to or otherwise made part of the IoT system100. In general, the passive IoT devices may include barcoded devices, Bluetooth devices, radio frequency (RF) devices, RFID tagged devices, infrared (IR) devices, NFC tagged devices, or any other suitable device that can provide its identifier and attributes to another device when queried over a short range interface. Active IoT devices may detect, store, communicate, act on, and/or the like, changes in attributes of passive IoT devices.

For example, a first passive IoT device may include a coffee cup and a second passive IoT device container of orange juice. Each may have an RFID tag or barcode. A cabinet IoT device may have an appropriate scanner or reader that can read the RFID tag or barcode to detect when the coffee cup has been added or removed from the cabinet IoT device. The refrigerator IoT device116may have an appropriate scanner or reader that can read the RFID tag or barcode to detect when the container of orange juice has been added or removed from the refrigerator IoT device116. In response to the cabinet IoT device detecting the removal of the coffee cup and the refrigerator IoT device116detecting the removal of the container of orange juice, the supervisor device160may receive one or more signals that relate to the activities detected at the cabinet IoT device and the refrigerator IoT device116. The supervisor device160may then infer that a user is drinking orange juice from the coffee cup and/or likes to drink orange juice from a coffee cup.

Although the foregoing describes the passive IoT devices as having some form of RFID tag or barcode communication interface, the passive IoT devices may include one or more devices or other physical objects that do not have such communication capabilities. For example, certain IoT devices may have appropriate scanner or reader mechanisms that can detect shapes, sizes, colors, and/or other observable features associated with the passive IoT devices to identify the passive IoT devices. In this manner, any suitable physical object may communicate its identity and attributes and become part of the IoT system100and be observed, monitored, controlled, or otherwise managed with the supervisor device160. Further, passive IoT devices may be coupled to or otherwise made part of the IoT system100inFIG. 1and observed, monitored, controlled, or otherwise managed in a substantially similar manner.

FIG. 2Aillustrates a high-level example of an IoT device200A in accordance with aspects of the disclosure. While external appearances and/or internal components can differ significantly among IoT devices, many IoT devices will have some sort of user interface, which may comprise a display and a means for user input. IoT devices without a user interface may be communicated with remotely over a wired or wireless network, such as air interface122inFIG. 1.

As shown inFIG. 2A, in an example configuration for the IoT device200A, an external casing of IoT device200A may be configured with a display226, a power button222, and two control buttons224A and224B, among other components, as is known in the art. The display226may be a touchscreen display, in which case the control buttons224A and224B may not be necessary. While not shown explicitly as part of IoT device200A, the IoT device200A may include one or more external antennas and/or one or more integrated antennas that are built into the external casing, including but not limited to Wi-Fi antennas, cellular antennas, satellite position system (SPS) antennas (e.g., global positioning system (GPS) antennas), and so on.

While internal components of IoT devices, such as IoT device200A, can be embodied with different hardware configurations, a basic high-level configuration for internal hardware components is shown as platform202inFIG. 2A. The platform202can receive and execute software applications, data and/or commands transmitted over a network interface, such as air interface122inFIG. 1and/or a wired interface. The platform202can also independently execute locally stored applications. The platform202can include one or more transceivers206configured for wired and/or wireless communication (e.g., a Wi-Fi transceiver, a Bluetooth transceiver, a cellular transceiver, a satellite transceiver, a GPS or SPS receiver, etc.) operably coupled to a processing system208including one or more processing devices, such as a microcontroller, microprocessor, application specific integrated circuit, digital signal processor (DSP), programmable logic circuit, or other data processing device. The processing system208can execute application programming instructions within a memory system212of the IoT device200A. The memory system212can include one or more of read-only memory (ROM), random-access memory (RAM), electrically erasable programmable ROM (EEPROM), flash cards, or any memory common to computer platforms. One or more input/output (I/O) interfaces214can be configured to allow the processing system208to communicate with and control from various I/O devices such as the display226, power button222, control buttons224A and224B as illustrated, and any other devices, such as sensors, actuators, relays, valves, switches, and the like associated with the IoT device200A.

Accordingly, an aspect of the disclosure can include an IoT device (e.g., IoT device200A) including the ability to perform the functions described herein. As will be appreciated by those skilled in the art, the various logic elements can be embodied in discrete elements, software modules executed on a processor (e.g., the processing system208) or any combination of software and hardware to achieve the functionality disclosed herein. For example, the transceiver206, the processing system208, the memory system212, and I/O interface214may all be used cooperatively to load, store and execute the various functions disclosed herein and thus the logic to perform these functions may be distributed over various elements. Alternatively, the functionality could be incorporated into one discrete component. Therefore, the features of the IoT device200A inFIG. 2Aare to be considered merely illustrative and the disclosure is not limited to the illustrated features or arrangement.

IoT device200A may also include an always-on vision sensor210. The always-on vision sensor210may be capable of detecting objects and/or gestures to allow the IoT device200A to more intelligently interact with a user and/or the environment, for example to allow IoT device200A to receive positive and negative user feedback from a user. In one example, the always-on vision sensor210may be capable of counting the number of human bodies that are within a room and, for example, instruct an HVAC system to turn on or off in response thereto. In another example, the always-on vision sensor210can detect user gestures as feedback as will be discussed further below. In one implementation, the vision sensor210includes a sensor element array (e.g., image sensor, camera, etc.), dedicated CV computation hardware, and a dedicated microprocessor. In various implementations, the dedicated CV computation hardware within always-on vision sensor210can perform CV computations in either the digital or analog domain, and examples of such CV computation hardware are known in the art. The CV computation hardware is dedicated CV computation hardware in the sense that it is hardware designed to have little or no functionality other than to compute CV features. In some implementations, the dedicated CV computation hardware may use combinations, sums, or averages of signals associated with blocks of sensor elements or pixels. Therefore, optionally, the always-on vision sensor210may also include two dimensional integration hardware. In some implementations, the integral image (which may be an integral image of only a portion or sample window of the image sampled by the sensor element array) generated by the two dimensional integration hardware can be stored in a hardware scanning window array, which may also optionally be included in the always-on vision sensor210. In one example, the hardware scanning window array includes a random-access memory (RAM) array or other form of analog or digital memory for storing the integral image. Returning to the dedicated CV computation hardware, for example, in a local binary pattern (LBP) implementation of dedicated CV computation hardware, CV computation hardware can include hardware that receives signal values corresponding to image signals—or combinations, sums, or averages of image signals (generated, for example, using an integral image)—and generates a digital LBP label based on the image signals. Although the description above referenced dedicated CV computation hardware as separate from the dedicated microprocessor, it is understood that in some implementations, dedicated CV computation hardware may be implemented in hardware within the dedicated microprocessor. Generating the CV features, such as the LBP labels discussed above, in dedicated hardware can reduce the power of the always-on vision sensor210compared to computing the CV features in a processor, for example a general purpose processor such as an application processor or even the dedicated microprocessor. However, the always-on vision sensor210may still include a dedicated microprocessor coupled to the CV computation hardware. The dedicated microprocessor receives the hardware-computed CV features from the CV computation hardware and can perform higher-level computer vision operations such as object-class detection (of which face detection can be regarded as a specific case), in which the task is to find the locations and sizes of all objects in an image that belong to a given class, as well as other computer vision operations. Furthermore, the dedicated microprocessor can provide control signals to various subcomponents of the always-on vision sensor210, including those mentioned above and those not mentioned for brevity. In some implementations, to perform the object-class detection or other computer vision operations, the dedicated microprocessor may use a cascade classifier algorithm to perform object-class detection, for example face detection, or gesture recognition. However, further power savings are possible by implementing the cascade classifier in hardware, to further reduce the computational burden on the microprocessor. The optional cascade classifier hardware includes a hardware implementation of a cascade classifier. The cascade classifier can be configured to detect the presence of a reference object (e.g., a human face, a particular human's face, a human body or upper body to determine a number of humans in a room, an animal face or body) or a reference gesture (which can be considered a sequence of reference objects in time), within the sample window stored in the scanning window array based on CV features computed by the dedicated CV computation hardware. In some implementations, the cascade classifier is trained using machine learning techniques on a data set of images including examples of the reference object the cascade classifier will be trained for and examples of non-objects, for example images of faces and non-faces. Details of the operation of always-on vision sensor210, including a hardware cascade classifier, can be found in U.S. Pat. Pub. Nos. 2016/0094814 and 2016/0110603, which are both incorporated herein in their entireties. In an alternative implementation, IoT device200A may not include always-on vision sensor210, but may instead be in communication with always-on vision sensor210, as is shown with reference to image sensor110inFIG. 1(where image sensor110is connected to a plurality of devices, which could include, for example, IoT device200A).

FIG. 2Billustrates a high-level example of a passive IoT device200B in accordance with aspects of the disclosure. In general, the passive IoT device200B shown inFIG. 2Bmay include various components that are the same and/or substantially similar to the IoT device200A shown inFIG. 2A, which was described in greater detail above. In particular, IoT device200B may include always-on vision sensor210. As such, for brevity and ease of description, various details relating to certain components in the passive IoT device200B shown inFIG. 2Bmay be omitted herein to the extent that the same or similar details have already been provided above in relation to the IoT device200A illustrated inFIG. 2A.

The passive IoT device200B shown inFIG. 2Bmay generally differ from the IoT device200A shown inFIG. 2Ain that the passive IoT device200B may not have a processing system208, memory system212, or certain other components. Instead, in one aspect, the passive IoT device200B may only include an I/O interface214or other suitable mechanism that allows the passive IoT device200B to be observed, monitored, controlled, managed, or otherwise known within a controlled IoT network. For example, in one aspect, the I/O interface214associated with the passive IoT device200B may include a barcode, Bluetooth interface, radio frequency (RF) interface, RFID tag, IR interface, NFC interface, or any other suitable I/O interface that can provide an identifier and attributes associated with the passive IoT device200B to another device when queried over a short range interface (e.g., an active IoT device, such as IoT device200A, that can detect, store, communicate, act on, or otherwise process information relating to the attributes associated with the passive IoT device200B). In particular implementations of IoT devices200A and200B, objects or gestures recognized by always-on vision sensor210may be communicated to an appropriate device such as supervisor device160ofFIG. 1or other appropriate component of IoT system for training the IoT system, or, in the case of IoT device200A with onboard processing capabilities, such feedback may be used for training locally without communicating with supervisor device. In alternative implementations, always-on vision sensor210may be integrated within supervisor device160, where supervisor device160receives the user feedback, while IoT device200A and200B are trained or receive configurations based on training from the supervisor device160.

Although the foregoing describes the passive IoT device200B as having some form of RF, barcode, or other I/O interface214, the passive IoT device200B may comprise a device or other physical object that does not have such an I/O interface214. For example, certain IoT devices may have appropriate scanner or reader mechanisms that can detect shapes, sizes, colors, and/or other observable features associated with the passive IoT device200B. The passive IoT device200B may be identified based on the detected observable features. In this manner, any suitable physical object may communicate its identity and attributes and be observed, monitored, controlled, or otherwise managed within a controlled IoT network. Returning to an earlier example, a coffee mug may have a recognizable shape, size, color, etc., and a cabinet IoT device may determine that the recognized coffee mug has been taken from or placed into the cabinet.

FIG. 3illustrates examples of UEs (i.e., client devices) in accordance with an aspect of the disclosure. Referring toFIG. 3, UE300A is illustrated as a calling telephone and UE300B is illustrated as a touchscreen device (e.g., a smart phone, a tablet computer, etc.). As shown inFIG. 3, an external casing of UE300A is configured with an antenna305A, display310A, at least one button315A (e.g., a PTT button, a power button, a volume control button, etc.) and a keypad330A among other components, as is known in the art. Also, an external casing of UE300B is configured with a touchscreen display305B, peripheral buttons310B,315B,320B and325B (e.g., a power control button, a volume or vibrate control button, an airplane mode toggle button, etc.), at least one front-panel button330B (e.g., a Home button, etc.), among other components, as is known in the art. While not shown explicitly as part of UE300B, the UE300B can include one or more external antennas and/or one or more integrated antennas that are built into the external casing of UE300B, including but not limited to Wi-Fi antennas, cellular antennas, satellite position system (SPS) antennas (e.g., global positioning system (GPS) antennas), and so on.

While internal components of UEs such as the UEs300A and300B can be embodied with different hardware configurations, a basic high-level UE configuration for internal hardware components is shown as platform302inFIG. 3. The platform302can receive and execute software applications, data and/or commands transmitted from the Internet130and/or other remote servers and networks (e.g., IoT server150, web URLs, etc.). The platform302can include a transceiver306operably coupled to a processing system308, including an application specific integrated circuit (ASIC), microprocessor, logic circuit, other data processing device, or any combination thereof. The processing system308or other processor executes the application programming interface (API)310layer that interfaces with any resident programs in the memory system312of the UEs300A and300B. The memory system312can be comprised of read-only or random-access memory (RAM and ROM), EEPROM, flash cards, or any memory common to computer platforms. The platform302also can include a local database314that can store applications not actively used in the memory system312, as well as other data. The local database314is typically a flash memory cell, but can be any secondary storage device as known in the art, such as magnetic media, EEPROM, optical media, tape, soft or hard disk, or the like.

Accordingly, an aspect of the disclosure can include a user equipment (UE) including the ability to perform the functions described herein, for example serve as a supervisor device similar to supervisor device160ofFIG. 1. Furthermore, UEs300A and/or300B may include an integrated always-on vision sensor210capable of receiving user feedback, including gestures from users for training an IoT system. As will be appreciated by those skilled in the art, the various logic elements can be embodied in discrete elements, software modules executed on a processor or any combination of software and hardware to achieve the functionality disclosed herein. For example, the processing system308, memory system312, API310and local database314may all be used cooperatively to load, store and execute the various functions disclosed herein and thus the logic to perform these functions may be distributed over various elements. Alternatively, the functionality could be incorporated into one discrete component. Therefore, the features of UEs300A and300B inFIG. 3are to be considered merely illustrative and the disclosure is not limited to the illustrated features or arrangement.

The wireless communication to and from the UEs300A and/or300B can be based on different technologies, such as CDMA, W-CDMA, time division multiple access (TDMA), frequency division multiple access (FDMA), Orthogonal Frequency Division Multiplexing (OFDM), GSM, or other protocols that may be used in a wireless communications network or a data communications network. As discussed in the foregoing and known in the art, voice transmission and/or data can be transmitted to the UEs using a variety of networks and configurations. Accordingly, the illustrations provided herein are not intended to limit the aspects of the disclosure and are merely to aid in the description of aspects of aspects of the disclosure.

FIGS. 4A-4Bgenerally illustrate different scenarios in which various IoT devices interact with a supervisor device, one another, and/or a user of the IoT system.

InFIG. 4A, a user401is depicted in a basement of a home equipped with an IoT system analogous to the IoT system100depicted inFIG. 1. An image sensor410(analogous to the image sensor110depicted inFIG. 1and/or always-on vision sensor210ofFIG. 2A) is provided in the basement, as is a lamp413(analogous to the lamp113depicted inFIG. 1). The image sensor410and the lamp413may be configured to interact with a supervisor device460(analogous to the supervisor device160depicted inFIG. 1). Although a direct wired connection is shown (analogous to the direct wired connected124depicted inFIG. 1), it will be understood that the image sensor410and the lamp413may communicate with the supervisor device460in any suitable manner set forth in the present disclosure. Additionally or alternatively, image sensor410and supervisory device460may be integrated into a single device. Also, in some implementations, supervisory device460may be integrated into the device which it controls, in this example, lamp413. It can be appreciated from the previous discussions of IoT devices200A,200B and UEs300A,300B that each of them may serve as a supervisory device460.

In the scenario depicted inFIG. 4A, the user401has descended the stairs into a dark basement. The image sensor410determines that the user401has entered the basement, and activates the lamp413. As noted above, the image sensor410may interact with the lamp413in any suitable manner, for example, directly using peer-to-peer communications or in tandem with the supervisor device460. The user401takes notice of the fact that the lamp413has been activated, and signals her approval with, for example, a hand gesture, a facial gesture, or any other suitable movement. The image sensor410and/or supervisor device460may be configured to recognize and interpret the gesture.

Given the ability to recognize and interpret gestures, the IoT system depicted inFIG. 4Acan obtain real-time feedback directly from the user401. In particular, when the user401gestures her approval, the IoT system may recognize the gesture and interpret the gesture as being a sign of approval.

Based on the positive feedback, the IoT system may be configured to conclude that the user401may prefer that the lamp413be activated when the user401enters the basement. As a result, the IoT system may in the future activate the lamp413when the user401enters the basement, or increase the likelihood that the activation of the lamp413will be triggered when the user401enters the basement.

InFIG. 4B, the user401is depicted in a sunroom having one or more large windows. In the scenario depicted inFIG. 4B, the user401has entered the sunroom, which is brightly illuminated by natural sunlight. The image sensor410determines that the user401has entered the sunroom, and activates the lamp413.

The user401concludes that it is wasteful to activate the lamp413in the sunroom on a bright, sunny day, and may disapprove of the activation of the lamp413. The user401signals her disapproval with a gesture, and the image sensor410and/or supervisor device460recognizes the gesture and interprets it as being a sign of disapproval. As a result, the IoT system may in the future avoid activation of the lamp413when the user401enters the sunroom, or decrease the likelihood that activation of the lamp413will be triggered when the user401enters the sunroom.

In one example scenario, the IoT system may be trained to activate the lamp413in the sunroom only if it is cloudy or dark. In some implementations, the IoT system may be able to determine whether there is sufficient sunlight in the sunroom using the solar panel411(analogous to the solar panel111depicted inFIG. 1). In other implementations, the IoT system may rely on a light sensor, an internal clock and/or a weather indicator received via the Internet130. Alternatively or additionally, the IoT system may recognize the correlation between the gesture of disapproval and the sunroom being already bright in the example ofFIG. 4B. For example, when the room is dark, and the lamp is activated, the user401has previously signaled her approval of the activation of the lamp413. As such, the IoT system recognized a correlation between approval for activation of the lamp413when the sunroom is dark, but disapproval for activation of the lamp413when the sunroom is bright. Hence, the IoT system may in the future avoid activation of the lamp413when the user enters the sunroom on a sunny day or when the ambient light in the room is sufficient and/or decrease the likelihood of doing so, but keep the likelihood of activating the lamp413the same when the sunroom is dark. Hence, the IoT system may take the inputs of all sensor data available to it, including, for example, an ambient light sensor.

Accordingly, the user401may train the IoT system by providing a mixture of positive user feedback (when it is sunny) and negative user feedback (when it is cloudy or dark). The training may be accomplished by recognizing the gestures of the user401and interpreting the recognized gesture. For example, identifying the recognized gesture as indicating positive user feedback, negative user feedback, or any other suitable type of feedback. The training may be promoted by taking note of contextual conditions (for example, sun exposure levels detected by the solar panel411), and reconfiguring the trigger for performing the lamp413activation in response to the contextual conditions.

FIG. 5generally illustrates a method500A for configuring and/or reconfiguring an IoT system to perform a function in response to contextual data. The method500A may be performed by, for example, one or more components of an IoT system analogous to the IoT system100depicted inFIG. 1. As will be discussed in greater detail below, the method500A may be reconfigured any number of times in accordance with aspects of the disclosure. Examples of methods that may emerge from the reconfiguration of the method500A include the method500B depicted inFIG. 7and the method500C depicted inFIG. 9.

At510, the method500A identifies one or more triggering conditions, determines whether the one or more triggering conditions holds, and proceeds to550when triggering occurs. For example, if a first condition C1holds (i.e., C1=TRUE), then the method500A proceeds to550(‘yes’ at510). If C1does not hold (i.e., C1=FALSE), then the method500A returns to510(‘no’ at510). Accordingly, C1=TRUE constitutes a triggering condition for proceeding to550.

As used herein, a condition may refer to any contextual condition sensed by the IoT system, whether through direct detection of the condition using an appropriate sensor within the IoT system, or data received from the network or the Internet130. The condition may be a characteristic of an environment in which the IoT system is situated (for example, C1=TRUE if the user401is present in the basement), or may be a detected change in condition (for example, an event or transition) that occurs within the environment (for example, the user401enters the basement, causing a presence indicator to transition from FALSE to TRUE).

For illustrative purposes, C1is described herein as being either TRUE or FALSE, although other arrangements are possible. For example, C1may be a temperature sensed by the thermostat114(for example, a value within a range of temperatures from 0°-200°), or a wattage generated by the solar panel411(for example, a value within a range of wattages from 0-1,000 W). In some implementations, the value of C1may be mapped to a TRUE/FALSE determination using a threshold. For example, C1may be set such that C1=TRUE if sensed temperature is greater than a threshold temperature of 75°, and C1=FALSE if sensed temperature is not greater than the threshold temperature.

At550, the method500A performs a function of the IoT system in response to the triggering at510(i.e., the determination that C1holds). To return to a previous example, C1may be a user401presence in the basement depicted inFIG. 4A. The IoT system may determine that the user401depicted inFIG. 4Ais present in the basement (i.e., C1=TRUE), thus triggering the IoT system to activate the lamp413in the basement.

At560, the method500A receives image data from an image sensor, for example, the image sensor410depicted inFIGS. 4A-4B. Image sensor410can be included as a stand-alone image sensor as depicted inFIGS. 4A-4B, but may additionally or alternatively be included in an IoT device similar to IoT devices200A,200B and/or200C (analogous to image sensor210), or in a mobile device capable to sending data to the IoT system such as UEs300A and/or300B. The image data may include, for example, a sequence of image frames. In some implementations, the image data may be received and processed by the image sensor410and/or at a supervisor device analogous to the supervisor device460depicted inFIGS. 4A-4B. In some implementations, the image data may be captured during a finite time window having a predetermined duration, for example, a time window that commences when the triggering at510occurs and terminates after a predetermined amount of time (or number of frames). The duration of the time window may reflect an amount of time within which user feedback is expected to be received, for example, twenty seconds.

At570, the method500A recognizes a gesture in the image data received at560. The IoT system may only be capable of recognizing a single gesture, or alternatively, any number of different gestures. The recognizing may be performed using any suitable image processing technique. In some implementations, the gesture may be present in a single image frame, for example, a smiling face, frowning face, a thumbs up, a thumbs down, or other gesture capable of being captured in a single image frame. In other implementations, the gesture may be a multi-component gesture having a sequence of different gesture components in different image frames, for example, a user nodding his or her head, moving his or her head from left to right, a hand gesture, or other multi-component gesture having a sequence of different gesture components in different image frames.

At580, the method500A interprets the recognized gesture to generate user feedback data. As noted above, the IoT system may be capable of recognizing any number of different gestures. The IoT system may be configured to distinguish between the any number of different gestures, and may be further configured to translate a particular recognized gesture into a particular type of feedback. In the example ofFIG. 4A, the IoT system recognizes a first gesture and interprets the recognized first gesture as signifying approval (i.e., positive user feedback). The result is positive user feedback data that may be generated and/or recorded in a memory system. In the example ofFIG. 4B, the IoT system recognizes a second gesture distinct from the first gesture and interprets the recognized second gesture as signifying disapproval (i.e., negative user feedback). The result is negative user feedback data that may be generated and/or recorded in a memory system. As will be discussed in greater detail below, the user feedback data (positive, negative, or otherwise) may be used to reconfigure the IoT system in accordance with aspects of the disclosure. To that end, the IoT system may store the generated user feedback data for later use.

At590, the method500A reconfigures the IoT system for future performances of the function at550based on the user feedback data, which in turn is generated using recognized gestures from one or more users. As such, the method500A includes reconfiguring the IoT system for a future performance of the function based on the recognized gesture of570.

In some implementations, the reconfiguring may comprise increasing a confidence level associated with the currently-used triggering condition(s). For example, in the example ofFIG. 4A, the user401enters the basement, the lamp413is activated, and the user401signals his or her approval to the image sensor410. The IoT system may track a correlation between instances of the performing of the function (activate lamp413if C1=TRUE, C1being user401presence) and instances of receiving of positive user feedback. The IoT system may track the correlation using a first correlation score S1. The value of S1may be between −1 and +1, wherein −1 represents a maximum negative correlation and +1 represents a maximum positive correlation. A value of S1=0 may indicate that the correlation between C1and approval of the performed function is unknown, ambiguous, and/or non-existent. When the user401signals his or her approval to the image sensor410, the IoT system may increase S1relative to its previous value (i.e., tending toward +1).

In other implementations, the reconfiguring may comprise decreasing a confidence level associated with the currently-used triggering condition(s). For example, in the example ofFIG. 4B, the user401enters the naturally-lit sunroom, the lamp413is activated, and the user401signals his or her disapproval to the image sensor410. When the user401signals his or her disapproval to the image sensor410, the IoT system may decrease S1relative to its previous value (i.e., tending toward −1).

As a result of one or more reconfigurings of the method500A, a new method500B may emerge, as will be discussed in greater detail below with reference toFIG. 7. In addition, or as an alternative, to increasing and/or decreasing S1, the reconfiguring may comprise commencement of monitoring of additional contextual conditions beyond the first condition C1, as will be discussed in greater detail below with reference toFIG. 6.

FIG. 6generally illustrates an example implementation of a reconfiguration algorithm. The reconfiguration algorithm depicted inFIG. 6may be an example implementation of the reconfiguring algorithm depicted at590ofFIG. 5.

At610, the reconfiguring algorithm, having recognized and interpreted a gesture (at570and580), determines whether the recognized gesture is positive user feedback or negative user feedback. (For simplicity of illustration, it will be assumed that the IoT system is capable of recognizing two gestures, the two gestures comprising a positive feedback gesture signifying approval and a negative feedback gesture signifying disapproval.) If the recognized gesture is positive (‘(+)’ at610), then the reconfiguring algorithm proceeds to612. If the recognized gesture is negative (‘(−)’ at610), then the reconfiguring algorithm proceeds to614.

At612, a first correlation score S1is increased. At614, by contrast, the first correlation score S1is decreased. As noted above, S1may reflect a confidence level associated with future performances of the function at550in response to the trigger that is currently used by the IoT system (i.e., C1=TRUE) to determine whether the function should be performed at550.

The increasing and decreasing may be at any suitable increment or decrement and may be based on any suitable technique. In some implementations, S1may increase or decrease at predetermined intervals, for example, ±0.1. In other implementations, a statistical correlation may be calculated. The statistical correlation may be based on any number of instances of feedback, for example, ten of the most recent instances, twenty of the most recent instances, or all previously recorded instances.

In some implementations, a correlation score may refer to an explicit value recorded in memory. In other implementations, the correlation score may be an implied value that is not directly retrievable. For example, the correlation score may be one or more factors related to a machine learning algorithm. In some implementations, the correlation score is based on weighted feedback. For example, more recent feedback may be weighted more heavily (e.g., assigned a higher weight coefficient) than more remote feedback (which may be assigned a lower weight coefficient).

As noted above, the recognition performed at610may be as simple as distinguishing between two (or fewer) recognized gestures. However, it will be understood that more complex determinations are possible in accordance with aspects of the disclosure, for example, recognition of three or more gestures. For example, a third recognized gesture may be interpreted as strong approval (resulting in a relatively greater increase of S1) and a fourth recognized gesture may be interpreted as strong disapproval (resulting in a relatively greater decrease of S1). In some implementations, if the user401does not make a recognized gesture, this may be interpreted as mild positive feedback signifying tacit approval (resulting in a correspondingly mild increase of S1).

At650, the reconfiguring algorithm may determine whether the first correlation score S1exceeds a confidence threshold. If S1exceeds the confidence threshold (‘yes’ at650), then the reconfiguring depicted at590inFIG. 5may be complete, and the method500A may return to510, as depicted inFIG. 5. If S1does not exceed the confidence threshold (‘no’ at650), then the reconfiguring algorithm may proceed to660before returning to510.

At660, the reconfiguring algorithm commences monitoring of a second condition C2. As will be understood fromFIG. 6, the IoT system initially assumes that knowledge of the first condition C1is sufficient for determining whether to perform the function at550. However, after a certain amount of negative feedback is received from the user401, the IoT system may lose confidence in C1=TRUE as the sole triggering condition used at510. This loss of confidence may be reflected in any suitable manner, for example, as a first correlation score S1that is decreased below the confidence threshold.

As noted above, the number of monitored conditions is limited only by the storage and processing powers of the IoT system. Accordingly, although the remainder of the disclosure makes reference to just two conditions, C1and C2, it will be understood that any number of conditions C3, C4, . . . CNmay be monitored.

In some implementations, the IoT system may have limited storage and/or processing powers. As a result, the IoT system may not commence monitoring of the second condition C2until it is necessary to improve the performance of the IoT system (i.e., when S1falls below the confidence threshold). This is the approach depicted inFIG. 6, as noted above. However, in an IoT system with excess storage and/or processing power, every monitorable condition may be monitored at all times and the results may be recorded and/or stored indefinitely, even if they have no known relevance at the time that they are recorded and/or stored to any function of the IoT system. Accordingly, instead of commencing to monitor the second condition C2(as at660), the IoT system may simply gather the already-recorded data relating to C2, C3, C4, . . . CN, or any combination thereof. The IoT system may then generate respective correlation scores S2, S3, S4, . . . SNbased on the recorded data and identify the best condition for determining whether the performance of the function at550should be triggered. Additionally or alternatively, the IoT system may also generate any number of combination correlation scores based on the recorded data and identify the best combination of conditions under which the function of550should be performed. The combination correlation scores may be based on couplets of different conditions (for example, C1and C2), but it will be understood that the combination correlation scores may also be based on triplets (C1, C2, and C3), quadruplets (C1, C2, C3, and C4), etc.

FIG. 7generally illustrates a method500B for configuring and/or reconfiguring an IoT system to perform a function in response to contextual data. The method500B may be performed by, for example, one or more components of an IoT system analogous to the IoT system100depicted inFIG. 1. The method500B may be analogous in some respects to the method500A, and may, in some implementations, constitute a result of the reconfiguring performed at590inFIG. 5and/orFIG. 6. For example, like the method500A, the method500B may include the determining at510, the performing of the function at550, the receiving of the image data at560, the recognizing at570, the interpreting at580, and the reconfiguring at590. However, unlike the method500A, the method500B does not proceed directly to the performing of the function at550in response to a determination that C1=TRUE (‘yes’ at510). Instead, the method500B proceeds to720before proceeding to the performing of the function at550.

At720, the method500B determines whether a second condition C2holds. The determining at720may further include recording and/or storage in a memory system of data associated with C2(for example, a particular value of C2). It will be understood that the determining at720, which has been inserted into the prior-use method500A in order to produce the current-use method500B, may have been inserted as a result of the reconfiguring algorithm depicted inFIG. 6.

As a result of one or more reconfigurings of the method500B, a new method500C may emerge, as will be discussed in greater detail below with reference toFIG. 9.

FIG. 8generally illustrates an example implementation of another reconfiguration algorithm. The reconfiguration algorithm depicted inFIG. 8may be an example implementation of the reconfiguring at590depicted inFIG. 7.

Once the IoT system commences monitoring of additional conditions (for example, C2as depicted inFIG. 7), the IoT system can also begin tracking a correlation score for each additional condition (for example, a second correlation score S2). InFIG. 8, the reconfiguring algorithm tracks S1and S2. The IoT system then uses the correlation scores to determine whether the triggering condition (which currently uses C1=TRUE as the sole triggering condition) can be refined (for example, by expressing the triggering condition in terms of C2or a combination of C1and C2). These refinements may increase the likelihood of receiving positive user feedback at560-580.

At810, the reconfiguring algorithm, having recognized and interpreted a gesture (at560-580), determines whether the recognized gesture is positive user feedback or negative user feedback. If the recognized gesture is positive (‘(+)’ at810), then the reconfiguring algorithm proceeds to812. If the recognized gesture is negative (‘(−)’ at810), then the reconfiguring algorithm proceeds to814.

At812, a first correlation score S1is increased, and then the recognition algorithm proceeds to820A. At814, by contrast, the first correlation score S1is decreased, and then the recognition algorithm proceeds to820B. As noted above, S1may reflect a confidence level associated with future performances of the function at550in response to the presently-used triggering condition (i.e., C1=TRUE).

At820A, the recognition algorithm determines whether the second condition C2is TRUE or FALSE. If C2=TRUE, then the reconfiguring algorithm proceeds to822. If C2=FALSE, then the reconfiguring algorithm proceeds to824.

At820B, the recognition algorithm makes the same determination as to whether the second condition C2is TRUE or FALSE, but proceeds differently from the determining at820A. In particular, if C2=FALSE, then the reconfiguring algorithm proceeds to822. If C2=TRUE, then the reconfiguring algorithm proceeds to824.

At822, a second correlation score S2is increased. At824, by contrast, the second correlation score S2is decreased. As noted above, S2may reflect a confidence level associated with future performances of the function at550in response to the trigger C2=TRUE.

At830, the recognition algorithm optionally determines one or more combination correlation scores Scombo1, Scombo2, Scombo3, and/or Scombo4corresponding to different combinations of conditions. For example, the one or more combination correlation scores may be associated with the particular combinations of conditions shown in Table 1.

As noted above, the one or more combination correlation scores Scombo1, Scombo2, Scombo3, and/or Scombo4may reflect a confidence level associated with future performances of the function at550in response to mixed triggers. For brevity, the details of the determining at830are not depicted inFIG. 8. However, it will be understood that any particular combination correlation score may be determined using a method similar to the increasing and/or decreasing of S2, as described above with respect to820A,820B,822, and824. For example, suppose that positive feedback is received at810. To determine Scombo2at830, the recognition algorithm may first determine whether ((C1=TRUE) AND (C2=FALSE))=TRUE.

If it is true that ((C1=TRUE) AND (C2=FALSE)), then Scombo2may be increased in an operation analogous to the increasing at812and/or the increasing at822. By contrast, if it is not true that ((C1=TRUE) AND (C2=FALSE)), then Scombo2may be decreased in an operation analogous to the decreasing at814and/or the decreasing at824.

At850, the reconfiguring algorithm may determine whether any of the alternative correlation scores (i.e., S2and/or ScomboX) exceeds a confidence threshold. The confidence threshold may be set to, for example, the first correlation score S1determined at812or814, or any other suitable value. If any of the alternative correlation scores exceed the confidence threshold (‘yes’ at850), then the reconfiguring algorithm may proceed to860before returning to510. If none of the additional correlation scores exceed the confidence threshold (‘no’ at850), then the method500A may return to510directly, as depicted inFIG. 8.

It will be understood that the determination at850is an inquiry into whether one of the additional conditions (for example, C2=FALSE) or a particular combination of conditions (for example, C1=TRUE and C2=FALSE) will be a better predictor of positive user feedback than the presently-used triggering condition, which relies solely on the first condition C1. If S2and/or ScomboXis greater than S1, then this may indicate that C1should not be the sole triggering condition. As will be discussed in greater detail below, the reconfiguring algorithm may reconfigure the IoT system such that at least one condition other than C1is used as a basis for determining whether to perform the function at550.

At860, the reconfiguring algorithm may reconfigure the triggering condition for future performances of the function (performed at550). For example, inFIG. 5, the method500A proceeds to perform the function at550in accordance with the triggering condition C1=TRUE. Likewise for the method500B depicted inFIG. 7. However, after the reconfiguring performed at860, one or more additional conditions (e.g., C2) may be checked before proceeding to perform the function at550. One possible result of such a reconfiguring is depicted inFIG. 9, as will be discussed in greater detail below.

FIG. 9generally illustrates a method500C for configuring and/or reconfiguring an IoT system to perform a function in response to contextual data. The method500C may be performed by, for example, one or more components of an IoT system analogous to the IoT system100depicted inFIG. 1. The method500C may be analogous in some respects to the method500A and/or the method500B, and may, in some implementations, constitute a result of the reconfiguring performed at590in one or more ofFIGS. 5-8. For example, like the method500A and the method500B, the method500C may include the determining at510, the performing of the function at550, the receiving of the image data at560, the recognizing at570, the interpreting at580, and the reconfiguring at590. However, unlike the method500A and the method500B, the method500C does not proceed to the performing of the function at550in response to a determination that C1=TRUE (‘yes’ at510). Instead, the method500C proceeds to920.

At920, the method500C determines whether a second condition C2holds. In particular, if C2=FALSE, then the method500A proceeds to550(‘yes’ at920). But if C2=TRUE, then the method500C returns to510(‘no’ at920). Accordingly, ((C1=TRUE) AND (C2=FALSE)=TRUE) constitutes a trigger for proceeding to550, and any other combination of C1and C2results in a return to510.

The determining at920may further include recording and/or storage in a memory system of data associated with C2(for example, a particular value of C2or a TRUE/FALSE condition of C2), similar to the determining at720depicted inFIG. 7.

Recall that inFIG. 8, the IoT system determined at850whether at least one additional correlation score (for example, Scombo2) exceeded a confidence threshold, and proceeded to860when the confidence threshold was exceeded (‘yes’ at850). Recall further that at860, the IoT system reconfigured the triggering condition for future performances of the function (performed at550). The method500C depicted inFIG. 9may represent one possible result of that reconfiguration. In the method500C, the triggering condition is no longer based solely on a determination that C1=TRUE (as in the method500A and the method500B). Instead, to proceed to the performing of the function at550, the IoT system must make two determinations: a first determination at510that C1=TRUE and a second determination at920that C2=FALSE.

We will now return to the example scenarios ofFIGS. 4A-4Bto illustrate how, in one particular scenario, an IoT system may be trained using gesture recognition.

In the scenario depicted inFIG. 4A, the user401has descended the stairs into a dark basement, and the IoT system activates the lamp413in response to a determination that the user401is present (C1=TRUE). Consider now how the IoT system will behave if it acts in accordance with the method500A depicted inFIG. 5. At510, the IoT system would determine at510whether the user401is present. When the method500A detects that the user401is present (C1=TRUE), the IoT system would proceed to550, where it would perform the function of activating the lamp413. The image sensor410would then receive image data (at560), and a gesture would be recognized in the image data (at570). The recognized gesture would be interpreted as positive user feedback (at580), and the IoT system would be reconfigured by increasing a confidence level associated with activation of the lamp413in response to a determination that the user401has entered the basement (for example, increasing the first confidence score S1as depicted at612inFIG. 6).

In the scenario depicted inFIG. 4B, the user401has entered a naturally-lit sunroom, and the IoT system activates the lamp413in response to a determination that the user401is present (C1=TRUE). Consider now how the IoT system will behave if it acts in accordance with the method500A depicted inFIG. 5. At510, the IoT system would determine at510whether the user401is present. When the method500A detects that the user401is present (C1=TRUE), it would proceed to550, where it would perform the function of activating the lamp413. The image sensor410would then receive image data (at560), and a gesture would be recognized in the image data (at570). The recognized gesture would be interpreted as negative user feedback (at580), and the IoT system would be reconfigured by decreasing a confidence level associated with activation of the lamp413in response to a determination that the user401has entered the sunroom (for example, decreasing the first confidence score S1as depicted at614inFIG. 6).

Consider now how the IoT system would behave if it performed the recognition algorithm depicted inFIG. 6. At610, the IoT system would recognize a gesture of approval every time the user401entered the sunroom when it was dark and/or cloudy. Moreover, the IoT system would recognize a gesture of disapproval every time the user401entered the sunroom on a sunny day. If there was enough negative feedback over a given period of time, then the first correlation score S1would decrease. For example, if roughly half of the feedback were positive feedback, and the other half of the feedback were negative feedback, then S1would tend toward zero (indicating no correlation). At a certain point in its decline toward zero, S1may fall below a confidence threshold, for example, +0.4. As a result, the IoT system would commence the monitoring of the second condition C2at660. The IoT system would abandon the method500A in favor of the method500B depicted inFIG. 7.

In the method500B depicted inFIG. 7, the activation of the lamp413at550is still performed in response to a determination that C1=TRUE. However, this approach has not led to consistent positive feedback, so C2is being monitored to determine if other approaches would be more successful.

Suppose that the second condition C2is a solar power level measured by the solar panel411as one example. Consider now how the IoT system would behave if it commenced monitoring of C2(as at720) while performing the recognition algorithm depicted inFIG. 8. For clarity of illustration, the sun exposure levels in the present example are mapped to a simple TRUE/FALSE determination in the present example. In particular, if a solar power level is at 100 W or above, then C2=TRUE, and if the solar power level is below 100 W, then C2=FALSE.

At810, the IoT system would recognize a gesture of approval when the user401entered the sunroom (C1=TRUE) at night (C2=FALSE). As a result, S1would be increased at812, S2would be increased at822, and Scombo2would be increased at830. However, some time later, the IoT system would recognize a gesture of disapproval when the user401entered the sunroom (C1=TRUE) when it was sunny (C2=TRUE). As a result, S1would be decreased at812, S2would be increased at822, and Scombo2would be increased at830. Over time, the first correlation score S1would steadily decrease toward zero, because the IoT system continues to receive mixed user feedback when C1=TRUE. By contrast, Scombo2will tend toward +1, indicating a strong likelihood of positive user feedback when (C1=TRUE) AND (C2=FALSE). Accordingly, at some point, the value of Scombo2may exceed the value of S1. In response to the determination at850that Scombo2exceeds the value of S1, the recognition algorithm proceeds to860. At860, the IoT system is reconfigured yet again, so as to behave in accordance with the method500C.

Because Scombo2has proven to be the best predictor of positive user feedback (by virtue of its value being nearest to +1), the triggering condition used in the method500C is reconfigured so as to adopt as the triggering condition the combination of conditions associated with Scombo2. In particular, the method500C proceeds to activate the lamp413only if C1is TRUE (user401present) and C2is FALSE (solar power level is below 100 W).

By reconfiguring the IoT system in response to feedback provided by the user, the IoT system proceeded from the method500A depicted inFIG. 5to the method500B depicted inFIG. 7, where it was trying to identify new and better methods. After some data-gathering, the IoT system proceeded from the method500B to the method500C depicted inFIG. 9, where better methods were put into practice. Unless the user's preferences change (or new relevant condition-measurement techniques are added to the IoT system), the IoT system will continue to perform the method500C. While the reconfiguring of the IoT system has been illustrated with reference toFIGS. 5-9, it is understood that other methods of incorporating user feedback can be used, including various machine learning approaches. The various approaches allow the IoT system to respond, over time, to positive and negative user feedback to allow IoT system to better control various devices and appliances in accordance with user preferences. WhileFIGS. 4A and 4B, and subsequent discussions with reference toFIGS. 5-9, were discussed in the context of an IoT system for activating a lamp, it is understood that an IoT system could use positive and negative user feedback in other contexts, including automatic faucets, automatic toilets, automatic soap dispensers, as well as the various devices referred to inFIG. 1such as thermostat114for regulating room temperature, refrigerator116, and washer and dryer118, to name some non-limiting examples. Also, while the disclosure herein has been with reference to an IoT system, a smart sensor system that includes a vision sensor (such as always-on vision sensor210ofFIG. 2A) capable of controlling a lamp or an HVAC system, or other kind of device, appliance, or system, may be reconfigured based on user feedback in the manner described above even it is not addressable via the internet and hence not, strictly speaking, an IoT device. While the foregoing disclosure shows illustrative aspects of the disclosure, it should be noted that various changes and modifications could be made herein without departing from the scope of the disclosure as defined by the appended claims. The functions, steps and/or actions of the method claims in accordance with the aspects of the disclosure described herein need not be performed in any particular order. Furthermore, although elements of the disclosure may be described or claimed in the singular, the plural is contemplated unless limitation to the singular is explicitly stated.

Various aspects are disclosed in the following description and related drawings to show specific examples relating to exemplary aspects of an IoT system. Alternate aspects will be apparent to those skilled in the pertinent art upon reading this disclosure, and may be constructed and practiced without departing from the scope or spirit of the disclosure. Additionally, well-known elements will not be described in detail or may be omitted so as to not obscure the relevant details of the aspects disclosed herein.

The terminology used herein describes particular aspects only and should not be construed to limit any aspects disclosed herein. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises,” “comprising,” “includes,” and/or “including,” when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. Similarly, the phrase “based on” as used herein does not necessarily preclude influence of other factors and should be interpreted in all cases as “based at least in part on” rather than, for example, “based solely on” or “based only on”.

Further, many aspects are described in terms of sequences of actions to be performed by, for example, elements of a computing device. It will be recognized that various actions described herein can be performed by specific circuits (for example, one or more general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof). It will be further recognized that the various actions described herein can be performed by executing program instructions. Additionally, the sequence of actions described herein can be considered to be embodied entirely within a memory system comprising any form of computer-readable storage medium (for example, RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art). The memory system may store and/or provide code for causing a processing system to perform the corresponding functionality described herein. Accordingly, those skilled in the art will appreciate that the various illustrative logical blocks, modules, circuits, and algorithms described in connection with the aspects disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. Whether such functionality is implemented as hardware or software or both depends upon the particular application and design constraints imposed on the overall system.