Home automation system including user interface operation according to user cognitive level and related methods

A home automation (HA) system may include a cloud server, HA operation devices within a senior living facility, and HA user interface devices for respective users within the senior living facility. Each HA user interface device may include a user input device, a display defining a user interface (UI), and a controller. The HA system may include HA hub devices within the senior living facility to provide communications for the cloud server, the HA user interface devices, and the HA operation devices. The controller may send user interaction data to the cloud server and operate the UI according to a user cognitive level. The cloud server may be configured to determine the user cognitive level based upon the user interaction data received from a given HA user interface device, and send the user cognitive level to the given HA user interface device.

RELATED APPLICATION

The present application is based upon provisional application Ser. No. 62/595,810, filed Dec. 7, 2017, the entire contents of which is hereby incorporated by reference.

TECHNICAL FIELD

The present embodiments are directed to the field of electronics, and more particularly to home automation systems and related methods.

BACKGROUND

There are a number of home automation systems and approaches that seek to permit automated control of electrical devices in a house. The popularity of home automation has been increasing due to the greater availability of smartphones and tablets. As noted in “The Problem With Home Automation's Internet Of Things (IoT)”, an article appearing in Forbes dated Sep. 26, 2013, home automation was typically for wealthy consumers with an expensive system to control lights, home theater, security, air conditioning, and home audio. This market has expanded with many do it yourself (DIY) products now available, and, although the products are useful, they may be difficult to aggregate. In other words, as explained in the article, difficulties could arise if a consumer bought a Nest thermostat, Kwikset door lock, Phillips Hue lighting device, Lutron light switch, Sonos audio system, and Belkin wireless plugs. The consumer would need to have multiple applications each requiring time to setup, learn, and use. Additionally, the article states that there is no easy way to make devices work together, such as if the consumer wanted to trigger one event using one device based on another event from another device.

Multiple communication protocols may also be problematic. In particular, different devices may operate using different communication protocols, for example, Wifi, Zigbee, Zwave, Insteon, Itron, RadioRA2, and others. This may create additional difficulties for home automation.

One approach to address these shortcomings is for the consumer, which may include a user and/or enterprise, to use a service and device aggregator that provides one application and a consolidated wireless adapter unit. The user would contract with such a provider for multiple years. Unfortunately, as noted in the article, the consumer may not benefit from the most advanced hardware and software.

Another approach, as noted in the Forbes article, is to provide a single application that attempts to consolidate disparate applications and consolidate wireless adaptors, for example, using each of the different communications protocols. However, such aggregation or single application approaches may present increased difficulty in usability. For example, if a given user cannot readily operate the one application or if the application is not readily accessible for any given user, operational difficulties may still exist. Accordingly, further improvements to the operation of home automation devices may be desirable.

SUMMARY

A home automation (HA) system may include a cloud server, a plurality of HA operation devices within a senior living facility, and a plurality of HA user interface devices for respective users within the senior living facility. Each of the plurality of HA user interface devices may include a user input device, a display defining a user interface (UI), and a controller coupled to the user input device and the display. The HA system may also include a plurality of HA hub devices within the senior living facility to provide communications for the cloud server, the plurality of HA user interface devices and the plurality of HA operation devices. The controller of each HA user interface device may be configured to send user interaction data to the cloud server and operate the UI according to a user cognitive level. The cloud server may be configured to determine a cognitive level of the user based upon the user interaction data received from a given HA user interface device and send the user cognitive level to the given HA user interface device.

The cloud server may be configured to decrease the user cognitive level based upon the user interaction data received from the given HA user interface device representing cognitive decline of the user, for example. The display and the input device may define a touch display, and the controller of each HA user interface device may be configured to acquire user interaction data from the touch display.

The user interaction data may include touch display user contact data. The user interaction data may include time periods between successive user contacts, for example.

The user input device may include a camera, and the user interaction data may include device-to-user distance data. The user input device may include an accelerometer, and the user interaction data may include device acceleration data, for example.

The user interaction data may include user game interaction data. The user cognitive level may determine at least one of a color adjustment within the UI, an adjustment in contrast within the UI, an adjustment in brightness within the UI, an adjustment in size within the UI, and a disablement of elements within the UI, for example.

A method aspect is directed to a method of operating a user interface (UI) according to a user cognitive level. The UI is defined by a display of a given home automation (HA) user interface device for a respective user within a senior living facility from among a plurality of HA user interface devices in an HA system. The HA system includes a cloud server, a plurality of HA operation devices within the senior living facility, and a plurality of HA hub devices within the senior living facility to provide communications for the cloud server, the plurality of HA user interface devices and the plurality of HA operation devices. The method may include using the HA user interface device to send user interaction data to the cloud server, the cloud server determining the user cognitive level based upon the user interaction data and to receive the user cognitive level from the cloud server. The method may also include using the HA user interface device to operate the UI according to the user cognitive level.

DETAILED DESCRIPTION

Referring initially toFIGS. 1 and 2, a home automation (HA) system20illustratively includes a cloud server33. The cloud server33may store media content for distribution, for example, documents, photos, videos, data feeds, etc. The cloud server33may also process operations for HA devices, as will be described in further detail below. The cloud server33may also function as a gateway to permit operation of HA operation devices31a-31nfrom outside a structure21or internal network (i.e., over the Internet).

The HA operation devices31a-31nof the HA system20are within the structure21. The structure21may be a senior living facility, for example, and include living rooms for members of the senior living facility. Of course, the structure21may be another type of structure, for example, a hospital, a home, a commercial, residential, and/or educational campus, etc.

The HA operation devices31a-31nmay include a light switch in a room in the structure21, a thermostat, a motion detector, an audio controller, a door lock, and a camera, for example. The HA operation devices31a-31nmay include other and/or additional types of devices.

The HA system20also includes HA user interface devices36a-36nfor respective users within the structure21. The HA user interface devices36a-36nmay be in the form of tablet computers, for example. Each HA user interface device36a-36nmay be another or different type of HA user interface device, for example, a smartphone, a laptop or personal computer, and/or television interface or streaming device.

Each HA user interface device36a-36nis configured to wirelessly communicate with at least one of the HA operation devices31a-31nto perform at least one HA operation. For example, a given HA user interface device36a-36nmay be used, based upon input thereto, to control a light switch in a room in the structure21, control a thermostat, communicate with a motion detector, control an audio controller, control a door lock, and communicate with a camera.

In the exemplary embodiment where an HA user interface device36a-36nis in the form of a tablet computer, the HA user interface device may include a portable housing41, wireless communication circuitry42carried by the portable housing, an input device54carried by the portable housing, and a display43carried by the housing. The display43and the input device54may together define a touch display55to accept a touch input. While the embodiments described herein refer to a touch display, it should be understood that embodiments may not include a touch display, but rather a physically separated display43and input device54, such as, for example, a pushbutton. A controller44is coupled to the wireless communications circuitry42and the display43. The controller44is configured to cooperate with the display43to display a user interface (UI)48, for example, a graphical user interface (GUI). Based upon input to the touch display55, the controller may cooperate with the wireless communications circuitry42to control operation of the HA operation devices31a-31n. Each HA user interface device36a-36nmay also be used for social media, playing games, and/or surfing the Internet. Each HA user interface device36a-36nmay also provide stored media content to the respective user for listening and/or viewing on a display43. In the example of a senior living center, the stored media content may include current weather data, a social calendar, a daily food menu for the cafeteria, photos and/or video from a current event, local and national news, etc. Of course, each HA user interface device36a-36nmay be used to perform other and/or additional functions within the HA system20.

The HA system20further includes HA hub devices34a-34nwithin the structure or senior living facility21. The HA hub devices34a-34nprovide communications for the cloud server33, the HA user interface devices36a-36n, and the HA operation devices31a-31n. More particularly, a respective HA hub device34a-34nmay be in each area of the structure21and communicate with a corresponding HA user interface device36a-36n. With respect to the senior living facility example noted above, each member of the senior living facility may have an HA user interface device36a-36nand an HA hub device34a-34nwithin their room. The HA user interface device36a-36nmay communicate with the HA hub device34a-34nto control respective HA operation devices31a-31n. The HA hub device34a-34nmay be considered a bridge between respective HA operation devices31a-31nand the corresponding HA user interface device36a-36n.

An HA user interface device36a-36nmay communicate with the cloud server33to retrieve updated data or media content from the Internet, for example, or uploaded to the cloud server. Of course, the HA user interface device36a-36nmay communicate with either or both of the corresponding HA hub device34a-34nand cloud server33to download or retrieve data and/or media. The HA user interface device36a-36nmay also upload data, for example, to the cloud server33. With respect to the senior living example, the HA user interface device36a-36nmay upload or communicate usage data and/or other data, for example, cognitive data to be used by staff for determining a user health level and/or a user activity level. Further details of an exemplary HA system are described in U.S. Patent Application Publication No. 2017/0006533, assigned to the present assignee, and the entire contents of which of herein incorporated by reference.

Further details of user interaction with the HA user interface devices36a-36n, and more particularly, the UI, will now be described. Returning again to the senior living facility example, it may be particularly desirable for staff of the senior living facility to determine if and when a user has undergone a cognitive change, which may be relatively difficult to uncover. For example, a cognitive change may be indicative of a broader health concern. Moreover, a member may be less desirous of using the HA user interface device36a-36nif they experience operational difficulties due to a cognitive change. To address this, a given user's interaction with the UI48of his or her HA user interface device36a-36nmay be used as a basis for adapting the UI. More particularly, the controller44of the given HA user interface device36a-36noperates the UI according to a user cognitive level, as will be described in further detail below, and through the touch display55and/or other elements and/or functions of the HA user interface device, monitors interaction with the UI by collecting user interaction data. Acquired user interaction data may be wirelessly communicated from the given HA user interface device36a-36nto the cloud server33for processing. It should be understood that while functions of the HA user interface devices36a-36nare described herein, the functions are performed by the controller44through cooperation with other respective elements or hardware.

The cloud server33may process the acquired user interaction data and communicate with the given HA user interface device36a-36nto change the UI. More particularly, the cloud server33determines the cognitive user level based upon the user interaction data received from the given HA user interface device36a-36n. The cognitive user level determined by the cloud server33may determine how and what UI changes to make based upon historical data for a given member and/or using machine learning techniques, as will be appreciated by those skilled in the art. For example, the cloud server33may decrease the user cognitive level based upon the user interaction data being representative over cognitive decline of the user.

Various examples of types of changes of the user cognitive level, for example, cognitive decline, and corresponding UI changes will now be described. Referring now additionally toFIG. 3, in one example, the controller44may cooperate with the touch display55to acquire touch display user contact data. More particularly, an input position of the user51on the touch display55may be detected. The detected input position may be compared to a position of an element52or a “button” displayed on the touch display55. The detected position and the position of the “button”52on the touch display55may be communicated to the cloud server33. The cloud sever33may compare the detected position to a historical detected position for the “button”52. In other words, if the user51or member typically presses on the edge of the button52, the cloud server33may determine that the edge press is outside a given tolerance based upon previous inputs and adjust or determine the user cognitive level based thereon. If, for example, the user51presses just to the side of the button52(but not on it) in several instances (e.g., 3-times), the cloud server33may determine the user cognitive level so that the size of the button on the touch display55is to be increased (i.e., the user cognitive level determines an adjustment in size in the UI48).

The cloud server33communicates instructions for the HA user interface device36ato display images, text, and inputs with a larger size. Of course, the cloud server33may operate in the reverse. That is, if the detected position of input closely matches the position of the button on the touch display55, the cloud server33may communicate instructions to the HA user interface device36ato reduce the size of the button on the touch display. As will be appreciated by those skilled in the art, updates to the UI48are user dependent rather than machine dependent.

Referring now additionally toFIG. 4, in another embodiment, a user's typing or input speed may be used as an indicator of cognitive decline. The controller44may cooperate with the touch display55to detect time periods between successive user contacts (i.e., a time delta between inputs from the user51to the touch display). The time between inputs to the touch display55may be communicated to the cloud server33. The cloud sever33may compare these time periods to historical time periods between inputs to the touch display55to adjust or determine the user cognitive level. If, for example, the time periods between inputs to the touch display55are increasing or have increased beyond a threshold time period, the cloud server33may determine user cognitive level associated with voice activation instead of typed input should be used. Thus, the cloud server33communicates the user cognitive level, for example, wirelessly, that causes the HA user interface device36ato switch to a voice command mode and/or remove a keyboard (e.g., icons, letters keys, etc.) from the touch display55. The corresponding HA user interface device36amay also remove the UI elements52or the keyboard from the touch display55and begin a voice recognition mode using an audio input device45(i.e., microphone), carried by the portable housing41and coupled to the controller44. Other and/or additional functions may also be activated or removed, for example, some keys (e.g., the shift key) from the keyboard on the touch display55may be removed and/or a different keyboard displayed on the touch display. Of course, similar to the embodiments described above, the cloud server33may operate in the reverse. That is, if the user's typing speed increases, then the cloud server33may add virtual keys or display a different more advanced keyboard on the touch display55.

Referring now additionally toFIG. 5, in another embodiment, the given HA user interface device36aincludes a camera46carried by the portable housing41and coupled to the controller44. The camera46may cooperate with the controller44to measure device-to-user distance data (i.e., the distance from the face of the user51to the device), for example. A larger or growingly large distance from the face of the user51may be an indication that the user may need reading glasses, for example. Accordingly, the cloud server33may determine the user cognitive level based upon a varying or changing distance over a given time period. The HA user interface device36a, based upon the user cognitive level sent from the cloud server33, may operate the UI48so that the size of elements52within the UI on the touch display55increase. While a camera46is described, it will be appreciated by those skilled in the art that other types of sensors for sensing the user's distance from touch display55may be used, for example, one or more proximity sensors.

Referring now additionally toFIG. 6, the HA user interface device36amay include an accelerometer47carried by the portable housing41and coupled to the controller44. The accelerometer47may cooperate with the controller44to acquire device acceleration data or to detect an acceleration of the portable housing41, which may correspond to a shaking motion or tremors of the user51, for example. Accordingly, the cloud server33may set the user cognitive level based upon the acceleration, and the user cognitive level received from the cloud server may cause the HA user interface device36ato increase the size of elements52within the UI48on the touch display55based upon an amount of shaking (e.g., an increased or increasing amount, an amount beyond a threshold).

Referring now additionally toFIG. 7, the user interaction data may include game interaction data. Acuity levels based upon games played on a given HA user interface device36a-36nmay be used as a basis for changing or updating the UI48on the touch display55. For example, if the user51is continuously losing in the game of solitaire, this may be an indication of a potential unaddressed health issue. More particularly, the given HA user interface device36amay communicate the win/loss data to the cloud server33. The cloud server33may generate an acuity score, for example, which may be generated across multiple games and may serve as the basis for the user cognitive level. The cloud server33may cause the HA user interface device36a-36nto adjust the UI48, for example, make the elements52of the UI larger. If the condition continues even after an update of the UI48, further updates to the UI can be made. The received feedback taken in combination with the types and amount of UI updates may be used to determine the existence of a potential health condition.

While several results of cognitive measures have been described, it will be appreciated that the cloud server33may communicate instructions for other and/or additional functions on the HA user interface device36a-36n. For example, through communication with the cloud server33, the user cognitive level may determine or cause the HA user interface devices36a-36nto add or remove functions, access to applications, communications capabilities, change colors within the UI48, increase or reduce contrast within the UI, adjust brightness within the UI, adjust size of elements52within the UI, and adjust touch sensitivity, etc.

Moreover, while the HA system20has been described with respect to the cloud server33making the UI48change or update determination based upon the inputs and communication from the given HA user interface device36a-36n, it will be appreciated that other devices within the HA system20may also determine whether to change and what type of changes to apply to the UI. For example, any given HA user interface device36a-36nmay measure the inputs and determine the UI48changes independent of the cloud server. The corresponding HA hub device34a-34nmay also perform these operations.

Interaction data communicated from the HA user interface devices36a-36nmay be processed by the cloud server33to generate an alert or notification, for example, of declining cognitive behavior. The notification may be an email, SMS message, or other notification and may be communicated, for example, wirelessly, to one or more of the user, senior living staff, medical staff, and/or family members.

The HA system20may be particularly advantageous for predicting changes in cognitive skills, for example, to identify declining health of a user. Additionally, elements of the UI are typically adjusted manually, which may, for a senior user, be relatively difficult. Thus, a senior user may not use or use less of a device that is difficult to operate. The system20may address this difficulty by adjusting elements of the UI based upon input over time.

A method aspect is directed to a method of operating a user interface (UI)48according to a user cognitive level, the UI defined by a display43of a home automation (HA) user interface device36a-36nfor a respective user within a senior living facility21from among HA user interface devices in an HA system20. The HA system includes a cloud server33, HA operation devices31a-31nwithin the senior living facility21, and HA hub devices34a-34nwithin the senior living facility to provide communications for the cloud server33, the HA user interface devices36a-36nand the HA operation devices. The method includes using the HA user interface device34ato send user interaction data to the cloud server33. The cloud server determines the user cognitive level based upon the user interaction data. The method also includes using the HA user interface device36ato receive the user cognitive level from the cloud server33, and operate the UI48according to the user cognitive level.