System and method to view occupant status and manage devices of building

A device to provide information to a visual interface that is mountable to a vehicle dashboard includes a memory configured to store device information indicative of controllable devices of a building and occupant data indicative of one or more occupants of the building. The device includes a processor configured to receive, in real-time, status information associated with the one or more occupants of the building. The status information includes at least one of dynamic location information or dynamic activity information. The processor is configured to generate an output to provide, at the visual interface device, a visual representation of at least a portion of the building and the status information associated with the one or more occupants. The processor is also configured to generate an instruction to adjust an operation of one or more devices of the controllable devices based on user input.

The present disclosure is generally related to an interface to control devices in a building.

II. DESCRIPTION OF RELATED ART

Advances in technology have resulted in smaller and more powerful computing devices. For example, there currently exist a variety of portable personal computing devices, including wireless telephones such as mobile and smart phones, tablets and laptop computers that are small, lightweight, and easily carried by users. These devices can communicate voice and data packets over wireless networks. Further, many such devices incorporate additional functionality such as a digital still camera, a digital video camera, a digital recorder, and an audio file player. Also, such devices can process executable instructions, including software applications, such as a web browser application, that can be used to access the Internet. As such, these devices can include significant computing capabilities.

A network of “smart” devices within a house, such as Internet-of-Things (IoT)-enabled devices, can be controlled via a “home dashboard” application. For example, a user can turn lights on or off or view feeds from security cameras using a home dashboard application running at a smartphone or another portable personal computing device. However, controlling devices from a remote location may cause problems for an occupant inside the house. To illustrate, turning off the lights in the house may be dangerous or inconvenient for an occupant in the house that is not expecting the lights to be turned off.

In a particular aspect, a device to provide information to a visual interface device that is mountable to a vehicle dashboard includes a memory configured to store device information indicative of controllable devices of a building and occupant data indicative of one or more occupants of the building. The device also includes a processor configured to receive, in real-time, status information associated with the one or more occupants of the building. The status information includes at least one of dynamic location information or dynamic activity information. The processor is configured to generate an output to provide, at the visual interface device, a visual representation of at least a portion of the building and the status information associated with the one or more occupants. The processor is also configured to generate an instruction to adjust an operation of one or more devices of the controllable devices based on user input.

In another particular aspect, a method for providing information associated with occupants of a building includes receiving, in real-time at a dashboard device, status information associated with one or more occupants of the building. The status information includes at least one of dynamic location information or dynamic activity information. The method includes generating, at a visual interface of the dashboard device, a visual representation of at least a portion of the building and of the status information associated with the one or more occupants. The method also includes receiving, at the dashboard device, user input related to control of one or more controllable devices of the building, and generating, at the dashboard device, an instruction to adjust an operation of the one or more controllable devices based on the user input.

In another particular aspect, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to receive, in real-time, status information associated with one or more occupants of the building. The status information includes at least one of dynamic location information or dynamic activity information. The instructions further cause the one or more processors to generate an output for a visual interface device to provide a visual representation of at least a portion of the building and the status information associated with the one or more occupants. The instructions further cause the one or more processors to receive user input regarding control of one or more controllable devices of the building and generate an instruction to adjust an operation of the one or more controllable devices based on the user input.

In another particular aspect, an apparatus to perform operations to provide information about occupants of a building includes means for receiving, in real-time, status information associated with one or more occupants of the building. The status information includes at least one of dynamic location information or dynamic activity information. The apparatus also includes means for providing a visual representation of at least a portion of the building and the status information associated with the one or more occupants. The apparatus includes means for receiving user input regarding control of one or more controllable devices of the building. The apparatus includes means for generating an instruction to adjust an operation of the one or more controllable devices based on the user input.

V. DETAILED DESCRIPTION

Systems and methods operable to view occupant status and manage devices in a building are disclosed. Because a remote user using a conventional home dashboard application (e.g., at a remote phone) to control devices that are within a house can result in inconveniences or safety concerns for occupants of the house, such as turning off the lights in the house when an occupant in the house is not expecting the lights to be turned off, conventional home dashboard applications are not well suited for use when occupants are in the building.

As described herein, devices and systems enable a user to view a visual representation of a house or other building with real-time status information regarding occupants of the building in addition to providing control of devices in the building. In some implementations, the occupant status information includes location and activity information of each occupant and also includes an emotional state and age estimate of each occupant. The occupant status information enables a user of the device to avoid dangers or inconveniences to occupants of the building that may otherwise be introduced by conventional home dashboard applications that do not provide the user with occupant status information. In this context, “real-time” refers to without significant delay. Since the status information pertains to occupants (e.g., humans) of a building, any delay introduced by communication latency, data processing, image or data buffering for playout, etc., is insignificant because such delays are too short for the location or activity of an occupant to have changed significantly and in many cases too short for typical human perception. Thus, “real-time” in the context of the status information indicates that the status information received is descriptive of occupant status within a timescale of human action, such as several seconds or up to one minute.

In addition, various interfaces are described including touchscreen and context-aware speech and gesture recognition to enable ease of use under differing conditions, such as when implemented as a car dashboard display, a mobile device application, or a wall-mounted building management display. In some implementations, access control based on user authorization protects privacy and security of occupants in building, controllable devices in the building, or both, from unauthorized users or viewers of the device.

Referring toFIG. 1, a particular illustrative aspect of a system operable to provide the status of occupants and to manage devices in a building is disclosed and generally designated100. The system100includes a device104that is coupled to a visual interface device140and to a microphone142. The device104is also coupled to a building150, such as via one or more wireless networks, one or more wired networks, or a combination thereof.

The building150includes a building management system154coupled to one or more controllable devices156and to one or more sensors158. The one or more sensors158are configured to detect activity related to one or more occupants152of the building150. In a non-limiting example, the building150is the home of a user102of the device104. However, in other implementations, the building150is another type of building, such as an office building, apartment building, or theater, as non-limiting examples. In an illustrative example, the building management system154is a multiple-unit control system, such as for use by an apartment manager.

In some implementations, the one or more sensors158include one or more microphones, cameras, infrared sensors, ultrasound sensors, other sensors, or a combination thereof. Occupant activity, such as movement between rooms in the building150, gestures, and speech, is detected by the one or more sensors158and provided to the building management system154. In some implementations, the building management system154is configured to interpret the sensor data to determine identities, ages, and emotional states of the occupants152and to transmit the occupant information to the device104as status information130. For example, information regarding an age and an emotional state of an occupant can assist the user102in determining what actions to perform via the device104, as described further below. In some implementations, some or all of the sensor data is instead transmitted to the device104and processed at the device104to generate a corresponding portion of the status information130. Alternatively, or in addition, generation of the status information130can be distributed between the building management system154and the device104.

The one or more controllable devices156are responsive to instructions from the building management system154. In an illustrative example, the one or more controllable devices156include a climate control device such as a central heat and air unit, a lighting device such as a dimmable ceiling light, an entertainment system such as an audio/video system, one or more other controllable devices, or any combination thereof.

The device104is configured to provide the user102with access to occupant information and control of devices in the building150via a visual “home dashboard”—type application. The device104is configured to provide a visual representation of the building150and the one or more occupants152, along with a visual representation of the status information130, to the visual interface device140to be displayed to the user102as a visual representation144. The device104is also responsive to inputs from the user102, such as commands spoken by the user102in user speech103that is captured by the microphone142, to translate the commands into instructions for controlling the one or more controllable devices156.

The device104includes a memory106and a speech recognition engine118that are coupled to a processor108. The memory106is configured to store executable instructions (e.g., program code) that are executable by the processor108to perform operations associated with the visual “home dashboard”—type application (e.g., receiving and displaying status updates related to the building150and receiving user input to control one or more of the controllable devices156). The memory106is also configured to store data associated with the one or more controllable devices156and the data associated with one or more occupants152, as described further with reference toFIG. 2.

The speech recognition engine118is configured to receive an audio signal143from the microphone142and to process the audio signal143to generate a user input119. In an illustrative example, the speech recognition engine118is configured to perform automatic speech recognition (ASR) to identify words spoken by the user102and to generate a text representation of the identified words. The user input119is provided to the processor108for processing as a user command to the visual dashboard-type application.

The processor108is configured to process the user input119to identify a user command from the user102and to respond to the user input119. As an illustrative example, the processor108can perform natural language processing (NLP) to map the user input119to an action to be performed with regard to the one or more controllable devices156. For example, a user input119that includes the sequence of words “increase the living room lights” can be mapped to an action that increases a brightness level of ceiling lights in a living room of the building150by twenty-five percent. The processor108generates and transmits an instruction132to the building management system154that identifies the action to be taken. The processor108can update an output117to visually present to the user102that the action has been performed, such as via a pop-up message or by adjusting an indicator within the visual representation144(e.g., adjusting an indicator of the lighting level of the living room to indicate the new lighting level).

In some implementations, the visual interface device140is mountable to, or integrated in, a vehicle dashboard, such as illustrated inFIGS. 7 and 10. Alternatively or in addition, in some implementations the visual interface device140is integrated in a mobile device, such as in a tablet device as illustrated inFIG. 6. In other implementations, the visual interface device140is implemented as a device within the building150, such as illustrated inFIG. 8. In some implementations the visual interface device140includes a touchscreen or other input mechanism to enable the user102to enter non-speech commands to the device104, such as depicted inFIG. 2. In some examples, the visual interface device140is configured to provide a holographic display that can provide different versions of the visual representation144for different viewers, such as to enable authorized users to view information that is restricted from the view of non-authorized users (e.g., based on positions of the users seated in a vehicle), such as described further with reference toFIGS. 10-12.

During operation, in a particular implementation in which the device104, the visual interface device140, and the microphone142are integrated in a car, the user102(e.g., a car driver) can view the visual representation144depicting occupants and conditions in the building150. The device104receives updates to the status information130, such as detected changes in location, activity, and emotional state of the one or more occupants152, and updates the visual representation144via the output117so that the visual representation144provides dynamic information regarding the one or more occupants152to the user102. As used herein, “dynamic” information indicates information that is updated in response to detection of a status change at the building150. The status change is indicated via the status information130, which may be transmitted to the device104in real-time upon detection of change, or may be periodically or continually transmitted to the device104and processed by the device104to detect the status change.

For example, the device104may maintain a record of each of the detected occupants and may update one or more records (e.g., in the memory106) based on changes indicated in the status information130, such as when an occupant has changed locations (e.g., a room) within the building150, changed an activity (e.g., the occupant has stopped watching television and is now participating in a telephonic or in-person conversation), or changed an emotional status. In response to a detected change of one or more of the occupants, the device104updates the output117to cause the visual interface device140to adjust the visual representation144to indicate the change. To illustrate, an icon representing an occupant may be removed from one location (e.g., a first room in a graphical map of the building150) and inserted in another location (e.g., a second room in the graphical map) in response to a change of location of the occupant in the building150. An example of displaying location, activity, and other status information is described further with reference toFIG. 5.

Based on the status information of the one or more occupants152, the user102can decide whether or how to adjust an operation of one or more of the one or more controllable devices156. For example, the user102can provide spoken commands to turn off lighting in rooms of the building150that are unoccupied. As another example, the user102can provide spoken commands to cause a home entertainment system to play soft, gentle music when one of the one or more occupants152is detected as being in an agitated or angry emotional state, or to play loud, motivational music (e.g., a march) upon detecting that one of the one or more occupants152has overslept. As yet another example, the status information130can indicate that a first occupant is happy and watching television in a living room and that a second occupant (e.g., a child) is awake and lying in a bed in a bedroom, and the user input119can include an instruction from the user102to lower a volume of the television to help the second occupant to fall asleep. In this example, the instruction (e.g., the instruction132) is transmitted to the building management system154to cause the television to lower the volume.

The spoken commands are captured by the microphone142, interpreted as the user input119by the speech recognition engine118, and processed by the processor108to generate instructions, such as the instruction132. The instructions are transmitted to the building management system154, which causes the instructions to be carried out by the designated one or more controllable devices156. Changes within the building150, such as detected by the sensors158, can be provided to the device104as updated status information130and displayed to the user102via the visual interface device140.

By providing the user102with the real-time status information130regarding occupants of the building150in addition to providing control of the devices156, the user102can effect changes in the building150remotely with reduced risk of inconveniencing the occupants152as compared to conventional home dashboard applications that do not provide a user with occupant status information. In implementations in which the occupant status information130includes location and activity information of each occupant and also includes an emotional state and age estimate of each occupant, more effective or appropriate actions may be selected by the user102by taking into account the occupant age and emotional state as compared to when the user102does not have such information.

Although in some implementations the device104can communicate with the building management system154which controls the one or more controllable devices156and interfaces with the one or more sensors158, in other implementations the device104is configured to interact with the one or more controllable devices156and the one or more sensors158absent a central management system (e.g., the building management system154may be omitted), such as in a distributed internet of things (IoT) implementation.

It should be noted that various functions performed by the system100ofFIG. 1are described as being performed by certain components or modules. However, this division of components and modules is for illustration only. In an alternate aspect, a function performed by a particular component or module may be divided amongst multiple components or modules. Moreover, in an alternate aspect, two or more components or modules ofFIG. 1may be integrated into a single component or module. Each component or module illustrated inFIG. 1may be implemented using hardware (e.g., a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a controller, etc.), software (e.g., instructions executable by a processor), or any combination thereof.

FIG. 2depicts an example of a system200that includes an implementation of the device104. The device104is coupled to the visual interface device140, the microphone142, and a sensor244. For example, the device104may be integrated into a component that is distinct and separate from the visual interface device140, the microphone142, and the sensor244. To illustrate, the visual interface device140may be integrated into a car dashboard device, the microphone142may be part of a microphone array in an interior of the car, and the sensor244may be integrated into another interior component of the car. The device104may be implemented in an electronics package of the car and coupled to the visual interface device140, the microphone142, and the sensor244via one or more wires or via wireless signaling.

The device104includes the memory106and the processor108. The memory106is configured to store instructions230that are executable by the processor108to perform functionality described herein. The memory106is further configured to store device information240indicative of the one or more controllable devices156of the building150. The memory106is also configured to store occupant data242indicative of one or more of the one or more occupants152of the building150.

The processor108is coupled to the memory106and includes an output generator260, an instruction generator262, and a user verifier264. In some implementations, one or more, or all, of the output generator260, the instruction generator262, or the user verifier264are implemented via execution of the instructions230at one or more processing cores of the processor108. In other implementations, at least a portion of one or more of the output generator260, the instruction generator262, or the user verifier264is implemented via dedicated hardware or circuitry specific to each particular application.

The processor108is configured to receive real-time status information130associated with the one or more occupants152of the building150. The status information130includes at least one of dynamic location information202or dynamic activity information204. As illustrated, the location information202and the activity information204can be included in displayed status information248provided to the user102via the visual interface device140, as described further below. In addition, the displayed status information248can include an estimated age206and emotional status208associated with each of the one or more occupants152.

The output generator260is configured to generate the output117to provide, at the visual interface device140, the visual representation144. The visual representation144graphically depicts at least a portion of the building150and the status information248associated with the one or more occupants152. For example, the visual representation144may illustrate a simplified view of one or more interior spaces of the building150and may include one or more indicators or icons representing the one or more occupants152in approximate locations within the building150, as described in further detail with reference toFIG. 5.

The instruction generator262is configured to cause one or more of the controllable devices156to adjust operation based on the user input119. For example, the user input119may represent a command received from the user102via an audible speech of the user102, a touch detection at a touchscreen246of the visual interface device140, or a gesture of the user102recognized as a command for the device104. In an illustrative implementation, the instruction132generated by the instruction generator262includes at least one of an instruction to cause a climate control device in the building150to adjust a temperature, an instruction to adjust a lighting level in at least one room of the building150, or an instruction to adjust playback settings of an entertainment system in the building150.

The device104includes the speech recognition engine118. The speech recognition engine118is coupled to the processor108and is configured to receive the audio signal143corresponding to the audible user speech103and to translate the audio signal143into the user input119. The microphone142is coupled to the speech recognition engine118and configured to capture the audible user speech103and to generate the audio signal143.

The device104also includes a gesture recognition unit218coupled to the processor108. The gesture recognition unit218is configured to receive a sensor signal243corresponding to a user gesture. The gesture recognition unit218is configured to translate the sensor signal243into the user input119. For example, the sensor244is coupled to the gesture recognition unit218and is configured to detect a user gesture and to generate the sensor signal243. To illustrate, the sensor244can include at least one of a camera, an infrared sensor, or an ultrasonic sensor. Examples of gestures that may be captured by the sensor244include a hand waving gesture, such as a side-to-side hand gesture, a hand rising gesture, a hand lowering gesture, an open hand, a closed hand, a number of extended digits, one or more other gestures, or a combination thereof. Other examples of gestures that may be captured by the sensor244include head gestures, such as head nodding or head shaking, facial gestures, or any combination thereof. The gesture recognition unit218is configured to interpret the sensor signal243, such as by processing the sensor signal243to form an input for a trained neural network that is configured to classify input signals as gestures.

The device104further includes the touch recognition unit220configured to receive a touch detection signal245corresponding to a user touch and to translate the touch detection signal245into the user input119. For example, the visual interface device140includes a touchscreen246that is coupled to the touch recognition unit220. The touchscreen246is configured to display the visual representation144and to generate the touch detection signal245. For example, when the visual representation144depicts a simplified graphical indication of the building150, the touchscreen246may detect a touched location while displaying the visual representation144, and the location may be interpreted by the touch recognition unit220to identify one or more controls, one or more icons, or one or more locations within the visual representation144that are to be manipulated by the user102.

The user verifier264is configured to determine whether a user of the device104is an authorized user and to selectively restrict control of the one or more controllable devices156based on the determination of whether the user is an authorized user (e.g., a user that is identified as being authorized to access information that is restricted to non-authorized users). For example, the user verifier264can be configured to determine, based on whether the user is an authorized user, whether to selectively restrict access to at least a portion of the one or more controllable devices156. As another example, the user verifier264may be configured to receive an identification of the user102, such as via a credential provided by the user102, a password, a facial recognition, a voice recognition, a challenge question, other vision-based or biometric-based information, or one or more mechanisms to verify an identity of the user102. The identified user102may be compared against a database of authorized users of the device104to determine access to information of the building150. In some implementations, the device104can be used to provide information regarding multiple buildings, such as a user's home, a user's office, and one or more locations. In such circumstances, the user verifier264may provide different levels of access, for various users, for each of the multiple buildings that may be accessed via the device104. In some implementations, restricting access to the status information248and to visibility into one or more portions of the building150may be based on whether the user102is a parent or authorized adult of the building150, such as when the building150corresponds to the home of the user102. In such cases, an observer of the visual representation144that is not an authorized user, such as a passenger in an automobile that includes the device104, may be unable to see specific information regarding portions of the building150or statuses of one or more occupants of the building150.

The device104also includes a receiver222and a transmitter224coupled to the processor108. The receiver222is configured to receive information, such as the status information130, and the transmitter224is configured to transmit instructions, such as the instruction132, to the one or more controllable devices156. Although the receiver222and the transmitter224ofFIG. 2are depicted as distinct components, in other implementations the receiver222and the transmitter224may be implemented as a transceiver.

As illustrated, the device104receives sensor signals (e.g., the audio signal143, the sensor signal243, and the touch detection signal245) from external sensors and processes the sensor signals to generate the user input119. In other implementations, however, the device104instead receives processed signals as the user input119, as described further inFIG. 3, or includes one or more sensors, such as the touchscreen246, the microphone142, the sensor244, or any combination thereof. AlthoughFIG. 2illustrates the device104as responsive to user input received via the touchscreen246, the microphone142, and the sensor244, in other implementations the device104is responsive to fewer modes of user input, such as a single mode of user input (e.g., touch detection only, user speech only, or gestures only) or is responsive to two modes of user input (e.g., touch detection and user speech, or touch detection and gestures, or user speech and gestures). Although three representative modes of user input are described, other modes of user input can be implemented in addition or, or in place of, the described modes. As an illustrative, non-limiting example, in some implementations a brain-computer interface, such as an electroencephalography (EEG)-based interface, can be used in place of, or in addition to, one or more of the touchscreen246, the microphone142, or the sensor244.

FIG. 3depicts an example300of the device104in an implementation that excludes the touch recognition unit220, the speech recognition engine118, the gesture recognition unit218, the receiver222, and the transmitter224ofFIG. 2. The device104includes an input recognition engine302that is implemented at the processor108and that is operable to perform functionality as described with one or more of the touch recognition unit220, the speech recognition engine118, or the gesture recognition unit218ofFIG. 2.

In the example300, the device104may be directly coupled to one or more sensors and the user input119may include sensor data, such as the touch detection signal245, the audio signal143, and the sensor signal243, which are processed at the input recognition engine302. Alternatively, the user input119may be received from user interface devices that are also configured to perform signal processing and recognition of sensor signals. For example, the microphone142may include or may be implemented in a device that includes a speech recognition engine to process an audio signal generated by the microphone142and to output the user input119to the device104. Thus, the device104may be implemented as a distinct component that may be coupled to one or more input devices, display devices, transmitting and receiving devices or network interfaces, or any combination thereof.

FIG. 4depicts an example of an implementation400of the device104in which the visual interface device140, the microphone142, the sensor244, the touch recognition unit220, the speech recognition engine118, and the gesture recognition unit218are integrated within the device104. For example, the implementation400can correspond to a mobile electronic device, such as a tablet computer, as described further with reference toFIG. 6. Alternatively, the device104may be implemented in a smart phone, a laptop computer, one or more other devices, or a combination thereof. For example, the microphone142may include a microphone array of a smart phone, the sensor244may include one or more cameras implemented in the smart phone, and the visual interface device140may correspond to a screen of the smart phone. The receiver222and the transmitter224may be coupled to one or more antennas, such as an array of antennas, and may correspond to components for cellular network communications, Institute for Electrical and Electronics Engineers (IEEE) 802.11-type communications, short-distance ad-hoc wireless network communications, such as Bluetooth-type communications, one or more radio technologies, or in combination thereof.

FIG. 5depicts an example of an implementation500of the visual representation144and includes a visual representation of at least a portion of the building150, illustrated as a representation502of a house, and various status indicators and user-selectable controls. In particular, the implementation500includes a skill or action selector504, a speech recognition component506, an action status508, a context indicator510, a gesture indicator512. The implementation500also includes controls514for operation of a controllable device, illustrated as playback controls516and518for an entertainment system. Status information520includes an indicator522of the emotional status208and an indicator524of the age or estimated age206.

The representation502depicts a bathroom530, a bedroom532, a living room534, a kitchen536, a garage538and a balcony540. Each of the locations530-540of the building150may include one or more of the sensors158(ofFIG. 1) to enable occupant detection, speech recognition, gesture detection, and other information gathering regarding occupants152within the sensor's particular location. For example, each of the locations530-540may include one or more microphones, cameras, other detectors, or a combination thereof, and may provide sensor information to enable the device104ofFIGS. 1-4to display status information of the one or more occupants152including each occupant's location information202, activity information204, age206, and emotional status208. Other environmental information may also be displayed in the representation502, such as an illumination indicator542that indicates a detected light level as a value between 100% (full illumination) and 0% (no detectable light) or a setting of an illumination device, such as a dimmable ceiling light, with 0% indicating the illumination device is off or fully dimmed, 50% indicating that the illumination device is 50% dimmed, and 100% indicating the illumination device is not dimmed.

The location information202for each detected occupant152may be indicated by display of an icon544or other indicator in a respective location530-540for the occupant. Although a single instance of the context indicator510, the indicator522, and the indicator524is illustrated for the single detected occupant that is represented by the icon544, in an implementation in which multiple occupants are identified within the building150, multiple instances of the context indicator510, the indicator522, and the indicator524(one for each of the detected occupants) may be displayed.

The action selector504is configured to provide a list of actions or “skills” that can be designated by the user102, such as a dropdown menu associated with the current context. For example, the user102may select, such as via pressing a touchscreen on the icon544, to indicate the user102desires to view or control a context of the occupant associated with the icon544. In the illustrated example, the context indicator510indicates that the occupant indicated by the icon544is watching television in the living room534. The dropdown menu of the action selector504provides selectable actions that are appropriate for the context of watching television or operating an entertainment system, such as adding a song. The user102may speak the command “lower the light,” and the speech recognition component506may display an interpretation of the received user speech via automatic speech recognition and in accordance with the context of watching television in the living room534, as “lower light in living room.” As another example, a “down” hand gesture may be detected (e.g., the user102lowering a hand from a higher position to a lower position) and interpreted in accordance with the context of watching television in the living room as “lower the television volume.” The active skill corresponding to the automatic speech recognition and the context is indicated as the action status508and is converted and sent as the instruction132.

The identified emotional status208may be interpreted via a combination of detection of gestures of the one or more occupants152, speech of the one or more occupants152, voice characteristics of the one or more occupants152, such as prosody, or combination thereof. For example, the processor108or the speech recognition engine118may be configured to receive and process recorded speech of one or more of the occupants152to detect one or more prosody elements, such as emphasis, tonality, pitch, speech rate, or one or more other elements. In an illustrative example, duration is one aspect of prosody: if the speech has longer or shorter duration than usual, the speaker is likely in an emotional state, such as sadness, un-satisfaction, hesitation, etc. Other prosody elements like tonality and pitch contour, with which dynamic range is higher or lower than usual may suggest the speaker is happy or frustrated. A relatively complex mapping correlation may exist between all features of prosody and detected emotion. Extracted prosody related features may be used as one of the inputs to an emotion analysis at the processor108. Alternatively, or in addition, a video capture of one or more of the occupants152may be received and processed by the processor108to detect one or more physical characteristics of the occupant, such as body language or facial expression. The processor108may perform facial expression detection, such as smiling or frowning, or body motion detection, such as nodding, shaking head, or shrugging, or a combination thereof, which may indicate emotional state. The processor108may be configured to compare the prosody features, the results of the video processing, or both, to one or more audio emotional models to generate an output indicating one or more confidence levels that one or more respective emotional states have been detected as being experienced by the user.

The estimated age206of the occupant can also be estimated based on factors used for emotion identification. For example, an age/gender detector can be trained based on supervised learning approaches. Training speech data of different ranges of age and gender (e.g., children, female adult under 50 years old, female adult 50 years old or older, male adult under 50 years old, etc) could be provided, the trained model can provide an estimated age range and gender upon receiving a speech input. The estimated age206can be used to determine whether or not the status information520associated with a particular occupant is to be displayed to a viewer of the visual representation144that is not an authorized user, such as described further with reference toFIGS. 7-14.

Although the implementation500includes the interface components504-512, in other implementations one or more, or all, of the interface components504-512is omitted for clarity of display. For example, the interface components504-512may instead be accessible to the user in response to selection via a menu option.

Although the implementation500includes the status information520below the representation502of the building150and the icon544displayed within the representation502of the building150, in other implementations the status information520is instead displayed in conjunction with the icon544. For example, a “happy face” visual element may be depicted on (e.g., integrated within or overlaid upon) the icon544to indicate an identified emotion as happy, and a relative size of the icon544may be used as an indicator of an estimated age of that particular occupant.

FIG. 6depicts a particular implementation600of the device104in a portable electronic device. For example, the device104may include a smart phone or a tablet computer device that includes the microphone142and the sensor244.

FIG. 7depicts an example of an implementation700of the device104integrated into a vehicle dashboard device, such as a car dashboard device702. The visual interface device140is mounted within the car dashboard device702to be visible to a driver of the car, and the visual interface device140may be also visible to one or more passengers of the vehicle. The visual interface device140may be integrated within the device104, such as in the implementation400ofFIG. 4, or may be separate from and coupled to the device104, such as in the implementation of the system200depicted inFIG. 2.

FIG. 8depicts an implementation800of the device104implemented as a home control device within the building150. As illustrated, the device104is as a wall-mounted device in the living room of the building150, which enables an occupant of the living room to view the location and status information associated with each of the rooms of the building150and each of the occupants of the building150.

FIG. 9depicts an example of an implementation900of a visual representation144in which access to a portion of the building150is restricted to a user of the device104. To illustrate, in an implementation in which a user of the device104is not an authorized user, in order to preserve privacy of the owner of the building150or of occupants of the private locations in the building150, private areas of the building150may not be displayed. As illustrated, the bathroom530and the bedroom532ofFIG. 5are blacked out to prevent a non-authorized user of the device104from acquiring information regarding the private areas and occupants within the private areas. Similarly, information regarding one or more of the occupants152may be removed, such as by not displaying the icon544and status information520in a situation in which a viewer of the visual representation144is determined to be unauthorized to view occupant information. For example, the icon544and status information520may be omitted when the viewer is not authorized and the occupant is a minor.

FIG. 10depicts an example of an implementation1000in which the user102is a driver of a car. The visual interface device140is mounted into a car dashboard1006and is visible to the user102and to a passenger1004.

In the implementation1000, the visual interface device140incorporates a holographic display which is capable of providing different visual displays for the user102associated with a first viewing direction1010and the passenger1004associated with a second viewing direction1012. For example, the device104may use facial recognition to determine that the user102is an authorized user but that the passenger1004is not an authorized user. As a result, the device104may generate the output117to the visual interface device140to cause the visual interface device140to provide to the user102with a visual representation144that includes all areas of the building150, the one or more controllable devices156, and the one or more occupants152, as illustrated inFIG. 11, and to provide to the passenger1004a reduced content version of the visual representation144that omits information regarding the private areas of the building150and information regarding the one or more occupants152, as illustrated inFIG. 12.

FIG. 11depicts an example1100of the visual representation144as seen by the user102ofFIG. 10, displaying unrestricted information in accordance with the implementation500ofFIG. 5. In contrast,FIG. 12depicts an example1200of the visual representation144that is visible to the passenger1004ofFIG. 10and that omits display of sensitive information, such as private locations in the building150and occupants of the building150, such as described with reference toFIG. 9.

By determining whether a viewer is an authorized user of the device and selectively allowing or disabling presentation of information regarding one or more portions of the building150, the one or more controllable devices156, the one or more occupants152, or any combination thereof, granular security and privacy may be provided. Additional information regarding situational display and authorization associated with the device104is described in further detail with reference toFIG. 14.

FIG. 13depicts an implementation of a method1300for providing information associated with occupants of a building. For example, the method1300can be performed by a “dashboard” device, such as the device104. The method1300includes receiving, at the dashboard device, real-time status information associated with one or more occupants of the building, at1302. The status information includes at least one of dynamic location information or dynamic activity information. In some implementations, the status information further includes at least one of an age or an emotional status of at least one of the occupants. In an example, the status information corresponds to the status information130received by the device104.

The method1300includes generating, at a visual interface of the dashboard device, a visual representation of at least a portion of the building and of the status information associated with the one or more occupants, at1304. For example, the visual interface device140displays the visual representation144depicted inFIG. 5that includes a cross-sectional graphical depiction of the building150, distinct status information for each detected occupant152, and user interface controls to enable a user of the device104to adjust operation of the controllable devices156in a single display (e.g., on a single screen or holographic image).

The method1300includes receiving, at the dashboard device, user input related to control of one or more controllable devices of the building, at1306. In some implementations, the user input is received via a speech recognition operation, such as via the speech recognition engine118operating on the audio signal143. In some implementations, the user input is received via at least one of a gesture recognition operation, such as via the gesture recognition unit218operating on the sensor signal243, or a touch detection operation, such as the touch recognition unit220operating on the touch detection signal245.

The method1300includes generating, at the dashboard device, an instruction to adjust an operation the one or more controllable devices based on the user input, at1308. For example, the processor108generates the instruction132to be transmitted to the building management system154ofFIG. 1.

In some implementations, the method1300includes determining whether a user of the dashboard device is an authorized user and selectively restricting at least one of control of the controllable devices or access to at least a portion of the status information based on whether the user is an authorized user. For example, the user verifier264may operate to determine whether one or more viewers of the visual representation144are authorized users.

In some implementations the dashboard device is in a vehicle, and generating the visual representation includes, in response to the vehicle having a single occupant and the single occupant being an authorized user, incorporating the status information and information regarding the controllable devices, without restriction, into the visual representation. In other implementations, the dashboard device is integrated into a dashboard of a vehicle, and generating the visual representation includes, in response to the vehicle having a single occupant and the single occupant being an unauthorized user, omitting the status information and omitting information regarding controllable devices in non-public areas from the visual representation, such as illustrated inFIG. 9. In response to the vehicle having multiple occupants and the vehicle being in an emergency situation, generating the visual representation may include incorporating the status information and information regarding the controllable devices, without restriction, into the visual representation. In response to the vehicle having multiple occupants and the vehicle not being in an emergency situation, generating the visual representation may include holographically displaying different screens for different occupants. Additional details regarding an example of controlling information displayed based on context and user authentication are described with reference toFIG. 14.

By providing the user with the real-time status information regarding occupants of the building, in addition to providing control of the devices within the building, the method1300enables the user to effect changes in the building remotely and with reduced risk of inconveniencing the occupants as compared to conventional home dashboard applications that do not provide a user with occupant status information. In implementations in which the occupant status information includes location and activity information of each occupant and also includes an emotional state and age estimate of each occupant, more effective or appropriate actions may be selected by the user by taking into account the occupant age and emotional state as compared to when the user does not have such information.

FIG. 14depicts an example of a method1400of determining one or more visual representations to display at the visual interface device140, such as in the car dashboard implementation depicted inFIG. 10. The method1400includes, at1402, determining the number of people inside the car. For example, one or more sensors may be used, such as seat sensors, cameras, infrared or ultrasound sensors, or any combination thereof, to determine a number and placement of occupants in the car. In response to determining that a single occupant is in the car, the method1400includes determining, at1404, whether the single occupant is an authorized user. For example, when the occupant is an authorized user, the visual representation144(e.g., the “dashboard”) provides unrestricted access to private locations, occupant data, and device control options, at1406. Alternatively, when the occupant is not an authorized user, the method1400includes, at1408, displaying the visual representation144to enable control of appliances in public areas, such as in the living room, kitchen, and the garage, and omitting occupant data and information regarding private areas.

Alternatively, when more than one person is detected inside the car, the method1400includes, at1410, determining whether an emergency situation is detected. In some implementations, the emergency situation corresponds to an emergency for the user of the device104(e.g., a car wreck that is detected via an accelerometer or other sensor(s) in the device104or that is detected by receiving a notification via wireless signaling from an emergency response system of the car). In some implementations, the emergency situation corresponds to an emergency for occupants of the building (e.g., an alarm is activated, such as a smoke alarm, a fire alarm, an air contaminant alarm, or a security alarm, as non-limiting examples, and a notification is sent to the device104(e.g., in the status information130) indicating detection of the emergency situation).

In response to determining that an emergency situation is detected, the method1400includes, at1412, displaying the visual representation144with unrestricted information. Otherwise, the method1400includes, at1414, displaying holographically different screens for different passengers. The different visual representations144can be based on whether each of the individual passengers is an authorized user or is not an authorized user, such as described previously with reference toFIGS. 10-12.

By identifying the number of passengers in a vehicle and selectively generating different holographic screens for different passengers, a privacy of occupants and devices in the interior of the building150may be protected from access by non-authorized users, while authorized users may have full access to status and control information. In emergency situations, all viewers (e.g., an emergency responder) are granted access to the status and control information.

Referring toFIG. 15, a block diagram of a particular illustrative example of a device (e.g., a wireless communication device, a home device, an internet of things (IOT) device, a voice-activated digital assistant, or a combination thereof) is depicted and generally designated1500. In various examples, the device1500includes fewer or more components than illustrated inFIG. 15. In an illustrative example, the device1500corresponds to the device104. In an illustrative example, the device1500performs one or more operations described with reference toFIGS. 1-14.

In a particular aspect, the device1500includes the processor108(e.g., a central processing unit (CPU)). The device1500may include one or more additional processors1510(e.g., one or more digital signal processors (DSPs)). The processors1510include a speech and music coder-decoder (CODEC)1508, an echo canceller1512, or both. The processor108is coupled via a transceiver1540to an antenna1542. The transceiver1540includes the receiver222, the transmitter224, or both.

Although the speech and music CODEC1508is illustrated as a component of the processors1510, in other examples one or more components of the speech and music CODEC1508are included in the processor108, a CODEC1534, another processing component, or a combination thereof. The device1500also includes the memory106. The device1500includes the visual interface device140coupled to a display controller1526. One or more speakers1536, one or more microphones142, or a combination thereof may be coupled to the CODEC1534. The CODEC1534may include a digital-to-analog converter (DAC)1502and an analog-to-digital converter (ADC)1504.

In a particular aspect, the CODEC1534may receive analog signals from the microphones142, convert the analog signals to digital signals using the analog-to-digital converter1504, and provide the digital signals to the speech and music CODEC1508, such as in a pulse code modulation (PCM) format. The speech and music CODEC1508may process the digital signals. In a particular aspect, the speech and music CODEC1508may provide digital signals to the CODEC1534. The CODEC1534may convert the digital signals to analog signals using the digital-to-analog converter1502and may provide the analog signals to the speakers1536.

The memory106may include the instructions230executable by the processor108, the processors1510, the CODEC1534, another processing unit of the device1500, or a combination thereof, to perform methods and processes disclosed herein, such as one or more operations described with reference toFIGS. 1-14. One or more components of the systems and devices described with reference toFIGS. 1-14may be implemented via dedicated hardware (e.g., circuitry), by a processor executing instructions (e.g., the instructions230) to perform one or more tasks, or a combination thereof. As an example, the memory106or one or more components of the processor108, the processors1510, and/or the CODEC1534includes a memory device, such as a random access memory (RAM), magnetoresistive random access memory (MRAM), spin-torque transfer MRAM (STT-MRAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, or a compact disc read-only memory (CD-ROM). The memory device includes instructions (e.g., the instructions230) that, when executed by a computer (e.g., a processor in the CODEC1534, the processor108, and/or the processors1510), causes the computer to perform one or more operations described with reference toFIGS. 1-14. As an example, the memory106or the one or more components of the processor108, the processors1510, or the CODEC1534is a computer-readable storage device that includes instructions (e.g., the instructions230) that, when executed by a computer (e.g., a processor in the CODEC1534, the processor108, and/or the processors1510), cause the computer perform one or more operations described with reference toFIGS. 1-14.

In a particular aspect, the device1500is included in a system-in-package or system-on-chip device1522, such as a mobile station modem (MSM). In a particular aspect, the processor108, the processors1510, the display controller1526, the memory106, the CODEC1534, and the transceiver1540are included in a system-in-package or the system-on-chip device1522. In a particular aspect, an input device1530, such as a touchscreen and/or keypad, and a power supply1544are coupled to the system-on-chip device1522. Moreover, in a particular aspect, as illustrated inFIG. 15, the visual interface device140, the input device1530, the speakers1536, the microphones142, the antenna1542, and the power supply1544are external to the system-on-chip device1522. However, each of the visual interface device140, the input device1530, the speakers1536, the microphones142, the antenna1542, and the power supply1544can be coupled to a component of the system-on-chip device1522, such as an interface or a controller. In an illustrative example, the device1500corresponds to a virtual assistant, a home appliance, a smart device, an internet of things (IoT) device, a communication device, a computer, a display device, a television, a gaming console, a music player, a radio, a video player, an entertainment unit, a personal media player, a digital video player, a camera, a navigation device, a mobile communication device, a smartphone, a cellular phone, a laptop computer, a tablet computer, a personal digital assistant, a display device, an optical disc player, a tuner, a decoder system, an encoder system, or any combination thereof.

In conjunction with the described aspects, an apparatus is disclosed that includes means for receiving real-time status information associated with one or more occupants of the building. The status information includes at least one of dynamic location information or dynamic activity information. For example, the means for receiving the real-time status information may correspond to the receiver222, the processor108, the transceiver1540, one or more other circuits or devices configured to receive the status information, or any combination thereof.

The apparatus includes means for providing a visual representation of at least a portion of the building and the status information associated with the one or more occupants. For example, the means for providing may correspond to the visual interface device140, the processor108, the output generator260, one or more other circuits or devices configured to provide the visual representation, or any combination thereof.

The apparatus includes means for receiving user input regarding control of one or more controllable devices of the building. For example, the means for receiving the user input may correspond to the microphone142, the speech recognition engine118, the touchscreen246, the touch recognition unit220, the sensor244, the gesture recognition unit218, the input device1530, the processor108, the input recognition engine302, one or more other circuits or devices configured to receive the user input, or any combination thereof.

The apparatus includes means for generating an instruction to adjust operation of the one or more controllable devices based on the user input. For example, the means for generating the instruction may correspond to the processor108, the instruction generator262, one or more other circuits or devices configured to generate the instruction, or any combination thereof.

In some implementations, the user input is received via a means for performing a speech recognition operation, such as the speech recognition engine118, the processor108, the input recognition engine302, one or more other circuits or devices configured to perform the speech recognition operation, or any combination thereof.

In some implementations, the user input is received via at least one of a means for performing a gesture recognition operation or a means for performing a touch detection operation. For example, the means for performing the gesture recognition operation may correspond to the sensor244, the gesture recognition unit218, the processor108, the input recognition engine302, one or more other circuits or devices configured to perform the gesture recognition operation, or any combination thereof. As another example, the means for performing the touch recognition operation may correspond to the touchscreen246, the touch recognition unit220, the processor108, the input recognition engine302, one or more other circuits or devices configured to perform the touch recognition operation, or any combination thereof.

In some implementations, the apparatus includes means for selectively restricting at least one of control of the controllable devices or access to at least a portion of the status information based on whether a user is an authorized user. For example, the means for selectively restricting may correspond to the processor108, the user verifier264, one or more other circuits or devices configured to selectively restrict at least one of control of the controllable devices or access to at least a portion of the status information based on whether a user is an authorized user, or any combination thereof.

As used herein, “coupled” may include communicatively coupled, electrically coupled, magnetically coupled, physically coupled, optically coupled, and combinations thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., a wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive electrical signals (digital signals or analog signals) directly or indirectly, such as via one or more wires, buses, networks, etc.

As used herein, “generating,” “calculating,” “using,” “selecting,” “accessing,” and “determining” may be used interchangeably. For example, “generating,” “calculating,” or “determining” a value, a characteristic, a parameter, or a signal may refer to actively generating, calculating, or determining a value, a characteristic, a parameter, or a signal or may refer to using, selecting, or accessing a value, a characteristic, a parameter, or a signal that is already generated, such as by a component or a device.

The previous description of the disclosed aspects is provided to enable a person skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.