Control device, display device, method, and program

There is provided a control device to improve convenience for a user by resolving or alleviating a disadvantage of a known voice interaction, the control device including: a device control unit configured to control one or more controlled devices; a voice notification unit configured to output user-oriented voice notification regarding at least the one controlled device; and a display control unit configured to cause a display device to display a message corresponding to the voice notification output by the voice notification unit.

CROSS REFERENCE TO PRIOR APPLICATION

This application is a National Stage Patent Application of PCT International Patent Application No. PCT/JP2016/079517 (filed on Oct. 4, 2016) under 35 U.S.C. § 371, which claims priority to Japanese Patent Application No. 2016-001670 (filed on Jan. 7, 2016), which are all hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to a control device, a display device, a method, and a program.

BACKGROUND ART

In the related art, technologies for utilizing voice for user interfaces in remote controllers controlling controlled devices in environments are known. For example, Patent Literature 1 discloses a technology for recognizing a voice command as a user input. Patent Literature 2 discloses a technology for outputting voice guidance for a manipulation by a user.

CITATION LIST

Patent Literature

DISCLOSURE OF INVENTION

Technical Problem

With advances in information processing technologies, interactions between machines and users have been further improved and diversified. However, there is still room for improvement in known voice interaction structures in view of reliability, timeliness, or selectivity of information delivery.

Solution to Problem

According to the present disclosure, there is provided a control device including: a device control unit configured to control one or more controlled devices; a voice notification unit configured to output user-oriented voice notification regarding at least the one controlled device; and a display control unit configured to cause a display device to display a message corresponding to the voice notification output by the voice notification unit.

In addition, according to the present disclosure, there is provided a display device to be connected to a control device controlling one or more controlled devices and outputting user-oriented voice notification regarding at least the one controlled device, the display device including: a display control unit configured to cause a display unit to display a message corresponding to the voice notification output by the control device in accordance with an instruction received from the control device.

In addition, according to the present disclosure, there is provided a control method performed by a control device controlling one or more controlled devices, the method including: outputting user-oriented voice notification regarding at least the one controlled device; and causing a display device to display a message corresponding to the output voice notification.

In addition, according to the present disclosure, there is provided a display control method performed by a display device connected to a control device controlling one or more controlled devices and outputting user-oriented voice notification regarding at least the one controlled device, the method including: causing a display unit to display a message corresponding to the voice notification output by the control device in accordance with an instruction received from the control device.

In addition, according to the present disclosure, there is provided a program causing a processor of a control device to function as: a device control unit configured to control one or more controlled devices; a voice notification unit configured to output user-oriented voice notification regarding at least the one controlled device; and a display control unit configured to cause a display device to display a message corresponding to the voice notification output by the voice notification unit.

In addition, according to the present disclosure, there is provided a program causing a processor of a display device connected to a control device controlling one or more controlled devices and outputting user-oriented voice notification regarding at least the one controlled device, to function as: a display control unit configured to cause a display unit to display a message corresponding to the voice notification output by the control device in accordance with an instruction received from the control device.

Advantageous Effects of Invention

According to the technology of the present disclosure, it is possible to further improve convenience for a user by mixing a voice interaction and a visual interaction in a preferred form. Note that the effects described above are not necessarily limitative. With or in the place of the above effects, there may be achieved any one of the effects described in this specification or other effects that may be grasped from this specification.

MODE(S) FOR CARRYING OUT THE INVENTION

In addition, the description will be made in the following order.

1. Overview of system

1-1. Example of assumed environment

1-3. Several problems

1-4. Introduction of auxiliary agent

2. Configuration example of controller

3. Configuration example of auxiliary agent

4. Flow of processes

4-1. Process of controlled device

4-2. Process of controller

4-3. Process of auxiliary agent

5. Cooperation with external server

1. OVERVIEW OF SYSTEM

[1-1. Example of Assumed Environment]

FIG. 1is an explanatory diagram illustrating an example of an environment in which an interaction system according to the present disclosure can be used. Referring to the left half ofFIG. 1, an environment1includes, for example, a living room of a home in which a user lives. In the living room, there are controlled devices10a,10b, and10c. The controlled device10ais an air conditioner that adjusts an indoor temperature of the living room. The controlled device10bis a television device that includes a display to display content of television broadcast. The controlled device10cis a curtain10cthat has an automatic opening and closing function. Referring to the right half ofFIG. 1, controlled devices10dand10einstalled outside of the home are further included in the environment1. The controlled device10dis a smart lock that manages locking and unlocking of a door. The controlled device10eis an exterior lamp that emits illumination light. Note that the controlled devices are not limited to the examples ofFIG. 1. In the environment1, there may be any kind of controlled device. In the following description, in a case in which it is not necessary to distinguish the controlled devices10ato10efrom each other, the controlled devices10ato10eare collectively referred to as the controlled devices10by omitting the letters suffixed to the reference numerals. The meaning of the omission of the letters suffixed to the reference numerals applies to other constituent elements.

An interaction system according to the present disclosure is used in such an environment in which there are one or more controlled devices10. AlthoughFIG. 1illustrates the shape of the home in which the user lives as an example of the environment, the interaction system according to the present disclosure may be used in any other environment such as an automobile, another vehicle, an office, a store, an education facility, or a public facility. In addition, one interaction system may be in a plurality of geographically separated environments (for example, a home and an automobile or a home and an office) in an integrated manner

Referring toFIG. 1, a communication device20and a control device30which is an example of a device which can be included in the interaction system are further illustrated. The communication device20relays communication between the controlled device10and the control device30. The communication device20may be, for example, any type of communication device such as a wireless local area network (LAN) access point, a router, or a LAN switch. In a case in which the control device30performs direct communication with the controlled device10(for example, via an ad hoc network or via a connection line in a device or between devices), the communication device20may not be included.

The control device30has a function of controlling one or more controlled devices10. In the present specification, a logical function of controlling the controlled device is referred to as a “controller.” In the example ofFIG. 1, the control device30includes a controller100. For example, the controller100controls the air conditioner10asuch that an indoor temperature is adjusted to a temperature appropriate for the user. In addition, the controller100controls the television device10bsuch that video content desired by the user is displayed, controls the curtain10bsuch that desired brightness of the room is realized, controls the smart lock10din accordance with a predetermined security rule, and controls ON/OFF or intensity of illumination light of the exterior lamp10e.

The controlled device10periodically reports a status associated with the controlled device10to the controller100in response to a request from the controller100or when a certain trigger condition is satisfied. For example, a status of the air conditioner10acan include an environment status such as a temperature measured by a temperature sensor and humidity measured by a humidity sensor in addition to an operation status of the device such as a target temperature, a wind volume, and a wind direction. In addition, in a case in which a certain event is detected, the controlled device10reports occurrence of the event to the controller100. For example, the television device10bmay report an approach to a broadcast time of a broadcast program to be recommended to a user as an event. In addition, the smart lock10dmay report locking or unlocking as an event.

The controller100may automatically control the controlled device in accordance with a status acquired from the controlled device or a sensor (not illustrated) and/or a control condition stored in advance. In addition, the controller100may control the controlled device in accordance with a user input acquired via a user interface. A control signal exchanged between the controller100and the controlled device10may be transmitted via a wireless channel such as infrared rays, visible rays, or radio waves or may be transmitted via a wired channel.

FIGS. 2A to 2Cillustrate examples of a relation between the controller and the controlled devices. In a first example illustrated inFIG. 2A, the controller100is a module that is mounted on the control device30separate from the controlled devices10aand10b. The controller100controls each of the controlled devices10aand10band receives a status report or an event report from the controlled devices10aand10b. In a second example illustrated inFIG. 2B, the controller100is mounted as a module that is contained in the controlled device10b. The controller100controls the controlled device10band monitors a status associated with the controlled device10b. In a third example illustrated inFIG. 2C, the controller100is mounted as a module that is contained in the controlled device10b. Further, the controlled device10bcommunicates with the controlled device10a. The controller100in the controlled device10bcontrols the controlled device10aand receives a status report or an event report from the controlled device10ain addition to controlling the controlled device10band monitoring the status associated with the controlled device10b. Note thatFIGS. 2A to 2Cillustrate the examples in which the controller100is mounted as a functional module that operates alone. An example in which functionality of the controller100is provided in cooperation with an external module (for example, an external server) will be further described later.

In an embodiment, the controller100performs an interaction with a user via a voice interface. Typically, the controller100outputs user-oriented voice notification regarding at least one controlled device10. The user can comprehend a status associated with the controlled device10, comprehend occurrence of an event, or determine whether control proposed by the controller100is approved by hearing the voice notification output from the controller100. The voice interface has better advantages than other types of user interfaces in that visual attention of the user is not necessary, even a user with low IT skills can easily become familiar with the voice interface, and a pseudo interaction similar to a human interaction through dialogue is enabled. Means for a user input to the controller100from the user may also be a voice interface. Instead of or in addition to the voice interface, the means for a user input may also be another type of user interface (for example, a mechanical interface such as a button or a switch, a touch interface, or a gesture interface).

The voice interface can be said to have the above-described good advantages, but also has several problems. First, there is a possibility of a user not being within a range in which voice arrives at the time of output of voice notification. In addition, even if a user is within a range at which voice arrives, the user may fail to hear voice notification or the user may not desire to respond to the voice notification at that time. Second, although repeatedly outputting voice notification of the same content does increase the probability that the content will be delivered to a user, a user who intentionally refrains from responding may be annoyed by the repetition. Third, for example, when much information accumulated while a user is away is output as voice notification, the user may be burdened by the long duration necessary for notification. In addition, only some of the long notification may be content which the user wants to know, but it is difficult for the user to simply select and gain only voice notification desired by the user.

[1-4. Introduction of Auxiliary Agent]

In an embodiment of the technology according to the present disclosure, additional functionality called an “auxiliary agent” in the present specification is introduced to resolve or at least reduce one or more of the above-described problems which can occur when the controller controlling the controlled devices provides a voice interface. The auxiliary agent assists the controller by providing a visual user interface to compensate for inconvenience of the voice interface provided by the controller. Note that the controller has a role of an agent itself in that control of a controlled device is relayed between the user and the controlled device. In addition, as will be described below, the auxiliary agent can also have a function of controlling a controlled device. Accordingly, the terms “controller” and “agent” in the present specification do not limit the technology according to the present disclosure and are used to separately refer to only two types of functionality for convenience.

Typically, the auxiliary agent causes the display device to display a message corresponding to voice notification output by the controller. The auxiliary agent may be mounted on the same device as a controller or may be mounted on another device communicating with the controller via a communication interface.

FIGS. 3A to 3Eillustrate examples of disposition of an auxiliary agent. In a first example illustrated inFIG. 3A, both the controller100and an auxiliary agent200are mounted to be integrated on the control device30separate from the controlled devices10. The auxiliary agent200can cause a display device40ato display a message on a screen. Here, the display device40ais a projector and the screen is a projection plane on a table.

In a second example illustrated inFIG. 3B, both the controller100and the auxiliary agent200are mounted to be integrated on the controlled device10b. The auxiliary agent200can cause the display device40bto display a message. Here, the display device40bis equivalent to a display of a television device.

In a third example illustrated inFIG. 3C, the auxiliary agent200is mounted on the controlled device10bseparate from the control device30containing the controller100, and the control device30(the controller100) and the controlled device10b(the auxiliary agent200) communicate with each other via communication interfaces35aand35b, respectively. The auxiliary agent200can cause the display device40bto display a message.

In a fourth example illustrated inFIG. 3D, the auxiliary agent200is mounted on a display terminal40cwhich is a device different from the control device30(the controller100) and the controlled device10. The control device30(the controller100) and the display terminal40c(the auxiliary agent20X)) communicate with each other via each communication interface. The auxiliary agent200can cause the display terminal40cto display a message.FIG. 3Dillustrates a tablet personal computer (PC) as an example of the display terminal40c, but a display terminal on which the auxiliary agent can be mounted is not limited to the foregoing example. For example, the auxiliary agent may be mounted on any other type of terminal such as a desktop PC, a laptop PC, a smartphone, a game terminal, a car navigation device, a head-mounted display, or a smartwatch.

In a fifth example illustrated inFIG. 3E, there are a plurality of auxiliary agents200a,200b,200c, and200d. The auxiliary agent200ais mounted to be integrated with the controller100on the control device30. The auxiliary agent200bis mounted on the controlled device10bthat is fixedly installed. The auxiliary agent200cis mounted on the display terminal40cwith mobility. The auxiliary agent200dis mounted on a display terminal40dwith mobility. Note that, in the present specification, the expression “with mobility” is assumed to include various movement forms such as carrying of a target terminal, mounting on a moving user, or implantation into the body of a moving user. The auxiliary agent may be mounted on an autonomous mobile machine (for example, a robot) that autonomously performs an operation such as walking, running, or flying.

Irrespective of disposition of the auxiliary agent200, as described above, the auxiliary agent200causes the display device to display a message corresponding to at least one user-oriented voice notification in a case in which the user-oriented voice notification is output by the controller100. Even in a case in which a user is located at a place distant from the controller100, the user can immediately confirm content of a message if a screen is near the user. In addition, the user can confirm the display content of the message at a desired timing or can respond to the content at a desired timing. In addition, since the auxiliary agent200provides a visual user interface as alternative means of the voice interface of the controller100, the controller100may not excessively repeat the voice notification. The visual user interface provided by the auxiliary agent200can display much information in various forms. Such a mixture of the voice interaction and the visual interaction can lead to an improvement in reliability, timeliness, or selectivity of information delivery, thereby improving convenience for the user. An example of such a more detailed configuration of the controller100and the auxiliary agent200will be described in detail in the next section.

2. CONFIGURATION EXAMPLE OF CONTROLLER

FIG. 4is a block diagram illustrating an example of a configuration of a logical function of the controller100according to an embodiment. The controller100includes a voice notification unit110, a voice recognition unit120, an image recognition unit130, a device control unit140, a storage unit150, and a display control unit160. In the example ofFIG. 4, the voice notification unit110is connected to a speaker31. The voice recognition unit120is connected to a microphone32. The image recognition unit130is connected to a camera33. The device control unit140is connected to one or more sensors34, the communication interface35, one or more input interfaces36, and an infrared interface37. The display control unit160is connected to the communication interface35, the display device40, and an announcement device41. The display device40may be, for example, a projector or a monitor. The announcement device41may be, for example, a light emitting diode (LED) lamp or a vibrator.

(1) Voice Notification Unit

The voice notification unit110and the voice recognition unit120have a role of performing voice interaction with a user by the controller100. Typically, the voice notification unit110outputs user-oriented voice notification regarding at least one controlled device10controlled by the controller100via the speaker31. For example, the voice notification unit110may generate voice notification using an automatic voice synthesis technology on the basis of notification text designated by the device control unit140. Instead, the voice notification unit110may acquire voice data corresponding to a notification number designated by the device control unit140or another piece of identification information from the storage unit150and may reproduce voice notification from the acquired voice data.

The voice notification output by the voice notification unit110may not be said to be necessarily heard and received by a target user. The voice notification unit110may repeat the same voice notification a plurality of times. Here, in the embodiment, after the voice notification is output by the voice notification unit110, a message corresponding to the voice notification is displayed by the auxiliary agent200. Therefore, the voice notification unit110does not repeat the same voice notification the excessive number of times.

In a case in which the voice recognition unit120or the image recognition unit130to be described below or another human detection sensor detects that there is a user nearby, the voice notification unit110may output additional voice notification for prompting the user to see the message (hereinafter referred to as reminding notification) when the auxiliary agent200is able to display the message corresponding to the previously output voice notification. Here, the message may be already displayed by the auxiliary agent200or may be displayed using any user input as a trigger. In addition, in a case in which the user hearing the reminding notification requests to receive the output voice notification (for example, by a voice command or another user input) instead of seeing the displayed message, the voice notification unit110may output the output voice notification again via the speaker31.

(2) Voice Recognition Unit

The voice recognition unit120recognizes content of a voice input via the microphone32using a voice recognition technology. For example, the voice recognition unit120recognizes a voice command spoken by the user. The voice command recognized by the voice recognition unit120can include a voice command for controlling the controlled device10. Typically, the voice command for controlling the controlled device10is formed by a name for identifying a target device and a keyword associated with control content (for example, “television”+“switch on”). In a case in which the controller100proposes specific control for the controlled device10, a voice command which is a response to the proposal may be formed by only a simple keyword with a meaning of an approval or a refusal. In addition, the voice command recognized by the voice recognition unit120may include a voice command directed to the controller100. For example, the voice recognition unit120may recognize a voice command indicating that the user requests to re-output the voice notification. In addition, the voice recognition unit120may recognize a voice command indicating that the user explicitly requests that the voice notification is transmitted to the auxiliary agent200. The voice recognition unit120can recognize an individual voice command, for example, by comparing a pattern of the voice command stored in advance by the storage unit150with a pattern of input voices.

Further, the voice recognition unit120may identify a user speaking a voice by comparing a spectrum obtained by analyzing a voice signal with a voiceprint of the individual user. In addition, the voice recognition unit120may merely recognize presence of the user located nearby without identifying an individual user.

(3) Image Recognition Unit

The image recognition unit130recognizes a situation of the environment1shown in a captured image input from the camera33using an image recognition technology. For example, the image recognition unit130may identify an individual user by recognizing the face of the user speaking a voice toward the controller100. In addition, the image recognition unit130may merely recognize presence of the user located nearby. In addition, the image recognition unit130may recognize a gesture command for controlling the controlled device10or a gesture command directed to the controller100by reading a gesture acted by the user. The gesture command may be a command corresponding to a motion or a shape of a hand or a finger of the user moved above, for example, a display image projected by a projector. The image recognition unit130can identify the user and recognize the presence of the user or recognize the gesture command, for example, by comparing an image feature amount stored in advance by the storage unit150with an image feature amount extracted from a captured image.

(4) Device Control Unit

The device control unit140controls one or more controlled devices10. According to a certain embodiment, the device control unit140controls the controlled device10acted on an environment status input from the sensor34so that the environment status satisfies a control target value. The environment status may be received from the controlled device10via the communication interface35. Hereinafter, examples of combinations of environment statuses and kinds of corresponding controlled devices are listed:an indoor temperature: an air conditioner, a heating device, and a cooling device;indoor humidity: air conditioner, humidifier, and dehumidifier;an environment volume: a television device and a media player;an environment illuminance: an illumination device, and a curtain; anda house dust-amount: an air cleaner.

According to another embodiment, the device control unit140decides a control target value or a target operation state of the controlled device10from the environment status acquired from the sensor34or the controlled device10and controls the controlled device10so that the decided target value or the target state is achieved. Hereinafter, examples of combinations of the environment status, kinds of corresponding controlled devices, and content of the control are listed:an indoor temperature: a water heater (water temperature);environment illuminance: a display (luminance) and a projector (luminance); anda house dust-amount: an autonomous cleaner (level, On/Off. and activity place).

According to still another embodiment, the device control unit140controls the controlled device10in a case in which a control condition in which a date or a time is dynamically learned or stored in advance in the storage unit150is satisfied. Hereinafter, examples of combinations of kinds of controlled devices and content of the control are listed:a television device: reproduction of a specific channel;a recorder: recording of images/sounds of contents;a curtain: opening or closing;an exterior lamp: On/Off and colors (for example, Christmas colors); andan automobile: starting an engine at time of attendance.

According to still another embodiment, the device control unit140controls the controlled device10depending on presence of a user determined on the basis of a recognition result input from the voice recognition unit120or the image recognition unit130. Hereinafter, examples of combinations of kinds of presence of a user, kinds of corresponding controlled devices, and content of the control are listed:a user returning back/going outside: an air conditioner (On/Off);a user returning back: a water heater (On); anda user absent: a smart lock (security level: high).

Note that such examples do not limit the content of the control to which the technology according to the present disclosure can be applied. The controller100may control any kind of controlled device10in accordance with any scheme.

The device control unit140instructs the voice notification unit110to output voice notification to a user on the basis of a report received from the controlled device10via the communication interface35. For example, the device control unit140may periodically receive a report of an environment status and/or an operation status from the controlled device10. In a case in which a predetermined user input (for example, a voice command, a gesture command, or any input from the input interface36) is detected, the device control unit140may request the controlled device10to give a status report. When a status report or an event report is received from the controlled device10, the device control unit140causes the voice notification unit110to output voice notification for delivering the reported status or event to the user.

The device control unit140may output the voice notification from the voice notification unit110only in a case in which a predetermined notification condition is satisfied. For example, the notification condition of the voice notification can be associated with one or more of the parameters listed below:an operation status of a controlled device;an environment status;presence of user;presence of specific user; anddate and/or time.

In the embodiment, a few of the voice notification instructed to be output from the device control unit140include content for requesting the user to respond. For example, the device control unit140starts controlling the controlled device10and requests the user to approve the control operation through voice notification before the control is changed or ends. Then, the device control unit140performs the action in a case in which the control operation is approved and does not perform the action in a case in which the control operation is rejected. In addition, the device control unit140may propose a control operation predicted to be preferable for the user to the user through voice notification. Content of the control operation preferable for the user may be set by the user and may be stored by the storage unit150. Instead, a control operation preferable for the user may be learned on the basis of a history of instructions or approvals from the user. The device control unit140performs the proposed control operation in a case in which the proposal is approved and does not perform the proposed control operation in a case in which the proposal is rejected.

In a case in which the control operation is approved or a control operation which it is not necessary for the user to approve is started, the device control unit140transmits a control signal to the target controlled device10via the communication interface35or the infrared interface37. The controlled device10operates in accordance with the received control signal. The controlled device10may report an operation result to the controller100.

After the device control unit140instructs the voice notification unit110to output the voice notification, the device control unit140further performs a visual interaction corresponding to the voice notification in cooperation with the display control unit160. The visual interaction corresponding to the voice notification will be described in detail later.

(5) Storage Unit

The storage unit150stores various kinds of data necessary for control of the controlled device10by the controller100and an interaction with the user. For example, the storage unit150stores user information which can include user identification information (an identifier, a nickname, and the like), user attribute information (an age, a sex, a preference, and the like), and setting information according to users (for example, a message selection condition and a display order). In addition, the storage unit150stores various kinds of control setting such as a notification condition for the voice notification and a control condition for controlling the controlled device10. Further, the storage unit150may store voice data for the voice notification output by the voice notification unit110. In addition, the storage unit150may store dictionary data for voice recognition by the voice recognition unit120. In addition, the storage unit150may store image feature amount data for image recognition by the image recognition unit130.

(6) Display Control Unit

The display control unit160causes the auxiliary agent200to display a message corresponding to the voice notification output by the voice notification unit110. In the embodiment, the display control unit160causes the auxiliary agent200to display a message corresponding to the voice notification after a certain time interval elapses from output of the voice notification. An instruction transmitted from the display control unit160to the auxiliary agent200may include a time stamp indicating a time at which the message is displayed. Instead, the instruction transmitted from the display control unit160to the auxiliary agent200may include a time stamp indicating a time at which the voice notification is output and information indicating the above-described time interval. In addition, in a case in which the instruction is transmitted after the elapse of the above-described time interval, the time information may not be included in the instruction. The instruction transmitted from the display control unit160to the auxiliary agent200may include another piece of information such as priority of the message.

FIG. 5Aillustrates an example of the voice notification. Referring toFIG. 5A, voice notification N11dedicated to a user U1is output from the controller100. The voice notification N11has content for proposing to watch broadcast contents recommended for the user U to the user U1. The user U1can approve or reject the proposal through the voice notification N11by a voice command or another user input. Here, the user U1does not respond to the voice notification N11since another work preoccupies the user U1or a volume of the voice notification N11is not sufficient.

FIG. 5Billustrates a first example of display of a message corresponding to voice notification performed after a certain time interval elapses from the output of the voice notification N11. Here, the auxiliary agent200is in the same device as the controller100and displays a message using the display device40. With reference toFIG. 5B, the auxiliary agent200displays a message M11corresponding to the voice notification N11in accordance with an instruction from the controller100. The controller100outputs new voice notification N12instead of repeating the voice notification N11.

FIG. 5Cillustrates a second example of display of a message corresponding to voice notification performed after a certain time interval elapses from an output of the voice notification N11. Here, the auxiliary agent200is in a device separate from the controller100and receives a message from the controller100via a communication interface. Referring toFIG. 5C, the auxiliary agent200displays a message M13corresponding to the voice notification N11on a display terminal used by the user U1in accordance with an instruction from the controller100.

The display control unit160may cause the auxiliary agent200to display a plurality of messages respectively corresponding to a plurality of pieces of output voice notification. In addition, the display control unit160may make an announcement for prompting the user to see a displayed message using the announcement device41. The announcement for prompting confirmation of a message may be, for example, vibration of a vibrator or turning on or flicking an LED lamp with a specific color or pattern.

FIG. 5Dillustrates an example of an announcement for prompting confirmation of a message corresponding to the voice notification. Here, the auxiliary agent200is in the same device as the controller100. Referring toFIG. 5D, the auxiliary agent200displays the message M11corresponding to the voice notification N11and a message M12corresponding to the voice notification N12in accordance with an instruction from the controller100. The controller100outputs reminding notification N14by voice. In addition, the auxiliary agent200announces presence of a message indicating a wait of a response by the user by turning on an LED lamp41a.

FIG. 6is an explanatory diagram illustrating a first scenario for display of a message corresponding to voice notification. In the first scenario, voice notification N21is first output from the controller100at time t21. Subsequently, at time t22after a time interval T1elapses from time t21, a message M21corresponding to the voice notification N21is displayed by the auxiliary agent200. The message M21disappears at time t23after time t22since a user input is detected. The reason why the message M21disappears is that a notification condition may not be satisfied due to expiration of a display period decided in advance or a change in a status to be described below.

In a case in which voice notification with content for requesting a response of a user is output, the display control unit160may cause the auxiliary agent200not to display a message corresponding to the voice notification when the response of the user is detected before the elapse of the time interval.FIG. 7is an explanatory diagram illustrating a second scenario for display of a message corresponding to the voice notification. Even in the second scenario, the voice notification N21is output from the controller100at time t21. Subsequently, before a time t22corresponding to an elapsed time point of the time interval T1comes, the user U1issues a voice command C21in response to the voice notification N21at time t24. In this case, the display control unit160comprehends that the necessity for displaying the message M21corresponding to the voice notification N21is lost now through recognition of the voice command. Thus, even when time122comes, the display control unit160causes the auxiliary agent200not to display the message M21.

The display control unit160may cause the auxiliary agent200not to display a message corresponding to the voice notification output in a case in which a status regarding the controlled device10satisfies a predetermined notification condition, in a case in which the status does not satisfy the notification condition before the time interval elapses.FIG. 8is an explanatory diagram illustrating a third scenario for display of a message corresponding to the voice notification. Even in the third scenario, the voice notification N21is output from the controller100at time t21. The voice notification N21has content for proposing that the user U1watches specific broadcast contents in the controlled device10band is output under the condition that the broadcast content is not being reproduced. Subsequently, before time t22corresponding to an elapsed time point of the time interval T1comes, the user U1manipulates the controlled device10bwithout using the controller100to start watching recommended broadcast contents at time t25. In this case, the display control unit160now comprehends that the notification condition of the voice notification N21is not satisfied through a report from the controlled device10b. Thus, even when time t22comes, the display control unit160causes the auxiliary agent200not to display the message M21.

As described above, the auxiliary agent200can be mounted using any of various types of display device. For example, any type of display device is integrated with the same device as the controller100. Other types of display devices are devices separate from the controller100. The devices separate from the controller100can be further classified into fixed types of devices and devices with mobility. The display control unit160may set a time interval between output of the voice notification and display of a corresponding message in accordance with a type regarding such an installation location or mobility of the display device displaying a message. The type regarding the installation location can be classified into, for example, the same device, the inside of an environment, or the outside of the same environment. The type of the mobility can be classified into, for example, non-mobility (fixation) and mobility.

FIG. 9is an explanatory diagram illustrating a fourth scenario for display of a message corresponding to the voice notification. In the fourth scenario, there are three auxiliary agents200. One of the auxiliary agents is integrated with the controller100in a device including the display device40a. Another auxiliary agent is mounted on the fixed display terminal40binstalled in the same environment as that of the controller100. The remaining auxiliary agent is mounted on the display terminal (mobile terminal)40cwith mobility. At time t31, voice notification N31is first output from the controller100. Subsequently, at time t35after a time interval T32elapses from time t31, a message M31corresponding to the voice notification N31is displayed by the display device40a. Subsequently, at time t36after a time interval T33elapses from time t31, a message M32corresponding to the voice notification N31is displayed on the fixed display terminal40b. The time interval T33is longer than the time interval T32. In a case in which a user input is detected or the notification condition is not satisfied before the time interval T33elapses, the message M32may not be displayed. Subsequently, at time t37after the time interval T34elapses from time t31, a message M33corresponding to the voice notification N31is displayed on the mobile terminal40c. The time interval T34is longer than the time intervals T32and T33. In the case in which the user input is detected or the notification condition is not satisfied before the time interval T34elapses, the message M33may not be displayed. In this way, by displaying a message with a time difference in order from the display device nearer from the controller100, it is possible to reduce inconvenience that a message is excessively frequently displayed for a user who is not in a controlled environment or a user who is not interested in the control.

In a case in which there are the plurality of auxiliary agents200as in the fourth scenario, the display control unit160may select an auxiliary agent (or a display device) that displays a message under a predetermined selection condition.

Hereinafter, examples of the selection condition are listed along with description of assumed cases. Note that the selection conditions may be combined:selection condition 1: a type of display device,

a message corresponding to voice notification related to only a user who is in a specific environment can be displayed by a display device installed to be fixed to the environment;selection condition 2: a current position of a display device,

a message corresponding to voice notification related to only a user who is in a specific environment is not displayed on a mobile terminal located at a location distant from the environment;selection condition 3: an attribute of a user,

a message corresponding to voice notification related to security is not displayed on a mobile terminal of a user of an age less than a threshold;selection condition 4: presence of a user,

any message is not displayed on a display device which is in an environment in which there is no user;selection condition 5: authority of a user,

a message corresponding to voice notification related to a certain controlled device can be displayed only on a display device designated by a user who has authority involved in control of the controlled device; andselection condition 6: priority of a message,

a message corresponding to voice notification to which high priority is given can be displayed preferentially or forcibly on a display device which is used by all users or a specific user irrespective of another condition.

FIG. 10is an explanatory diagram illustrating a fifth scenario for display of a message corresponding to the voice notification. In the fifth scenario, there are three auxiliary agents200. One of the auxiliary agents is integrated with the controller100in a device including the display device40a. Another auxiliary agent is mounted on the mobile terminal40cwhich is used by a user U2. The remaining auxiliary agent is mounted on the mobile terminal40dwhich is used by the user U1. At time t41, voice notification N41is first output from the controller100. Subsequently, at time t45after a time interval T42elapses from time t41, the message M41corresponding to the voice notification N41is displayed by the display device40a. Subsequently, at time t46after a time interval T43elapses from time t41, a message M42corresponding to the voice notification N41is displayed on the mobile terminal40c. The time interval T43is longer than the time interval T42. On the other hand, the message corresponding to the voice notification N41is not displayed on the mobile terminal40dirrespective of the elapse of the time. The reason why the message corresponding to the voice notification N41is not displayed is that, for example, the user U1is not in a specific environment, the age of the user U1is less than the threshold, or the user U1does not have authority involved in the voice notification N41. In this way, by selectively deciding a device that displays a message corresponding to the voice notification under the predetermined selection condition, it is possible to suppress display of an unnecessary message and ensure safety associated with control of the controlled device.

In a case in which a user input (for example, a voice command) for requesting the auxiliary agent200to transmit notification is detected, the display control unit160may instruct the auxiliary agent200to display a message corresponding to the novice notification. In this case, irrespective of the above-described time interval, the message corresponding to the voice notification can be displayed by the auxiliary agent200. In a case in which a user is a situation in which the user may not return an appropriate response although the user hears the voice notification, a timing at which the user responds to the notification can be postponed utilizing the auxiliary agent20X).

3. CONFIGURATION EXAMPLE OF AUXILIARY AGENT

FIG. 11is a block diagram illustrating an example of a configuration of a logical function of the auxiliary agent200according to an embodiment. The auxiliary agent200includes a user interface unit210and a display control unit220. In the example ofFIG. 11, the user interface unit210is connected to one or more input interfaces46. The display control unit220is connected to the communication interface35and the display device40.

The user interface unit210has a role of performing an interaction with a user by the auxiliary agent200. For example, the user interface unit210detects a user input via the input interface46and outputs a user input signal to the display control unit220. The input interface46may include any input means such as a touch sensor, a keyboard, a keypad, a button, a switch, a camera, a microphone, a human detection sensor, a vibration sensor, or an attitude sensor. The user interface unit210may recognize a voice command using a voice recognition technology as the user input or may recognize a gesture command from a captured image, as in the controller100.

The display control unit220is connected to the controller110via a connection line inside the same device or the communication interface35. In a case in which voice notification is output toward a user by the controller100, the display control unit220causes the display device40to display a message corresponding to the voice notification in accordance with an instruction received from the controller100. In the embodiment, after a certain time interval elapses from output of voice notification, the display control unit220causes the display device40to display a message corresponding to the voice notification.

FIG. 12is an explanatory diagram illustrating an example of a basic configuration of a message window according to an embodiment. Referring toFIG. 12, a message window W1that has a plurality of message items for displaying a plurality of messages is illustrated. The message window W1has a tab format and includes a first tab61, a second tab62, and a third tab63which can be mutually switched and displayed. The first tab61is an “All Messages” tab for displaying all effective message items. The second tab62is a “Proposals” tab for displaying message items associated with proposal from the controller100to a user. The third tab63is a “Status” tab for displaying message items associated with statuses collected by the controller100.

Note that the configuration of the message window W1illustrated inFIG. 12is merely an example and a window that has another configuration may be provided. For example, the number of message items displayed in the message window may be the same as the number of controlled devices controlled or managed by the controller100(that is, one message item is provided for one controlled device). Thus, it is easy for a user to ascertain general situations of many controlled devices which are under environments while avoiding complication of display in the window. In addition, only one message item may be displayed for a plurality of controlled devices belonging to the same class (for example, one message item is provided for a plurality of illumination devices). In addition, a GUI item for inclusively returning an approval or a refusal in response to a plurality of proposing messages may be provided in the window. Content of the message may not match content of voice notification. For example, text of a message may be changed depending on a type of display device or an attribute of a target user. For junior users, text of a message may be processed or simplified so that the junior users can easily understand content of the message.

FIG. 13is an explanatory diagram illustrating an example of switching of tabs on the message window W1illustrated inFIG. 12. Referring toFIG. 13, the second tab62corresponding to the “Proposal” tab is displayed in the message window W1as a result obtained when a user touches an index portion of the second tab62. The second tab62includes message items51and53, that is, only message items for displaying messages of proposals from the controller100to the user. The user can immediately recognize only a message requesting a response of the user and return an appropriate response (for example, an approval or a refusal of the proposal) through the selective display. A proposal in which a response of the user is not detected despite elapse of a predetermined time duration from display of a message may be treated as being approved of the proposal, or the proposal may be treated as being rejected.

The display control unit220can decide a display order of the message items in the window in order of time stamps of corresponding messages in the above-described message window W1. Here, the time stamp may indicate a time at which a corresponding message is displayed or received (from the controller100) by the auxiliary agent200for the first time or may indicate a time at which corresponding voice notification is output from the controller100. In the example ofFIG. 12, the message items in the message window W1are lined up in a descending order of the time stamps (newer items are located higher).

The display control unit220may cause the display device40to selectively display messages received from the controller100instead of causing the display device40to display all the received messages in the window. For example, the display control unit220may select the message items to be displayed in the message window in accordance with one of selection conditions 1 to 6 described above or a combination of two or more of selection conditions 1 to 6 (that is, in accordance with a type or a current position of the display device displaying the messages, an attribute, presence, or authority of the user, or priority of messages) in relation to the display control unit160of the controller100.

The display control unit220may preferentially display a message in which high priority is set in the message window irrespective of the above-described display order.FIG. 14is an explanatory diagram illustrating an example of display of a message item with high priority. Referring toFIG. 14, a message item58is displayed in a form superimposed on the message items51and53in a message window W2. The message item58displays a message for notifying a user of an event indicating a door is unlocked in the controlled device10dwhich is a smart lock. The priority may be granted to each message on the basis of setting registered in advance in the controller100or the auxiliary agent200(for example, setting of each controlled device, each kind of voice notification, or each kind of message for each controlled device). When preferential or forcible display is performed, for example, a message which it is preferable for a user to necessarily confirm, for example, in terms of security or the like can be reliably delivered to the user.

FIG. 15Ais an explanatory diagram illustrating another example of the display of the message item. InFIG. 15A, the message window W1described with reference toFIG. 13is illustrated again. A menu item “History” in a popup menu65displayed by touching an icon of the message item51is touched by a user. In addition, as a result, a history window W3in which a history of messages regarding the controlled device10corresponding to the message item51is shown chronologically is displayed. By additionally providing a window in which only a message unique to one device is displayed, it is easy to also deliver details such as a change in a status in a specific controlled device or an event occurrence situation to the user while simply maintaining content of a main message window.

FIG. 15Bis an explanatory diagram illustrating another example of the display of the message item. InFIG. 15B, a message window W4is illustrated as an example which can be displayed by the auxiliary agent200mounted particularly on a mobile terminal. The message window W4has message items71,72, and73. The message items71,72, and73are not all the messages received from the controller100and are items selected by the display control unit220under a predetermined selection condition (for example, items set by the user so that the items are displayed even when the user goes outside). In addition, the message item71indicates that two proposals from the controller100are reserved. For example, when the user touches a button “Read” included in the message item71, two message items that respectively display content of the proposals can be loaded into the message window W4. When such selected messages are displayed, an uncomplicated interaction suitable for a behavior situation or the degree of interest of the user can be realized.

FIG. 16is an explanatory diagram illustrating an example of a chronological change in the display of the message window. Referring toFIG. 16, a message item81that has a latest time stamp is first displayed in a message window W51at time T=t51. In the message window W51, message items82and83are displayed in addition to the message item81. The message item81has content in which a proposal to the user with regard to control of the controlled device10ais indicated. The message item82has content in which a proposal to the user with regard to control of the controlled device10bis indicated.

At time T=t52, the user approves the proposal from the controller100via the message item82by touching a button in the message item82. In response to this user input, the message item82disappears from the message window W52and a new message item84is displayed instead on the message item81. The message item84has content indicating a change in a status of the controlled device10boccurring as a result obtained when the user approves the proposal in the message item82.

At time T=t53, for example, it is assumed that a notification condition of voice notification corresponding to the message item81is satisfied as a result obtained when the user manipulates the controlled device10a(without being involved in the auxiliary agent200). Then, the message item81disappears from the message window W52and a new message item85is displayed instead on the uppermost stage. The message item85has content indicating a change in a status of the controlled device10aoccurring as a result obtained when the user manipulates the controlled device10a.

At time T=t54, a new message item86is displayed on the uppermost stage using occurrence of a new event in the controlled device10das an opportunity. The message item86has content indicating a change in an operation status detected in the controlled device10das an event. In a case in which high priority is set in a status or an event regarding the controlled device10d, the message item86can be preferentially displayed in the message window in a mode in which the message item86can be distinguished from the other message items (a display attribute).

The message window may have a role of a portal used for the user to unitarily manipulate one or more controlled devices10in an environment. For example, in a case in which a predetermined user input on a message item in the message window is detected, the display control unit220may cause the display device40to display a remote control window for remotely manipulating the controlled device10identified by the message item.

FIG. 17is an explanatory diagram illustrating an example of transition from a message window to a remote control window On the upper left ofFIG. 17, the same message window W51as the message window illustrated inFIG. 16is illustrated again. The message window W51has message items81,82, and83. For example, when the user performs flicking on the message item81corresponding to the controlled device10a, the remote control window W52for manipulating the controlled device10ais displayed. When the user performs flicking on the message item82, a remote control window W53for manipulating the controlled device10bis displayed. When the user performs flicking on the message item83, a remote control window W54for manipulating the controlled device10cis displayed. When such a user interface is provided, the user can reject a proposal (or another notification) from the controller100) and the user can also perform desired control quickly using the proposal as an opportunity in a case in which the proposal of the control is offered from the controller100and the proposal is not appropriate.

4. FLOW OF PROCESSES

[4-1. Process of Controlled Device]

FIG. 18is a flowchart illustrating an example of a flow of a process performed by the controlled device10according to an embodiment. Referring toFIG. 18, the controlled device10first acquires environment statuses (step S110). For example, the air conditioner10acan acquire a status such as environment temperature and environment humidity. Note that all kinds of controlled devices10may not necessarily acquire environment statuses. Accordingly, step S110may be omitted.

Next, the controlled devices10perform autonomous operations (step S112). For example, the air conditioner10achanges temperature of sent air on the basis of a difference between the environment temperature and a target temperature in a case in which the air conditioner10ais operating. The television device10bstarts receiving, reproducing, or recording a target broadcast program on the basis of reservation data of watching reservation or recording reservation. The curtain10cis changed from a closed state to an open state or from the open state to the closed state on the basis of setting of a timer.

Subsequently, the controlled devices10determine whether a timing of a status report comes (step S114). The timing of the status report can be set in advance periodically or can be set to a specific time. In a case in which the controlled devices10determine that the timing of the status report comes, the controlled devices10transmit the environment statuses or the operation statuses to the controller100(step S116).

Subsequently, the controlled devices10determine whether an event report is performed (step S118). For example, the smart lock10dmay determine unlocking of a door during a period set by the user or a failure of a trial of the unlocking of the door by a certain person as an event to be reported to the controller100. In a case in which the controlled devices10determine that the event report is performed, the controlled devices10transmit the event report to the controller100(step S120).

Subsequently, the controlled devices10determine whether a control signal is received from the controller100or another device (step S122). In a case in which the control signal is received, the controlled devices10perform operations in accordance with the received control signal (step S124). For example, the exterior lamp10eis turned on in a case in which a control signal for giving an instruction to turn the exterior lamp10eon is received from the controller100.

[4-2. Process of Controller]

FIG. 19is a flowchart illustrating an example of a flow of a process performed by the controller100according to an embodiment. Referring toFIG. 19, the device control unit140first collects environment statuses and operation statuses from one or more controlled devices10and one or more sensors34(step S130). The device control unit140may acquire, for example, supplementary information such as an electronic program table or weather data in addition to the statuses from an external server.

Subsequently, the voice recognition unit120detects presence of a user by performing voice recognition and identifies the user or detects a voice command spoken by the user (step S132). In addition, the image recognition unit130can detect the presence of the user by performing image recognition and identify the user or detect a gesture command.

Subsequently, the device control unit140determines whether the user-oriented voice notification is output by determining a related notification condition with regard to each controlled device10(step S134). This determination can be based on indexes such as the environment statuses and operation statuses collected in step S130, the presence of the user acquired in step S132, and a date. When the device control unit140determines that one or more user pieces of—oriented voice notification are output, the device control unit140causes the voice notification unit110to output the pieces of voice notification in order via the speaker31(step S136).

Subsequently, the display control unit160determines whether messages corresponding to the voice notification output by the voice notification unit110is displayed by the auxiliary agent200(step S136). In a case in which the display control unit160determines what the messages are displayed by the auxiliary agent200, the display control unit160selects one or more display devices which are to display the messages (step S140) or the display control unit160decides the time interval of each of the selected display devices on the basis of, for example, the types of display devices, presence of the user, or priority of the messages (step S142).

Subsequently the display control unit160determines whether the time interval decided in step S142has elapsed with regard to the individual message to be displayed (step S144). In a case in which a response of the user to be requested is detected or the corresponding notification condition is not satisfied before the time interval elapses, the display control unit160withdraws the display of the messages. In a case in which the time interval has elapsed and the withdrawal of the display of the messages does not end (step S146), the display control unit160instructs the auxiliary agent200to display the messages corresponding to the voice notification (step S148).

Subsequently, the device control unit140determines whether the controlled devices10are controlled (step S150). For example, in a case in which a user input for giving an instruction to control the controlled devices10is detected, the device control unit140decides to control the controlled devices10. Here, the user input can be detected as a voice command, a gesture command, or another input by the controller100or can be detected through signaling (for example, an approval of proposal of a control operation) from the auxiliary agent200. When the device control unit140decides to control the controlled devices10, the device control unit140transmits a control signal to the controlled devices10(step S152).

[4-3. Process of Auxiliary Agent]

FIG. 20is a flowchart illustrating an example of a flow of a process performed by an auxiliary agent200according to an embodiment. Referring toFIG. 20, the display control unit220first receives a message display instruction from the controller100via the connection line in the same device or the communication interface35(step S210).

Subsequently, the display control unit220generates a message item corresponding to a message instructed to be displayed (after a time interval allocated to the message elapses) or updates an existing message item so that a newly instructed message is displayed (step S212).

Subsequently, the display control unit220forms a latest message window by disposing one or more valid message items in a window (step S214). For example, the display control unit220can filter messages to be displayed under a predetermined selection condition and rearrange the message items in a predetermined display order.

Subsequently, the display control unit220causes the display device40to display the message window formed by the one or more valid message items (updates the display in a case in which a message window has already been displayed) (step S220).

In addition, the user interface unit210monitors a user input via, for example, the input interface46(step S220). When the user input is detected by the user interface unit210, the display control unit220performs an operation in accordance with the detected user input (step S222). The operation performed herein may be any operation, for example, switching of tabs in the message window, display of a remote control window for manipulating the designated controlled device10, or signaling for delivering an approval or a refusal of proposal to the controller100.

5. COOPERATION WITH EXTERNAL SERVER

In the previous sections, the example in which the interaction system includes the single controller100and one or more auxiliary agents200has been described. However, the present disclosure is not limited to the example. The controller100may be embodied by a plurality of devices that cooperate with each other.

FIG. 21Ais an explanatory diagram illustrating a first example of cooperation with an external server. Referring toFIG. 21A, in addition to a controller100awhich is in the environment1, an external server5communicating with the controller100avia a network3is illustrated. Then, a controller100boperates on the external server5. The controller100aincludes, for example, the voice notification unit110, the device control unit140, and the display control unit160in the functional block of the controller100illustrated inFIG. 4. On the other hand, the controller100bincludes the voice recognition unit120and the image recognition unit130in the functional block of the controller100illustrated inFIG. 4. The controller100atransmits a voice signal acquired via the microphone32or voice feature data extracted from the voice signal to the controller100band entrusts recognition of a voice command to the controller100b. In addition, the controller100atransmits an image signal acquired via the camera33or image feature data extracted from the image signal to the controller100band entrusts image recognition to the controller100b. The controller100performs the entrusted voice recognition or image recognition using abundant processing resources and massive dictionary data and returns a performed result to the controller100a. The controller100acontrols an interaction with a user and the controlled devices10on the basis of the recognition result returned from the controller100b. Note that distribution of functions of the controllers100aand100bis not limited to the example described herein.

The controller100bmay have a role of controlling access from a terminal outside the environment1or to the controller100aor other devices in the environment1. In addition, a separate device (for example, a home gateway (not illustrated)) from the external server5may have a role of the access control.

FIG. 21Bis an explanatory diagram illustrating a second example of cooperation with an external server. Referring toFIG. 21B, the controller100awhich is in the environment1, the external server5communicating with the controller100a, and the controller100boperating on the external server5are illustrated again. In the second example, the controller100aincludes all the units in the functional block of the controller100illustrated inFIG. 4. The controller100balso includes all the units in the functional block of the controller100illustrated inFIG. 4. Here, processing resources and storage resources available in the controller100aare less than processing resources and storage resources available in the controller100b. For example, when a load of processes such as voice recognition, image recognition, control of the controlled devices, or display control exceeds a capacity, the controller100aentrusts some of the processes to the controller100b.When the load of the processes is low, the controller100aperforms the processes alone.

The embodiments of the technology according to the present disclosure have been described above with reference toFIGS. 1 to 21B. According to the foregoing embodiments, the user-oriented voice notification regarding at least one controlled device is output from the controller that controls one or more controlled devices and the message corresponding to the voice notification is displayed by the display device. Accordingly, even when the voice notification does not arrive at the user or the user fails to hear the voice notification, the user can confirm the content of the displayed message later and respond to the message as necessary. By guaranteeing such complementary interaction means, the controller that mainly performs a voice interaction may not excessively repeat the user-oriented voice notification. By mixing such a voice interaction and such a visual interaction, it is possible to improve reliability, timeliness, or selectivity of information delivery, and thus it is possible to improve convenience for a user.

In addition, according to the foregoing embodiments, a message corresponding to voice notification is displayed after a certain time interval elapses from output of the voice notification. Accordingly, a time suspension necessary for the user to take an appropriate action in response to the voice notification is ensured. Then, it is possible to avoid a situation in which a message corresponding to voice notification is unnecessarily displayed in a case which an objective of the voice notification is achieved because of a rapid response of the user or a change in statues of the controlled devices or effectiveness of content of voice notification becomes unnecessary.

In addition, according to the foregoing embodiments, the time interval until the individual display device displays the message can be decided dynamically in accordance with the type of display device that displays the message. Accordingly, for example, only in a case in which a user who is in an environment in which the controlled devices are installed first is caused to pay attention to notification but an appropriate response may not be obtained nevertheless, it is possible to realize a step-by-step interaction for attracting user's attention otherwise.

In addition, according to the foregoing embodiments, the message corresponding to the voice notification is displayed along with information for identifying the corresponding controlled device in the message window in which one or more message items are listed. Accordingly, the user can simply ascertain general situations of the plurality of controlled devices which are under environments through the message window. In addition, the user can later return a response which may not have been returned at the time of outputting the voice notification through a GUI item provided on a message item. In addition, by changing a message to be displayed in accordance with setting of each display terminal, it is possible to reduce a possibility that an important message or a message in which a user is particularly interested is missed.

Note that according to the foregoing embodiments, a main interaction between the controller and a user is realized through a voice interaction and a visual interaction is complementarily provided. However, the present disclosure is not limited to this example. Another type of interaction means (for example, subtitles for displaying information serially or a gesture interaction such as sign language) that has a similar nature to voice interaction may be adopted instead of the voice interaction. In the technology according to the present disclosure, it is possible to complement a defect of the other type of interaction means through a visual interaction.

The various processes described in the present specification may be realized using one of software, hardware, and a combination of software and hardware. A program of the software is stored in advance in, for example, a storage medium (a non-transitory media) that is provided inside or outside of each device. Then, for example, each program is read to a random access memory (RAM) at the time of execution and is executed by a processor such as a central processing unit (CPU).

In addition, the processes described with reference to the flowcharts in the present specification may not necessarily be executed in the orders indicated in the flowcharts. Some of the steps of the processes may be performed in parallel. In addition, additional steps of processes may be adopted or some of the steps of the processes may be omitted.

A control device including:

a device control unit configured to control one or more controlled devices:

a voice notification unit configured to output user-oriented voice notification regarding at least the one controlled device: and

a display control unit configured to cause a display device to display a message corresponding to the voice notification output by the voice notification unit.

The control device according to (1), in which the display control unit causes the display unit to display the message corresponding to the voice notification after a certain time interval elapses from the output of the voice notification.

The control device according to (2),

in which the voice notification includes content for requesting a response of a user, and

in a case in which the reaction is detected before the elapse of the time interval, the display control unit causes the display device not to display the message corresponding to the voice notification.

The control device according to (2) or (3),

in which the voice notification is output in a case in which a status regarding the controlled device satisfies a predetermined notification condition, and

in a case in which the status does not satisfy the predetermined notification condition before the elapse of the time interval, the display control unit causes the display device not to display the message corresponding to the voice notification.

The control device according to any one of (2) to (4), in which the display control unit sets the time interval in accordance with a type regarding an installation location or mobility of the display device displaying the message.

The control device according to any one of (1) to (5), further including:

a recognition unit configured to recognize presence of a user near the control device,

in which, in a case in which the recognition unit recognizes presence of the user, the voice notification unit outputs additional voice notification for prompting the user to see the message when the display device is able to display the message corresponding to the voice notification.

The control device according to any one of (1) to (6), in which the display device is a device separate from the control device.

The control device according to any one of (1) to (7), in which the one or more controlled devices include at least one of a first controlled device containing the control device and a second controlled device separate from the control device.

(9) A display device to be connected to a control device controlling one or more controlled devices and outputting user-oriented voice notification regarding at least the one controlled device, the display device including:

a display control unit configured to cause a display unit to display a message corresponding to the voice notification output by the control device in accordance with an instruction received from the control device.

The display device according to (9), in which the display control unit causes the display unit to display the message corresponding to the voice notification after a certain time interval elapses from the output of the voice notification by the control device.

The display device according to (9) or (10),

in which the display control unit causes the display unit to display a message window that has one or more message items, and

each message item displays information for identifying a corresponding controlled device and the message regarding the controlled device.

The display device according to (11), in which the display control unit selects a message item to be displayed in the message window in accordance with a type or current position of the display device, an attribute, presence, or authority of a user, or priority of the message.

The display device according to (11) or (12),

in which the message window has the message item corresponding to each of the one or more controlled devices, and

the display control unit decides a display order of the message items in the message window in order of time stamps of corresponding messages.

The display device according to (13), in which the display control unit causes a message in which high priority is set to be preferentially displayed in the message window irrespective of the display order.

The display device according to any one of (11) to (14), in which, in a case in which a predetermined user input for a certain message item in the message window is detected, the display control unit causes the display unit to display a remote control window for remotely manipulating a controlled device identified by using the message item.

The display device according to any one of (11) to (15), in which a number of message items displayed in the message window is the same as a number of controlled devices.

A control method performed by a control device controlling one or more controlled devices, the method including:

outputting user-oriented voice notification regarding at least the one controlled device; and

causing a display device to display a message corresponding to the output voice notification.

A display control method performed by a display device connected to a control device controlling one or more controlled devices and outputting user-oriented voice notification regarding at least the one controlled device, the method including:

causing a display unit to display a message corresponding to the voice notification output by the control device in accordance with an instruction received from the control device.

A program causing a processor of a control device to function as:

a device control unit configured to control one or more controlled devices;

a voice notification unit configured to output user-oriented voice notification regarding at least the one controlled device; and

a display control unit configured to cause a display device to display a message corresponding to the voice notification output by the voice notification unit.

A program causing a processor of a display device connected to a control device controlling one or more controlled devices and outputting user-oriented voice notification regarding at least the one controlled device, to function as:

a display control unit configured to cause a display unit to display a message corresponding to the voice notification output by the control device in accordance with an instruction received from the control device.

REFERENCE SIGNS LIST