Patent Description:
Home appliances are devices used for user convenience. The home appliances, such as an image display device, an air conditioner, a washing machine, a refrigerator, etc., which are used in specific spaces at home or office, perform their unique functions and operations by a user's manipulation. For example, the image display device, such as a TV, a monitor, or a projector, has the function of displaying images viewed by a user, and the user may view broadcast programs using the image display device. Further, the air conditioner has the function of heating or cooling the indoor air, the refrigerator has the function of keeping foods fresh, the washing machine has the function of washing the clothes, the cleaner has a cleaning function, cooking equipment has the function of cooking foods, and the like.

Generally, a user controls the home appliance by directly manipulating a button provided on a main body of the home appliance or by using a remote control device, such as a remote control and the like, to avoid inconvenience in moving to the main body. However, even when using the remote control device, it is also inconvenient for the user to check and manipulate a manipulation key for each function, and in a dark indoor space, it is difficult for the user to identify the manipulation key, such that a light-emitting device should be further provided for the remote control device or the user is required to manipulate a separate lighting device.

In addition, there is also a problem in that when wishing to control the home appliance, the user may not identify the position of a remote control device or the remote control device may be lost, such that there is no way of remotely controlling the home appliance. In order to solve the problem, research has been actively conducted recently on methods to control the home appliances by using a speech recognition technology.

Generally, when controlling the home appliance by using the speech recognition technology, the user activates a speech command receiving function by voice wake-up and the like, and then utters speech to input a command. In this case, by directly recognizing and processing the input speech, the home appliance may perform an operation according to a command corresponding to the speech, or upon transmitting speech to a server capable of processing speech, the home appliance may perform an operation according to a command corresponding to the speech received from the server.

However, even in the case where the speech recognition technology is used, upon completing a speech input from the user, the home appliance terminates the activated speech command receiving function and merely performs an operation according to a command corresponding to the input speech, such that every time the user wishes to control the home appliance, it is inconvenient for the user to reactivate the speech command receiving function of the home appliance by voice wake-up and the like, to utter a speech command.

For instance, <CIT> relates to an electronic device in a network and is capable of receiving a voice input via a microphone and is configured to extend an end point detection (EPD) time until when a voice input is expected, particularly when a last word of the voice input corresponds to a preset word. <CIT> relates to exchanging information between a user and an application. During such conversation a command and a keyword are detected that are spoken by the user, and an application is accessed in response to the detected command and information is retrieved from the application in response to the keyword. <NPL>", relates to contents of elements that induce a natural conversation between humans and computers.

It is an object of the present disclosure to solve the above and other problems.

It is another object of the present disclosure to provide a server and a system including the same, in which the server may automatically reactivate a speech command receiving function of a home appliance according to words included in speech input from a user, and may further receive the user's speech.

In accordance with an aspect of the present disclosure, the above and other objects can be accomplished by providing a server, according to claim <NUM>. For instance such server can include: a communicator configured to perform communication through a network; a storage configured to store data on at least one predetermined word; and a controller, wherein upon receiving an input signal, including data on speech, from a first electronic device through the communicator, the controller performs speech recognition of speech included in the input signal, and determines whether a last word of the speech corresponds to any one of the at least one predetermined word; in response to there being a word corresponding to the last word of the speech among the at least one predetermined word, the controller transmits a first response signal, including data on a response to the speech except for the last word and data including at least one query for inducing input of additional speech different from the speech, to the first electronic device through the communicator; and in response to there being no word corresponding to the last word of the speech among the at least one predetermined word, the controller transmits a second response signal, including the data on the response to the speech, to the first electronic device through the communicator. The data on the response to the speech includes information corresponds to an answer to the speech.

In accordance with another aspect of the present disclosure, the above and other objects can be accomplished by providing a system, according to claim <NUM>.

The additional range of applicability of the present disclosure will become apparent from the following detailed description. However, it should be understood that the detailed description and specific embodiments are merely given by way of example.

To clearly and briefly describe the present disclosure, a part irrelevant to the description will be omitted in the drawings, and the same or like reference numerals in the specification denote the same elements.

The suffixes "module" and "unit" of elements herein are used for convenience of description and do not have any distinguishable meanings or functions. Thus, the "module" and the "unit" may be used interchangeably.

It should be understood that the terms "comprise", "include", "have", etc. when used in this specification, specify the presence of stated features, integers, steps, operations, elements, components, or combinations of them but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

It will be understood that, although the terms "first", "second", etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element.

<FIG> is a diagram illustrating a system according to various embodiments of the present disclosure.

Referring to <FIG>, a system <NUM> includes at least one home appliance <NUM> and a server <NUM>. The system <NUM> may include a relay device, i.e., an access point (AP).

According to various embodiments of the present disclosure, the home appliance, electronic device, electronic equipment, and the like may be used interchangeably. While <FIG> illustrates an image display device 100a, an air conditioner 100b, an air purifier 100c, a washing machine 100d, and a refrigerator 100e as examples of the home appliance <NUM>, the present disclosure is not limited thereto.

The AP <NUM> may be a network assisted device which allows devices, communicating with the home appliance <NUM> and accessing a local network provided by the AP <NUM>, to mutually transmit and receive data or to access a network <NUM> such as the Internet.

The home appliance <NUM> may communicate with the server <NUM> via a network <NUM> provided by the AP <NUM>, and may transmit and receive signals, including data, with the server <NUM>. The server <NUM> may process data received from the home appliance <NUM>.

The home appliances <NUM> may mutually transmit and receive data via the local network provided by the AP <NUM>. The home appliances <NUM> may mutually transmit and receive data through short-range wireless communication, such as Wireless Fidelity (Wi-Fi), Bluetooth, Bluetooth Low Energy (BLE), ZigBee, Near Field Communication (NFC), and the like.

<FIG> is an internal block diagram illustrating an image display device as an example of the home appliance of <FIG>. The home appliance <NUM> according to various embodiments of the present disclosure may include all or part of the components illustrated in <FIG>, and may further include components according to each of unique functions and operations of the home appliance <NUM>.

Referring to <FIG>, the image display apparatus 100a as an example of the home appliance <NUM> may include a broadcast receiver <NUM>, an external device interface <NUM>, a network interface <NUM>, a storage <NUM>, a user input interface <NUM>, an input unit <NUM>, a controller <NUM>, a display <NUM>, an audio output unit <NUM>, and/or a power supply unit <NUM>.

The broadcast receiver <NUM> may include a tuner <NUM> and a demodulator <NUM>.

The tuner <NUM> may select a broadcast signal corresponding to a channel selected by a user or broadcast signals corresponding to all prestored channels from among broadcast signals received through an antenna (not shown) or a cable (not shown). The tuner <NUM> may convert a selected broadcast signal into an intermediate frequency (IF) signal or a baseband video or audio signal.

For example, if the selected broadcast signal is a digital broadcast signal, the tuner <NUM> may convert the selected broadcast signal into a digital IF signal (DIF), and if the selected broadcast signal is an analog broadcast signal, the tuner <NUM> may convert the selected broadcast signal into an analog baseband video or audio signal CVBS/SIF. That is, the tuner <NUM> may process digital broadcast signals or analog broadcast signals. The analog baseband video or audio signal CVBS/SIF output from the tuner <NUM> may be directly input to the controller <NUM>.

In addition, the tuner <NUM> may sequentially select broadcast signals of all the broadcast channels stored through a channel memory function from among the received broadcast signals and may convert the selected broadcast signals into IF signals or baseband video or audio signals.

Further, the tuner <NUM> may include a plurality of tuners for receiving broadcast signals of a plurality of channels. Alternatively, the tuner <NUM> may be a single tuner that simultaneously receives broadcast signals of a plurality of channels.

The demodulator <NUM> may receive a digital IF signal DIF converted by the tuner <NUM> and may demodulate the digital IF signal.

Upon performing demodulation and channel decoding, the demodulator <NUM> may output a stream signal TS. In this case, the stream signal may be a multiplexed video signal, audio signal or data signal.

The stream signal output from the demodulator <NUM> may be input to the controller <NUM>. Upon performing demultiplexing, video/audio signal processing, etc., the controller <NUM> may output an image to the display <NUM> and may output sound to the audio output unit <NUM>.

The external device interface <NUM> may transmit or receive data to or from a connected external device. To this end, the external device interface <NUM> may include an A/V input/output unit (not shown).

The external device interface <NUM> may be connected by wire or wirelessly to external devices, such as a digital versatile disc (DVD) player, a Blu-ray player, a game console, a camera, a camcorder, a computer (laptop), a set-top box, etc., and may perform input/output operations for external devices.

The A/V input/output unit may receive video and audio signals of an external device. For example, the A/V input/output unit may include an Ethernet terminal, a USB terminal, a composite video banking sync (CVBS) terminal, a component terminal, a S-video terminal (analog), a digital visual interface (DVI) terminal, a high definition multimedia interface (HDMI) terminal, a mobile high-definition link (MHL) terminal, an RGB terminal, a D-SUB terminal, an IEEE <NUM> terminal, an SPDIF terminal, a liquid HD terminal, and the like. A digital signal input through such terminals may be transmitted to the controller <NUM>. In this case, an analogue signal input through the CVBS terminal and the S-video terminal may be converted into a digital signal by an analogue/digital conversion unit (not shown), to be transmitted to the controller <NUM>.

The external device interface <NUM> may include a wireless communicator (not shown) for short-range wireless communication with other electronic devices. Through the wireless communicator, the external device interface <NUM> may exchange data with an adjacent mobile terminal. For example, the external device interface <NUM> may receive, from the mobile terminal, device information, running application information, application images, and the like in a mirroring mode.

The external device interface <NUM> may perform short-range wireless communication by using Bluetooth, Radio Frequency Identification (RFID) communication, infrared Data Association (IrDA) communication, Ultra-Wideband (UWB), ZigBee, and the like.

The network interface <NUM> may provide an interface for connecting the home appliance <NUM> to a wired/wireless network including an Internet network.

The network interface <NUM> may include a communication module (not shown) for communication with the wired/wireless network <NUM>. For example, the network interface <NUM> may include a communication module for Wireless LAN (WLAN; Wi-Fi), Wireless broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), and the like.

The network interface <NUM> may transmit or receive data to or from other users or other electronic devices through a connected network or another network linked to the connected network.

The network interface <NUM> may receive web content or data provided by a content provider or a network operator. That is, the network interface <NUM> may receive the web content or data, such as movies, advertisements, games, VOD, broadcast signals, etc., as well as information related thereto, which are provided by content providers or network providers through the network.

The network interface <NUM> may receive update information and an update file of firmware provided by network operators, and may transmit data to internet or content providers or network operators.

The network interface <NUM> may select a desired application from among a plurality of applications open to the public, and may receive the selected application through a network.

The storage <NUM> may store programs for processing and controlling each signal within the controller <NUM>, and may store signal-processed video, audio or data signals. For example, the storage <NUM> may store applications designed to perform various operations which may be processed by the controller <NUM>, and in response to a request from the controller <NUM>, the storage <NUM> may selectively provide some of the stored applications.

The programs and the like stored in the storage <NUM> are not particularly limited, as long as the programs may be executed by the controller <NUM>.

The storage <NUM> may perform the function of temporarily storing video, audio or data signals received from an external device through the external device interface <NUM>.

The storage <NUM> may store information on predetermined broadcast channels through a channel memory function.

While <FIG> illustrates an example in which the storage <NUM> is provided separately from the controller <NUM>, the scope of the present disclosure is not limited thereto, and the storage <NUM> may be included in the controller <NUM>.

The storage <NUM> may include at least one of a volatile memory (e.g., DRAM, SRAM, SDRAM, etc.) and a non-volatile memory (e.g., flash memory, hard disk type memory (HDD), solid-state drive (SSD), etc.). In various embodiments of the present disclosure, the storage <NUM> and the memory may be used interchangeably.

The user input interface <NUM> may transmit a signal, input by a user, to the controller <NUM> or may transmit a signal, input from the controller <NUM>, to the user. For example, the user input interface <NUM> may transmit/receive a user input signal, such as power on/off, channel selection, screen setup, etc., to/from a remote control device <NUM>, and may deliver a user input signal input through a local key (not shown), such as a power key, a channel key, a volume key, or a setup value, to the controller <NUM>, or may deliver a user input signal, input from a sensor unit (not shown), which senses a user's gesture, to the controller <NUM>, or may transmit a signal from the controller <NUM> to the sensor unit.

The input unit <NUM> may be provided on one side of a main body of the home appliance <NUM>. For example, the input unit <NUM> may include a touchpad, a physical button, and the like.

The input unit <NUM> may receive various user commands associated with the operation of the image display device 100a, and may transmit a control signal corresponding to the input command to the controller <NUM>.

The input unit <NUM> may include at least one microphone (not shown), and may receive a user's speech through the microphone.

The controller <NUM> may include at least one processor, and by using the included processor, the controller <NUM> may control the overall operation of the image display device 100a. Here, the processor may be a general processor such as a central processing unit (CPU). It is also possible that the processor is a dedicated device, such as an ASIC, or other hardware-based processor.

The controller <NUM> may demultiplex a stream input through the tuner <NUM>, the demodulator <NUM>, the external device interface <NUM>, or the network interface <NUM>, or may process demultiplexed signals to generate and output signals for video or audio output.

The display <NUM> may convert a video signal, a data signal, an OSD signal, and a control signal processed by the controller <NUM> or a video signal, a data signal and a control signal received from the external device interface <NUM> to generate driving signals.

The display <NUM> may include a display panel (not shown) including a plurality of pixels.

The plurality of pixels included in the display panel may have RGB sub-pixels. Alternatively, the plurality of pixels included in the display panel may have RGBW sub-pixels. The display <NUM> may convert the video signal, data signal, OSD signal, and control signal processed by the controller <NUM> to generate signals for driving the plurality of pixels.

The display <NUM> may be a Plasma Display Panel (PDP), a Liquid Crystal Display (LCD), an Organic Light Emitting Diode (OLED), a flexible display, or the like, and may also be a three-dimensional (3D) display. The 3D display <NUM> may be divided into an autostereoscopic display and a glasses-type display.

Further, the display <NUM> may be configured as a touchscreen to be used as an input device in addition to an output device.

The audio output unit <NUM> receives an audio signal processed by the controller <NUM>, and outputs the audio signal as a sound.

The image signal, processed by the controller <NUM>, may be input to the display <NUM> to be displayed as an image corresponding thereto. Further, The image signal processed by the controller <NUM> may be input to an external output device through the external device interface <NUM>.

The audio signal processed by the controller <NUM> may be output as a sound to the audio output unit <NUM>. Further, the audio signal processed by the controller <NUM> may be input to an external output device through the external device interface <NUM>.

Although not illustrated in <FIG>, the controller <NUM> may include a demultiplexer, a video processor, a frame rate converter, a formatter, an audio processor, a data processor, and the like.

Besides, the controller <NUM> may control the overall operation of the home appliance <NUM>. For example, the controller <NUM> may control the tuner <NUM> to tune in to a broadcast channel selected by a user or a prestored channel.

In addition, the controller <NUM> may control the image display device 100a according to a user command input through the user input interface <NUM> or an internal program.

Further, the controller <NUM> may control the display <NUM> to display images. Here, the images displayed on the display <NUM> may be still images or moving images and may be 2D images or 3D images.

Moreover, the controller <NUM> may control a predetermined 2D object to be displayed in an image displayed on the display <NUM>. For example, the object may be at least one of an accessed web screen (newspaper, magazine, etc.), an electronic program guide (EPG), various menus, a widget, an icon, a still image, a moving image, and text.

In addition, the image display device 100a may further include an image capturing unit (not shown). The image capturing unit may capture images of a user. The image capturing unit may be implemented with one camera, but is not limited thereto, and may be implemented with a plurality of cameras. Further, the image capturing unit may be embedded in the home appliance <NUM> on the top of the display <NUM>, or may be provided separately. Image information captured by the image capturing unit may be input to the controller <NUM>.

The controller <NUM> may recognize a user's position based on the images captured by the image capturing unit. For example, the controller <NUM> may identify a distance (z-axis coordinates) between the user and the image display device 100a. In addition, the controller <NUM> may identify x-axis coordinates and y-axis coordinates corresponding to a user's position in the display <NUM>.

The controller <NUM> may sense a user's gesture based on the images captured by the image capturing unit or the respective signals sensed by the sensor unit, or a combination thereof.

The power supply unit <NUM> may supply power throughout the image display device 100a. Particularly, the power supply unit <NUM> may supply power to the controller <NUM> implemented in the form of a system on chip (SOC), the display <NUM> for image display, the audio output unit <NUM> for audio output, and the like.

Specifically, the power supply unit <NUM> may include a converter for converting AC power into DC power and a DC/DC converter (not shown) for changing a DC power level.

The remote control device <NUM> transmits a user input to the user input interface <NUM>. To this end, the remote controller <NUM> may use Bluetooth, radio frequency (RF) communication, infrared (IR) communication, Ultra Wideband (UWB), ZigBee and the like. Furthermore, the remote control device <NUM> may receive video, audio or data signals output from the user input interface <NUM>, and may display the received signals or may output the same as sound through the remote control device <NUM>.

In addition, the aforementioned image display device 100a may be a fixed type or movable digital broadcast receiver capable of receiving digital broadcast.

The block diagram of the image display device 100a illustrated in <FIG> is merely a block diagram for an embodiment of the present disclosure. Components of the block diagram may be integrated, added or omitted according to specifications of the image display device 100a implemented in practice.

That is, two or more components may be combined or one component may be divided into two or more components as needed. Furthermore, a function executed in each block is for description of an embodiment of the present disclosure, and a specific operation or device of each block is not intended to limit the scope of the present disclosure.

<FIG> is an example of an internal block diagram illustrating the server of <FIG>.

Referring to <FIG>, the server <NUM> may include a communicator <NUM>, a storage <NUM>, and/or a controller <NUM>.

The communicator <NUM> may include at least one communication module for connection with a wired/wireless network <NUM>, and may access the network <NUM> to communicate with the home appliance <NUM>.

The storage <NUM> may store programs for processing and controlling each signal within the controller <NUM>, and may store processed data signals. For example, the storage <NUM> may store applications designed to perform various operations which may be processed by the controller <NUM>, and in response to a request from the controller <NUM>, the storage <NUM> may selectively provide some of the stored applications. The programs stored in the storage <NUM> are not particularly limited, as long as the programs may be executed by the controller <NUM>.

The storage <NUM> may include at least one of a volatile memory (e.g., DRAM, SRAM, SDRAM, etc.) and a non-volatile memory (e.g., flash memory, hard disk type memory (HDD), solid-state drive (SSD), etc.).

In addition, the storage <NUM> may include at least one database.

The storage <NUM> may include a word database <NUM> including data on various words used for speech recognition.

The storage <NUM> may include an account database <NUM> including data on an account of at least one user. For example, the data on a user's account may include a user account identifier, a device identifier of at least one home appliance <NUM> registered to an account, a serial number, a MAC Address, and the like.

The storage <NUM> may include an operation history database <NUM> including data on an operation history of at least one home appliance <NUM>. For example, the data on the operation history may include a surrounding environment (e.g., temperature, humidity, etc.) or an operation history of the home appliance <NUM> over time.

The storage <NUM> may store data, models, algorithms, etc., which are pre-learned by machine learning such as deep learning and the like.

The controller <NUM> may be connected to each component of the server <NUM>, and may control the overall operation of each component. The controller <NUM> may transmit and receive data with each component of the server <NUM>.

The controller <NUM> transmits and receives signals including data with the home appliance <NUM> through the communicator <NUM>.

The controller <NUM> performs speech recognition of speech included in a signal received from the home appliance <NUM>. For example, the controller <NUM> may perform speech recognition by using natural language processing (NLP), and may extract words included in the speech. Here, natural language processing (NLP) may refer to technology that mechanically analyzes human languages and makes it into a form that can be understood by electronic devices.

Based on a speech recognition result, the controller <NUM> may extract a command included in the speech and may generate a response to the command and/or an additional query. In this case, the controller <NUM> generates a response to the command and/or an additional query based on at least one database included in the storage <NUM>.

The controller <NUM> transmits a signal, including data on the response and/or the additional query, a control command, etc., to the home appliance <NUM>.

Further, although not illustrated herein, the server <NUM> may be composed of at least one sub-server. For example, the server <NUM> may include a speech recognition server configured to receive a signal, including data on speech, from the home appliance <NUM> and to perform speech recognition of the speech to extract words included in the speech, a response generation server configured to generate a response to the speech and/or an additional query, a control server configured to transmit a signal, including data on the response and/or the additional query, a control command, etc., to the home appliance <NUM>, and the like.

<FIG> and <FIG> are flowcharts illustrating an operating method of a server according to an embodiment of the present disclosure.

Referring to <FIG>, the server <NUM> receives a signal including data on speech (hereinafter referred to as an input signal) from the home appliance <NUM> in operation S401.

In operation S402, the server <NUM> performs speech recognition of the speech included in the input signal. For example, the server <NUM> may extract words included in the speech by using NLP.

In operation S403, the server <NUM> determines whether a word corresponding to the last part of the speech is a predetermined word. Here, the predetermined word may include a conjunction (e.g., and, so, but, or, yet, etc.) that joins together preceding and following words, phrases, or clauses, a meaningless word (e.g., filler word) present in the middle of a sentence or in a last portion of a sentence, etc., and a predetermined command (e.g., wait, repeat, etc.).

For example, if speech included in the input signal received from the home appliance <NUM> is "what is the current temperature? and," the server <NUM> may extract words, "current," "temperature," "what," and "and," and may determine whether the word "and" corresponding to the last part of the speech is a predetermined word.

If the word corresponding to the last part of the speech is the predetermined word, the server <NUM> generates a response to the remaining part of the speech, except the last part, in operation S404. For example, in the speech "what is the current temperature? and" included in the input signal, the server <NUM> may generate a response to the current temperature in response to the remaining part, i.e., "what is the current temperature?," except "and" corresponding to the last part of the speech.

If a word corresponding to the last part of the speech is the predetermined word, the server <NUM> generates an additional query in operation S405. In this case, based on at least one database included in the storage <NUM>, the server <NUM> may generate an additional query regarding the generated response to the speech, which will be described below with reference to <FIG>.

Referring to <FIG>, based on the operation history database <NUM>, the server <NUM> may verify whether there is at least one operation history associated with the response in operation S501, among operation histories of a first electronic device 100a, to which the input signal is transmitted, in the home appliance <NUM>.

For example, if the generated response to the speech is a response to the current temperature, the server <NUM> may verify whether there is an operation history of the first electronic device 100a which is associated with: an operation of the first electronic device 100a performed at a temperature corresponding to the current temperature; an operation performed by the first electronic device 100a after the response to the current temperature is transmitted to the first electronic device 100a; the content of additional speech received from the first electronic device 100a after the response to the current temperature is transmitted to the first electronic device 100a; an operation performed by the first electronic device 100a after the response to the additional speech is transmitted to the first electronic device 100a; and the like.

If there is at least one operation history of the first electronic device 100a associated with the generated response to the speech, the server <NUM> may generate an additional query (hereinafter referred to as a first additional query) about the first electronic device 100a in operation S502 based on the operation history of the first electronic device 100a.

In operation S503, the server <NUM> may check an operating state of at least one second electronic device 100b, except the first electronic device 100a to which the input signal is transmitted, in the home appliance <NUM>. Here, the second electronic device 100b may be a home appliance <NUM> registered to a user's account, to which the first electronic device 100a is registered.

Further, the at least one second electronic device 100b may transmit a signal, including data on the operating state, to the server <NUM> at predetermined intervals and/or in response to a transmission request of the server <NUM>.

In operation S504, the server <NUM> may verify whether there is at least one second electronic device 100b, which is in an operating state associated with the response, in the at least one second electronic device 100b. For example, if the generated response to the speech is the response to the current temperature, the server <NUM> may verify whether there is at least one second electronic device 100b, which is in an operating state associated with the response, based on the operation history regarding: an operation of the second electronic device 100b performed at a temperature corresponding to the current temperature; an operation performed by the second electronic device 100b after the response to the current temperature is transmitted to the second electronic device 100b; the content of additional speech received from the second electronic device 100b after the response to the current temperature is transmitted to the second electronic device 100b; an operation performed by the second electronic device 100b after the response to the additional speech is transmitted to the second electronic device 100b; and the like.

If there is at least one second electronic device 100b which is in an operating state associated with the generated response to the speech, the server <NUM> may generate an additional query (hereinafter referred to as a second additional query) about the second electronic device 100b in operation S505 based on the operation history of the second electronic device 100b.

Although not illustrated herein, the server <NUM> may generate an additional query about the response based on data pre-learned by machine learning such as deep learning and the like.

Referring back to <FIG>, the server <NUM> transmits a response signal (hereinafter referred to as a first response signal), including the data on the response generated in operation S404 and the data on the at least one additional query generated in operation S405, to the home appliance <NUM> in operation S406. In this case, the first response signal may further include a control command for activating a speech command receiving function of the home appliance <NUM>. The home appliance <NUM> receiving the first response signal may turn on a microphone included in the input unit <NUM>, and may wait for a speech input from a user.

If the first and second additional queries are not generated in operation S405, the server <NUM> may include data on a predetermined additional query in the first response signal. Here, the predetermined additional query may include a query, e.g., "do you need any more information?" and the like, to induce a user to give additional speech.

Further, the server <NUM> may further include a control command for controlling a word, corresponding to the last part of the speech, to be output between an output of the response to the speech and an output of at least one additional query. As described above, when the home appliance <NUM> outputs the word, corresponding to the last part of the speech uttered by a user, while including the word in the response, the home appliance <NUM> may promote a user's sense of closeness for the home appliance <NUM> and a speech recognition technology.

In operation S407, the server <NUM> may verify whether an input signal is further received from the home appliance <NUM> within a predetermined period of time after the server <NUM> transmits the first response signal to the home appliance <NUM>.

In this case, if the input signal is received from the home appliance <NUM> within the predetermined period of time after the first response signal is transmitted, the server <NUM> may return to operation S402 to perform speech recognition of speech included in the additionally received input signal.

If the input signal is not received from the home appliance <NUM> within the predetermined period of time after the first response signal is transmitted, the server <NUM> may transmit a termination signal including a predetermined termination message to the home appliance <NUM> in operation S408. Here, the termination message may be a message notifying a user of termination of the activated speech command receiving function.

Further, if a word corresponding to the last part of the speech is not the predetermined word, the server <NUM> generates a response corresponding to the entire part of the speech in operation S409.

In addition, the server <NUM> transmits a response signal (hereinafter referred to as a second response signal) including data on the response generated in S409 to the home appliance <NUM> in operation S410. In this case, unlike the first response signal, the second response signal includes no control command for activating the speech command receiving function of the home appliance <NUM>, such that the home appliance <NUM> receiving the second response signal may output only a response included in the second response signal.

<FIG> is a flowchart illustrating an operating method of a system according to an embodiment of the present disclosure. Detailed description of portions overlapping those described above with reference to <FIG> and <FIG> will be omitted.

Referring to <FIG>, the first electronic device 100a may receive a user's speech in operation S601. For example, upon receiving speech including a predetermined wake-up word through a microphone of the input unit <NUM>, the first electronic device 100a may activate the speech command receiving function, and may receive the speech, received while the speech command receiving function is activated, as a speech command.

In this case, the first electronic device 100a may be maintained in a standby state in which the first electronic device 100a neglects a speech input except the predetermined wake-up word.

Further, the first electronic device 100a may wait for the input of a speech command for a predetermined period of time after receiving the predetermined wake-up word, or may receive continuous speech including the predetermined wake-up word and the speech command.

In operation S602, the first electronic device 100a may transmit an input signal, including data on the speech, to the server <NUM>. In this case, the data on the speech may include only a speech command which is input after a predetermined wake-up word is received, or may include both the predetermined wake-up word and the speech command.

In operation S603, the server may perform speech recognition of the speech included in the input signal. For example, the server <NUM> may extract words included in the speech by using NLP.

In operation S604, by verifying the last part of the speech, the server <NUM> may determine whether a word corresponding to the last part of the speech is a predetermined word.

In operation S605, the server <NUM> may generate a response to the speech. For example, if the word corresponding to the last part of the speech is the predetermined word, the server <NUM> may generate a response corresponding to a remaining part of the speech except the last part. The following description of <FIG> will be given using an example in which the word corresponding to the last word of the speech is the predetermined word.

In operation S606, the server <NUM> may transmit a request for transmitting an operating state to at least one second electronic device 100b, except the first electronic device 100a to which the input signal is transmitted, among the home appliances <NUM>. In operation S607, in response to the request from the server <NUM>, the at least one second electronic device 100b may transmit a signal including data on the operating state to the server <NUM>.

Further, as illustrated herein, even when not receiving the request from the server <NUM>, the at least one second electronic device 100b may transmits the signal, including the data on the operating state, to the server <NUM> at predetermined intervals and/or in response to the occurrence of a predetermined event.

In operation S608, the server <NUM> may generate an additional query about the generated response to the speech based on an operation history of the first electronic device 100a and/or an operation history of the second electronic device 100b.

In operation S609, the server <NUM> may transmit a first response signal, including data on the response to the speech and data on the additional query, to the first electronic device 100a. In this case, if the word corresponding to the last part of the speech is the predetermined word, the server <NUM> may include a control command for activating a speech command receiving function in the first response signal.

In operation S610, upon receiving the first response signal from the server <NUM>, the first electronic device 100a may output the response and the additional query which are included in the first response signal. For example, the first electronic device 100a may sequentially output speech corresponding to the response and speech corresponding to the additional query through the audio output unit <NUM>. In this case, the first electronic device 100a may output the word, corresponding to the last part of the speech received in operation S601, between the output of the speech corresponding to the response and the output of the speech corresponding to the additional query.

In addition, according to the control command for activating the speech command receiving function included in the first response signal, the first electronic device 100a may activate the speech command receiving function even when there is no input of the predetermined wake-up word, and may receive the speech command included in speech uttered by a user.

Further, in operation S611, when the first response signal is transmitted, the server <NUM> may count an elapsed time after the first response signal is transmitted.

In operation S612, if the input signal is not transmitted from the home appliance <NUM> until a predetermined time has elapsed after transmission of the first response signal, the server <NUM> may transmit a termination signal, including a predetermined termination message, to the home appliance <NUM>.

In this case, the first electronic device 100a receiving the termination signal may output speech, corresponding to the predetermined termination message included in the termination signal, through the audio output unit <NUM>, and may terminate the speech command receiving function.

<FIG> are diagrams referred to in explaining various embodiments of the present disclosure.

Referring to <FIG>, a user <NUM> may utter speech <NUM>, including "Hi, LG" which is a predetermined wake-up word and "what is the current indoor temperature?" corresponding to a speech command, toward an air conditioner as an example of the home appliance <NUM>.

In this case, the home appliance <NUM> may activate the speech command receiving function in response to the predetermined wake-up word, "Hi, LG," and may transmit data corresponding to the speech command, "what is the current indoor temperature?", to the server <NUM>.

In addition, in the speech corresponding to the speech command uttered by the user, "what is the current indoor temperature?", a word corresponding to the last part of the speech is not a predetermined word, such that the server <NUM> may transmit a response signal including a response to the speech, "what is the current indoor temperature?", to the home appliance <NUM>.

Further, the home appliance <NUM> may output speech <NUM>, corresponding to the response to the current temperature included in the response signal received from the server <NUM>, through the audio output unit <NUM>. In this case, the word corresponding to the last part of the speech, "what is the current indoor temperature?", is not the predetermined word, such that the response signal received from the server <NUM> does not include the control command for activating the speech command receiving function. Accordingly, upon outputting only the speech <NUM> corresponding to the response to the current temperature, the home appliance <NUM> may be maintained in a standby state.

Further, referring to <FIG>, the user <NUM> may utter speech <NUM> including the predetermined wake-up word, "Hi, LG," and "what is the current indoor temperature? and" corresponding to a speech command.

In this case, in the speech corresponding to the speech command uttered by the user, "what is the current indoor temperature? and", the last part of the speech "and" is a conjunction corresponding to a predetermined word, such that the server <NUM> may transmit a response signal including a response to the speech, "what is the current indoor temperature?", and an additional query to the home appliance <NUM>.

In addition, if there is no operation history of the home appliance <NUM>, which is associated with the response to the current temperature, the server <NUM> may include a predetermined additional query, "do you need any more information?", in the response signal.

Further, the home appliance <NUM> may output the speech <NUM>, corresponding to the response to the current temperature, and the speech <NUM> corresponding the predetermined additional query through the audio output unit <NUM>. In this case, the home appliance <NUM> may output "and", which is the last part of the speech uttered by a user, between the response to the current temperature and the predetermined additional query.

Referring to <FIG>, if there is an operation history of the home appliance <NUM>, which is associated with the response to the current temperature, for example, if there is an operation history regarding an operation of decreasing the set temperature from the current temperature of <NUM>, the server <NUM> may generate an additional query, "would you like to lower the temperature a little?", based on the operation history of the home appliance <NUM>, and may include the generated additional query in the response signal.

In addition, the home appliance <NUM> may output the speech <NUM>, corresponding to the response to the current temperature, and the speech <NUM> corresponding to the additional query through the audio output unit <NUM>, and may output "and", which is the last part of the speech uttered by the user, between the response to the current temperature and the additional query.

Referring to <FIG>, while the image display device as an example of the home appliance <NUM> outputs a screen for displaying content, the user <NUM> may utter speech <NUM>, including a predetermined wake-up word, "Hi, LG", and "what are the clothes worn by the man? and", which corresponds to a speech command, toward the home appliance <NUM>.

In this case, the home appliance <NUM> may activate the speech command receiving function according to the predetermined wake-up word, "Hi, LG", and may transmit data on "what are the clothes worn by the man? and", which corresponds to the speech command, to the server <NUM>.

In addition, in the speech, "what are the clothes worn by the man? and", corresponding to the speech command uttered by the user, the last part of the speech, "and", is a conjunction corresponding to the predetermined word, such that the server <NUM> may transmit a response signal, including a search result for clothes as a response to "what are the clothes worn by the man?" and an additional query, to the home appliance <NUM>.

In this case, if there is an operation history of the home appliance <NUM> which is associated with a search for clothes, for example, upon outputting a search result for clothes, if the server <NUM> receives a request for an additional search from the user or if there is an operation history regarding an operation of performing an additional search in response to the request, the server <NUM> may generate an additional query, "which one would you choose between tops and bottoms?", and may include the generated additional query in the response signal.

The home appliance <NUM> may output screens <NUM> and <NUM> displaying the search result for clothes on the display <NUM>, and may output speech <NUM> corresponding to the additional query including "and", which is the last part of the speech uttered by the user, through the audio output unit <NUM>.

In addition, upon outputting the speech <NUM> corresponding to the additional query, the home appliance <NUM> may activate the speech command receiving function, and the user may utter speech <NUM> including "the top, t-shirt" as a response to the additional query.

Further, referring to <FIG>, as the last part of the speech <NUM> further uttered by the user does not correspond to the predetermined word, the server <NUM> performs speech recognition of the speech <NUM> corresponding to the response to the additional query, and then may transmit a response signal, including only a search result for the top, t-shirt as a response to the speech <NUM>, to the home appliance <NUM>.

Moreover, as the response signal received from the server <NUM> does not include a control command for activating the speech command receiving function, the home appliance <NUM> outputs only a screen <NUM> for displaying the search result for the top> t-shirt, and then may be maintained in a standby state.

Referring to <FIG>, while a washing machine as an example of the home appliance <NUM> washes the clothes, the user <NUM> may utter speech <NUM>, including the predetermined wake-up word, "Hi, LG", and "how much time is left until the laundry is done? um", toward the washing machine which is the first electronic device 100a.

In this case, the first electronic device 100a may activate the speech command receiving function in response to the predetermined wake-up word, "Hi, LG", and may transmit to the server <NUM> data on the speech, "how much time is left until the laundry is done? um", which corresponds to the speech command.

Further, in the speech, "how much time is left until the laundry is done? um", corresponding to the speech command uttered by the user, the last part of the speech, "um", corresponds to a predetermined filler word, such that the server <NUM> may transmit a response signal including a remaining time as a response to "how much time is left until the laundry is done?" and an additional query to the first electronic device 100a.

In this case, if there is an operation history of the first electronic device 100a which is associated with the remaining time, for example, if there is an operation history regarding an operation of adding a rinsing function after the remaining time has elapsed and the laundry is done, the server <NUM> may generate an additional query, "would you like to add one more rinse cycle?", based on the operation history of the first electronic device 100a and may include the generated additional query in the response signal.

In addition, the first electronic device 100a may output speech <NUM>, including a response to the remaining time, "there are <NUM> minutes to go before the laundry is done", and the additional query, "would you like to add one more rinse cycle?", through the audio output unit <NUM> and may further output, "um", which is the last part of the speech uttered by the user, between the response to the remaining time and the additional query.

Further, referring to <FIG>, the server <NUM> may check an operating state of the second electronic device 100b, and may verify whether there is at least one second electronic device 100b being in an operating state related to the response to the remaining time. For example, in the operation history of the image display device as an example of the second electronic device 100b, if there is an operation history regarding an operation of pausing, by a user, playback of content output through the image display device after the laundry is done, the server <NUM> may generate an additional query, "would you like to have the TV paused in <NUM> minutes?", based on the operation history of the second electronic device 100b, and may include the generated additional query in the response signal.

In addition, the first electronic device 100a may output speech <NUM> including the response to the remaining time, "there are <NUM> minutes to go before the laundry is done", the additional query about the first electronic device 100a, "would you like to add one more rinse cycle?", and the additional query about the second electronic device 100b, "would you like to have the TV paused in <NUM> minutes?", through the audio output unit <NUM>.

As described above, according to various embodiments of the present disclosure, after a response to the speech is provided, the speech command receiving function of the home appliance <NUM> may be reactivated automatically according to words included in the speech input from a user, such that the user may control the home appliance <NUM> in a simple and convenient manner without repetitive operations, such as voice wake-up and the like.

Furthermore, according to various embodiments of the present disclosure, not only the speech command receiving function of the home appliance <NUM> may be reactivated automatically, but also an additional query on the response may be provided for a user, thereby naturally inducing the user to make an additional utterance.

In addition, according to various embodiments of the present disclosure, an additional query, regarding an operation expected to be performed following a response, may be generated based on the operation history of the home appliances 100a and 100b, thereby providing a user with an additional query which is further relevant to a user's intention, usage pattern, surrounding environment, and the like.

The accompanying drawings are used to assist in easy understanding of various technical features and it should be understood that the embodiments presented herein are not limited by the accompanying drawings.

Meanwhile, an operating method of the server and the system including the same according to the present disclosure can be realized as a processor-readable code written on a recording medium readable by a processor included in the server and the system including the same. The processor-readable recording medium may be any type of recording device in which data is stored in a processor-readable manner. Examples of the processor-readable recording medium include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disc, an optical data storage, and a carrier wave, e.g., data transmission through the Internet. The processor-readable recording medium can be distributed over a plurality of computer systems connected to a network so that a processor-readable code is written thereto and executed therefrom in a decentralized manner.

Claim 1:
A server (<NUM>), comprising:
a communicator (<NUM>) configured to perform communication through a network (<NUM>);
a storage (<NUM>) configured to store data (<NUM>) on at least one predetermined word; and
a controller (<NUM>) configured to:
upon receiving (S401, S602) an input signal, including data on speech, from a first electronic device (100a) through the communicator (<NUM>), perform (S402, S603) speech recognition of speech included in the input signal, and determine whether a last word of the speech corresponds to any one of the at least one predetermined word,
characterized in that the controller (<NUM>) is further configured to:
in response to there being a word corresponding to the last word of the speech among the at least one predetermined word, transmit (S406, S609) a first response signal, including data on a response to the speech except for the last word and data including at least one query for inducing input of additional speech different from the speech, to the first electronic device (100a) through the communicator (<NUM>), and
in response to there being no word corresponding to the last word of the speech among the at least one predetermined word, transmit (S410) a second response signal, including the data on the response to the speech, to the first electronic device (100a) through the communicator (<NUM>),
wherein the data on the response to the speech includes information (<NUM>, <NUM>, <NUM>) corresponding to an answer to the speech.