ELECTRONIC DEVICE FOR PROVIDING OPERATION STATE INFORMATION OF HOUSEHOLD ELECTRICAL DEVICE AND METHOD OF OPERATING THE SAME

Provided are an electronic device for providing operation state information of a household electrical device without network function and a method of operating the electronic device. Provided is an electronic device receiving a first sound signal output from a household electrical device through a microphone, detecting the first sound signal as a wake-up signal, identifying first operation state information corresponding to the first sound signal from memory, and outputting a notification message representing the first operation state information.

BACKGROUND

The disclosure relates to an electronic device for providing information representing an operation state of a household electrical device, and a method of operating the electronic device. More specifically, the disclosure relates to an electronic device for identifying a sound signal output from a household electrical device having no network function and outputting, to a user, operation state information of the household electrical device based on a result of the identifying, and a method of operating the electronic device.

2. Description of Related Art

With the development of multimedia technology and network technology, various services are provided to users through electronic devices. Particularly, with the development of voice recognition technology, users can provide a voice input (for example, an utterance) to electronic devices and receive response messages according to the voice inputs through a service providing agent. Lately, with the development of internet of things (IoT) technology, information about operation states of household electrical devices in a home is provided in the form of notifications from the household electrical devices, or household electrical devices are controlled by voice inputs from users.

Household electrical devices are used for a long time after being purchased. Because old household electrical devices do not have a network function, such as Wi-Fi or Bluetooth, an electronic device (for example, an artificial intelligence (AI) speaker) having an IoT function cannot be connected to such old household electrical devices through a network. When a network connection between an electronic device and an old household electrical device cannot be established, the electronic device can neither notify an operation state of the household electrical device nor control the household electrical device, which deteriorates user convenience. It is difficult to install a network module in an old household electrical device. Also, although a network module is additionally installed, related software for applying the IoT function needs to be developed.

Accordingly, because it is difficult to use the IoT function in an old household electrical device, a user who wants to use the IoT function in an old household electrical device has no choice but to replace the old household electrical device with a new model having the IoT function, which is an additional cost.

SUMMARY

The disclosure relates to an electronic device for providing information about an operation state of a household electrical device without network function by using a sound signal output from the household electrical device, and a method of operating the electronic device.

An embodiment of the disclosure provides an electronic device for detecting a sound signal output from a household electrical device, and outputting operation state information corresponding to the detected sound signal. An embodiment of the disclosure provides an electronic device for transmitting operation state information corresponding to a sound signal output from a household electrical device to a user's mobile terminal such that the operation state information is output in the form of a notification message through the mobile terminal.

In order to solve the above-described technical problems, an embodiment of the disclosure provides an electronic device for identifying an operation state of a household electrical device and outputting operation state information. An electronic device according to an embodiment of the disclosure includes: a memory storing a plurality of sound signals output from at least one household electrical device as a wake-up signal, and storing a plurality of pieces of operation state information respectively representing a plurality of operation states of the at least one household electrical device, the plurality of operation states respectively related to the plurality of sound signals; and a processor, wherein the processor is configured to receive a first sound signal output from a first household electrical device from among the plurality of sound signals through a microphone, identify the first sound signal as a wake-up signal, identify first operation state information corresponding to the first sound signal based on the plurality of pieces of operation state information, and control an output interface to output a notification message representing the first operation state information.

According to an embodiment of the disclosure, the memory may further store at least one word uttered by a user as the wake-up signal, and the processor may be further configured to receive a voice input uttered by a user through the microphone, and detect the voice input as the wake-up signal by identifying the at least one word from the voice input.

According to an embodiment of the disclosure, after detecting the voice input as the wake-up signal, the processor may be further configured to control a communication interface to transmit a second sound signal received from the first household electrical device to a server, and receive second operation state information of the first household electrical device corresponding to the second sound signal from the server through the communication interface.

According to an embodiment of the disclosure, the processor may be further configured to calculate similarities between the first sound signal and the plurality of sound signals stored in the memory, determine a sound signal having a greatest similarity from among the plurality of sound signals, as a sound signal corresponding to the first sound signal, and identify the first operation state information corresponding to the determined sound signal from among the plurality of pieces of operation state information.

According to an embodiment of the disclosure, the processor may be further configured to control a communication interface to transmit the first operation state information to a mobile terminal of a user, the mobile terminal connected to the electronic device through a wireless communication network, and output a notification message representing the first operation state information through the mobile terminal.

According to an embodiment of the disclosure, the memory may further store a plurality of pieces of follow-up operation information that are recommended in connection with the plurality of pieces of operation state information, and the processor may be further configured to identify first follow-up operation information related to the first operation state information and output the first follow-up operation state information.

According to an embodiment of the disclosure, the processor may be further configured to receive a new sound signal output from a new household electrical device newly registered in a server and new operation state information corresponding to the new sound signal from the server through a communication interface, and store device identification information of the new household electrical device, the new sound signal, and the new operation state information in the memory.

According to an embodiment of the disclosure, the processor may be further configured to transmit a registration request including user account information and device identification information of a new household electrical device to a server through a communication interface, and the new household electrical device may be registered in the server by connecting the device identification information of the new household electrical device to the user account information.

According to an embodiment of the disclosure, the processor may be further configured to receive update data for updating a wake-up signal sensing module from the server through the communication interface to perform a wake-up operation by detecting a new sound signal output from the new household electrical device, and update the wake-up signal sensing module by using the update data.

In order to solve the above-described technical problems, another embodiment of the disclosure provides a method of operating an electronic device to identify operation state information from a sound signal output from a household electrical device. An operation method according to an embodiment of the disclosure includes: receiving a first sound signal output from a first household electrical device; identifying the first sound signal as a wake-up signal; identifying first operation state information corresponding to the first sound signal, based on a plurality of pieces of operation state information stored in a memory; and outputting a notification message representing the first operation state information, wherein the memory stores a plurality of sound signals output from at least one household electrical device as a wake-up signal, and stores a plurality of pieces of operation state information respectively representing a plurality of operation states of the at least one household electrical device, the plurality of operation states respectively related to the plurality of sound signals.

According to an embodiment of the disclosure, the memory may store at least one word uttered by a user as the wake-up signal, and the method may further include: receiving a voice input uttered by a user; and detecting the voice input as the wake-up signal by identifying the at least one word from the voice input.

According to an embodiment of the disclosure, the method may further include: after detecting the voice input as the wake-up signal, transmitting, to a server, a second sound signal received from the first household electrical device, and receiving second operation state information of the first household electrical device corresponding to the second sound signal from the server.

According to an embodiment of the disclosure, the method may further include calculating similarities between the first sound signal and the plurality of sound signals stored in the memory, wherein the identifying of the first operation state information may include: determining a sound signal having a greatest similarity from among the plurality of sound signals, as a sound signal corresponding to the first sound signal; and identifying the first operation state information corresponding to the determined sound signal from among the plurality of pieces of operation state information.

According to an embodiment of the disclosure, the method may further include: transmitting the first operation state information to a mobile terminal connected to the electronic device through a wireless communication network to output a notification message representing the first operation state information through the mobile terminal.

According to an embodiment of the disclosure, the memory may further store a plurality of pieces of follow-up operation information that are recommended in connection with the plurality of pieces of operation state information, and the method may further include: identifying first follow-up operation information related to the first operation state information from the memory, and outputting the first follow-up operation information.

According to an embodiment of the disclosure, the method may further include: transmitting a registration request including user account information and device identification information of a new household electrical device to a server, wherein the new household electrical device may be registered in the server by connecting the device identification information of the new household electrical device to the user account information.

According to an embodiment of the disclosure, the method may further include: receiving a new sound signal output from the new household electrical device and new operation state information corresponding to the new sound signal from the server; and storing the new sound signal of the new household electrical device and the new operation state information in the memory.

According to an embodiment of the disclosure, the method may further include: receiving update data for updating a wake-up signal sensing module from the server to perform a wake-up operation by detecting a new sound signal output from the new household electrical device; and updating the wake-up signal sensing module by using the update data.

To solve the above-described technical problem, another embodiment of the disclosure provides a computer-readable recording medium storing a program for executing the method on a computer.

In order to solve the above-described technical problems, another embodiment of the disclosure provides a server including: a communication interface configured to perform data communication with the electronic device or a mobile terminal through a wired or wireless communication network; a memory storing a plurality of sound signals output from at least one household electrical device and a plurality of pieces of operation state information respectively representing a plurality of operation states respectively related to the plurality of sound signals; and a processor, wherein the processor is configured to receive a voice input uttered by a user and a sound signal output by a household electrical device from the electronic device through the communication interface, separate the voice input from the sound signal, identify operation state information corresponding to the sound signal from the memory, and control the communication interface to transmit the identified operation state information to a mobile terminal of the user.

DETAILED DESCRIPTION

Although general terms being widely used in embodiments of the present specification were selected as terminology used in the disclosure while considering the functions of the disclosure, they may vary according to intentions of one of ordinary skill in the art, judicial precedents, the advent of new technologies, and the like. Terms arbitrarily selected by the applicant of the disclosure may also be used in a specific case. In this case, their meanings will be described in detail in the detailed description of the disclosure. Hence, the terms must be defined based on the meanings of the terms and the contents of the entire specification, not by simply stating the terms themselves.

It is to be understood that the singular forms “a,” “an,” and “the” include plural referents unless the context clearly dictates otherwise. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the disclosure belongs.

It will be understood that when a certain part “includes” a certain component, the part does not exclude another component but can further include another component, unless the context clearly dictates otherwise. As used herein, the terms “unit” or “module” refers to a unit that can perform at least one function or operation, and may be implemented as hardware, software, or a combination of hardware and software.

The expressions “configured to” used in the disclosure may be replaced with, for example, “suitable for”, “having the capacity to”, “designed to”, “adapted to”, “made to”, or “capable of” according to cases. The term “configured to” may not necessarily mean “specifically designed to”. Instead, in some cases, the expression “system configured to” may mean that the system is “capable of . . . ” together with another device or components. For example, the phrase “processor configured to perform A, B, or C” may be a dedicated processor (for example, an embedded processor) for performing the corresponding operation, or a generic-purpose processor (for example, a central processing unit (CPU) or an application processor) capable of performing the corresponding operations by executing one or more software programs stored in a memory.

In the disclosure, an ‘electronic device’ may be a device for detecting a sound signal output from a household electrical device as a wake-up signal, and outputting operation state information corresponding to the sound signal. The electronic device may include a network function, and be connected to a server or a mobile terminal through a wired or wireless communication network. According to an embodiment of the disclosure, the electronic device may be an artificial intelligence (AI) speaker that performs a voice recognition function. However, the electronic device is not limited to an AI speaker, and the electronic device may be, for example, a tablet personal computer (PC), a laptop PC, a workstation, a wearable device, a home camera, or mobile medical equipment.

In the disclosure, a ‘household electrical device’ may be an electronic device that is used in home. The household electrical device may include at least one of, for example, a television (TV), a refrigerator, an air cleaner, an air conditioner, an audio system, an induction, a washing machine, a clothes care apparatus, a dryer, a dish washer, an oven, a microwave, an air fryer, an automatic pet feeding machine, a water purifier, a light, or a bidet, but is not limited thereto. According to an embodiment of the disclosure, the household electrical device may output a sound signal representing an operation state. For example, the household electrical device may output a sound signal representing an error or a failure, as well as state information representing starting, finishing, or progressing of a certain function.

In the disclosure, a ‘sound signal’ may be a signal obtained by converting sound falling within an audible frequency range into an electrical waveform. According to an embodiment of the disclosure, the sound signal may have a format of a wave file, but is not limited thereto. According to an embodiment of the disclosure, a sound signal may be output from a household electrical device.

In the disclosure, a ‘wake-up signal’ may be a sound signal for activating an electronic device for switching to a voice recognition mode for performing a voice recognition function. When the electronic device operating in a standby mode or a sleep mode receives a wake-up signal, the electronic device may perform a wake-up operation for switching to a voice recognition mode for receiving a voice input from a user. According to an embodiment of the disclosure, the electronic device may wake up by sensing a wake-up signal through a wake-up signal sensing module. The wake-up signal may include a voice signal, such as, for example, ‘Hi Bixby’ or ‘Hello Galaxy’, etc. However, the disclosure is not limited thereto. For example, the wake-up signal may be a sound signal output from a household electrical device. In this case, the sound signal may have been registered in advance as a wake-up signal by a user.

Hereinafter, embodiments of the disclosure will be described in detail with reference to the accompanying drawings so that the disclosure may be readily implemented by one of ordinary skill in the technical field to which the disclosure pertains. However, the disclosure may be implemented in various different forms, and is not limited to the embodiments of the disclosure described herein.

Hereinafter, the embodiments of the disclosure will be described in detail with reference to the accompanying drawings.

FIG.1illustrates an operation, by an electronic device1000, of outputting operation state information of a household electrical device2000based on a sound signal output from the household electrical device2000, according to an embodiment of the disclosure.

Referring toFIG.1, the electronic device1000may be a device for detecting a wake-up signal and performing an operation or function corresponding to the wake-up signal. According to an embodiment of the disclosure, the electronic device1000may be an AI speaker. However, the electronic device1000is not limited to an AI speaker, and may be various kinds of devices including a microphone. For example, the electronic device1000may be a tablet PC, a laptop PC, a workstation, a wearable device, a home camera, or mobile medical equipment.

The household electrical device2000may output a sound signal representing an operation state. According to an embodiment of the disclosure, the household electrical device2000may output a sound signal representing an error or a failure, as well as state information representing starting, finishing, or progressing of a function. The sound signal may be at least one of effect sound, bell sound, a melody, music, or a voice signal. According to an embodiment of the disclosure, as illustrated inFIG.1, the household electrical device2000may be a washing machine, and include a speaker2100for outputting a sound signal representing washing completion.

The electronic device1000may receive a sound signal output from the speaker2100of the household electrical device2000, and identify operation state information corresponding to the received sound signal. According to an embodiment of the disclosure, the electronic device1000may compare the sound signal received from the household electrical device2000with a pre-registered sound signal, and, when the received sound signal is identical to the pre-registered sound signal or similarity between the received sound signal and the pre-registered sound signal exceeds a preset threshold value according to a result of the comparing, the electronic device1000may wake up by detecting the sound signal received from the household electrical device2000as a wake-up signal. According to an embodiment of the disclosure, when the sound signal received from the household electrical device2000is not similar to the pre-registered sound signal, the electronic device1000may ignore the sound signal without performing any operation of responding to the sound signal.

The electronic device1000may compare the sound signal detected as the wake-up signal with a plurality of sound signals stored in advance in database, and identify a sound signal that is most similar to the sound signal received as the wake-up signal from among the plurality of sound signals stored in the database according to a result of the comparing. The electronic device1000may identify operation state information corresponding to the identified sound signal, and obtain the operation state information from the database. According to the embodiment of the disclosure, as illustrated inFIG.1, the electronic device1000may obtain ‘washing completion’ information as the operation state information corresponding to the received sound signal.

The electronic device1000may output a notification message representing the operation state information. The electronic device1000may output a notification message in the form of a voice through a speaker1500(seeFIGS.2A and3). According to an embodiment of the disclosure, the electronic device1000may output the notification message representing the operation state information, together with information representing a subject of the operation. According to the embodiment of the disclosure, as illustrated inFIG.1, the electronic device1000may output a voice message “washing is completed by the washing machine” as the notification message representing the operation state information of ‘washing completion’.

According to an embodiment of the disclosure, the electronic device1000may transmit the operation state information to a mobile terminal4000. The mobile terminal4000may be at least one of a smart phone, a tablet PC, a digital camera, an e-book terminal, personal digital assistants (PDA), a portable multimedia player (PMP), or a MPEG audio layer-3 (MP3) player, but is not limited thereto. For example, the mobile terminal4000may be a wearable device. The wearable device may be a device worn by a user, and include at least one of an accessory type device (for example, a watch, a ring, a bracelet, an anklet, a necklace, glasses, and contact lenses), a head-mounted device (HMD), a device combined with fabrics or clothes (for example, electronic clothes), a body attaching type device (for example, a skin pad), or a bio-implantable type device (for example, an implantable circuit). According to the embodiment of the disclosure, the mobile terminal4000may be a smart phone.

The mobile terminal4000may receive the operation state information from the electronic device1000, and output a notification message4100representing the received operation state information. According to an embodiment of the disclosure, the mobile terminal4000may display the notification message4100representing the operation state information of the household electrical device2000in a form of a pop-up message. According to another embodiment of the disclosure, the mobile terminal4000may output a notification message in a form of a voice representing the operation state information of the household electrical device2000through the speaker2100.

A device such as an AI speaker having an IoT function may be not connected to an old household electrical device without network function. When an IoT device is not connected to an old household electrical device without network function, it may be impossible to provide a notification about an operation state of the household electrical device or control the household electrical device, which deteriorates user convenience. It may be difficult to additionally install a network module in an old household electrical device, and although a network module is additionally installed, related software may need to be additionally developed to apply the IoT function, resulting in a reduction of efficiency. For a device having an IoT function to provide a notification about an operation state of a household electrical device or control the household electrical device, there may be no choice but to replace an old household electrical device with a latest household electrical device. In this case, a lot of cost may be needed.

Because the electronic device1000according to an embodiment of the disclosure detects a sound signal output from the household electrical device2000, identifies operation state information corresponding to the sound signal, and outputs a notification message representing the operation state information, the electronic device1000may provide an IoT function related to the household electrical device2000although the household electrical device2000without network function. Accordingly, the electronic device1000according to an embodiment of the disclosure may improve usage satisfaction and convenience. Also, because the electronic device1000according to an embodiment of the disclosure outputs a notification message representing operation state information of the existing household electrical device2000, additionally purchasing a new IoT device may be not needed, which improves economical utility by removing cost burden and increases utilization of functions of the household electrical device2000.

FIG.2Aillustrates an embodiment of the electronic device1000outputting operation state information of the household electrical device2000based on a sound signal output from the household electrical device2000and transmitting the operation state information to the mobile terminal4000, according to an embodiment of the disclosure.

FIG.2Ais a view for describing data transmission/reception and operation execution between components, and arrows illustrated inFIG.2Arepresent transmission/reception directions of data.

Referring toFIG.2A, the electronic device1000may include a microphone1100, a wake-up signal sensing module1310, a communication interface1400, a speaker1500, and a wake-up signal database1600. InFIG.2A, essential components for describing functions and/or operations of the electronic device1000are illustrated. However, components included in the electronic device1000are not limited to those illustrated inFIG.2A. The components of the electronic device1000will be described in detail with reference toFIG.3, below.

The electronic device1000may receive a sound signal output from the speaker2100of the household electrical device2000through the microphone1100. A ‘sound signal’ received through the microphone1100may be at least one of effect sound, bell sound, a melody, music, or a voice signal. The microphone1100may perform a pre-processing operation of removing noise (for example, a non-acoustic component) from a sound signal output from the household electrical device2000. The microphone1100may transmit the sound signal to the wake-up signal sensing module1310.

The wake-up signal sensing module1310may be a module configured to detect a predefined sound signal among sound signals output from the household electrical device2000, as a wake-up signal. The wake-up signal sensing module1310may compare the sound signal detected as the wake-up signal with a plurality of sound signals1610-1to1610-nstored in advance in the wake-up signal database1600, and identify a sound signal corresponding to the sound signal received from the household electrical device2000from among the plurality of sound signals1610-1to1610-naccording to a result of the comparing. The wake-up signal sensing module1310may extract operation state information mapped to the identified sound signal from among a plurality of pieces of operation state information1620-1to1620-nstored in advance in the wake-up signal database1600.

The wake-up signal database1600may be database storing a sound signal obtained in advance before the sound signal is received from the household electrical device2000, and operation state information of the household electrical device2000, corresponding to the sound signal obtained in advance. The plurality of sound signals1610-1to1610-nand the plurality of pieces of operation state information1620-1to1620-nrespectively related to the plurality of sound signals1610-1to1610-nmay be stored in the wake-up signal database1600. According to an embodiment of the disclosure, the plurality of sound signals1610-1to1610-nmay be paired with the corresponding pieces of the plurality of pieces of operation state information1620-1to1620-nand then stored in the wake-up signal database1600. According to an embodiment of the disclosure, the plurality of sound signals1610-1to1610-nand the plurality of pieces of operation state information1620-1to1620-nmay be stored in key-value types in the wake-up signal database1600. For example, the plurality of sound signals1610-1to1610-nmay be stored as keys in the wake-up signal database1600, and the corresponding pieces of the plurality of pieces of operation state information1620-1to1620-nmay be stored as values in the wake-up signal database1600.

The plurality of sound signals1610-1to1610-nmay have a format of a wave file, but is not limited thereto. However, the plurality of sound signals1610-1to1610-nmay be stored in a format of a binary data file encoded in a binary format in the wake-up signal database1600.

The plurality of pieces of operation state information1620-1to1620-nmay further include device information (for example, device identification information, a device type, etc.) of the household electrical device2000, in addition to information about operation states of the household electrical device2000. For example, first operation state information1620-1may be washing start information of a washing machine, and second operation state information1620-2may be washing completion information of a dish washer.

According to an embodiment of the disclosure, the plurality of pieces of operation state information1620-1to1620-nmay include information about errors or failures of the household electrical device2000, in addition to information about operation states of the household electrical device2000. For example, n-th operation state information1620-nmay be information about a door failure of a washing machine.

According to an embodiment of the disclosure, the plurality of sound signals1610-1to1610-nand the plurality of pieces of operation state information1620-1to1620-nmay be obtained by a user input received through a registration process, and stored in the wake-up signal database1600, but is not limited thereto. According to another embodiment of the disclosure, the plurality of sound signals1610-1to1610-nand the plurality of pieces of operation state information1620-1to1620-nmay be obtained from a server.

The wake-up signal sensing module1310may measure similarities between the sound signal received through the microphone1100and the plurality of sound signals1610-1to1610-nstored in advance in the wake-up signal database1600. According to an embodiment of the disclosure, the wake-up signal sensing module1310may perform pre-processing, such as frequency sampling, offset processing, etc., on the received sound signal, and convert the pre-processed sound signal into an acoustic feature vector by using at least one of mel-frequency cepstrum (MFCCs), filter-bank (FBANK), or feature space maximum likelihood linear regression (fMLLR). The wake-up signal sensing module1310may classify the acoustic feature vector into a predefined sound signal by using a classifier, and measure similarities to the plurality of sound signals1610-1to1610-nbased on a probability value representing a result of the classifying. The wake-up signal sensing module1310may measure similarities between sound signals by using an open source algorithm such as, for example, kaldi.

However, a method of measuring similarities between a received sound signal and the plurality of sound signals1610-1to1610-nin the wake-up signal sensing module1310is not limited to the above-described method. For example, the wake-up signal sensing module1310may use an algorithm for calculating a cross-correlation between wavelengths of frequencies of sound signals, cosine similarity, or a known method of measuring similarities between sound signals.

The wake-up signal sensing module1310may identify a sound signal having a greatest similarity from among the plurality of sound signals1610-1to1610-nstored in the wake-up signal database1600.

The wake-up signal sensing module1310may identify operation state information mapped to correspond to the identified sound signal from among the plurality of pieces of operation state information1620-1to1620-nstored in the wake-up signal database1600. For example, when the sound signal having the greatest similarity is a first sound signal1610-1among the plurality of sound signals1610-1to1610-n, the wake-up signal sensing module1310may identify first operation state information1620-1mapped in a key-value type to the first sound signal1610-1.

The wake-up signal sensing module1310may extract the identified operation state information, and provide the operation state information to the speaker1500or transfer the operation state information to the communication interface1400.

The speaker1500may output a notification message representing the operation state information provided from the wake-up signal sensing module1310. According to an embodiment of the disclosure, the speaker1500may output a voice message representing the operation state information, but is not limited thereto. However, the speaker1500may output notification sound including at least one of effect sound, bell sound, melody, music, or a song, set in advance according to the operation state information.

The communication interface1400may transmit the operation state information to the mobile terminal4000of the user through a wired or wireless communication network. The communication interface1400may transmit the operation state information to the mobile terminal4000by using at least one data communication network of, for example, wired local area network (LAN), wireless LAN, wireless-fidelity (Wi-Fi), Bluetooth, zigbee, Wi-Fi Direct (WFD), Bluetooth Low Energy (BLE), wireless broadband internet (Wibro), world interoperability for microwave access (WiMAX), shared wireless access protocol (SWAP), wireless gigabit alliance (WiGig), or radio frequency (RF) communication.

The mobile terminal4000may display the notification message4100representing the operation state information received from the communication interface1400of the electronic device1000. According to an embodiment of the disclosure, the mobile terminal4000may display the notification message4100in a form of a pop-up message, but is not limited thereto. However, the mobile terminal4000may output a voice notification message representing the operation state information through the speaker1500.

The mobile terminal4000may be a device registered in advance in the server with the same user account as that of a user of the electronic device1000.

FIG.2Billustrates an embodiment of the electronic device1000outputting operation state information of the household electrical device2000based on a sound signal output from the household electrical device2000and outputting a notification message through the mobile terminal4000, according to an embodiment of the disclosure.

Referring toFIG.2B, the electronic device1000may include the microphone1100, the wake-up signal sensing module1310, the communication interface1400, and the speaker1500. InFIG.2B, essential components for describing functions and/or operations of the electronic device1000are illustrated. However, components included in the electronic device1000are not limited to those illustrated inFIG.2B. For example, the electronic device1000may further include the wake-up signal database1600(seeFIG.2A).FIG.2Bis a view for describing data transmission/reception and operation execution between components, and arrows illustrated inFIG.2Brepresent transmission/reception directions of data.

The electronic device1000may receive a voice input from a user through the microphone1100. The ‘voice input’ may be a voice uttered by the user. The voice input may include a wake-up voice. The ‘wake-up voice’ may be a voice instruction which is received by the wake-up signal sensing module1310and wakes the electronic device1000up to perform a voice recognition function. The ‘wake-up voice’ may include, for example, ‘Hi Bixby’ or ‘Hello Galaxy’, etc. In the embodiment illustrated inFIG.2B, the microphone1100may receive a user's voice input “Hi Bixby! Let me know today's weather”.

According to an embodiment of the disclosure, the microphone1100may obtain a voice signal by converting the received voice input into a sound signal and removing noise (for example, a non-acoustic component) from the sound signal. The microphone1100may transmit the voice signal to the wake-up signal sensing module1310.

The wake-up signal sensing module1310may detect a wake-up signal (for example, a signal corresponding to Hi Bixby) included in the voice signal obtained from the microphone1100. As the wake-up signal sensing module1310detects the wake-up signal, the wake-up signal sensing module1310may switch a standby mode of the electronic device1000to a voice recognition mode for activating the voice recognition function. After the electronic device1000is switched to the voice recognition mode, the wake-up signal sensing module1310may provide the voice signal to the communication interface1400.

The communication interface1400may transmit the voice signal obtained from the wake-up signal sensing module1310to a server3000. Functions and/or operations of the communication interface1400have been described above with reference toFIG.2A, and therefore, redundant descriptions will be omitted.

After the microphone1100receives the voice input from the user, the microphone1100may receive a sound signal output from the household electrical device2000.

After the wake-up signal sensing module1310detects the wake-up signal included in the voice input received from the user and provides the voice signal to the communication interface1400, the wake-up signal sensing module1310may terminate operation. After the operation of the wake-up signal sensing module1310terminates, the sound signal of the household electrical device2000, received through the microphone1100, may be transferred to the communication interface1400. The communication interface1400may transmit the sound signal to the server3000.

The server3000may include sound signal database3400. Components included in the server3000are not limited to those illustrated inFIG.2B. The components of the server3000will be described in detail with reference toFIG.14.

The sound signal database3400may be database storing a plurality of sound signals3410-1to3410-nrelated to household electrical devices and a plurality of pieces of operation state information3420-1to3420-nof the household electrical devices respectively related to the plurality of sound signals3410-1to3410-n. According to an embodiment of the disclosure, the plurality of sound signals3410-1to3410-nand the plurality of pieces of operation state information3420-1to3420-nmay be stored in key-value types in the sound signal database3400.

The server3000may measure similarities between the sound signal received from the electronic device1000and the plurality of sound signals3410-1to3410-nstored in advance in the sound signal database3400, and identify a sound signal having a greatest similarity from among the plurality of sound signals3410-1to3410-n. A method of measuring similarities between sound signals in the server3000may be the same as that performed by the wake-up signal sensing module1310as described above with reference toFIG.2A, and therefore, redundant descriptions will be omitted. The server3000may identify operation state information mapped to correspond to the identified sound signal. For example, when the sound signal having a greatest similarity to the sound signal received from the electronic device1000among the plurality of sound signals3410-1to3410-nis a second sound signal3410-2, the server3000may identify second operation state information3420-2mapped to the second sound signal3410-2in a key-value type. The server3000may extract the identified operation state information and transmit the operation state information to the electronic device1000.

The electronic device1000may receive the operation state information from the server3000through the communication interface1400. The communication interface1400may provide the operation state information to the speaker1500, and the speaker1500may output a notification message representing the operation state information.

The server3000may transmit the operation state information to the mobile terminal4000. The mobile terminal4000may display a notification message4100representing the operation state information, received from the server3000. According to an embodiment of the disclosure, the mobile terminal4000may display the notification message4100in a form of a pop-up message.

Although not illustrated inFIG.2B, the server3000may generate a response message with respect to a voice signal (that is, a user's utterance) received from the communication interface1400of the electronic device1000. According to an embodiment of the disclosure, the server3000may convert the received voice signal into computer-readable text by using an automatic speech recognition (ASR) model, detect a domain, an intent, and a parameter included in the text by using a natural language understanding (NLU) model, and generate a response message corresponding to the domain, the intent, and the parameter by using a natural language generation (NLG) model. The server3000may transmit the response message to the electronic device1000. The electronic device1000may receive the response message through the communication interface1400, and output the received response message through the speaker1500.

In the embodiment illustrated inFIG.2A, the electronic device1000may detect a sound signal output from the household electrical device2000through the wake-up signal sensing module1310, as a wake-up signal, obtain operation state information corresponding to the sound signal, and output the operation state information. Because the embodiment illustrated inFIG.2Ais an embodiment in which the electronic device1000operates in an on-device state without any intervention of the server3000(seeFIG.2B), network cost may be not added, and high processing speed may be achieved. Also, because the electronic device1000operates according to a sound signal from the household electrical device2000stored in advance in the wake-up signal database1600through a registration procedure, user convenience may be improved.

The embodiment illustrated inFIG.2Bmay correspond to a case in which a voice input uttered by a user, in addition to a sound signal from the household electrical device2000, is received, unlike the embodiment illustrated inFIG.2A. According to the embodiment ofFIG.2B, the electronic device1000may perform a wake-up operation for switching from the standby mode (or a sleep mode) to the voice recognition mode by a wake-up signal included in a voice input received from a user, and, after the electronic device1000wakes up, the electronic device1000may transmit all received voice signals and sound signals to the server3000. A sound signal output from the household electrical device2000may be transmitted to the server3000, and the server3000may transmit operation state information corresponding to the sound signal to the electronic device1000. In the embodiment illustrated inFIG.2B, because the electronic device1000uses a communication network, network cost may be generated. The server3000may store a relatively larger amount of sound signals and related operation state information, than an on-device. Accordingly, in the embodiment illustrated inFIG.2B, sound signals that may be not identified in the embodiment ofFIG.2Amay be identified, and the corresponding operation state information may be provided. Accordingly, recognition accuracy of a sound signal output from the household electrical device2000may be improved.

FIG.3illustrates a block diagram of the electronic device1000according to an embodiment of the disclosure.

The electronic device1000may be a device for detecting a wake-up signal and performing an operation or function corresponding to the wake-up signal. According to an embodiment of the disclosure, the electronic device1000may be an AI speaker, but is not limited thereto. The electronic device1000may be a tablet PC, a laptop PC, a workstation, a wearable device, a home camera, or mobile medical equipment.

Referring toFIG.3, the electronic device1000may include the microphone1100, a processor1200, a memory1300, the communication interface1400, the speaker1500, and the wake-up signal database1600. The microphone1100, the processor1200, the memory1300, the communication interface1400, the speaker1500, and the wake-up signal database1600may be electrically and/or physically connected to each other.

The microphone1100may receive a sound signal or a voice signal. According to an embodiment of the disclosure, the microphone1100may receive a sound signal output from the household electrical device2000(seeFIGS.1,2A, and2B). The ‘sound signal’ may be at least one of effect sound, bell sound, a melody, music, or a voice signal.

According to an embodiment of the disclosure, the microphone1100may receive a voice input uttered by a user. The microphone1100may obtain a voice signal by converting the received voice input into a sound signal and removing noise (for example, a non-acoustic component) from the sound signal.

The microphone1100may provide the sound signal or the voice signal to the processor1200.

The processor1200may execute one or more instructions of a program stored in the memory1300. The processor1200may be configured as a hardware component for performing arithmetic, logic, and input/output operations and signal processing. The processor1200may be configured as at least one of, for example, a central processing unit (CPU), a microprocessor, a graphic processing unit (GPU), application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), or field programmable gate arrays (FPGAs), but is not limited thereto.

InFIG.3, the processor1200is illustrated to be a single element. However, the processor1200is not limited to this. According to an embodiment of the disclosure, the processor1200may be configured as one or plurality of processors.

According to an embodiment of the disclosure, the processor1200may be configured as a dedicated hardware chip that performs AI training.

The memory1300may store a program including instructions for outputting operation state information corresponding to the sound signal output from the household electrical device2000, based on the sound signal. The memory1300may store instructions and program codes that are readable by the processor1200. In the following embodiments, the processor1200may be implemented by executing the instructions or program codes stored in the memory1300.

The memory1300may be configured as at least one type of storage medium of, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, Secure Digital (SD) or eXtreme Digital (XD) memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), or an optical disk.

The memory1300may store data about the wake-up signal sensing module1310. A ‘module’ included in the memory1300means a unit for processing a function or operation that is performed by the processor1200, and may be implemented as software, such as instructions or program codes.

In the following embodiments, the processor1200may be implemented by executing the instructions or program codes stored in the memory1300.

The wake-up signal sensing module1310may be a module configured to detect a wake-up signal from a sound signal or a voice signal. As a wake-up signal is detected by the wake-up signal sensing module1310, the electronic device1000may perform a wake-up operation for switching from the standby mode (or sleep mode) to the voice recognition mode.

According to an embodiment of the disclosure, the wake-up signal sensing module1310may compare the sound signal detected as the wake-up signal with a plurality of sound signals stored in advance in the wake-up signal database1600, and identify a sound signal corresponding to the received sound signal from among the plurality of sound signals, according to a result of the comparing.

The processor1200may receive a sound signal or a voice signal through the microphone1100. When a sound signal output from the household electrical device2000is received through the microphone1100, the processor1200may compare the received sound signal with a plurality of sound signals stored in advance in the wake-up signal database1600and identify a sound signal corresponding to the received sound signal from among the plurality of sound signals by executing instructions or program codes of the wake-up signal sensing module1310.

According to an embodiment of the disclosure, the processor1200may measure similarities between the sound signal received through the microphone1100and the plurality of sound signals stored in the wake-up signal database1600, and identify a sound signal corresponding to the sound signal received through the microphone1100from among the plurality of sound signals based on the similarities. According to an embodiment of the disclosure, the processor1200may perform pre-processing, such as frequency sampling, offset processing, etc., on the received sound signal, and convert the pre-processed sound signal into an acoustic feature vector. The processor1200may convert the sound signal into the acoustic feature vector by using at least one of, for example, mel-frequency cepstrum (MFCCs), filter-bank (FBANK), or feature space maximum likelihood linear regression (fMLLR). The processor1200may calculate probability values with which the acoustic feature vector is classified into the plurality of sound signals stored in advance in the wake-up signal database1600, by using a classifier. The classifier may be at least one of, for example, logistic regression, support vector machine (SVM), decision tree, or random forest. The processor1200may measure similarities between sound signals by using an open source algorithm such as kaldi.

The processor1200may extract at least one sound signal of which a calculated probability value exceeds a preset threshold value, and identify a sound signal having a greatest probability value among the extracted at least one sound signal.

However, a method of measuring similarities between a sound signal received through the microphone1100and a plurality of sound signals stored in the wake-up signal database1600in the processor1200is not limited to the above-described method. According to another embodiment of the disclosure, the processor1200may measure similarities between sound signals by using an algorithm for calculating a cross-correlation between wavelengths of frequencies of sound signals, cosine similarity, or a known method of measuring similarities between sound signals.

The processor1200may identify operation state information corresponding to the sound signal identified from the wake-up signal database1600. The wake-up signal database1600may store a plurality of sound signals and a plurality of pieces of operation state information for the respective sound signals. The wake-up signal database1600may store the plurality of pieces of operation state information respectively corresponding to the plurality of sound signals. According to an embodiment of the disclosure, the plurality of sound signals and the plurality of pieces of operation state information may be stored in key-value types in the wake-up signal database1600. For example, the plurality of sound signals may be stored as keys in the wake-up signal database1600, and the corresponding pieces of the plurality of pieces of operation state information may be stored as values in the wake-up signal database1600. The processor1200may identify operation state information mapped to the identified sound signal in a key-value type from the wake-up signal database1600. An embodiment of identifying operation state information from the wake-up signal database1600based on similarities between sound signals in the processor1200will be described in detail with reference toFIG.8, below.

According to an embodiment of the disclosure, the processor1200may extract the identified operation state information from the wake-up signal database1600and provide the extracted operation state information to the communication interface1400. The communication interface1400may transmit the operation state information to the mobile terminal4000such that the operation state information is output in a form of a notification message on the mobile terminal4000.

According to another embodiment of the disclosure, the processor1200may provide the operation state information to the speaker1500.

The processor1200may provide follow-up operation information related to the operation state information to the communication interface1400or the speaker1500. According to an embodiment of the disclosure, the wake-up signal database1600may further store information about a follow-up operation that may be recommended as an operation following each of the plurality of pieces of operation state information, in addition to the plurality of sound signals and the plurality of pieces of operation state information. In the wake-up signal database1600, the plurality of sound signals, the plurality of pieces of operation state information, and a plurality of pieces of follow-up operation information may be paired with each other and stored. According to an embodiment of the disclosure, the processor1200may extract follow-up operation information corresponding to the identified operation state information from among the plurality of pieces of follow-up operation information stored in the wake-up signal database1600, and transmit the extracted follow-up operation information to the mobile terminal4000of the user through the communication interface1400or output the follow-up operation information in a form of a voice message through the speaker1500. An embodiment of providing follow-up operation information in the processor1200will be described in detail with reference toFIGS.9to12.

According to an embodiment of the disclosure, the electronic device1000may register a new household electrical device in the server3000. The processor1200may register the new household electrical device by transmitting device identification information of the new household electrical device to the server3000, through an application. The new household electrical device may be registered in the server3000with the same account as that of a user logged in the electronic device1000. Device identification information and a sound signal of the new household electrical device may have not been stored in the wake-up signal database1600. In this case, the processor1200may transmit, to the server3000, a signal requesting the server3000to provide device information of the new household electrical device and a sound signal of the new household electrical device. According to an embodiment of the disclosure, the processor1200may request the server3000to provide update data for updating the wake-up signal sensing module1310through the communication interface1400, and receive the update data for updating the wake-up signal sensing module1310from the server3000. The update data may include, for example, device identification information of the new household electrical device, a new sound signal that is output by the new household electrical device, and new operation state information corresponding to the new sound signal. The processor1200may update the wake-up signal sensing module1310by using the update data received from the server300. An embodiment of updating the wake-up signal sensing module1310by using update data received from the server3000in the processor1200will be described in detail with reference toFIG.13, below.

However, the disclosure is not limited to the embodiment, and the processor1200may receive a new sound signal output from the new household electrical device by using the microphone1100, and store the received new sound signal in the wake-up signal database1600. In this case, the processor1200may store the new sound signal output from the new household electrical device, together with device identification information of the new household electrical device, in the wake-up signal database1600.

When a voice input including a wake-up signal uttered by a user, for example, a voice, such as ‘Hi Bixby’ or ‘Hello Galaxy’, is received through the microphone1100, the processor1200may detect Hi Bixby included in the voice input as a wake-up signal by using instructions or program codes related to the wake-up signal sensing module1310. After the wake-up signal is detected, the processor1200may wake up, and after the processor1200wakes up, a sound signal of the household electrical device2000, received through the microphone1100, may be transmitted to the server3000(seeFIG.2B) through the communication interface1400without being detected or processed by the processor1200.

The processor1200may receive operation state information corresponding to a sound signal from the server3000through the communication interface1400. For example, the server3000may compare a sound signal received from the electronic device1000with the plurality of sound signals3410-1to3410-nstored in the sound signal database3400(seeFIG.2B), and identify a sound signal corresponding to the received sound signal. The server3000may search operation state information corresponding to the identified sound signal in the sound signal database3400. The server3000may transmit the searched operation state information to the electronic device1000.

The communication interface1400may be configured to perform data transmission/reception to/from the server3000or the mobile terminal4000through a wired or wireless communication network. The communication interface1400may perform data communication with the server3000or the mobile terminal4000by using at least one data communication network of, for example, a wired LAN, a wireless LAN, Wi-Fi, Bluetooth, zigbee, Wi-Fi Direct (WFD), Bluetooth Low Energy (BLE), wireless broadband internet (Wibro), world interoperability for microwave access (WiMAX), shared wireless access protocol (SWAP), wireless gigabit alliance (WiGig), or RF communication.

The communication interface1400may transmit the operation state information to the server3000or the mobile terminal4000through a control by the processor1200. According to an embodiment of the disclosure, the communication interface1400may transmit a sound signal or a voice signal to the server3000through a control by the processor1200.

The speaker1500may output an audio signal. The speaker1500may output a notification message representing the operation state information through a control by the processor1200. According to an embodiment of the disclosure, the speaker1500may output a voice message representing the operation state information, but is not limited thereto. However, the speaker1500may output notification sound including at least one of effect sound, bell sound, a melody, music, or a song, set in advance according to the operation state information.

Although not illustrated inFIG.3, the electronic device1000may further include a display. According to an embodiment of the disclosure, the display may display the notification message in a form of text or a graphic user interface (GUI) through a control by the processor1200. The display may be configured as at least one of, for example, a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT-LCD), an organic light-emitting diode (OLED) display, a flexible display, a three Dimensional (3D) display, or an electrophoretic display.

The wake-up signal database1600may be a storage medium storing a plurality of sound signals and a plurality of pieces of operation state information related to the plurality of sound signals. The wake-up signal database1600may be configured as a non-volatile memory. The non-volatile memory may be a storage medium capable of storing and maintaining information even when no supply voltage is supplied and using the stored information when a supply voltage is supplied. The non-volatile memory may include at least one of, for example, a flash memory, a hard disk, a solid state drive (SSD), a multimedia card micro type, a card type memory (for example, SD or XD memory), ROM, a magnetic disk, or an optical disk. InFIG.3, the wake-up signal database1600is illustrated to be a separate component from the memory1300, but is not limited thereto. According to an embodiment of the disclosure, the wake-up signal database1600may be a single component included in or integrated into the memory1300, but is not limited thereto. However, the wake-up signal database1600may be implemented in a form of an external memory not included in the electronic device1000, or as a web-based storage medium connected to the electronic device1000via the communication interface1400through wired/wireless communication.

FIG.4illustrates a flowchart of a method of operating the electronic device1000according to an embodiment of the disclosure.

In operation S410, the electronic device1000may receive a first sound signal output from the household electrical device2000. According to an embodiment of the disclosure, the electronic device1000may receive the first sound signal output from the speaker2100of the household electrical device2000through the microphone. The ‘first sound signal’ received by the electronic device1000through the microphone may be at least one of effect sound, bell sound, a melody, music, or a voice signal.

According to an embodiment of the disclosure, the electronic device1000may perform a pre-processing operation of removing noise (for example, a non-acoustic component) from the first sound signal output from the household electrical device2000.

In operation S420, the electronic device1000may detect the first sound signal as a wake-up signal. According to an embodiment of the disclosure, the first sound signal may have been registered in advance as a wake-up signal. The electronic device1000may detect the first sound signal as a wake-up signal by comparing the first sound signal with a sound signal registered in advance as a wake-up signal. When the first sound signal is identical to the sound signal registered in advance or a similarity between the first sound signal and the sound signal registered in advance exceeds a preset threshold value according to a result of the comparing, the electronic device1000may detect the first sound signal received from the household electrical device2000as a wake-up signal. As the electronic device1000detects the first sound signal as a wake-up signal, the electronic device1000which operates in the standby mode or the sleep mode may wake up to be switched to the voice recognition mode of performing the voice recognition function.

According to another embodiment of the disclosure, when the first sound signal received from the household electrical device2000is not similar to the sound signal registered in advance, the electronic device1000may ignore sound signal without performing any operation responding to the first sound signal.

In operation S430, the electronic device1000may identify first operation state information corresponding to the first sound signal from the wake-up signal database1600(seeFIGS.2A and3). According to an embodiment of the disclosure, the electronic device1000may compare the first sound signal with the plurality of sound signals1610-1to1610-n(seeFIG.2A) stored in advance in the wake-up signal database1600by using the wake-up signal sensing module1310(seeFIGS.2B and3), and identify a sound signal corresponding to the first sound signal received from the household electrical device2000in operation S410from among the plurality of sound signals1610-1to1610-naccording to a result of the comparing. According to an embodiment of the disclosure, the electronic device1000may measure similarities between the first sound signal received from the household electrical device2000in operation S410and the plurality of sound signals1610-1to1610-nstored in advance in the wake-up signal database1600, and identify a sound signal having a greatest similarity from among the plurality of sound signals1610-1to1610-n.

The electronic device1000may identify first operation state information mapped to correspond to the identified sound signal from among the plurality of pieces of operation state information1620-1to1620-n(seeFIG.2A) stored in advance in the wake-up signal database1600.

In operation S440, the electronic device1000may output a notification message representing the identified first operation state information. According to an embodiment of the disclosure, the electronic device1000may output a voice message representing the operation state information through the speaker1500(seeFIG.3), but is not limited thereto. However, the disclosure is not limited to this, and the electronic device1000may output notification sound including at least one of effect sound, bell sound, a melody, music, or a song, set in advance according to the operation state information, through the speaker1000.

According to an embodiment of the disclosure, the electronic device1000may display the notification sound in a form of text or a GUI.

FIG.5illustrates a flowchart of a method of operating the electronic device1000according to an embodiment of the disclosure, the household electrical device2000, and the mobile terminal4000.

FIG.5illustrates an embodiment in which the electronic device1000receives a sound signal output from the household electrical device2000without receiving a voice input uttered by a user.

In operation S510, the household electrical device2000may output a sound signal according to an operation state. The household electrical device2000may output a sound signal about state information representing starting, finishing, or progressing of a function and/or operation, or a sound signal representing an error or a failure. The sound signal output from the household electrical device2000may be at least one of effect sound, bell sound, a melody, music, or a sound signal. For example, when the household electrical device2000is a washing machine, the household electrical device2000may output a sound signal representing washing completion. When the household electrical device2000is a refrigerator, the household electrical device2000may output a sound signal representing door opening. When the household electrical device2000is a dryer, the household electrical device2000may output an error sound signal representing freezing of a drain hose.

In operation S520, the electronic device1000may receive the sound signal output from the household electrical device2000. According to an embodiment of the disclosure, the electronic device1000may receive the sound signal output from the household electrical device2000through the microphone. The microphone may perform a pre-processing operation of removing noise (for example, a non-acoustic component) from the received sound signal.

In operation S530, the electronic device1000may perform a wake-up operation by detecting the sound signal as a wake-up signal. According to an embodiment of the disclosure, when a sound signal is received from the household electrical device2000, the electronic device1000which operates in the standby mode or the sleep mode may perform a wake-up operation for performing a voice recognition function.

In operation S540, the electronic device1000may identify a sound signal corresponding to the sound signal output from the household electrical device2000from the wake-up signal database1600(seeFIGS.2A and3). According to an embodiment of the disclosure, the electronic device1000may measure similarities between the sound signal received from the household electrical device2000in operation S520and the plurality of sound signals1610-1to1610-nstored in advance in the wake-up signal database1600(seeFIGS.2A and3), and identify a sound signal having a greatest similarity from among the plurality of sound signals1610-1to1610-n.

In operation S550, the electronic device1000may identify operation state information mapped to correspond to the sound signal identified from the wake-up signal database1600. According to an embodiment of the disclosure, the electronic device1000may identify operation state information mapped to correspond to the identified sound signal from among the plurality of pieces of operation state information1620-1to1620-n(seeFIG.2A) stored in advance in the wake-up signal database1600. The electronic device1000may extract the operation state information from the wake-up signal database1600.

In operation S560, the electronic device1000may transmit the extracted operation state information to the mobile terminal4000. The electronic device1000may transmit the operation state information to the mobile terminal4000by using at least one data communication network of, for example, Wi-Fi, Bluetooth, zigbee, WFD, BLE, Wibro, WiMAX, SWAP, WiGig, or RF communication.

In operation S570, the mobile terminal4000may receive the operation state information from the electronic device1000, and output a notification message representing the operation state information. According to an embodiment of the disclosure, the mobile terminal4000may display the notification message in a form of a pop-up message, but is not limited thereto. However, the mobile terminal4000may output a voice notification message representing the operation state information through the speaker1500(seeFIG.3).

FIG.6illustrates a flowchart of a method of operating the electronic device1000according to an embodiment of the disclosure, the household electrical device2000, the server3000, and the mobile terminal4000.

FIG.6illustrates an embodiment in which the electronic device1000receives a sound signal output from the household electrical device2000after receiving a voice input uttered by a user.

InFIG.6, the electronic device1000and the mobile terminal4000may be devices registered in the server3000with the same user account.

In operation S610, the electronic device1000may perform a wake-up operation by receiving a wake-up voice from a user. The ‘wake-up’ voice may be a voice instruction for causing the electronic device1000being in a standby mode or a sleep mode to be switched to a voice recognition mode to perform a voice recognition function. The electronic device1000may receive a wake-up voice, such as, for example, ‘Hi Bixby’ or ‘Hello Galaxy’, uttered by a user.

The electronic device1000may perform a wake-up operation by detecting the wake-up voice as a wake-up signal. According to an embodiment of the disclosure, the electronic device1000may compare the received wake-up voice with a wake-up signal registered in advance by using the wake-up signal sensing module1310(seeFIGS.2B and3), and, when the electronic device1000determines that the received wake-up voice is identical to the wake-up signal registered in advance according to a result of the comparing, the electronic device1000may perform a wake-up operation.

In operation S611, the electronic device1000may receive a voice input from a user. According to an embodiment of the disclosure, the electronic device1000may receive a voice input by using the microphone1100(seeFIGS.2B and3). The ‘voice input’ may be a voice uttered by a user. The voice input may be, for example, “Let me know today's weather”.

In operation S612, the electronic device1000may convert the voice input into a voice signal. According to an embodiment of the disclosure, the electronic device1000may obtain a voice signal by converting the voice input received through the microphone1100into a sound signal and removing noise (for example, a non-acoustic component) from the sound signal.

In operation S613, the electronic device1000may transmit the voice signal to the server3000. The electronic device1000may transmit the voice signal to the server3000by using at least one data communication network of, for example, Wi-Fi, Bluetooth, zigbee, WFD, BLE, Wibro, WiMAX, SWAP, WiGig, or RF communication.

In operation S620, the household electrical device2000may output a sound signal according to an operation state. The household electrical device2000may output a sound signal about state information representing starting, finishing, or progressing of a function and/or operation, or a sound signal representing an error or a failure. Operation S620may be the same as operation S510illustrated inFIG.5, and therefore, redundant descriptions will be omitted.

In operation S630, the electronic device1000may receive a sound signal output from the household electrical device2000. The received sound signal may be a sound signal registered in advance as a wake-up signal. Operation S630may be the same as operation S520illustrated inFIG.5, and therefore, redundant descriptions will be omitted.

In operation S631, the electronic device1000may transmit the sound signal to the server3000. According to an embodiment of the disclosure, the electronic device1000may transmit the received sound signal to the server3000without detecting the sound signal as a wake-up signal. The electronic device1000may transmit the sound signal to the server3000by using at least one data communication network of, for example, Wi-Fi, Bluetooth, zigbee, WFD, BLE, Wibro, WiMAX, SWAP, WiGig, or RF communication.

In operation S640, the server3000may identify a sound signal corresponding to the sound signal received from the electronic device1000from the sound signal database3400(seeFIG.2B). The server3000may include the sound signal database3400. The sound signal database3400may store the plurality of sound signals3410-1to3410-n(seeFIG.2B) and the plurality of pieces of operation state information3420-1to3420-n(seeFIG.2B) for the respective sound signals3410-1to3410-n. According to an embodiment of the disclosure, the plurality of sound signals3410-1to3410-nand the plurality of pieces of operation state information3420-1to3420-nmay have been stored in key-value types in the sound signal database3400.

The server3000may measure similarities between the sound signal received from the electronic device1000and the plurality of sound signals3410-1to3410-nstored in advance in the sound signal database3400, and identify a sound signal having a greatest similarity from among the plurality of sound signals3410-1to3410-n.

In operation S641, the server3000may identify operation state information mapped to correspond to the sound signal identified from the sound signal database3400. According to an embodiment of the disclosure, the server3000may extract operation state information stored in a key-value type with the identified sound signal in the sound signal database3400. For example, when the sound signal having a greatest similarity to the sound signal received from the electronic device1000from among the plurality of sound signals3410-1to3410-nstored in advance in the sound signal database3400is a first sound signal, the server3000may extract first operation state information mapped to the first sound signal in a key-value type.

In operation S642, the server3000may transmit the operation state information to the electronic device1000.

In operation S643, the server3000may transmit the operation state information to the mobile terminal4000. InFIG.6, the server3000is illustrated to transmit operation state information to both the electronic device1000and the mobile terminal4000, but is not limited thereto. According to an embodiment of the disclosure, the server3000may perform operation S642of transmitting the operation state information to the electronic device1000, before operation S643of transmitting operation state information to the mobile terminal4000. Alternatively, the server3000may perform operation S643before operation S642.

In operation S650, the electronic device1000may output a notification message. The electronic device1000may output a notification message representing the operation state information. According to an embodiment of the disclosure, the electronic device1000may output a voice message representing the operation state information through the speaker1500(seeFIG.3), but is not limited thereto. However, the electronic device1000may output notification sound including at least one of effect sound, bell sound, a melody, music, or a song, set in advance according to the operation state information, through the speaker1500.

In operation S660, the mobile terminal4000may output a notification message representing the operation state information received from the server3000. According to an embodiment of the disclosure, the mobile terminal4000may display a notification message in a form of a pop-up message, but is not limited thereto. However, the mobile terminal4000may output a voice notification message representing the operation state information through the speaker1500.

According to an embodiment of the disclosure, the server3000may generate a notification message representing the operation state information. In this case, the server3000may generate a notification message representing the operation state information by using a NLG model3316(seeFIG.14). The generated notification message may be configured in a form of text. In this case, the server3000may transmit the notification message configured as text to the electronic device1000and the mobile terminal4000. Each of the electronic device1000and the mobile terminal4000may output the notification message received from the server3000.

FIG.7illustrates a flowchart of a method of operating the electronic device1000according to an embodiment of the disclosure, the household electrical device2000, the server3000, and the mobile terminal4000.

InFIG.7, a case in which the electronic device1000receives a voice input uttered by a user after receiving a sound signal output from the household electrical device2000will be described as an example. For example, the electronic device1000may receive ending sound (wake-up signal) of a washing machine, and then receive a wake-up voice (for example, Hi Bixby) and a voice input (for example, Let me know today's weather) from a user.

InFIG.7, the electronic device1000and the mobile terminal4000may be devices registered in advance in the server3000with the same user account.

In operation S710, the household electrical device2000may output a sound signal according to an operation state. The household electrical device2000may output a sound signal about state information representing starting, finishing, or progressing of a function and/or operation, or a sound signal representing an error or a failure. Operation S710may be the same as operation S510illustrated inFIG.5, and therefore, redundant descriptions will be omitted.

In operation S720, the electronic device1000may receive a sound signal output from the household electrical device2000. Operation S720may be the same as operation S520illustrated inFIG.5, and therefore, redundant descriptions will be omitted.

In operation S721, the electronic device1000may perform a wake-up operation by detecting the sound signal as a wake-up signal. Operation S721may be the same as operation S530illustrated inFIG.5, and therefore, redundant descriptions will be omitted.

In operation S722, the electronic device1000may determine whether a voice input uttered by a user is received. The voice input may be, for example, “Let me know today's whether”.

When no voice input is received from the user, the electronic device1000may identify a sound signal corresponding to the sound signal received from the household electrical device2000from the wake-up signal database1600(seeFIGS.2A and3), in operation S723.

In operation S724, the electronic device1000may identify operation state information mapped to correspond to the sound signal identified from the wake-up signal database1600.

Operations S723and S724may be the same as operations S540and S550illustrated inFIG.5, and therefore, redundant descriptions will be omitted.

When the electronic device1000receives a voice input uttered from the user, the electronic device1000may transmit the sound signal and a voice signal to the server3000, in operation S725. The electronic device1000may transmit the sound signal and the voice signal to the server3000by using at least one data communication network of, for example, Wi-Fi, Bluetooth, zigbee, WFD, BLE, Wibro, WiMAX, SWAP, WiGig, or RF communication.

In operation S730, the server3000may separate the sound signal from the voice signal. According to an embodiment of the disclosure, the server3000may separate the sound signal from the voice signal by extracting the voice signal through a noise canceling function.

In operation S731, the server3000may identify a sound signal corresponding to the separated sound signal from the sound signal database3400. According to an embodiment of the disclosure, the server3000may compare the sound signal separated in operation S730with the plurality of sound signals3410-1to3410-n(seeFIG.2B) stored in advance in the sound signal database3400(seeFIG.2B), and identify a sound signal corresponding to the separated sound signal according to a result of the comparing. According to an embodiment of the disclosure, the server3000may measure similarities between the sound signal separated in operation S730and the plurality of sound signals3410-1to3410-nstored in advance in the sound signal database3400, and identify a sound signal having a greatest similarity from among the plurality of sound signals3410-1to3410-n.

In operation S732, the server3000may identify operation state information mapped to correspond to the sound signal identified from the sound signal database3400. According to an embodiment of the disclosure, the server3000may extract operation state information mapped to the identified sound signal in a key-value type from among the plurality of pieces of operation state information3420-1to3420-n(seeFIG.2B) stored in advance in the sound signal database3400. For example, when the sound signal having the greatest similarity to the sound signal received from the electronic device1000from among the plurality of sound signals3410-1to3410-nstored in advance in the sound signal database3400is a first sound signal, the server3000may extract first operation state information mapped to the first sound signal in a key-value type.

In operation S733, the server3000may transmit the operation state information to the electronic device1000.

In operation S734, the server3000may transmit the operation state information to the mobile terminal4000.

In operation740, the electronic device1000may output a notification message representing the operation state information. Operation S740may be the same as operation S650illustrated inFIG.6, and therefore, redundant descriptions will be omitted.

In operation S750, the mobile terminal4000may output a notification message representing the operation state information. Operation S750may be the same as operation S660illustrated inFIG.6, and therefore, redundant descriptions will be omitted.

In the embodiment illustrated inFIG.7, the electronic device1000may determine whether to process a sound signal output from the household electrical device2000, according to whether a voice input is received from a user after a wake-up voice is received.

More specifically, when a voice input is received from a user, the electronic device1000may transmit both the sound signal and the voice signal to the server3000in operation S725, and the server3000may identify operation state information corresponding to the sound signal and transmit the operation state information to the electronic device1000. The electronic device1000may output a notification message representing the operation state information received from the server3000.

When no voice input is received from the user in operation S722, the electronic device1000may identify, as an on-device, operation state information corresponding to the sound signal from the wake-up signal database1600, without any intervention of the server3000, and output a notification message representing the identified operation state information.

FIG.8illustrates a flowchart of a method, performed by the electronic device1000, of identifying operation state information corresponding to a sound signal output from a household electrical device, according to an embodiment of the disclosure.

In operation S810, the electronic device1000may calculate similarities between the received first sound signal and the plurality of sound signals1610-1to1610-n(seeFIG.2A) stored in advance in the wake-up signal database1600(seeFIGS.2A and3). According to an embodiment of the disclosure, the electronic device1000may perform pre-processing, such as frequency sampling, offset processing, etc., on a sound signal received from the household electrical device2000, and convert the pre-processed sound signal into an acoustic feature vector. For example, the electronic device1000may convert the sound signal into the acoustic feature vector by using at least one of mel-frequency cepstrum (MFCCs), filter-bank (FBANK), or feature space maximum likelihood linear regression (fMLLR). The electronic device1000may calculate probability values with which the acoustic feature vector is classified into the plurality of sound signals1610-1to1610-n(seeFIG.2A) stored in advance in the wake-up signal database1600, by using a classifier. The classifier may be at least one of, for example, logistic regression, SVM, decision tree, or random forest. The electronic device1000may calculate similarities between the sound signal received from the household electrical device2000and the plurality of sound signals1610-1to1610-nstored in advance in the wake-up signal database1600, based on the calculated probability values. The probability values may be values ranging from 0 to 1. For example, a probability value being closer to 1 means a higher similarity.

However, a method of calculating a similarity of a sound signal in the electronic device1000is not limited to the above-described method. According to an embodiment of the disclosure, the electronic device1000may calculate similarities between sound signals by using an algorithm for calculating a cross-correlation between wavelengths of frequencies of sound signals, cosine similarity, or a known method of measuring similarities between sound signals.

In operation S820, the electronic device1000may compare the calculated similarities with a preset threshold value.

When a calculated similarity exceeds the preset threshold value, the electronic device1000may perform a wake-up operation by detecting the first sound signal as a wake-up signal, in operation S830. According to an embodiment of the disclosure, there may be one or more sound signals of which a calculated similarity to the first sound signal among the plurality of sound signals1610-1to1610-nstored in the wake-up signal database1600exceeds the preset threshold value.

According to an embodiment of the disclosure, when a similarity between the first sound signal and the plurality of sound signals1610-1to1610-nstored in the wake-up signal database1600exceeds the preset threshold value, the electronic device1000which operates in the standby mode or the sleep mode may perform a wake-up operation for switching from the standby mode to the voice recognition function.

In operation S840, the electronic device1000may determine a sound signal having a greatest similarity from among the plurality of sound signals1610-1to1610-nstored in the wake-up signal database1600. The electronic device1000may extract at least one sound signal of which a similarity calculated in operation S810exceeds the preset threshold value from among the plurality of sound signals1610-1to1610-n. According to an embodiment of the disclosure, the electronic device1000may identify a sound signal having a greatest similarity from among the extracted at least one sound signal.

The electronic device1000may determine the identified sound signal to be a sound signal corresponding to the sound signal received from the household electrical device2000.

In operation S850, the electronic device1000may identify operation state information corresponding to the identified sound signal from among the plurality of pieces of operation state information1620-1to1620-n(seeFIG.2A) stored in the wake-up signal database1600. The wake-up signal database1600may store the plurality of pieces of operation state information1620-1to1620-nmapped to the respective sound signals1610-1to1610-n. According to an embodiment of the disclosure, the plurality of sound signals1610-1to1610-nand the plurality of pieces of operation state information1620-1to1620-nmay be stored in key-value types in the wake-up signal database1600. The electronic device1000may identify operation state information mapped to the identified sound signal in a key-value type from the wake-up signal database1600.

When the calculated similarity is smaller than or equal to the preset threshold value according to the result of the comparing, the electronic device1000may be maintained in the standby mode. For example, when a sound signal output from a refrigerator has not been stored in the wake-up signal database1600, the electronic device1000may determine that the calculated similarity is smaller than or equal to the preset threshold value, and be maintained in the standby mode without waking up.

FIG.9illustrates an operation, by the electronic device1000, of outputting operation state information of the household electrical device2000and recommendation information for a follow-up operation based on a sound signal output from the household electrical device2000and transmitting the operation state information and the recommendation information for the follow-up operation to the mobile terminal4000, according to an embodiment of the disclosure.

Referring toFIG.9, the electronic device1000may receive a sound signal output from the speaker2100of the household electrical device2000, and output a notification message910representing operation state information corresponding to the received sound signal and a follow-up operation recommendation message920. An embodiment in which the electronic device1000wakes up by detecting a sound signal received from the household electrical device2000as a wake-up signal, and outputs the notification message910representing operation state information corresponding to the sound signal may be the same as that described above with reference toFIGS.1to8, and therefore, redundant descriptions will be omitted.

Follow-up operation information may be information about an operation recommended for a user as a follow-up operation related to the operation state information. For example, when the household electrical device2000is a ‘washing machine’ and operation state information is ‘door failure of the washing machine’, follow-up operation information may be ‘information about failure inspection and repair of the washing machine’. For example, the follow-up operation information may include ‘phone number of the A/S center’ or ‘location information of the A/S center’.

The electronic device1000may output the follow-up operation recommendation message920for recommending follow-up operation information for a user. In the embodiment illustrated inFIG.9, the electronic device1000may output, as the follow-up operation recommendation message920, a message “Would you like to take the failed part for inspection and repair? The phone number of the A/S center is 1588-XXXX. The nearest A/S center is at XX, XXX-ro, XX-gu, Seoul”.

The electronic device1000may transmit the operation state information and the follow-up operation information to the mobile terminal4000. The mobile terminal4000may receive the operation state information and the follow-up operation information from the electronic device1000, and output a notification message4100representing the received operation state information and a follow-up operation recommendation message4200for recommending a follow-up operation. According to an embodiment of the disclosure, the mobile terminal4000may display the notification message4100representing the operation state information of the household electrical device2000and the follow-up operation recommendation message4200according to the operation station information, in a form of a pop-up message. According to an embodiment of the disclosure, the mobile terminal4000may display the follow-up operation recommendation message4200in a form of a ‘shortcut link’ for executing a related application capable of performing the follow-up operation. In the embodiment illustrated inFIG.9, the mobile terminal4000may display, as the notification message4100representing the operation state information of the household electrical device2000, “failure of the washing machine door is detected”, and display, as the follow-up operation recommendation message4200, a phone application shortcut link4210of “call the washing machine A/S center” and a map application shortcut link4220of “search the address of the washing machine A/S center”.

According to another embodiment of the disclosure, the mobile terminal4000may output a voice message for representing operation state information of the household electrical device2000and recommending a follow-up operation through the speaker2100.

The electronic device1000according to an embodiment of the disclosure, as illustrated inFIG.9, may improve user convenience by outputting the follow-up operation recommendation message920for recommending a follow-up operation according to an operation state of the household electrical device2000, together with the notification message910representing operation state information of the household electrical device2000based on a sound signal output from the household electrical device2000. Particularly, by informing a phone number or address of an A/S center when any problem in operation state of the household electrical device2000, such as a stop, failure, error, etc. of the household electrical device2000, occurs, a user may intuitively detect a follow-up operation without himself/herself searching a related follow-up operation, resulting in an improvement of efficiency and satisfaction.

A method in which the electronic device1000outputs the follow-up operation recommendation message920related to the operation state information, in addition to the notification message910representing the operation state information according to the sound signal output from the household electrical device2000will be described in detail with reference toFIGS.10to12, below.

FIG.10illustrates an embodiment of the electronic device1000outputting operation state information of the household electrical device2000and recommendation information for a follow-up operation based on a sound signal output from the household electrical device2000, and transmitting the operation state information and the recommendation information for the follow-up operation to the mobile terminal4000, according to an embodiment of the disclosure.

Referring toFIG.10, the electronic device1000may include the microphone1100, the wake-up signal sensing module1310, the communication interface1400, the speaker1500, and the wake-up signal database1600. InFIG.10, essential components for describing functions and/or operations of the electronic device1000are illustrated. However, components included in the electronic device1000are not limited to those illustrated inFIG.10.

The electronic device1000may receive a sound signal output from the speaker2100of the household electrical device2000through the microphone1100. A ‘sound signal’ received through the microphone1100may be at least one of effect sound, bell sound, a melody, music, or a voice signal.

The wake-up signal sensing module1310may be a module configured to detect a sound signal output from the household electrical device2000as a wake-up signal. The wake-up signal sensing module1310may compare the sound signal detected as the wake-up signal with the plurality of sound signals1610-1to1610-nstored in advance in the wake-up signal database1600, and identify a sound signal corresponding to the sound signal received from the household electrical device2000from among the plurality of sound signals1610-1to1610-naccording to a result of the comparing. The wake-up signal sensing module1310may extract operation state information and follow-up operation information mapped to correspond to the identified sound signal from among the plurality of pieces of operation state information1620-1to1620-nand the plurality of pieces of follow-up operation information1630-1to1630-nstored in advance in the wake-up signal database1600.

The wake-up signal database1600may store a sound signal of the household electrical device2000, obtained in advance, operation state information of the household electrical device2000, indicated by the sound signal, and information about a follow-up operation that is recommended in connection with the operation state information. The plurality of sound signals1610-1to1610-n, the plurality of pieces of operation state information1620-1to1620-nrespectively related to the plurality of sound signals1610-1to1610-n, and the plurality of pieces of follow-up operation information1630-1to1630-nrespectively related to the plurality of pieces of operation state information1620-1to1620-nmay be stored in the wake-up signal database1600. According to an embodiment of the disclosure, the plurality of sound signals1610-1to1610-nmay be mapped as keys and stored in the wake-up signal database1600, and the related pieces of the plurality of pieces of operation state information1620-1to1620-nand the plurality of pieces of follow-up operation information1630-1to1630-nmay be mapped as values and stored in the wake-up signal database1600.

The plurality of pieces of follow-up operation information1630-1to1630-nmay be information about an operation recommended for a user as a follow-up operation related to an operation state of the household electrical device2000. According to an embodiment of the disclosure, the plurality of pieces of follow-up operation information1630-1to1630-nmay be obtained by a user input received through a registration process, and stored in the wake-up signal database1600, but is not limited thereto. However, the disclosure is not limited to, and the plurality of pieces of follow-up operation information1630-1to1630-nmay be obtained from the server3000.

As described above, the plurality of pieces of follow-up operation information1630-1to1630-nmay be mapped to respectively correspond to the plurality of sound signals1610-1to1610-nand the plurality of pieces of operation state information1620-1to1620-n, and then stored in the wake-up signal database1600. According to an embodiment of the disclosure, one piece of operation state information may be mapped to at least one piece of follow-up operation information. For example, when operation state information is ‘door failure of the washing machine’, follow-up operation information may include ‘phone number of the A/S center’ and ‘location information of the A/S center’.

The wake-up signal sensing module1310may identify operation state information corresponding to the sound signal output from the household electrical device2000from the wake-up signal database1600, and extract the operation state information. An embodiment of identifying operation state information corresponding to a sound signal received from the household electrical device2000in the wake-up signal sensing module1310may be the same as that described above with reference toFIGS.1to8, and therefore, detailed descriptions will be omitted.

The wake-up signal sensing module1310may identify follow-up operation information mapped to correspond to the identified operation state information from the wake-up signal database1600. For example, when a first sound signal1610-1is identified as a signal corresponding to the sound signal received from the household electrical device2000, the wake-up signal sensing module1310may identify first operation state information1620-1and first follow-up operation information1630-1mapped to correspond to the first sound signal1610-1from the wake-up signal database1600. The wake-up signal sensing module1310may extract the identified first operation state information1620-1and the identified first follow-up operation information1630-1from the wake-up signal database1600.

The wake-up signal sensing module1310may provide the extracted operation state information and the extracted follow-up information to the speaker1500or the communication interface1400.

The speaker1500may output a notification message representing the operation state information provided from the wake-up signal sensing module1310and a follow-up operation recommendation message for recommending a follow-up operation. According to an embodiment of the disclosure, the speaker1500may output a voice message representing operation state information and follow-up operation information.

The communication interface1400may transmit the operation state information and the follow-up operation information to the mobile terminal4000of the user through a wired or wireless communication network.

The mobile terminal4000may receive the operation state information and the follow-up operation information from the communication interface1400of the electronic device1000, and output a notification message representing the operation state information and a follow-up operation recommendation message4200for recommending a follow-up operation. According to an embodiment of the disclosure, the mobile terminal4000may display the follow-up operation recommendation message4200in a form of a ‘shortcut link’ for executing a related application capable of performing the follow-up operation. For example, the mobile terminal4000may display, as the notification message4100representing the operation state information of the household electrical device2000, “failure of the washing machine door is detected”, and display, as the follow-up operation recommendation message4200, a phone application shortcut link4210of “call the washing machine A/S center” and a map application shortcut link4220of “search the address of the washing machine A/S center”.

FIG.11illustrates a flowchart of a method of operating the electronic device1000according to an embodiment of the disclosure, the household electrical device2000, and the mobile terminal4000.

FIG.11illustrates an embodiment of receiving a sound signal output from the household electrical device2000and providing operation state information and follow-up operation information corresponding to the received sound signal, in the electronic device1000. In the embodiment illustrated inFIG.11, the electronic device1000may process a sound signal as an on-device without any intervention of the server3000.

In operation S1110, the household electrical device2000may output a sound signal according to an operation state. The household electrical device2000may output a sound signal about state information representing starting, finishing, or progressing of a function and/or operation, or a sound signal representing an error or a failure. Operation S1110may be the same as operation S510illustrated inFIG.5, and therefore, redundant descriptions will be omitted.

In operation S1120, the electronic device1000may receive the sound signal output from the household electrical device2000.

In operation S1130, the electronic device1000may perform a wake-up operation by detecting the sound signal as a wake-up signal.

In operation S1140, the electronic device1000may identify a sound signal corresponding to the sound signal output from the household electrical device2000, from the wake-up signal database1600(seeFIG.10).

In operation S1150, the electronic device1000may identify operation state information mapped to correspond to the sound signal identified from the wake-up signal database1600.

Operations S1120to S1150may be the same as operations S520to S550illustrated inFIG.5, and therefore, redundant descriptions will be omitted.

In operation S1160, the electronic device1000may identify follow-up operation information mapped to correspond to the operation state information from the wake-up signal database1600. The wake-up signal database1600may store the plurality of sound signals1610-1to1610-n(seeFIG.10), the plurality of pieces of operation state information1620-1to1620-n(seeFIG.10), and the plurality of pieces of follow-up operation information1630-1to1630-n(seeFIG.10). According to an embodiment of the disclosure, the plurality of sound signals1610-1to1610-nand the plurality of pieces of operation state information1620-1to1620-nmay be mapped in key-value types and then stored in the wake-up signal database1600. For example, the plurality of sound signals1610-1to1610-nmay be mapped as keys and stored in the wake-up signal database1600, and the corresponding pieces of the plurality of pieces of operation state information1620-1to1620-nand the plurality of pieces of follow-up operation information1630-1to1630-nmay be mapped as values and stored in the wake-up signal database1600.

The electronic device1000may identify follow-up operation information mapped to correspond to the operation state information identified in operation S1150from the wake-up signal database1600. For example, when the identified operation state information is first operation state information, the electronic device1000may identify first follow-up operation information mapped to correspond to the first operation state information from the wake-up signal database1600.

In operation S1170, the electronic device1000may transmit the operation state information and the follow-up operation information to the mobile terminal4000. The electronic device1000may transmit the operation state information and the follow-up operation information to the mobile terminal4000by using at least one data communication network of, for example, Wi-Fi, Bluetooth, zigbee, WFD, BLE, Wibro, WiMAX, SWAP, WiGig, or RF communication.

In operation S1180, the mobile terminal4000may receive the operation state information and the follow-up operation information from the electronic device1000, and output a notification message representing the operation state information and a follow-up operation recommendation message for recommending a follow-up operation. According to an embodiment of the disclosure, the mobile terminal4000may display the notification message and the follow-up operation recommendation message in a form of a pop-up message, but is not limited thereto. However, the mobile terminal4000may output a voice notification message representing the operation state information and follow-up operation information through the speaker1500(seeFIG.3).

FIG.12illustrates a flowchart of a method of operating the electronic device1000according to an embodiment of the disclosure, the household electrical device2000, the server3000, and the mobile terminal4000.

FIG.12illustrates an embodiment of receiving a voice input uttered by a user, then receiving a sound signal output from the household electrical device2000, and providing operation state information and follow-up operation information corresponding to the sound signal in the electronic device1000.

InFIG.12, the electronic device1000and the mobile terminal4000may be devices registered in advance in the server3000with the same user account.

In operation S1210, the electronic device1000may perform a wake-up operation by receiving a wake-up voice from the user. The ‘wake-up voice’ may be a voice instruction for causing the electronic device1000being in a standby mode or a sleep mode to be switched to a voice recognition mode for performing a voice recognition function. The ‘wake-up voice’ may include, for example, ‘Hi Bixby’ or ‘Hello Galaxy’, etc.

The electronic device1000may perform a wake-up operation by detecting the wake-up voice as a wake-up signal. According to an embodiment of the disclosure, the electronic device1000may compare the received wake-up voice with a wake-up signal registered in advance by using the wake-up signal sensing module1310(seeFIGS.2B and3), and when the electronic device1000determines that the received wake-up voice is identical to the wake-up signal registered in advance, the electronic device1000may perform a wake-up operation.

In operation S1211, the electronic device1000may receive a voice input from the user. According to an embodiment of the disclosure, the electronic device1000may receive the voice input by using the microphone1100(seeFIGS.2B and3). The ‘voice input’ may be a voice uttered by the user. The voice input may be, for example, “Let me know today's weather”.

In operation S1212, the electronic device1000may convert the voice input into a voice signal. According to an embodiment of the disclosure, the electronic device1000may obtain a voice signal by converting the voice input received from the microphone1100(see FIG.10) into a sound signal and removing noise (for example, a non-acoustic component) from the sound signal.

In operation S1213, the electronic device1000may transmit the voice signal to the server3000. In operation51214, the electronic device1000may deactivate wake-up signal sensing module.

In operation51220, the household electrical device2000may output a sound signal according to an operation state. The household electrical device2000may output a sound signal about state information representing starting, finishing, or progressing of a function and/or operation, or a sound signal representing an error or a failure.

In operation51230, the electronic device1000may receive the sound signal output from the household electrical device2000.

In operation S1231, the electronic device1000may transmit the sound signal to the server3000. According to an embodiment of the disclosure, the electronic device1000may transmit the received sound signal to the server3000, without operating the wake-up signal sensing module1310. The electronic device1000may transmit the sound signal to the server3000by using at least one data communication network of, for example, Wi-Fi, Bluetooth, zigbee, WFD, BLE, Wibro, WiMAX, SWAP, WiGig, or RF communication.

In operation51240, the server3000may identify operation state information and follow-up operation information corresponding to the sound signal. The server3000may include the sound signal database3400. According to an embodiment of the disclosure, the sound signal database3400may store a plurality of sound signals, a plurality of pieces of operation state information, and a plurality of pieces of follow-up operation information. According to an embodiment of the disclosure, the plurality of sound signals, the plurality of pieces of operation state information, and the plurality of pieces of follow-up operation information may be mapped in key-value types and stored in the sound signal database3400. For example, the plurality of sound signals may be stored as keys in the sound signal database3400, and the corresponding pieces of the plurality of pieces of operation state information and the plurality of pieces of follow-up operation information may be stored as values in the sound signal database3400.

The server3000may measure similarities between the sound signal received from the electronic device1000and the plurality of sound signals stored in advance in the sound signal database3400, and identify a sound signal having a greatest similarity from among the plurality of sound signals. The server3000may identify operation state information and follow-up operation information mapped to correspond to the sound signal identified by the sound signal database3400. According to an embodiment of the disclosure, the server3000may extract operation state information and follow-up operation information mapped to the identified sound signal in a key-value type and stored in the sound signal database3400. For example, when a sound signal having a greatest similarity to the sound signal received from the electronic device1000from among the plurality of sound signals stored in advance in the sound signal database3400is a first sound signal, the server3000may extract first operation state information and first follow-up operation information mapped to the first sound signal in a key-value type.

In operation51241, the server3000may transmit the operation state information and the follow-up operation information to the electronic device1000. In operation S1250, the electronic device1000may output notification message and follow-up operation recommendation message.

In operation51242, the server3000may transmit the operation state information and the follow-up operation information to the mobile terminal4000. In operation S1260, the mobile terminal4000may output notification message and follow-up operation recommendation message. InFIG.12, the server3000is illustrated to transmit the operation state information and the follow-up operation information to both the electronic device1000and the mobile terminal4000, but is not limited thereto.

FIG.13illustrates an embodiment of the electronic device1000updating the wake-up signal sensing module1310by using update data received from the server3000, according to an embodiment of the disclosure.

Referring toFIG.13, the electronic device1000may include the processor1200, the wake-up signal sensing module1310, the communication interface1400, and the wake-up signal database1600, and the server3000may include a communication interface3100, a processor3200, wake-up signal sensing module update data3320, and the sound signal database3400. In the electronic device1000and the server3000illustrated inFIG.13, essential components for describing a function and/or operation for updating the wake-up signal sensing module1310of the electronic device1000are illustrated. However, components included in the electronic device1000and the server3000are not limited to those illustrated inFIG.13.

The wake-up signal database1600may store the plurality of sound signals1610-1to1610-nand the plurality of pieces of operation state information1620-1to1620-n. The plurality of sound signals1610-1to1610-nand the plurality of pieces of operation state information1620-1to1620-n, stored in the wake-up signal database1600, may be obtained from a user input received from a user through a registration procedure and stored.

The server3000may include the sound signal database3400storing the plurality of sound signals3410-1to3410-n(3412) and the plurality of pieces of operation state information3420-1to3420-n(3422). According to an embodiment of the disclosure, a first sound signal3410-1to a n-th sound signal3410-n, stored in the sound signal database3400, may be the same as a first sound signal1610-1to a n-th sound signal1610-n, stored in the wake-up signal database1600of the electronic device1000. According to an embodiment of the disclosure, first operation state information3420-1to n-th operation state information3420-n, stored in the sound signal database3400, may be the same as first operation state information1620-1to n-th operation state information1620-n, stored in the wake-up signal database1600of the electronic device1000.

The server3000may further store a sound signal of a new household electrical device and operation state information related to the sound signal. Herein, the ‘new household electrical device’ may be a household electrical device newly purchased or obtained by a user of the electronic device1000and then registered in the server3000through the electronic device1000.

The electronic device1000may perform a registration procedure of logging in with a user account and then registering device identification information (for example, a device id, a device serial number, etc.) of the new household electrical device in a user account of the server3000. The new household electrical device may include a speaker without network function and outputs a sound signal representing an operation state.

The server3000may store a sound signal output from the new household electrical device and operation state information represented by the sound signal, through the registration procedure. In the embodiment illustrated inFIG.13, the sound signal database3400of the server3000may store a new sound signal3412output from the new household electrical device, and new operation state information3422of the new household electrical device, indicated by the new sound signal3412. The new operation state information3422stored in the sound signal database3400may further include device information about the new household electrical device that outputs the new sound signal3412. For example, the new operation state information3422may include device identification information (for example, device id) or information about a device type of the new household electrical device. A new sound signal output from the new household electrical device and new operation state information indicated by the new sound signal may have not been stored in the wake-up signal database1600of the electronic device1000by a user.

The server3000may store the wake-up signal sensing module update data3320for updating the wake-up signal sensing module1310of the electronic device1000in the memory1330. The wake-up signal sensing module update data3320may be data for updating the wake-up signal sensing module1310to the latest version such that the wake-up signal sensing module1310of the electronic device1000detects the new sound signal3412output from the new household electrical device to identify the new operation state information3422which is operation state information of the new household electrical device, indicated by the new sound signal3412.

When a user purchases a new household electrical device or registers a new household electrical device in the server3000through the electronic device1000, a sound signal output from the new household electrical device may have not been stored in the wake-up signal database1600of the electronic device1000. In this case, although a sound signal is output from the new household electrical device, the wake-up signal sensing module1310of the electronic device1000may fail to detect the output sound signal as a wake-up signal, and accordingly, the electronic device1000may fail to identify operation state information of the new household electrical device.

In operation S1310, the processor1200of the electronic device1000may transmit an update request signal for requesting an update of the wake-up signal sensing module1310to the server3000through the communication interface1400. According to an embodiment of the disclosure, the electronic device1000may receive a user input requesting an update of firmware or software constructing the wake-up signal sensing module1310, and transmit an update request query to the server3000according to the reception of the user input, but is not limited thereto. According to another embodiment of the disclosure, the processor1200may transmit a query requesting an update of the wake-up signal sensing module1310to the server3000every preset time period (for example, one hour, one day, one week, one month, etc.).

When the electronic device1000transmits an update request signal to the server3000, the electronic device1000may transmit version information of the wake-up signal sensing module1310, together with the update request signal, to the server3000.

In operation S1320, the server3000may transmit the wake-up signal sensing module update data3320to the electronic device1000through the communication interface3100, in response to the update request signal received from the electronic device1000. According to an embodiment of the disclosure, the processor3200of the server3000may check version information of the wake-up signal sensing module1310, received from the electronic device1000, and compare the version information of the wake-up signal sensing module1310with version information of the wake-up signal sensing module update data3320. When the wake-up signal sensing module1310is not the latest version according to a result of the comparing, the processor3200may control the communication interface3100to transmit the wake-up signal sensing module update data3320to the electronic device1000.

The wake-up signal sensing module update data3320may include the latest version of firmware or software for detecting the new sound signal3412to identify the new operation state information3422. According to an embodiment of the disclosure, the wake-up signal sensing module update data3320may access the sound signal database3400to obtain the new sound signal3412and the new operation state information3422, but is not limited thereto. However, the wake-up signal sensing module update data3320may include the new sound signal3412and the new operation state information3422.

In operations S1310and S1320, the communication interface1400of the electronic device1000and the communication interface3100of the server3000may transmit/receive data through a wired or wireless communication network. The electronic device1000and the server3000may transmit and receive a query signal or update data by using at least one data communication network of, for example, wired LAN, wireless LAN, Wi-Fi, Bluetooth, zigbee, WFD, BLE, Wibro, WiMAX, SWAP, WiGig, or RF communication.

In operation51330, the electronic device1000may obtain the new sound signal3412and the new operation state information3422from the wake-up signal sensing module update data3320received from the server3000, and store the obtained new sound signal3412and the obtained new operation state information3422in the wake-up signal database1600. According to an embodiment of the disclosure, the electronic device1000may store, in addition to the new sound signal3412and the new operation state information3422, device information of the new household electrical device, for example, device identification information and device type information of the new household electrical device in the wake-up signal database1600.

In operation51340, the processor1200of the electronic device1000may receive the wake-up signal sensing module update data3320through the communication interface1400.

In operation51350, the processor1200of the electronic device1000may update the wake-up signal sensing module1310by using the wake-up signal sensing module update data3320.

In the embodiment illustrated inFIG.13, the electronic device1000may update the wake-up signal sensing module1310to the latest version by using the wake-up signal sensing module update data3320received from the server3000. By updating the wake-up signal sensing module1310, the electronic device1000may sense the new sound signal3412output from the new household electrical device as a wake-up signal, and identify the new operation state information3422indicated by the new sound signal3412. The user who has purchased a new household electrical device without network function may detect operation state information of the new household electrical device according to a sound signal from the new household electrical device through the electronic device1000, without paying additional cost. Accordingly, the embodiment of the disclosure may improve user convenience and satisfaction without any financial burden.

FIG.14illustrates a block diagram of components of a server according to an embodiment of the disclosure.

The server3000may receive a sound signal from the electronic device1000, identify operation state information corresponding to the received sound signal, and transmit the identified operation state information to the electronic device1000, but is not limited thereto. However, the server3000may receive a voice signal generated by a user's utterance from the electronic device1000, interpret the received voice signal by using a natural language processing module3330, and transmit a response message according to a result of the interpretation on the voice signal to the electronic device1000.

Referring toFIG.14, the server3000may include the communication interface3100, the processor3200, the memory3300, and the sound signal database3400.

The communication interface3100may be configured to perform data transmission/reception to/from the electronic device1000or the mobile terminal4000through a wired or wireless communication network. The communication interface3100may perform data communication with the electronic device1000or the mobile terminal4000by using at least one data communication network of, for example, a wired LAN, a wireless LAN, Wi-Fi, Bluetooth, zigbee, WFD, BLE, Wibro, WiMAX, SWAP, WiGig, or RF communication.

According to an embodiment of the disclosure, the communication interface3100may receive at least one of a sound signal or a voice signal from the electronic device1000, through a control by the processor3200. The sound signal may be a signal output from the household electrical device2000, and the voice signal may be a signal converted from a voice input uttered by a user.

According to an embodiment of the disclosure, the communication interface3100may transmit operation state information to the electronic device1000or the mobile terminal4000through a control by the processor3200.

The processor3200may execute one or more instructions of a program stored in the memory3300. The processor3200may be configured with a hardware component for performing arithmetic, logic, and input/output operations and signal processing. The processor3200may be configured as at least one of, for example, a CPU, a microprocessor, a GPU, ASICs, DSPs, DSPDs, PLDs, or FPGAs, but is not limited thereto.

According to an embodiment of the disclosure, the processor3200may be configured as a dedicated hardware chip for performing AI training.

The memory3300may store instructions and program codes that are readable by the processor1200. In the following embodiments of the disclosure, the processor3200may be implemented by executing the instructions or program codes stored in the memory3300.

The memory3300may be configured with at least one type of storage medium of, for example, a flash memory type, a hard disk type, a multimedia card micro type, card type memory (for example, SD or XD memory), RAM, SRAM, ROM, EEPROM, and PROM, or an optical disk.

The memory3300may store instructions or program codes for each of an operation state identifying module3310, the wake-up signal sensing module update data3320, and the natural language processing module3330. A ‘module’ included in the memory3300means a unit for processing a function or operation that is performed by the processor3200, and may be implemented with software, such as instructions or program codes.

The operation state identifying module3310may be a module configured to identify an operation state of the household electrical device2000by detecting a sound signal received from the electronic device1000. According to an embodiment of the disclosure, the operation state identifying module3310may be configured to compare the sound signal with a plurality of sound signals stored in advance in the sound signal database3400, and identify a sound signal corresponding to the received sound signal from among the plurality of sound signals according to a result of the comparing. The operation state identifying module3310may be configured to identify operation state information corresponding to the identified sound signal among a plurality of pieces of operation state information stored in the sound signal database3400.

The processor3200may receive a sound signal or a voice signal from the electronic device1000through the communication interface3100. The processor3200may compare the received sound signal with the plurality of sound signals stored in advance in the sound signal database3400by executing the instructions or program codes of the operation state identifying module3310, thereby identifying a sound signal corresponding to the received sound signal from among the plurality of sound signals. According to an embodiment of the disclosure, the processor3200may measure similarities between the received sound signal and the plurality of sound signals stored in the sound signal database3400, and identify a sound signal corresponding to the sound signal received from the electronic device1000from among the plurality of sound signals based on the similarities. A method of measuring similarities between sound signals and identifying the corresponding sound signal in the processor3200of the server3000may be the same as that performed by the processor1200of the electronic device1000illustrated inFIG.3, and therefore, redundant descriptions will be omitted.

The processor3200may identify operation state information corresponding to the sound signal identified from the sound signal database3400. The sound signal database3400may store a plurality of sound signals, and a plurality of pieces of operation state information respectively corresponding to the respective sound signals. According to an embodiment of the disclosure, the plurality of sound signals and the plurality of pieces of operation state information may be stored in key-value types in the sound signal database3400. For example, the plurality of sound signals may be stored as keys in the sound signal database3400, and the corresponding pieces of the plurality of pieces of operation state information may be stored as values in the wake-up signal database1600. The processor3200may identify operation state information mapped to the identified sound signal in a key-value type from the sound signal database3400.

The processor3200may extract the identified operation state information from the sound signal database3400, and provide the extracted operation state information to the communication interface3100. The processor3200may control the communication interface3100to transmit the extracted operation state information to the electronic device1000.

According to an embodiment of the disclosure, the processor3200may generate a notification message representing the operation state information. The processor3200may generate a notification message representing the operation state information of the household electrical device2000by using a NLG model3338. For example, when the processor3200identifies operation state information ‘washing completion of the washing machine’ from the sound signal database3400, the processor3200may generate a notification message “Washing is completed by the washing machine” representing washing completion. As another example, when the processor3200identifies operation state information ‘door opening of the refrigerator’ from the sound signal database3400, the processor3200may generate a notification message “The door of the refrigerator opened. Close the door of the refrigerator” having an intent of close the door of the refrigerator.

The notification message generated by the processor3200may be data in a form of text. The processor3200may control the communication interface3100to transmit text data including the notification message to the electronic device1000or the mobile terminal4000of the user.

The wake-up signal sensing module update data3320may be data for updating the wake-up signal sensing module1310(seeFIG.13) of the electronic device1000. The wake-up signal sensing module update data3320may be data for updating the wake-up signal sensing module1310to the latest version such that the wake-up signal sensing module1310of the electronic device1000detects a new sound signal output from a new household electrical device to identify new operation state information which is operation state information of the new household electrical device, indicated by the new sound signal. The wake-up signal sensing module update data3320has been described above with reference toFIG.13, and therefore, redundant descriptions will be omitted.

The natural language processing module3330may be a module configured to detect a voice signal received from the electronic device1000and provide a response message according to the detected voice signal. The ‘voice signal’ may be a signal converted from a voice input uttered by a user and received by the electronic device1000. The voice signal may be received from the electronic device1000. The natural language processing module3330may include an ASR model3332, a NLU model3334, a dialogue management (DM) model, and a NLG model3338.

The ASR model3332may be a model configured to convert a voice signal into a form of text that is readable by the processor3200. The processor3200may perform ASR by executing instructions or program codes of the ASR model3332. The processor3200may perform ASR to convert a voice signal received from the electronic device1000into text.

The NLU model3334may be a model configured to interpret the text converted by the ASR model3332and detect an intent according to a result of the interpretation on the text. The ‘intent’ may be information representing an intention included in the user's utterance, and include a request, an inquiry, operation execution, etc. from the user. The processor3200may obtain intent information from the text by executing instructions or program codes constructing the NLU model3334. For example, when the text is “Let me know today's weather”, the processor3200may obtain intent information ‘request for weather information’. According to an embodiment of the disclosure, the NLU model3334may obtain information about a domain or parameter of the text, in addition to the intent, by interpreting the text.

The NLU model3334may be configured as a rule-based model, but is not limited thereto. According to an embodiment of the disclosure, the NLU model3334may be configured as an AI model. The NLU model3334may be configured as at least one of, for example, convolutional neural network (CNN), recurrent neural network (RNN), support vector machine (SVM), linear regression, logistic regression, Naïve Bayes, random forest, decision tree, or k-nearest neighbor algorithm, or a combination thereof.

The DM model3336may be configured to receive at least one piece of data of a domain, an intent, or a parameter output from the NLU model3334and output a related response message. According to an embodiment of the disclosure, the DM model3336may provide a response or an answer message according to an intention of a user's voice input. The processor3200may obtain a response message with respect to a user's intent by executing instructions or program codes configuring the DM model3336.

The NLG model3338may be a model configured to generate a response message received from the DM model3336in a form of a natural language. The processor3200may generate a response message in a form of a natural language by executing instructions or program codes configuring the NLU model3334. For example, the processor3200may generate a response message such as “The volume of the TV is raised by one level”.

The processor3200may transmit the generated response message to the electronic device1000through the communication interface3100. According to an embodiment of the disclosure, the processor3200may transmit the response message to the user's mobile terminal through the communication interface3100.

FIG.15illustrates a flowchart of a method of operating the server3000according to an embodiment of the disclosure.

In operation S1510, the server3000may receive a voice signal uttered by a user and a sound signal output from the household electrical device2000, from the electronic device1000. The ‘sound signal’ may be a signal output from the household electrical device2000, and the ‘voice signal’ may be a signal converted from a voice input uttered by the user.

In operation51520, the server3000may separate the voice signal from the sound signal. According to an embodiment of the disclosure, the server3000may extract the voice signal by using a noise canceling function, and separate the sound signal from the voice signal.

In operation S1530, the server3000may identify operation state information corresponding to the sound signal from the sound signal database3400(seeFIG.14). According to an embodiment of the disclosure, the server3000may compare the sound signal received in operation S1510with the plurality of sound signals3410-1to3410-n(seeFIG.2B) stored in advance in the sound signal database3400, and identify a sound signal corresponding to the sound signal received from the household electrical device2000from among the plurality of sound signals3410-1to3410-naccording to a result of the comparing. According to an embodiment of the disclosure, the server3000may measure similarities between the sound signal received from the electronic device1000in operation S1510and the plurality of sound signals3410-1to3410-nstored in advance in the sound signal database3400, and identify a sound signal having a greatest similarity from among the plurality of sound signals3410-1to3410-n.

The server3000may identify operation state information mapped to correspond to the identified sound signal from among the plurality of pieces of operation state information3420-1to3420-n(seeFIG.2B) stored in advance in the sound signal database3400.

In operation S1540, the server3000may transmit the identified operation state information to the mobile terminal4000. According to an embodiment of the disclosure, the server3000may transmit the operation state information to the electronic device1000, as well as the mobile terminal4000.

InFIG.15, the server3000is illustrated to transmit operation state information to the mobile terminal4000, but is not limited thereto. According to an embodiment of the disclosure, the server3000may generate a notification message representing the operation state information identified in operation S1530by using the NLG model3338(seeFIG.14). The server3000may transmit text data including the notification message to the mobile terminal4000of the user or the electronic device1000.

The program that is executed by the electronic device1000described in the present specification may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. The program may be performed by all systems that can execute computer-readable instructions.

Software may include computer programs, codes, instructions, or one or more combinations thereof, for independently or collectively instructing or configuring a processing device to operate as desired.

The software may be implemented as a computer program including instructions stored in computer-readable storage media. Examples of the computer-readable storage media may be a magnetic storage medium (for example, ROM, RAM, a floppy disc, a hard disc, etc.), an optical readable medium (for example, compact disc-read only memory (CD-ROM), digital versatile disc (DVD)), etc. The computer-readable recording medium may be distributed to computer systems over a network, in which computer-readable codes may be stored and executed in a distributed manner. The medium may be readable by a computer, stored in a memory, and executed by a processor.

The computer-readable storage media may be provided in a form of non-transitory storage media. Herein, ‘non-transitory’ means that the storage media do not include a signal and current and are tangible, without meaning that data is semi-permanently or temporarily stored in the storage media. For example, the ‘non-transitory storage media’ may include a buffer in which data is temporarily stored.

Also, a program according to the embodiments disclosed in this specification may be included in a computer program product and provided. The computer program product may be traded as goods between a seller and a purchaser as a commodity.

The computer program product may include a software (S/W) program or a computer-readable storage medium storing an S/W program. For example, the computer program product may include an S/W program product (for example, a downloadable application) electronically distributed through a manufacturing company of an electronic device or an electronic market (for example, Google Play Store or App Store). For electronic distribution, at least one part of the software program may be stored in a storage medium or temporarily created. In this case, the storage medium may be a storage medium of a server of a manufacturing company of a vehicle or the electronic device1000, a server of an electronic market, or a relay server that temporarily stores an S/W program.

The computer program product may include a storage medium of the server3000(seeFIGS.2B and15) or a storage medium of the electronic device1000in a system configured with the electronic device1000, the server3000(seeFIGS.2B and15), and another electronic device (for example, the mobile terminal4000(seeFIGS.2A and2B)). Alternatively, when there is a third device (for example, the mobile terminal4000) communicatively connected to the electronic device1000, the computer program product may include storage media of the third device. Alternatively, the computer program product may include an S/W program itself transmitted from the electronic device1000to another electronic device or the third device, or from the third device to the electronic device1000.

In this case, one of the electronic device1000, the other electronic device, and the third device may execute the computer program product to perform the method according to the embodiments of the disclosure. Alternatively, two or more of the electronic device1000, the other electronic device, and the third device may execute the computer program product to distribute and perform the method according to the embodiments of the disclosure.

For example, the electronic device1000may execute a computer program product stored in the memory1300(seeFIG.3) to control another electronic device communicatively connected to the electronic device1000to perform the method according to the embodiments of the disclosure.

As another example, the third device may execute a computer program product to control an electronic device communicatively connected to the third device to perform the method according to the embodiments of the disclosure.

When the third device executes a computer program product, the third device may download the computer program product from the electronic device1000and execute the downloaded computer program product. Alternatively, the third device may execute a computer program product provided in a pre-loaded state to perform the method according to the embodiments of the disclosure.

Although the embodiments of the disclosure have been described above in connection with limited embodiments and drawings, it shall be understood that various changes and modifications can be made from the above description by those skilled in the art to which the disclosure pertains. For example, an appropriate result can be achieved even if the technologies described above are performed in different orders from the method described herein, and/or the described components such as the computer system, the module, etc. are coupled or combined in different forms from the method described herein, or replaced or substituted by other components or equivalents thereof.