Patent Description:
An electronic device having a voice recognition function may receive a user utterance voice to be recognized through a microphone, and perform a recognition operation on the received user utterance voice. Long-distance voice recognition uses a multi-channel microphone to perform and process voice recognition regardless of a user's location. Most electronic devices having a remote voice recognition function employs a technology of using two or more microphones to receive sound signals and separate the sound signals. In this case, the separated sound signals may include voices and sounds output from the surroundings, for examples, speakers such as TVs and audio devices in addition to the user utterance voice.

However, when the electronic device performing the voice recognition uses a multi-channel microphone at a long distance in an environment where various peripheral devices are simultaneously used in the vicinity, voices and sound signals other than the user utterance voice are mixed, so there is a problem in that it is difficult to distinguish only the user utterance voice and the voice recognition performance deteriorates.

A speaker recognition technology may be applied to classify signals other than the user voice that affect the voice recognition performance. The speaker recognition technology has a problem in that a relatively large amount of computation and resources are inevitably used, the performance of the speaker recognition device is greatly affected, and user's personal information needs to be stored and used. EP<NUM>8o82o, <CIT>, <CIT> disclose a voice recognition device, wherein <CIT> and <CIT> disclose the usage of direction information or the duration of a determined direction for distinguishing user voice utterances from other emitted audio signals to improve voice recognition performance.

The disclosure is to provide an electronic device capable of improving voice recognition performance even at a long distance by easily distinguishing a user utterance voice from a sound signal, a method for controlling the same, and a storage medium in which a computer program is stored.

According to aspects of the invention, there is provided an electronic device according to claim <NUM> and a method according to claim <NUM>.

The above and other aspects, features, and advantages of certain embodiments of the disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings.

In the drawings, the same reference numbers or signs refer to components that perform substantially the same function, and the size of each component in the drawings may be exaggerated for clarity and convenience. However, the technical idea and the core configuration and operation of the disclosure are not limited only to the configuration or operation described in the following examples. In describing the disclosure, if it is determined that a detailed description of the known technology or configuration related to the disclosure may unnecessarily obscure the subject matter of the disclosure, the detailed description thereof will be omitted.

In the disclosure, an expression "have", "may have", "include", "may include", or the like, indicates existence of a corresponding feature (for example, a numerical value, a function, an operation, a component such as a part, or the like), and does not exclude existence of an additional feature.

In the disclosure, an expression "A or B", "at least one of A and/or B", "one or more of A and/or B", or the like, may include all possible combinations of items enumerated together. For example, "A or B", "at least one of A and B", or "at least one of A or B" may indicate all of (<NUM>) a case in which at least one A is included, (<NUM>) a case in which at least one B is included, or (<NUM>) a case in which both of at least one A and at least one B are included.

In embodiments of the disclosure, terms including ordinal numbers such as first and second are used only for the purpose of distinguishing one component from other components, and singular expressions include plural expressions unless the context clearly indicates otherwise.

In addition, in the embodiment of the disclosure, terms such as 'top', 'bottom', 'left', 'right', 'inside', 'outside', 'inner surface', 'outer surface', 'front', and 'rear' are defined based on the drawings, and shapes or positions of each component are not limited thereby.

An expression "configured (or set) to" used in the disclosure may be replaced by an expression "suitable for", "having the capacity to" "designed to", "adapted to", "made to", or "capable of" depending on a situation. A term "configured (or set) to" may not necessarily mean "specifically designed to" in hardware. Instead, an expression "an apparatus configured to" may mean that the apparatus may "do" together with other apparatuses or components. For example, a "sub-processor configured (or set) to perform A, B, and C" may mean a dedicated processor (for example, an embedded processor) for performing the corresponding operations or a generic-purpose processor (for example, a central processing unit (CPU) or an application processor) that may perform the corresponding operations by executing one or more software programs stored in a memory.

An electronic device <NUM> according to various example implementations of the disclosure may include, for example, at least one of a smartphone, a tablet PC, a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a medical device, a camera, and a wearable device that recognize voice commands. In some embodiments, the electronic device <NUM> may include at least one of, for example, a television, a Blu-ray player, a digital video disk (DVD) player, an audio player, a refrigerator, an air conditioner, a cleaner, an oven, a microwave oven, a washing machine, an air cleaner, a set-top box, a home automation control panel, a security control panel, a media box, a game console, an electronic dictionary, an electronic key, a camcorder, and a digital photo frame.

In other example implementations, the electronic device <NUM> may include at least one of various medical devices (for example, various portable medical measuring devices (such as a blood glucose meter, a heart rate meter, a blood pressure meter, a body temperature meter, or the like), a magnetic resonance angiography (MRA), a magnetic resonance imaging (MRI), a computed tomography (CT), a photographing device, an ultrasonic device, or the like), a navigation device, a global navigation satellite system (GNSS), an event data recorder (EDR), a flight data recorder (FDR), an automobile infotainment device, a marine electronic equipment (for example, a marine navigation device, a gyro compass, or the like), avionics, a security device, an automobile head unit, an industrial or household robot, a drone, an automatic teller's machine of a financial institute, a point of sales (POS) of a shop, and Internet of things (IoT) devices (for example, a light bulb, various sensors, an electric or gas meter, a sprinkler system, a fire alarm, a thermostat, a street light, a toaster, an exercise equipment, a hot water tank, a heater, a boiler, and the like).

In the disclosure, the term "user" may refer to a person using the electronic device <NUM> or a device (for example, an artificial intelligence electronic device) using the electronic device <NUM>.

<FIG> is a schematic diagram illustrating a voice recognition scenario of an electronic device according to a first example of the disclosure. The electronic device <NUM> may be implemented as an AI speaker that performs an operation by recognizing a voice uttered by a user. Obviously, the electronic device <NUM> is not limited to the AI speaker, but may be implemented as various electronic devices capable of identifying and recognizing a voice command uttered by the user.

Referring to <FIG>, the electronic device <NUM> may be controlled by the remote control <NUM> and may be connected to the server <NUM> through a network. A peripheral device <NUM> such as a TV may be disposed around the electronic device <NUM>. Obviously, the peripheral device <NUM> may include an audio device, a notebook computer, a smartphone, a cleaner, an air conditioner, a personal computer (PC), and the like.

User A can utter a voice query or a command on the electronic device <NUM>. At this time, the peripheral device <NUM> such as a TV may output another voice through a speaker together with a user utterance voice A.

The electronic device <NUM> may perform an operation of receiving the user utterance voice A and a sound signal S in which a speaker output voice B and/or sound C output from the peripheral device <NUM> is mixed to recognize and identify the user utterance voice A only In addition, the electronic device <NUM> may perform an operation of controlling the electronic device <NUM> according to the recognized user utterance voice A, such as searching for a query, outputting a search result, and controlling an IoT device.

The remote control <NUM> may transmit an IR signal for controlling the electronic device <NUM> or the acquired sound signal S to the electronic device <NUM>. The remote control <NUM> may transmit the sound signal S received by the remote control <NUM> to the electronic device <NUM> as it is. Alternatively, the remote control <NUM> may identify only the user utterance voice A from the sound signal S received by the remote control <NUM> and transmit the identified user utterance voice A to the electronic device <NUM>. Alternatively, the remote control <NUM> may identify only the user utterance voice A from the sound signal S received by the remote control <NUM> and transmit the recognized result to the electronic device <NUM>.

The server <NUM> may receive content and the sound signal S as they are, or only the user utterance voice A identified among the sound signals from each electronic device <NUM>. The server <NUM> may provide services such as identifying and recognizing the user utterance voice A from the sound signal S provided by the electronic device <NUM> or recognizing the received user utterance voice A. The server <NUM> may be implemented as one or more servers for each service.

<FIG> is a block diagram illustrating a configuration of the electronic device <NUM> of <FIG>.

Referring to <FIG>, the electronic device <NUM> may include an interface unit <NUM> capable of receiving or transmitting, for example, video and audio data from various external devices such as the remote control <NUM>, the server <NUM>, a set-top box, and a mobile device. The interface unit <NUM> may indirectly receive sound from various external devices, for example, the remote control <NUM> or the mobile device.

The interface unit <NUM> may include wired interface units <NUM> to <NUM>, shown as wired interface units <NUM> in <FIG>, and wireless interface units <NUM> to <NUM>, shown as wireless interface units <NUM> in <FIG>.

The wired interface unit <NUM> may include a terrestrial/satellite broadcasting antenna connection tuner, a connection interface for a cable broadcasting cable, and the like for receiving a broadcast signal.

The wired interface unit <NUM> may include HDMI, DP, DVI, Component, S-Video, composite (RCA terminal), and the like for connecting an image device.

The wired interface unit <NUM> may include a USB interface or the like for a general-purpose electronic device connection.

The wired interface unit <NUM> may include a connection interface for an optical cable device.

The wired interface unit <NUM> may include an audio device connection interface such as a headset, earphones, and external speakers.

The wired interface unit <NUM> may include a connection interface for wired network devices such as Ethernet.

The wireless interface unit <NUM> may include a connection interface for wireless network devices such as Wi-Fi, Bluetooth, ZigBee, Z-wave, RFID, WiGig, WirelessHD, ultra-wide band (UWB), wireless USB, and near field communication (NFC).

The wireless interface unit <NUM> may include an IR transmission/reception module for transmitting and/or receiving a remote control signal.

The wireless interface <NUM> may include a connection interface for mobile communication devices such as <NUM> to <NUM>.

The interface unit <NUM> may include a dedicated communication module for performing communication exclusively for each of the mobile device and the server.

The interface unit <NUM> may include a common communication module and the like for performing communication with both the mobile device and the server. For example, both the mobile device and the server may perform communication through the Wi-Fi module.

The interface unit <NUM> may include an input interface unit and an output interface unit. In this case, the input interface unit and the output interface unit may be integrated into one module or may be implemented as separate modules.

The electronic device <NUM> may include a first microphone <NUM> as a sound receiver that directly receives sound.

The first microphone <NUM> may receive the user utterance voice A, the voice B and/or sound C output from the peripheral device <NUM>, and the sound signal S in which other noises are mixed. Here, the user utterance voice A may include various voice commands for controlling the electronic device <NUM> or a device connected to the electronic device <NUM> through a wired or wireless network, for example, an IoT device.

The electronic device <NUM> may include a detector <NUM> capable of detecting a user in the vicinity.

The detector <NUM> may include a camera that photographs an image. The detector <NUM> may acquire a user image photographed by the camera. When a plurality of user utterance voices are received, the detector <NUM> may select a true user utterance voice A from the plurality of user utterance voices by referring to information such as a user's gaze direction, a user's face, and a user's mouth shape.

The detector <NUM> may include an infrared sensor, an ultrasonic sensor, or the like. The detector <NUM> may detect a user through an infrared sensor, an ultrasonic sensor, or the like, and may identify the user utterance voice A by comparing whether the location of the input voice matches the location of the detected user. The user detection may be performed through a login of a user account.

The electronic device <NUM> may include a first memory <NUM> capable of storing various data.

The first memory <NUM> is a storage medium readable by a computer and stores data without limitation. The first memory <NUM> is accessed by the first processor <NUM>, and performs operations such as reading, writing, modifying, deleting, and updating data by the first processor <NUM>.

The first memory <NUM> may store various types of information and contents received through the interface unit <NUM> from the remote control <NUM>, the server <NUM>, the USB, the wirelessly connected mobile device, or the like.

The data stored in the first memory <NUM> may include, for example, a voice recognition module (voice recognition engine) that performs an operation of removing echo and noise from the sound signal S, separating the sound signal S into a plurality of sound source signals, detecting signal characteristics of each sound source signal, identifying the user utterance voice A from the plurality of sound signals, and recognizing the user utterance voice A.

The voice recognition module can be excluded from the electronic device <NUM>. At this time, the received sound signal S may be transmitted to the server (voice recognition server) <NUM>.

The server (voice recognition server) <NUM> may be a speech to text (STT) server that has only a function of identifying the user utterance voice A from the sound signal S and converting the user utterance voice A related data into appropriate text or a main server that also performs the STT server function. The STT server may again transmit the voice recognition result data back to the electronic device <NUM> or directly transmit the voice recognition result data to another server.

The electronic device <NUM> and the server <NUM> may perform, for example, a preprocessing function to remove echo and noise, a function of separating a plurality of sound sources, a function of detecting signal characteristics of each sound source, a function of identifying a user utterance voice A, and a function of recognizing the identified user utterance voice A alone or separately.

The first memory <NUM> may include an operating system, various applications executable on the operating system, image data, additional data, and the like.

The first memory <NUM> includes a nonvolatile memory in which a control program is installed, and a volatile memory in which at least a part of the installed control program is loaded.

The first memory <NUM> may include at least one storage medium of a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, an SD or XD memory, or the like), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk.

The electronic device <NUM> may include an output interface <NUM> that displays an image or outputs a voice.

The output interface <NUM> may include a display that displays an image based on the processed image signal or a speaker that outputs a voice based on the processed voice signal.

After recognizing the user utterance voice A, the display may interpret the user's answer, query, or command and output the result of performing the operation as an image or UI.

The implementation scheme of the display is not limited, and the display may be implemented in various display panels such as liquid crystal, plasma, a light-emitting diode, an organic light-emitting diode, a surface-electron gun conduction electron-emitter, carbon nano-tube, and nano-crystal.

The display may additionally include an additional component according to the implementation method. For example, the display may include an LCD panel, an LCD panel driver for driving the LCD panel, and a backlight unit for supplying light to the LCD panel.

After recognizing the user utterance voice A, the speaker may interpret the user's answer, query, or command and output the result of performing the operation as a voice.

The electronic device <NUM> may include a first processor <NUM> that controls each component.

The first processor <NUM> may execute the voice recognition module stored in the first memory <NUM> to remove the echo and noise from the acquired sound signal S using the interface unit <NUM> or the first microphone <NUM>, separate a plurality of sound source signals from the sound signal S based on analysis of an independent vector, detect the signal characteristics of each sound source signal, identify the user utterance voice A from the plurality of sound source signals based on the predefined information based on the correlation between the detected signal characteristics and the user utterance voice, recognize the user's answer, query, or command included in the user utterance voice A, and perform the operation according to the user's answer, query, or command.

The separation of the plurality of sound source signals may be performed using the first microphone <NUM> and a blind source separation (BSS) technology to separate the sound source signals for each channel.

The signal characteristics of each sound source signal may include volumes (lufs and lkfs), a zero crossing rate (ZCR), and a size of an average volume in a certain section, and information on the begin of speech (BOS) and the end of speech (EOS).

For the identification of the user utterance voice A, the correlation between the detected signal characteristics and the user utterance voice may be predefined as follows.

First, since the voice has a low ZCR and a large average volume, a sound source signal having the ZCR lower than a first threshold and the average volume greater than a second threshold among the plurality of sound source signals is selected as the user utterance voice A, and the sound source signals corresponding to non-voice may be excluded.

The ZCR refers to a rate at which a signal changes from positive (+) to negative (-) or from negative (-) to positive (+), and may be defined as in Equation <NUM> below.

Here, S is a signal of length T, and II (StSt-<NUM> < o) is an indication function. Therefore, when a value obtained by multiplying the signal value of the current sample signal St and the signal value of the previous sample signal St-<NUM> is negative, <NUM> may return, and when the value is o or positive, o may return. As a result, if such an operation is performed and added as much as the length of the signal, the number of times the sound source signal crosses o may be obtained.

The first threshold of the ZCR may be set to be the number of times the sound source signal within the predetermined frame section crosses o.

The second threshold of the average volume may be set as the size (dB) of the average volume of the sound source signal within a predetermined frame period.

Here, the first threshold of the ZCR and the second threshold of the average volume may be set by collecting, analyzing and learning repeatedly measured ZCR and average volume for various sound source signals.

Table <NUM> shows an example of extracting the user utterance voice by applying, for each of the plurality of sound source signals A to C, condition <NUM> to check whether the ZCR is less than a preset first threshold and the average volume exceeds a preset second threshold. As illustrated in Table <NUM>, it may be seen that the user utterance voice corresponds to the sound source signal A that satisfies the condition <NUM>, among the plurality of sound source signals A to C.

Second, since the sound source output from the speaker is at an even volume to eliminate the inconvenience of suddenly decreasing or increasing the sound, when there is no arbitrary volume control, the sound source signal of the channel in which the volume in the certain section, for example, the size of the loudness unit relative to full scale (lufs) or loudness KWeighted relative to full scale (lkfs) remains constant may be identified as the speaker output voice. On the other hand, in the user utterance voice A, the size (lufs or lkfs) in the certain section is not kept constant. Here, a third threshold representing the degree of change in the volume (lufs or lkfs) may be set as a sum of volumes exceeding or below an average volume (lufs or lkfs) in a certain section.

Table <NUM> shows an example of extracting the user utterance voice by applying, for each of the plurality of sound source signals A to C, condition <NUM> to check whether the volume (lufs or lkfs) in a certain section is uneven. As illustrated in Table <NUM>, it may be seen that the user utterance voice corresponds to the sound source signal A that satisfies condition <NUM>, among the plurality of sound source signals A to C.

Third, since the speaker output voice from the speaker of the peripheral device <NUM> is output for a relatively long time unlike the utterance voice A, and thus the begin of speech (BOS) and the end of speech (EOS) may not be detected and the utterance is continuously maintained, the sound source signal in which the BOS and the EOS are detected may be selected as the user utterance voice A.

Table <NUM> shows an example of extracting the user utterance voice by applying, for each of the plurality of sound source signals A to C, condition <NUM> to check whether the BOS and EOS are detected in a certain section. As illustrated in Table <NUM>, it may be seen that the user utterance voice corresponds to the sound source signal A that satisfies the condition <NUM>, among the plurality of sound source signals A to C.

The identification of the user utterance voice A may be performed by combining two or more of the three conditions described above.

Table <NUM> shows an example of extracting the user utterance voice by combining the conditions <NUM> and <NUM> described above. As illustrated in Table <NUM>, it may be seen that the user utterance voice corresponds to the sound source signal A that satisfies all the conditions <NUM> and <NUM>, among the plurality of sound source signals A to C.

Table <NUM> shows an example of extracting the user utterance voice by combining the above-described conditions <NUM> and <NUM>. As illustrated in Table <NUM>, it may be seen that the user utterance voice corresponds to the sound source signal A that satisfies all the conditions <NUM> and <NUM>, among the plurality of sound source signals A to C.

Table <NUM> shows an example of extracting the user utterance voice by combining all the conditions <NUM> to <NUM> described above. As illustrated in Table <NUM>, it may be seen that the user utterance voice corresponds to the sound source signal A that satisfies all the conditions <NUM> to <NUM>, among the plurality of sound source signals A to C.

The recognition of the user utterance voice A may include processes of natural language understanding (NLU), natural language generation (NLG), text-to-sentence, user's answer, query or command recognition.

The recognition of the user's answer, query, or command may be performed based on the voice recognition model generated by collecting, processing, analyzing, and learning data related to voice recognition in advance.

In the voice recognition model, data collection, processing, and various model learning processes necessary for voice recognition using the same may be performed by the data processing and model generation. The data processing may collect, select, and process data to generate a voice corpus, information on word pronunciation, and a sentence corpus. In addition, in the model generation, acoustic modeling, pronunciation modeling, and language modeling are performed using data-processed information, so that a phoneme adaptation model, a pronunciation dictionary model, and a language model may be generated, respectively.

In particular, the first processor <NUM> may collect data for generating the voice recognition model, and may perform at least a part of data analysis, processing, and generation of result information using at least one of machine learning, a neural network, or a deep learning algorithm as a rule-based or artificial intelligence algorithm.

For example, the first processor <NUM> may perform functions of a learning unit and a recognition unit. The learning unit may perform, for example, a function of generating a trained neural network, and the recognition unit may perform a function of recognizing (or reasoning, predicting, estimating, and determining) data using the trained neural network. The learning unit may generate or update the neural network. The learning unit may obtain learning data to generate the neural network. For example, the learning unit may acquire the training data from the first memory <NUM> or the outside. The learning data may be data used for learning a neural network.

Before learning the neural network using the learning data, the learning unit may perform a preprocessing operation on the obtained learning data, or select data to be used for learning from among a plurality of learning data. For example, the learning unit may process or filter the learning data in a predetermined format, or process data in a form suitable for learning by adding/removing noise. The trained neural network may be constituted by a plurality of neural networks (or layers). The nodes of the plurality of neural networks have weights, and the plurality of neural networks may be connected to each other so that an output value of one neural network is used as an input value of other neural networks. Examples of the neural networks may include models such as a convolutional neural network (CNN), a deep neural network (DNN), a recurrent neural network (RNN), a restricted Boltzmann machine (RBM), a deep belief network (DBN), a bidirectional recurrent deep neural network (BRDNN), and deep Q-networks.

Meanwhile, the recognition unit may acquire target data in order to recognize the content of the user's answer, query, or command from the user utterance voice. The target data may be acquired from the first memory <NUM> or from the outside. The target data may be data to be recognized by the neural network. Before applying the target data to the trained neural network, the recognition unit may perform the preprocessing operation on the obtained target data, or select data to be used for recognition from among the plurality of target data. For example, the recognition unit may process or filter the target data in a predetermined format, or process data in a form suitable for recognition by adding/removing noise. The recognition unit may obtain an output value output from the neural network by applying the preprocessed target data to the neural network. According to various embodiments, the recognition unit may acquire a probability value (or a reliability value) together with the output value.

The first processor <NUM> includes at least one general-purpose processor that loads at least a part of the control program from the nonvolatile memory in which the control program is installed into the volatile memory and executes the loaded control program, and may be implemented as, for example, a central processing unit (CPU), an application processor (AP), or a microprocessor.

The first processor <NUM> may include a single core, a dual core, a triple core, a quad core, or a multiple-number core thereof. A plurality of first processors <NUM> may be provided. The first processor <NUM> may include, for example, a main processor and a sub processor operating in a sleep mode (for example, a mode in which only standby power is supplied). In addition, the processor, the ROM, and the RAM can be interconnected via an internal bus.

The first processor <NUM> may be implemented as a form included in a main SoC mounted on a PCB embedded in the electronic device <NUM>. In another embodiment, the main SoC may further include an image processor.

The control program may include a program(s) implemented in at least one of a BIOS, a device driver, an operating system, firmware, a platform, and an application program (application). The application program may be pre-installed or pre-stored at the time of manufacturing of the electronic device <NUM>, or may be installed based on data of the application program received from the outside when used later. The data of the application program may be downloaded to the electronic device <NUM> from an external server such as an application market. Such a server is an example of a computer program product, but is not limited thereto.

The remote control <NUM> may include a user input interface <NUM>, a second microphone <NUM>, a first communication interface <NUM>, and a second processor <NUM>.

The remote control <NUM> may be implemented as an IR remote control that transmits <NUM>-bit control information based only on the IR signal, a multi-brand remote (MBR) that transmits user input information input by, for example, buttons, voice, touch, drag, and the like, as an IR signal, a Bluetooth signal, a Wi-Fi signal, and the like, mobile terminals such as a smartphone in which a remote application (app) is installed, or the like.

The user input interface <NUM> may receive a button input through various function key buttons, a touch or drag input through a touch sensor, a voice input through the second microphone <NUM>, a motion input through a motion sensor, and the like.

The second microphone <NUM> may receive the sound signal. In this way, an analog signal of the received sound signal may be converted into a digital signal and transmitted to a control target, for example, electronic device <NUM> through the first communication interface <NUM>, for example, a Bluetooth communication module, a WiFi communication module, an infrared communication module, or the like. When the remote control <NUM> is implemented as a mobile terminal such as a smartphone having a voice recognition function, the input voice input may be transmitted to the electronic device <NUM> in the form of a control signal recognized through the voice recognition.

The analog sound signal received by the second microphone <NUM> may be converted into a digital signal and transmitted to the electronic device <NUM> through, for example, Bluetooth.

The first communication interface <NUM> may transmit the analog sound signal input from the second microphone <NUM> as a digital sound signal to the electronic device <NUM>.

In order to perform wireless communication, the first communication interface <NUM> may be configured to perform at least one communication of IR, radio frequency (RF), Wi-fi, Bluetooth, ZigBee, ultra-wide band (UWB), wireless USB, and near field communication (NFC).

The second processor <NUM> may control each component of the remote control <NUM>. The second processor <NUM> may transmit a control command corresponding to a button input, a touch input, a drag input, and a motion input to the electronic device <NUM> through the first communication interface <NUM>.

The second processor <NUM> may convert the analog voice signal input through the second microphone <NUM> into a digital sound signal and transmit the digital sound signal to the electronic device <NUM> through the first communication interface <NUM>. When the remote control <NUM> has the voice recognition function, the second processor <NUM> may recognize the input voice signal to transmit the corresponding command or query text to the electronic device <NUM> through the first communication interface <NUM>.

<FIG> is a block diagram illustrating a configuration of an electronic device <NUM> according to a second example of the disclosure. The electronic device <NUM> according to the second example may output, to the external device <NUM>, voice or video content or a control signal according to a result of performing voice recognition. For example, the electronic device <NUM> may output an image to a display device, audio to an audio device, and a control signal to an IoT device.

Obviously, the electronic device <NUM> according to the second example may include a display that displays a simple notification, a control menu, or the like.

Unlike the electronic device according to the first example illustrated in <FIG>, the electronic device <NUM> according to the second example may transmit the voice, image, or control signal according to the result of performing the voice recognition to the external device <NUM> connected to the interface unit <NUM>.

<FIG> is a block diagram illustrating a configuration of the server <NUM> according to an example of the disclosure.

The server <NUM> may be a voice recognition server or a content server. The server <NUM> may include both the voice recognition server and the content server.

The server <NUM> may include a second communication interface <NUM> that transmits and receives various data by performing network communication with an external device.

The second communication interface <NUM> performs network communication with a plurality of electronic devices <NUM>-<NUM> to <NUM>-n. The electronic devices <NUM>-<NUM> to <NUM>-n may transmit the sound signal itself to the server <NUM> when there is no function of identifying the user utterance voice from the sound signal. The electronic devices <NUM>-<NUM> to <NUM>-n may transmit the identified user utterance voice to the server <NUM> when there is a function of identifying the user utterance voice from the sound signal and there is no recognition function.

The second communication interface <NUM> may receive the voice recognition model used when each of the plurality of electronic devices <NUM>-<NUM> to <NUM>-n recognizes the user utterance voice.

The second communication interface <NUM> may transmit or update the voice recognition model and the like collected, learned, and generated under the control of the third processor <NUM> to the electronic devices <NUM>-<NUM> to <NUM>-n.

The second communication interface <NUM> may transmit the voice recognition result to the electronic device <NUM> that has transmitted the sound signal or the user utterance voice signal for the voice recognition among the electronic devices <NUM>-<NUM> to <NUM>-n.

In order to perform wireless communication with, for example, the plurality of electronic devices <NUM>-<NUM> to <NUM>-n, the second communication interface <NUM> may include an RF circuit that transmits/receives a radio frequency (RF) signal, and may be configured to perform at least one communication of Wi-fi, Bluetooth, ZigBee, ultra-wide band (UWB), wireless USB, and near field communication (NFC). The second communication interface <NUM> may perform wired communication with the plurality of electronic devices <NUM>-<NUM> to <NUM>-n and other devices through a wired local area network (LAN). Communications can be implemented in various other communication schemes in addition to connection units including a connector or a terminal for wired connection.

The server <NUM> may include a second memory <NUM> capable of storing various data without any limitation.

The second memory <NUM> may store a voice recognition module (voice recognition engine) when the server <NUM> is the voice recognition server.

The voice recognition module (voice recognition engine), for example, has all the preprocessing function of removing echo and noise, the function of separating a plurality of sound sources, the function of detecting signal characteristics of each sound source, the function of identifying the user utterance voice A, and the function of recognizing the identified user utterance voice A, or may perform only the function of recognizing the user utterance voice A.

The server <NUM> may include a third processor <NUM> that controls each component.

The third processor <NUM> may recognize the user utterance voice based on the voice recognition module (voice recognition engine) and the voice recognition model stored in the second memory <NUM>.

The third processor <NUM> may generate the voice recognition model by collecting, processing, analyzing, and learning data related to the voice recognition in advance.

The third processor <NUM> may generate the voice recognition model and distribute the generated voice recognition model to the electronic devices <NUM>-<NUM> to <NUM>-n.

<FIG> is a block diagram illustrating a configuration of recognizing and correcting the user utterance voice in the electronic device <NUM> according to an example of the disclosure, and <FIG> is a flowchart illustrating the method for recognizing and correcting the user utterance voice.

Referring to <FIG>, the electronic device <NUM> may include a multi-channel sound signal input module <NUM>, a preprocessing module <NUM>, a user utterance voice identification module <NUM>, a voice recognition module <NUM>, and an operation execution module <NUM>.

The multi-channel sound signal input module <NUM> receives a sound signal using the first microphone <NUM> or via a third-party device. The sound signal may be a mixture of various sound source signals such as the user's speech voice to be recognized, a voice and sound output from the surrounding speaker, a sound, noise, and echo from a musical instrument or an object, and the like.

The preprocessing module <NUM> may remove noise and echo from the sound source signals included in the sound signal. The preprocessing module <NUM> may include a noise filter and an acoustic echo suppression.

The noise filter may remove noise using a noise source sensor and a line remover. The noise source sensor may include an acceleration sensor and a vibration sensor. The line remover may remove noise using linear prediction that predicts a subsequent signal by linearly combining a previous signal.

The acoustic echo suppression may prevent the speaker sound from being fed back to the microphone input again and transmitted in the echo by a method for comparing the speaker output signal with the microphone input signal and then extracting the speaker output signal from the microphone input signal.

The user utterance voice identification module <NUM> may identify the user utterance voice among the plurality of sound source signals by the method as illustrated in <FIG>.

In step S11, the user utterance voice identification module <NUM> may separate the sound signal, from which the noise and echo are removed, into the plurality of sound source signals using a blind source separation (BSS) technology.

In step S12, the user utterance voice identification module <NUM> may detect signal characteristics for each of the plurality of separated sound source signals. The signal characteristics may include volumes (lufs and lkfs), a zero crossing rate (ZCR), and a size of an average volume in a certain section, and information on the begin of speech (BOS) and the end of speech (EOS) which are useful to identify the user utterance voice. The begin of speech (BOS) and the end of speech (EOS) of the voice segment may be detected by voice activity detection (VAD).

In step S13, the user utterance voice identification module <NUM> may identify a sound source signal corresponding to the user utterance voice based on the predefined association information between the detected signal characteristics and the user utterance voice.

The voice recognition module <NUM> may recognize the user's answer, query, or command included in the user utterance voice identified based on the voice recognition model.

The voice recognition model may be created by collecting, processing, analyzing and learning data related to the voice recognition in advance.

The operation execution module <NUM> may perform an operation according to the user's answer, query, or command corresponding to the recognized user utterance voice. For example, the user's answer, query, or command may include search, self-control of the electronic device <NUM>, control of an IoT device connected through a network, and the like.

<FIG> is a block diagram illustrating the voice recognition method by the electronic device <NUM> and a server <NUM> according to another example of the disclosure. Since the voice recognition module requires a large database (DB) to increase the voice recognition rate, it may be difficult for a small electronic device <NUM> having low performance to have a voice recognition function.

Referring to <FIG>, the electronic device <NUM> may include the multi-channel sound signal input module <NUM>, the preprocessing module <NUM>, the user utterance voice identification module <NUM>, and the operation execution module <NUM>, and the server <NUM> may include a voice recognition module <NUM>. Description of each module is similar to the description with reference to <FIG> and <FIG>, and therefore will be omitted.

After recognizing the user's answer, query, or command of the user utterance voice identified by the voice recognition module <NUM>, the server <NUM> may transmit the recognition result to the operation execution module <NUM> of the electronic device <NUM>.

As another example, the user utterance voice identification module <NUM> may be excluded from the electronic device <NUM> and provided in the server <NUM>.

As another example, the subject executing the recognized user's answer, query or command may be the server <NUM> or the third-party device other than the electronic device <NUM>.

<FIG> is a diagram illustrating a scenario for the user utterance voice and the voice (sound) output from the peripheral device <NUM>.

Referring to <FIG>, the AI speaker <NUM> may receive a voice command utterance of "turn up a volume" to the AI speaker <NUM> from which a user A outputs a song, receive a conversation that a character speaks "turn down a volume" on a TV <NUM>-<NUM> that airs a drama, and receive "noise" generated from a surrounding cleaner <NUM>-<NUM>. As a result, when simultaneously receiving the "turn up a volume" that is the utterance voice command of the user A", the "turn down a volume that is the output conversation of the TV <NUM>-<NUM>, and the "noise" of the cleaner <NUM>-<NUM>, the AI speaker <NUM> may perform the operation by excluding the "noise" of the cleaner and the "turn down a volume" that is the output conversation of the TV and identifying only the sound that the user A utters.

<FIG> is a flowchart illustrating a voice recognition method according to a first example of the disclosure.

In step S21, the multi-channel sound signal input module <NUM> receives a sound signal using the first microphone <NUM> or via the third-party device. The sound signal may be a mixture of various sound source signals such as the user's speech voice to be recognized, a voice and sound output from the surrounding speaker, a sound, noise, echo and the like from a musical instrument or an object.

In step S22, the preprocessing module <NUM> may remove the noise and echo from the sound source signals included in the sound signal.

In step S23, the user utterance voice identification module <NUM> may separate the sound signal, from which the noise and echo are removed, into the plurality of sound source signals using the blind source separation (BSS) technology.

In step S24, the user utterance voice identification module <NUM> may detect the volumes (lufs and lkfs) for each of the plurality of separated sound source signals.

In step S25, for each of the sound source signal, the user utterance voice identification module <NUM> may determine the sound source signal in a certain section as the speaker output voice when a rate of change of the size of the volume (lufs and lkfs) in the certain section is equal to or lower than the third threshold, and determine the sound source signal in a certain section as the user utterance voice when the rate of change is greater than the third threshold.

In step S26, the voice recognition module <NUM> may recognize the user's answer, query, or command included in the identified user utterance voice based on the voice recognition model.

<FIG> is a flowchart illustrating a voice recognition method according to a second example of the disclosure.

In step S31, the multi-channel sound signal input module <NUM> receives a sound signal using the first microphone <NUM> or via the third-party device. The sound signal may be a mixture of various sound source signals such as the user's speech voice to be recognized, a voice and sound output from the surrounding speaker, a sound, noise, echo and the like from a musical instrument or an object.

In step S32, the preprocessing module <NUM> may remove the noise and echo from the sound source signals included in the sound signal.

In step S33, the user utterance voice identification module <NUM> may separate the sound signal, from which the noise and echo are removed, into the plurality of sound source signals using the blind source separation (BSS) technology.

In step S34, the user utterance voice identification module <NUM> may detect the zero-crossing rate (ZCR) and the average volume for each of the plurality of separated sound source signals.

In step S35, the user utterance voice identification module <NUM> may, for each sound source signal, determine the sound source signal as the user utterance voice when the detected ZCR of the sound source signal is lower than the first threshold and the average volume of the sound source signal is greater than the second threshold.

In step S36, the voice recognition module <NUM> may recognize the user's answer, query, or command included in the identified user utterance voice based on the voice recognition model.

<FIG> is a flowchart illustrating a voice recognition method according to a third example of the disclosure.

In step S41, the multi-channel sound signal input module <NUM> receives a sound signal using the first microphone <NUM> or via the third-party device. The sound signal may be a mixture of various sound source signals such as the user's speech voice to be recognized, a voice and sound output from the surrounding speaker, a sound, noise, echo and the like from a musical instrument or an object.

In step S42, the preprocessing module <NUM> may remove the noise and echo from the sound source signals included in the sound signal.

In step S43, the user utterance voice identification module <NUM> may separate the sound signal, from which the noise and echo are removed, into the plurality of sound source signals using the blind source separation (BSS) technology.

In step S44, the user utterance voice identification module <NUM> may detect the begin of speech (BOS) and the end of speech (EOS) for each of the plurality of separated sound source signals by the voice activity detection (VAD).

In step S45, the user utterance voice identification module <NUM> may, for each sound source signal, determine whether the begin of speech (BOS) and the end of speech (EOS) of the voice segment are detected. When the begin of speech (BOS) and the end of speech (EOS) of the voice segment are detected, the sound source signal may be determined as the user utterance voice, and when the begin of speech (BOS) and the end of speech (EOS) of the voice segment are not detected, the sound source signal may be determined as the speaker output voice.

In step S46, the voice recognition module <NUM> may recognize the user's answer, query, or command included in the identified user utterance voice based on the voice recognition model.

<FIG> is a flowchart illustrating a voice recognition method according to an embodiment of the disclosure.

In step S51, the multi-channel sound signal input module <NUM> receives a sound signal using the microphone <NUM> or via the third-party device. The sound signal may be a mixture of various sound source signals such as the user's speech voice to be recognized, a voice and sound output from the surrounding speaker, a sound, noise, echo and the like from a musical instrument or an object.

In step S52, the preprocessing module <NUM> may remove the noise and echo from the sound source signals included in the sound signal.

In step S53, the user utterance voice identification module <NUM> may separate the sound signal, from which the noise and echo are removed, into the plurality of sound source signals using the blind source separation (BSS) technology.

In step S54, the user utterance voice identification module <NUM> may detect the begin of speech (BOS) and the end of speech (EOS) for each of the plurality of separated sound source signals by the voice activity detection (VAD).

In step S55, the user utterance voice identification module <NUM> determines each sound source signal as the speaker output voice when the ZCR is greater than or equal to the first threshold and the average volume is lower than or equal to the second threshold, and determine each sound source signal as the user utterance voice when the ZCR is lower than the first threshold and the average volume is greater than the second threshold, thereby performing step S56.

In step S56, the user utterance voice identification module <NUM> may determine each sound source signal as the speaker output voice when the rate of change of the size of the volume (lufs and lkfs) in the certain section is equal to or lower than the third threshold, and determine each sound source signal as the user utterance voice when the rate of change is greater than the third threshold, thereby performing step S57.

In step S57, the user utterance voice identification module <NUM> may determine whether the begin of speech (BOS) and the end of speech (EOS) of the voice segment of each sound source signal are detected. The user utterance voice identification module <NUM> may finally determine, as the user utterance voice, the sound source signal in which the begin of speech (BOS) and the end of speech (EOS) of the voice segment are detected.

In step S58, the voice recognition module <NUM> may recognize the user's answer, query, or command included in the identified user utterance voice based on the voice recognition model.

<FIG> is a schematic diagram illustrating the voice recognition scenario of the electronic device <NUM> according to the second example of the disclosure.

Referring to <FIG>, the electronic device <NUM> maybe connected to the server <NUM> through the network. The peripheral device <NUM> such as the TV <NUM> may be disposed around the electronic device <NUM>.

While the user A1 utters the voice query or command on the electronic device <NUM>, the peripheral device <NUM>, such as a TV, outputs a speaker output voice B, and other users A2 and A3, respectively, may communicate with user utterance voice A2 and user utterance voice A3. Accordingly, the electronic device <NUM> may receive the sound signal S in which a plurality of user utterance voices A1, A2, and A3 and the speaker output voice B and/or sound C, noise, echo, and the like output from the peripheral device <NUM> are mixed.

<FIG> is a flowchart illustrating the voice recognition method of the electronic device <NUM> of <FIG>.

In step S61, the multi-channel sound signal input module <NUM> receives the sound signal S using the microphone or via the third-party device. The sound signal S may be a mixture of various sound source signals such as the user utterance voice (A1) to be recognized, the surrounding user conversation voices A2 and A3, the surrounding speaker output voice B and the sound C, a sound D, noise N, echo E from a musical instrument or an object, and the like.

In step S62, the preprocessing module <NUM> may remove the noise and echo from the sound source signals included in the sound signal.

In step S63, the user utterance voice identification module <NUM> may use the blind source separation (BSS) technology that may separate the sound signal, from which the noise and echo are removed, into the plurality of sound signals, that is, the user utterance voice (A1), the surrounding user conversation voice A2, the surrounding user conversation voice A3, the surrounding speaker output voice B and sound C, and the sound D from the musical instrument or object.

In step S64, for each of the plurality of separated sound source signals, the user utterance voice identification module <NUM> may detect the begin of speech (BOS) and the end of speech (EOS) by voice activity detection (VAD), and the zero-crossing rate (ZCR), the average volume, the volumes (lufs and lkfs).

In step S65, the user utterance voice identification module <NUM> may identify a the user voice utterance voices A1, A2 and A3, based on the detection performed in step S64.

In step S66, the user utterance voice identification module <NUM> may identify the user utterance voice A1 among the plurality of user utterance voices A1, A2, and A3.

The method for identifying a user utterance voice A1 among a plurality of user utterance voices A1, A2, and A3 may identify the user utterance voice A1 using the detector <NUM>, for example, a camera, an infrared sensor, an ultrasonic sensor, and the like. For example, the detector <NUM> may photograph three users A1, A2, and A3 with a camera, and then exclude two users (A2, A3) through face recognition, gaze direction recognition, mouth shape recognition of a user, and the like.

The method for identifying a user utterance voice A1 among a plurality of user utterance voices A1, A2, and A3 may identify the user utterance voice A1 based on a user identification model and user registration or login of a user account. The user identification model may pre-store a user-specific voice characteristic pattern by registering a user in advance or logging in to a user account. For example, among the users A1, A2, and A3 of the plurality of identified user utterance voices A1, A2, and A3, only users A1 are registered or logged in, and when other users A2 and A3 are not registered, the user utterance voice A1 corresponding to the voice characteristic pattern of the registered or logged-in users A1 may be selected preferentially.

When all the users A1, A2, and A3 of the plurality of identified user utterance voices A1, A2, and A3 are registered or logged in, the user utterance voice A1 that is suitable for the user's answer, query, or command format may be selected by recognizing all the users A1, A2, and A3 in the voice recognition process of the next step S67 and then referring to the user utterance voices A1, A2, and A3.

When two or more of texts of each of the user utterance voices A1, A2, and A3 are suitable for the user's answer, query, or command format, a selection UI may be displayed on the display or a voice may be output to the speaker so that the user may select the two or more texts. In this case, the electronic device <NUM> may display the message "<NUM>) search request and <NUM>) volume up request have been recognized. Please select the desired request" or output the message as a voice.

When all the texts of each of the user utterance voices A1, A2, and A3 is not suitable for the user's answer, query, or command format, the additional operation may not be executed.

In addition, a method for identifying a user utterance voice A1 among a plurality of user utterance voices A1, A2, and A3 may use a user identification model based on data used in previous voice recognition. The user identification model may be generated by collecting, processing, analyzing, and learning a plurality of user utterance patterns, for example, sentences, vocabulary, and speech, for each user, from a previously recognized result.

In step S67, the voice recognition module <NUM> may perform the recognition of the identified user utterance voice A1 among the plurality of user utterance voices A1, A2, and A3.

The user identification modules that identifies the user utterance voice A according to the embodiment of the disclosure may be implemented as a computer program product stored in the first memory <NUM> as a computer-readable storage medium or a computer program product transmitted and received through network communication. In addition, the above-described user identification modules may be implemented as a computer program alone or integrated.

As described above, the electronic device according to the disclosure may separate the sound signal into the plurality of sound source signals, and then simply identify the user utterance voice included in the sound signal based on predefined information on the correlation whether the signal characteristics of each sound source signal, such as the change in size of the volume lufs or lkfs, the high and low of the ZCR, the voice start, and the begin of speech and the end of speech (BOS and EOS) are detected and the user utterance voices.

As a result, the electronic device according to the disclosure may improve remote voice recognition performance by simply identifying the user utterance voice among the sound signals mixed with the plurality of voices.

Claim 1:
An electronic device (<NUM>) for distinguishing a user utterance voice from other sound signals, comprising:
a sound receiver (<NUM>); and
a processor (<NUM>) configured to:
separate a sound signal (S) obtained through the sound receiver (<NUM>) into a plurality of sound source signals based on Blind Source Separation (BSS), and
identify a sound source signal, among the plurality of sound source signals identified to have characteristics satisfying predefined conditions as corresponding to a user voice,
characterized in that the predefined conditions include:
a predefined condition to identify whether a volume of the sound source signal over a certain period of time is constant,
a predefined condition to identify whether the zero crossing rate (ZCR) of the sound source signal is lower than a first threshold and an average volume of the sound source signal is greater than a second threshold, and
a predefined condition to identify whether a beginning of speech (BOS) and an end of speech (EOS) are detected in the sound source signal, and
the processor is configured to:
identify the sound source signal, where the volume over the certain period of time is not constant, the ZCR is lower than the first threshold and the average volume is greater than the second threshold, and the BOS and the EOS are detected, as corresponding to the user voice, and
identify the sound source signal, where the volume over the certain period of time is constant, the ZCR is greater than the first threshold and the average volume is lower than the second threshold, and the BOS and the EOS are not detected, as not corresponding to the user voice.