Patent Description:
There has been a research to improve accuracy of voice recognition. In particular, research on an acoustic model and a language model forming a voice recognition model has been actively developed. The acoustic model and the language model may be updated by training. In this case, accurate data for training for a correct answer may be required to improve voice recognition.

The speech recognition engine may be implemented as a neural network model using an artificial intelligence (AI) algorithm. An artificial intelligence (AI) system is a computer system which realizes a human-level intelligence that the machine learns, judges, and becomes smart, unlike the existing rule and statistics-based smart system. As the use of AI systems improves, a recognition rate and understanding or anticipation of a user's taste may be performed more accurately. As such, existing rule-based smart systems are gradually being replaced by deep learning-based AI systems.

AI technology is composed of machine learning (deep learning) and elementary technologies which utilizes machine learning. Machine learning is an algorithm technology that is capable of classifying or learning characteristics of input data. Element technology is a technology that simulates functions such as recognition and judgment of a human brain using machine learning algorithms such as deep learning. Machine learning is composed of technical fields such as linguistic understanding, visual understanding, reasoning, prediction, knowledge representation, motion control, or the like.

Examples of various fields in which AI technology is applied are as identified below. Linguistic understanding is a technology for recognizing, applying, and/or processing human language or characters and includes natural language processing, machine translation, dialogue system, question and answer, voice recognition or synthesis, and the like. Visual understanding is a technique for recognizing and processing objects as human vision, including object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image enhancement, and the like. Inference prediction is a technique for judging and logically inferring and predicting information, including knowledge-based and probability-based inference, optimization prediction, preference-based planning, recommendation, or the like. Knowledge representation is a technology for automating human experience information into knowledge data, including knowledge building (data generation or classification), knowledge management (data utilization), or the like. Motion control is a technique for controlling the autonomous running of the vehicle and the motion of the robot, including motion control (navigation, collision, driving), operation control (behavior control), or the like.

<CIT> discloses that to improve performance of these neural networks, they may be updated/retrained during run time by training the neural network based on the output of a speech recognition system or based on the output of the neural networks themselves. <CIT> relates to transcribing a spoken communication using a language model obtained based on information obtained about a different communication. <CIT> relates to apparatus and methods for speech to text conversion using automatic speech recognition. <CIT> relate to systems and methods of speech recognition for hearing impaired individuals. <CIT> concerns speech recognition.

A speech recognition model for recognizing speech that is input via an electronic device may generally be generated using data for training collected in an optimal environment. For example, a speech recognition model may be trained using data for training collected in an optimal experimental environment where soundproofing is good or is not interrupted by an obstacle.

However, there may be a variety of variables in the environment using the actual speech recognition model. For example, an environment in which a user utters a voice toward an electronic device may be noisy, and a user may utter a voice from a far distance towards the electronic device. Alternatively, there may be an obstacle between the user and the electronic device.

In this case, a speech recognition model trained in an optimal environment may not provide optimal performance in an actual speech utterance environment. Accordingly, there is a need to update the speech recognition model to take into account the actual utterance environment, and thus, optimal data for training that takes into account the actual utterance environment may be required to train the speech recognition model.

If there are a plurality of electronic devices that may respond to a user's utterance, data for training may be required for each speech recognition model in order to train the speech recognition models used by the plurality of electronic devices. Accordingly, a method of using data for training that may be representative to quickly and effectively update the respective speech recognition models may be required.

According to the disclosure, a correct answer (for example, a voice recognition result) corresponding to a voice uttered by a user may be obtained using an electronic device (for example, a first electronic device) used as a representative. In this case, it is possible to efficiently update (or train) the voice recognition model used by another electronic device (for example, the second electronic device) by using the voice data for the user voice and the voice recognition result of the user voice as data for training. In addition, the data for training may be used to train voice recognition models used by each of a plurality of other electronic devices (for example, the second electronic device and the third electronic device) together.

In addition, a place (for example, on a sofa, or the like. ) where user utterance frequently occurs may be determined to some extent. In this case, an electronic device (for example, the first electronic device) located closest to the user may recognize the user voice more accurately and obtain a correct voice recognition result corresponding thereto, than other electronic devices (for example, the second electronic device or the third electronic device) located at a long distance.

In this case, if an electronic device located near a user shares the voice recognition result obtained by the user with other electronic devices remotely located, the remotely located other electronic devices may obtain a correct voice recognition result corresponding to the voice recognized from the far distance. If the other electronic devices use the voice recognition result as data for training, the speech recognition model that each of the other electronic devices uses may be effectively trained.

In addition, various effects that are grasped directly or indirectly may be provided through the disclosure.

It includes various specific details to assist in that understanding.

The terms and words used in the following description are not limited to the bibliographical meanings, but, are merely used by the inventor to enable a clear and consistent understanding of the disclosure. Accordingly, it should be apparent to those skilled in the art that the following description of various embodiments of the disclosure is provided for illustration purpose only and not for the purpose of limiting the disclosure, while the invention is defined by the appended claims.

The terminology used herein may be used to describe various elements, but the elements should not be limited by the terms. The terms are only used to distinguish one element from another.

The term such as "first" and "second" used in various embodiments may modify various elements, but does not limit the corresponding elements. For example, the term does not limit an order and/or importance of the corresponding elements. The term may be used to distinguish one element from another.

For example, a first element may be referred to as a second element, and, similarly, a second element may be referred to as a first element, without departing from the scope of the various embodiments of the disclosure.

In the specification, when a part is "connected to" another part, it is not only the case where a part is "directly connected", but also is "electrically connected" to the other element by interposing another element therebetween. Further, it should be noted that any portion is "connected" to another portion includes where some portion is in a state capable of performing data communication via signal transmission and reception with another portion.

Some embodiments of the disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented in various numbers of hardware and/or software configurations that perform particular functions. For example, the functional blocks of the disclosure can be implemented by one or more microprocessors, or can be implemented by circuit configurations for a given function. Also, for example, the functional blocks of the disclosure can be implemented in various programming or scripting languages. The functional blocks can be implemented as an algorithm executed in one or more processors. In addition, the disclosure can employ techniques for electronic environment setting, signal processing, and/or data processing. The terms "element," "means," and "configuration" can be used broadly, and are not limited to mechanical and physical configurations.

In addition, the connection lines or connecting members between the components shown in the drawings are functional connections and/or physical or circuit connections. In an actual device, connection among elements may be represented by various functional connection, physical connection, or circuitry connection which are added or may be replaced.

In the embodiment, the term "speaker" may mean a person that uttered any speech, and the term "listener" may refer to a person who hears the uttered speech. In the conversation of a plurality of people, a speaker and a listener change from moment to moment and there may be confusion in distinguishing each person as the speaker and the listener. Thus, in the disclosure, based on the language used by each person, the person who uses the first language may be referred to as "the first speaker" and the person who uses the second language may be referred to as "the second speaker.

The term "text" in the embodiments described in this disclosure may refer to data that aims to convey the meaning to the other party. A "text" may include data in a natural language format such as a character, a symbol, a word, a phrase, a sentence, a diagram, a chart, or the like, data in a form of an artificial language character arrangements, or audio data. For example, "text" may mean that speech uttered by a speaker is represented as text or voice data.

Any of the instances of the term "voice recognition" in present disclosure may be substituted with the term "speech recognition" such that any aspect of the disclosure described with reference to voice recognition may equally be applicable to speech recognition. In addition, any aspect of the disclosure described with reference to voice recognition may equally be applicable to a combination of voice recognition and speech recognition. Further, any of the instances of the term "speech recognition" in present disclosure may be substituted with the term "voice recognition" such that any aspect of the disclosure described with reference to speech recognition may equally be applicable to voice recognition. Also, any aspect of the disclosure described with reference to speech recognition may equally be applicable to a combination of speech recognition and voice recognition.

Hereinafter, the disclosure will be further described with reference to the drawings.

<FIG> is a view of a system according to an embodiment of the disclosure.

A system <NUM> of <FIG> may include a first electronic device <NUM>, a second electronic device <NUM>, a third electronic device <NUM>, a first server <NUM>, and a second server <NUM>.

The first to third electronic devices <NUM>-<NUM>, for example, an electronic device in accordance with various embodiments of the disclosure, may include at least one of, for example, smartphones, tablet PCs, mobile phones, video telephones, electronic book readers, desktop PCs, laptop PCs, netbook computers, workstations, servers, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a medical device, a camera, or a wearable device. A wearable device may include at least one of the accessory type (e.g., a watch, a ring, a bracelet, a bracelet, a necklace, a pair of glasses, a contact lens or a head-mounted-device (HMD)); a fabric or a garment-embedded type (e.g., an electronic clothing); a body-attached type (e.g., a skin pad or a tattoo); or a bio-implantable circuit. In some embodiments, the electronic device may include at least one of, for example, a television, a digital video disk (DVD) player, audio, refrigerator, cleaner, ovens, microwaves, washing machines, air purifiers, set top boxes, home automation control panels, security control panels, media box (e.g.: Samsung HomeSyncTM, Apple TVTM, or Google TVTM), game consoles (e.g., XboxTM, PlayStationTM), electronic dictionary, electronic key, camcorder, or electronic frame.

In other embodiments, the electronic device may include at least one of a variety of medical devices (e.g., various portable medical measurement devices such as a blood glucose meter, a heart rate meter, a blood pressure meter, or a temperature measuring device), magnetic resonance angiography (MRA), magnetic resonance imaging (MRI), computed tomography (CT), a photographing device, or ultrasonic wave device, etc.), navigation system, global navigation satellite system (GNSS), event data recorder (EDR), flight data recorder (FDR), automotive infotainment devices, marine electronic equipment (e.g., marine navigation devices, gyro compasses, etc.), avionics, security devices, car head units, industrial or domestic robots, drone, automated teller machine (ATM)s of financial institutions, points of sale of stores, or Internet of Things (IoT) devices (e.g., light bulbs, sensors, sprinkler devices, fire alarms, thermostats, street lights, toasters, exercise equipment, hot water tanks, heater, boiler, etc.).

In another embodiment, the electronic device may include at least one of a furniture, a building/structure or a portion of a vehicle, an electronic board, an electronic signature receiving device, a projector, or various metrology instruments (e.g., water, electricity, gas, or radio wave measuring equipment, or the like). The electronic device according to an embodiment of the disclosure is not limited to the aforementioned devices. In this disclosure, the term user may refer to a person who uses an electronic device or a device (example: artificial intelligence electronic device) that uses an electronic device.

Referring to <FIG>, a user <NUM> (or a speaker) is illustrated. The user <NUM> may utter a voice at a specific time (e.g., at specific time or during specific time). For example, the user <NUM> may utter voice <NUM>, "what time is it now?". In this case, the first electronic device <NUM> located at a first distance (e.g., short distance) from the user <NUM> may obtain first voice data for the voice <NUM> uttered by the user <NUM> at the specific time through a microphone provided in the first electronic device <NUM> (①). The first voice data may be, for example, a digital signal obtained by converting the voice that is an analog signal uttered by the user <NUM> through a converter. At this time, the second electronic device <NUM> located at a second distance (e.g., middle distance) from the user <NUM> may also obtain second voice data for the voice <NUM> uttered by the user <NUM> at the specific time through a microphone provided in the second electronic device <NUM> (②). In addition, the third electronic device <NUM> located at a third distance (e.g., long distance) from the user <NUM> may also obtain the third voice data for the voice <NUM> uttered by the user <NUM> at the specific time through a microphone provided in the third electronic device <NUM> (③).

In this case, the first voice data, the second voice data, and the third voice data may be different from each other. For example, specification of microphones provided in the first electronic device <NUM>, the second electronic device <NUM>, and the third electronic device <NUM> are different from each other and thus, the voice data may be different from each other. In addition, the location of each of the first electronic device <NUM>, the second electronic device <NUM>, and the third electronic device <NUM> may be different from each other. For example, in <FIG>, the position at which the second electronic device <NUM> obtains the voice <NUM> is farther than the position where the first electronic device <NUM> obtains the voice <NUM>, and thus, when compared with the first voice data, the second voice data obtained by the second electronic device <NUM> may include time difference or phase difference, less amplitude, or more noise. Similarly, the location at which the third electronic device <NUM> obtains the voice <NUM> may be farther than the location where the second electronic device <NUM> obtains the voice <NUM>, when compared with the first voice data, the third voice data obtained by the third electronic device <NUM> may include more time difference or phase difference, less amplitude, or more noise.

Referring to <FIG>, when the first electronic device <NUM> obtains the first voice data, the first electronic device <NUM> may obtain a voice recognition result corresponding to the obtained first voice data. For example, the first electronic device <NUM> may apply the voice recognition result corresponding to the obtained first voice data to the speech recognition model provided in the first electronic device <NUM> and obtain the voice recognition result. Alternatively, the first electronic device <NUM> may transmit the first voice data to the first server <NUM> that may communicate with the first electronic device <NUM> (④). The first electronic device <NUM> may obtain the voice recognition result corresponding to the first voice data from the first server <NUM> (⑤).

To be specific, the first electronic device <NUM> may, by applying the obtained first voice data to the first electronic device <NUM> or the voice recognition model provided in the first server <NUM>, obtain text data as a result of the voice recognition result. The acoustic model may include information related to vocalization, and the language model may include information on unit phoneme information and a combination of unit phoneme information. The voice recognition module may convert the utterance of the user <NUM> into text data using the information related to the vocalization and information on the unit phoneme.

As another example, the first electronic device <NUM> may apply the obtained first voice data to the voice recognition model and the natural language understanding module provided in the first electronic device <NUM> or the first server <NUM> and obtain the user's intension as the voice recognition result.

As an embodiment, the natural language understanding module may recognize the intention of a user by performing syntactic analysis or semantic analysis. Grammatical analysis may divide the user input in grammatical units (e.g., words, phrases, morphemes, or the like), and may grasp which grammatical elements the divided units may have. The semantic analysis may be performed using semantic matching, rule matching, formula matching, or the like. Accordingly, the natural language understanding module may acquire domain, intent, or parameter (or slot) for expressing the intent by the user input.

As another embodiment, the natural language understanding module may determine user intention and parameters using the matching rule divided into a domain, an intention, and a parameter (or a slot) for grasping the intention. For example, the one domain (e.g., an alarm) may include a plurality of intents (e.g., alarm setting, alarm cancellation, or the like), and one intention may include a plurality of parameters (e.g., time, repetition times, alarm sound, or the like). The plurality of rules may include, for example, one or more mandatory element parameters. The matching rule may be stored in a natural language understanding database (NLU DB).

As a still another embodiment, the natural language understanding module may grasp the meaning of a word extracted from a user input using a linguistic characteristic (e.g., a grammatical element) such as a morpheme or a phrase, and determine a user intention by matching the grasped meaning with the domain and the intention. For example, the natural language understanding module may determine the user's intention by calculating how many words extracted from user input are included in each domain and intention. According to one embodiment, the natural language understanding module may determine the parameters of the user input using words that become a basis for understanding the intent. According to one embodiment, the natural language understanding module may determine the user's intention using the natural language recognition database in which the linguistic characteristic for grasping the intention of the user input is stored.

As still another embodiment, the natural language module may determine a user's intention using a personal language model (PLM). For example, the natural language understanding module may determine the user's intention using personalized information (e.g., contact list, music list). The personalized language model may be stored in, for example, natural language recognition database.

As another embodiment, the first electronic device <NUM> may apply the obtained first voice data to the voice recognition model and the natural language understanding module provided in the first electronic device <NUM> or the first server <NUM>, and obtain a path rule as the voice recognition result.

As an embodiment, the natural language understanding module may generate a path rule based on the intent and a parameter of a user input. For example, the natural language understanding module may select an application to be executed based on the intention of a user input, and determine an action to be performed in the selected application. The natural language understanding module may generate the path rule by determining a parameter corresponding to the determined action. According to one embodiment, the path rule generated by the natural language understanding module may include an application to be executed, an action (e.g., at least one state) to be executed in the application, and information about parameters for executing the action.

As another embodiment, the natural language understanding module may generate one path rule or a plurality of path rules based on the intention and parameter of the user input. For example, the natural language understanding module may receive a path rule set corresponding to the first electronic device <NUM> from a pass planner module, and determine the path rule by mapping the intention and the parameter of the user input to the received path rule set.

As still another embodiment, the natural language understanding module may generate one or a plurality of path rules by determining an application to be executed based on the intention and parameter of the user input, an action to be executed in an application, and a parameter for executing the action. For example, the natural language understanding module may generate the path rule by arranging the application to be executed or the action to be executed by the application in an ontology or graph model according to the intention of the user input using the information of the first electronic device <NUM>.

As still another embodiment, the natural language understanding module may select at least one path rule from among a plurality of generated path rules. For example, the natural language understanding module may select an optimal path rule from among the plurality of path rules. For example, the natural language understanding module may select a plurality of path rules when only some actions are specified based on user utterance. The natural language understanding module may determine one path rule among the plurality of path rules by an additional input of the user.

The first electronic device <NUM> may display voice recognition information <NUM> associated with the voice recognition result corresponding to the obtained first voice data on a screen. The first electronic device <NUM> may transmit the voice recognition result (e.g., text data, user's intention or path rule) corresponding to the obtained first voice data to the second electronic device <NUM> located at the second distance which is relatively farther than the location of the first electronic device <NUM> from the user <NUM> (⑥). In addition, the first electronic device <NUM> may transmit the obtained voice recognition result to the third electronic device <NUM> located at the third distance which is farther than the location of the second electronic device <NUM> from the user <NUM> (⑦).

The second electronic device <NUM> or the third electronic device <NUM> that obtains the voice recognition result corresponding to the first voice data associated with the voice <NUM> uttered by the user <NUM> may use the voice recognition result as a part of the data for training (or learning label, similar correct answer, or the like) for training of the voice recognition model.

The second electronic device <NUM> may form the second voice data for the voice uttered by the user <NUM> obtained in operation (<NUM>) and the voice recognition result corresponding to the first voice data received in operation ⑥ into a set and use it as data for training. For example, when the voice recognition model is located in the second electronic device <NUM>, the second electronic device <NUM> may update the voice recognition model by applying the {second voice data, voice recognition result obtained from the first electronic device <NUM>} as the data for training to the voice recognition model (⑧).

The third electronic device <NUM> may form the third voice data for the voice uttered by the user <NUM> obtained in operation ③ and the voice recognition result corresponding to the first voice data obtained in operation ⑦ into a set and use it as data for training. For example, when the voice recognition model is located in the second server <NUM>, the third electronic device <NUM> may transmit {third voice data, voice recognition result obtained from the first electronic device <NUM>} as the data for training to the second server <NUM> where the voice recognition model is located (⑨). The second server <NUM> may apply the received data for training to the voice recognition model and update the voice recognition model stored in the second server <NUM> (⑩).

According to the disclosure, the second electronic device <NUM> or the third electronic device <NUM> may use the voice data obtained by the second electronic device <NUM> or the third electronic device <NUM>, and the voice recognition result obtained by the first electronic device <NUM> as the data for training. In this case, the second electronic device <NUM> or the third electronic device <NUM> may determine whether the voice data and the voice recognition result are suitable to be used as the data for learning, and if it is determined that the data is suitable for data for training, may use the voice data and the voice recognition result as the data for training.

For example, in a situation where the voice uttered by the user <NUM> is a voice related to one function (e.g., a call function, a camera function, or the like) of the first electronic device <NUM>, the second electronic device <NUM> may obtain the voice data regarding the voice through a microphone provided in the second electronic device <NUM>, and obtain the voice recognition result corresponding to the voice from the first electronic device <NUM>. In this case, if the second electronic device <NUM> does not support the function, the second electronic device <NUM> may ignore the voice data and the voice recognition result corresponding to the voice without using them as the data for training. For this purpose, the first electronic device <NUM> may also transmit information on the target function (or target application) using the voice recognition result to the second electronic device <NUM> together with the voice recognition result corresponding to the voice. The second electronic device <NUM> may determine whether the second electronic device <NUM> supports information about a target function, and if it is determined that the second electronic device <NUM> does not support the function, the second electronic device <NUM> may ignore the voice data regarding the voice and the voice recognition result corresponding to the voice without using them as data for training.

The second electronic device <NUM> or the second server <NUM> may train the learning network model (or data recognition model) using the aforementioned data for training according to, for example, supervised learning method or unsupervised learning method based on AI algorithm. The learning network model may be, for example, a model based on neural network. The learning network model may include a plurality of network nodes having a weight. In the deep learning model, a plurality of network nodes is located at different depths (or layers), and may transmit and receive data according to a convolution connection relationship. Examples of learned determination models include, but are not limited to, Deep Neural Network (DNN), Recurrent Neural Network (RNN), and Bidirectional Recurrent Deep Neural Network (BRDNN).

The data learning module provided in the second server <NUM> or the second electronic device <NUM> which trains the data for training may be implemented as software modules or at least one hardware chip form. The data learning module may be manufactured in the form of an exclusive-use hardware chip for AI, or a general purpose processor (e.g., a CPU or an application processor) or a graphics-only processor (e.g., a GPU) and may be mounted on various electronic devices described above. Herein, the exclusive-use hardware chip for AI may be a dedicated processor for probability calculation, and may have a higher parallel processing performance than related art general purpose processors, so it may quickly process computational tasks in AI such as machine learning.

As an embodiment, when there is a plurality of pre-constructed learning network models, the data learning module may determine the learning network model having a large relevance with the input data for training as the learning network model to be trained.

As an embodiment, the data learning module may also train the learning network model using, for example, a learning algorithm including an error back-propagation method or a gradient descent.

As an embodiment, the data learning module may train, for example, the learning network model through the supervised learning with data for training as an input value. Alternatively, the data learning module, by learning by itself without a specific supervision, may train the learning network model through unsupervised learning to detect criterion. The data learning module may train the learning network model through reinforcement learning using, for example, feedback on whether the result of learning is correct.

As an embodiment, when there is a basic learning network model which is pre-trained in the second electronic device <NUM> or the second server <NUM>, the second electronic device <NUM> or the second server <NUM> may update the basic learning network model using the aforementioned learning data. Accordingly, the learning network model which is optimized and adapted to the environment where the second electronic device <NUM> is located is generated, and thus, the voice recognition result of the second electronic device <NUM> may be significantly improved.

<FIG> is a view of a system in a situation not considering the first electronic device <NUM> according to an embodiment of the disclosure.

Referring to <FIG>, the voice recognition model used by the second electronic device <NUM> may be the voice recognition model which is updated using {the second voice data, voice recognition result obtained from the first electronic device <NUM>} as the data for training, as described above, and the voice recognition model used by the third electronic device <NUM> may be the voice recognition model which is updated using {the third voice data, voice recognition result obtained from the first electronic device <NUM>} as the data for training as described in <FIG>.

Referring to <FIG>, the user <NUM> may utter voice <NUM> at the time different from the specific time when the voice <NUM> is uttered. The voice <NUM>, for example, may have the same or similar pronunciation, phoneme, meaning, or the like, as the voice <NUM>. For example, if the voice <NUM> of the specific time is "what time is it now?", the voice <NUM> uttered at the different times may be, for example, "what time is it now?", "what time now?", "what time?", or the like. As an embodiment, after a trigger voice (e.g., "refrigerator," "speaker," "Bixby," or the like) for activating a voice recognition function of at least one of the second electronic device <NUM> and the third electronic device <NUM> is uttered, the voice <NUM> may be uttered.

As an embodiment, when the second electronic device <NUM> is triggered in response to the voice <NUM> of the user <NUM>, the second electronic device <NUM> may obtain fourth voice data with respect to the voice <NUM> uttered through a microphone (⑪). When the fourth voice data is obtained, the second electronic device <NUM> may apply the fourth voice data to the voice recognition model updated in operation ⑧ and obtain the voice recognition result corresponding to the voice <NUM> (⑫). The second electronic device <NUM> may output voice recognition information <NUM> related to the obtained voice recognition result. For example, the voice recognition information <NUM> may be "the time is ~ o'clock" as a reply to the uttered voice <NUM>.

As another embodiment, when the third electronic device <NUM> is triggered in response to the voice <NUM> of the user <NUM>, the third electronic device <NUM> may obtain fifth voice data with respect to the voice <NUM> uttered through the microphone (⑬). The third electronic device <NUM> may transmit the obtained fifth voice data to the second server <NUM> that may communicate with the third electronic device <NUM> (⑭). The third electronic device <NUM> may obtain the voice recognition result corresponding to the fifth voice data from the second server <NUM> (⑮). The third electronic device <NUM> may display the voice recognition information <NUM> related to the obtained voice recognition result on a screen.

Accordingly, in general, the second electronic device <NUM> or the third electronic device <NUM> located far from the user <NUM> has a problem that it is difficult to accurately recognize the voice uttered by the user <NUM>, but according to the disclosure, the voice recognition model may be trained using the accurate voice recognition result together with the voice of the user <NUM> at the far distance, so that the second electronic device <NUM> or the third electronic device <NUM> may provide an accurate voice recognition result even for the voice uttered by the user <NUM> at a far distance.

<FIG> is a view of a system according to another embodiment of the disclosure.

Referring to <FIG>, the first electronic device <NUM>, the second electronic device <NUM>, the third electronic device <NUM>, the first server <NUM>, and the second server <NUM> respectively correspond to the first electronic device <NUM>, the second electronic device <NUM>, the third electronic device <NUM>, the first server <NUM>, and the second server <NUM> of <FIG> and <FIG>, and will not be further described.

Referring to <FIG>, when the user <NUM> utters the voice <NUM>, the first electronic device <NUM> may sense the voice <NUM> uttered by the user <NUM> at the specific time using a microphone provided in the first electronic device <NUM> (①-<NUM>). When the voice <NUM> is sensed, the first electronic device <NUM> may transmit a wake-up command to control to obtain the voice data for the voice <NUM> to the second electronic device <NUM> and the third electronic device <NUM> (①-<NUM>, ①-<NUM>).

The first electronic device <NUM> may obtain the first voice data for the voice <NUM> (①-<NUM>), obtain the voice recognition result corresponding to the first voice data, and transmit the voice recognition result to the second electronic device <NUM> and the third electronic device <NUM> (⑥, ⑦).

In the meantime, the second electronic device <NUM> which received the wake-up command may obtain the second voice data for the voice <NUM> (②), and the third electronic device <NUM> which received the wake-up command may obtain the third voice data for the voice <NUM> (③). For example, when the wake-up command is received during the operation in a low power mode, the second electronic device <NUM> or the third electronic device <NUM> may operate in a normal mode, turn on the microphone, and obtain the second voice data and the third voice data for the voice <NUM>, respectively.

When the voice recognition corresponding to the first voice data is received from the first electronic device <NUM>, the second electronic device <NUM> may update the voice recognition model using the second voice data for the voice <NUM> and the received voice recognition result as the data for training, and the third electronic device <NUM> may update the voice recognition model using the third voice data for the voice <NUM> and the voice recognition result received from the first electronic device <NUM> as the data for training. The operation ⑧ to operation ⑩ correspond to operations ⑧ to ⑩ of <FIG>, and duplicate description will be omitted.

Referring to <FIG>, the first electronic device <NUM>, the second electronic device <NUM>, the third electronic device <NUM>, the first server <NUM>, and the second server <NUM> of <FIG> respectively correspond to the first electronic device <NUM>, the second electronic device <NUM>, the third electronic device <NUM>, the first server <NUM>, and the second server <NUM> of <FIG> and <FIG>, respectively, and duplicate description will be omitted. At this time, a situation in which the first electronic device <NUM>, the second electronic device <NUM>, and the third electronic device <NUM> are located may be different from <FIG>, <FIG>, and <FIG>.

Referring to <FIG>, the first electronic device <NUM> and the second electronic device <NUM> may be at the same or similar distance from the user <NUM>, unlike <FIG>, <FIG>, and <FIG>. However, a peripheral environment of the first electronic device <NUM> and the second electronic device <NUM> may be different from each other. For example, the first electronic device <NUM> may be located in the inside in a room, and the second electronic device <NUM> may be located nearby a window. In this case, ambient noise at the second electronic device <NUM> may be higher than the ambient noise at the first electronic device <NUM> due to noise (for example, traffic noise, or the like) from the outside of the window.

In this case, the ambient noise at the first electronic device <NUM> is relatively low and thus the first electronic device <NUM> may obtain the first voice data for the voice <NUM> (①), transmit the first voice data to the first server <NUM> (③), and obtain the voice recognition result corresponding to the voice <NUM> (④). The first electronic device <NUM> may transmit the obtained voice recognition result to the second electronic device <NUM> at which the ambient noise is relatively higher (⑤).

The second electronic device <NUM> having relatively higher ambient noise may obtain the second voice data for the voice <NUM> when the user <NUM> utters voice (②), and obtain the voice recognition result corresponding to the voice <NUM> from the first electronic device <NUM> (⑤). The second electronic device <NUM> may transmit the obtained second voice data and the voice recognition result obtained from the first electronic device <NUM> as the data for training to the second server (⑥), and control so that the second server <NUM> may update the voice recognition model using the received data for training (⑦).

Accordingly, the second server <NUM> may be able to train the voice recognition model using the accurate voice recognition result even for the voice <NUM> received in an environment with high ambient noise. When the user <NUM> utters voice in a situation where the first electronic device <NUM> not present, the second electronic device <NUM> located in an environment with high ambient noise may be able to provide the voice recognition result corresponding to the voice uttered by the user <NUM> using the updated voice recognition model.

<FIG> is a block diagram illustrating a configuration of an electronic device according to an embodiment of the disclosure.

Referring to <FIG>, an electronic device <NUM> may be any one of the first electronic device <NUM>, the second electronic device <NUM>, and the third electronic device <NUM> as described in <FIG>, <FIG>, <FIG>, and <FIG>.

The electronic device <NUM> may include a processor <NUM>, a microphone <NUM>, a communicator <NUM>, and a memory <NUM>. However, the electronic device <NUM> may be implemented by more components than those illustrated in <FIG>. For example, the electronic device <NUM> according to an embodiment may be included in at least one of a home appliance, a mobile computing device, and a server, or may be connected to at least one of the home appliance, a mobile computing device, and a server wirelessly or by wire. Detailed examples of the processor <NUM>, the microphone <NUM>, the communicator <NUM>, and the memory <NUM> may correspond to a processor <NUM>, an audio module <NUM>, a communication module <NUM>, and a memory <NUM> of <FIG>, respectively.

The mic (or microphone) <NUM> may directly receive voice of the user <NUM> (or speaker). The microphone <NUM> may receive voice of the user <NUM> and provide the processor <NUM> with the voice data associated with the voice of the user <NUM>.

The processor <NUM> may execute a program stored in the memory <NUM> and control at least one other component (e.g., hardware or software component) of the electronic device <NUM> connected to the processor <NUM>.

In one embodiment, the processor <NUM> may obtain the first voice data for the voice uttered by the user <NUM> at a specific time through the microphone <NUM>. In addition, when the voice uttered by the user <NUM> is detected, the processor <NUM> may transmit a wakeup command to control the second electronic device <NUM> to obtain second voice data about the voice through the communicator <NUM>.

The processor <NUM> may obtain a voice recognition result corresponding to the obtained first voice data. For example, the processor <NUM> may transmit the obtained first voice data to an external voice recognition server through the communicator <NUM>. In response to the transmission of the first voice data, the processor <NUM> may transmit, through the communicator <NUM>, the voice recognition result to the second electronic device that has acquired the second voice data for the voice uttered by the user <NUM> at the specific time so that the second electronic device may use the voice recognition result as data for training the voice recognition model.

As an embodiment, the processor <NUM> may transmit time information related to voice to the second electronic device. The time information related to voice may include, for example, at least one of a time when the processor <NUM> recognizes a start of the voice or a time when the processor <NUM> recognizes an end of the voice.

<FIG> is a view illustrating a voice input to a first electronic device and a second electronic device according to an embodiment of the disclosure.

Specifically, referring to <FIG>, the user <NUM> may start utterance of the voice <NUM> at time t0 and end the utterance of the voice <NUM> at time t5. In this case, as shown in <FIG>, there may be a difference between the time when the voice signal arrives at the first electronic device <NUM> located at a short distance from the user <NUM> and the time when the voice signal arrives at the second electronic device <NUM> located at a long distance from the user <NUM>.

For example, the first electronic device <NUM> may recognize the start of the voice <NUM> of the user <NUM> at a time t1, and may recognize the end of the voice <NUM> of the user <NUM> at a time t6. The first electronic device <NUM> that recognizes the voice <NUM> of the user <NUM> at a time t1 may transmit a wake-up command of the microphone of the second electronic device <NUM> to the second electronic device <NUM> located far from the user <NUM>. The second electronic device <NUM> that receives the wake-up command may recognize the start of the voice <NUM> of the user <NUM> from time t2, and recognize the end of the voice <NUM> of the user <NUM> at time t7.

In this case, since the voice recognition result provided by the first electronic device <NUM> to the second electronic device <NUM> corresponds to voices from time t1 to time t6, there may be difference with the voices which the second electronic device <NUM> recognize from time t2 to time t7. Accordingly, the first electronic device <NUM> may transmit the time information related to the voice corresponding to the voice recognition result and the time information on which the wake-up command is transmitted to the second electronic device <NUM> together with the voice recognition result.

The second electronic device <NUM> may correct the second voice data obtained by the second electronic device <NUM>, wherein the second voice data to be used as data for training, using at least one of the voice recognition result, time information related to the voice, and time information at which the wake-up command is transmitted. For example, the second electronic device <NUM> may restore the missing second voice data according to delayed time of the voice due to difference in distance, voice recognition after receiving the wake-up command to correspond to the voice recognition result.

As another example, the second electronic device <NUM> may extract only a portion corresponding to the second voice data obtained by the second electronic device <NUM> among the voice recognition result received from the first electronic device <NUM>. The extracted voice recognition result and the second speech data may be used as data for training. Specifically, the voice recognized by the first electronic device <NUM> may be "Annyeonghaseyo", and the voice recognized by the second electronic device <NUM> may be "Haseyo" recognized after receiving the wake-up command. In this case, the second electronic device <NUM> may extract only a portion corresponding to "Haseyo" from the voice recognition result received from the first electronic device <NUM> and use the portion as data for training.

When the corrected second voice data is obtained, the second electronic device <NUM> may use the corrected second voice data and the voice recognition result corresponding to the first voice data obtained from the first electronic device <NUM> as data for training the voice recognition model.

In one embodiment, the processor <NUM> may obtain, through the microphone <NUM>, second voice data regarding the voice uttered by the user <NUM> at a specific time. For example, the processor <NUM> may receive a wake-up command for obtaining the second voice data from the first electronic device <NUM> through the communicator. In response to the wake-up command, the processor <NUM> may obtain the second voice data regarding the voice uttered by the user at a specific time through the microphone <NUM>.

The processor <NUM> may receive, from the first electronic device which acquires the first voice data for the voice uttered by the user <NUM> at the specific time, the voice recognition result corresponding to the first voice data through the communicator <NUM>.

The processor <NUM> may use the second voice data and the voice recognition result corresponding to the first voice data as data for training the voice recognition model.

In addition, the processor <NUM> may receive time information related to voice from the first electronic device <NUM> through the communicator <NUM>. In this case, the processor <NUM> may correct the obtained second voice data by using time information related to the voice. In addition, the processor <NUM> may use the corrected second voice data and a voice recognition result corresponding to the first voice data as data for training the voice recognition model.

In addition, when the voice recognition model is updated using the second voice data and the voice recognition result corresponding to the first voice data, the processor <NUM> may obtain third voice data associated with voice data which a user newly utters at a different time than the specific time, through the microphone <NUM>. The processor <NUM> may obtain the voice recognition result corresponding to the third voice data by applying the obtained third voice data to the updated voice recognition model using the data for training.

In addition, when the first voice data regarding the voice uttered by the user at a specific time is obtained, the processor <NUM> of the first electronic device <NUM> may transmit the obtained first voice data to the second electronic device <NUM> through the communicator <NUM>. While the third voice data for the voice uttered by the user and another user is being obtained, the processor <NUM> of the second electronic device <NUM> may obtain predicted voice data for the voice of the user from the third voice data using the obtained first voice data, and use the predicted voice data and the voice recognition result corresponding to the first voice data as data for training the voice recognition model.

<FIG> is a flowchart between a first electronic device and a second electronic device according to an embodiment of the disclosure.

Referring to <FIG>, the first electronic device <NUM> may sense a voice uttered by a user at a specific time at operation <NUM>. As the voice is sensed, the first electronic device <NUM> may transmit a wake-up command to the second electronic device <NUM> to control the second electronic device <NUM> to obtain second voice data for the voice at operation <NUM>. In addition, as the voice is sensed, the first electronic device <NUM> may obtain first voice data of the voice uttered by the user at the specific time through the microphone provided in the first electronic device <NUM> at operation <NUM>. The first electronic device <NUM> may obtain a voice recognition result corresponding to the obtained first voice data at operation <NUM>. For example, the first electronic device <NUM> may apply the first voice data to a voice recognition model provided in the first electronic device <NUM> to obtain a voice recognition result. Alternatively, the first electronic device <NUM> may obtain the voice recognition result by applying the first voice data to the voice recognition model provided in the external server. The first electronic device <NUM> may transmit the obtained voice recognition result to the second electronic device <NUM> at operation <NUM>.

The second electronic device <NUM> that has received the wake-up command may obtain the second voice data for the voice uttered by the user at the specific time through the microphone provided in the second electronic device <NUM> at operation <NUM>. The second electronic device <NUM> may update the voice recognition model used by the second electronic device <NUM> by using the obtained second voice data and the voice recognition result corresponding to the first voice data as data for training at operation <NUM>. For example, the second electronic device <NUM> may update the voice recognition model by applying the data for training to the voice recognition model provided in the second electronic device <NUM>. Alternatively, the second electronic device <NUM> may update the voice recognition model by applying the data for training to a voice recognition model provided in an external server.

The second electronic device <NUM> may obtain third voice data regarding the voice uttered by the user at a specific time different from the specific time through the microphone provided in the second electronic device <NUM> at operation <NUM>. In this case, the second electronic device <NUM> may obtain a voice recognition result corresponding to the third speech data by applying the obtained third voice data to the voice recognition model updated in operation <NUM> at operation <NUM>.

<FIG> is a flowchart of a first electronic device according to an embodiment of the disclosure.

Referring to <FIG>, the first electronic device <NUM> may be capable of communicating with the second electronic device <NUM>. In this case, the first electronic device <NUM> may exist in a situation in which the user's voice is relatively well recognized compared to the second electronic device <NUM>. For example, the first electronic device <NUM> may be an electronic device located at a first distance from the user, and the second electronic device <NUM> may be located at a second distance relatively farther than the first distance. Alternatively, the ambient noise at the first electronic device <NUM> may be lower than the ambient noise at the second electronic device <NUM>. Alternatively, the performance of the microphone of the first electronic device <NUM> may be higher than that of the microphone of the second electronic device <NUM>.

In this situation, the first electronic device <NUM> may obtain first voice data regarding voice uttered by the user at a specific time through a microphone provided in the first electronic device at operation <NUM>. In this case, when the voice uttered by the user is detected, the first electronic device <NUM> may transmit a wakeup command to the second electronic device <NUM> to control the second electronic device <NUM> to obtain the second voice data.

In one embodiment, the first electronic device <NUM> may transmit the first voice data obtained by the first electronic device <NUM> to the second electronic device <NUM>. The second electronic device <NUM> may obtain predicted voice data to be used for the data for training by correcting the third voice data obtained by the second electronic device <NUM> using the received first voice data. For example, the third voice data may be third voice data for the voice uttered by the user and other users. In this case, the second electronic device <NUM>, by using the first voice data, may obtain from the third voice data the predicted voice data for the user's voice which is expected to be obtained when the voice uttered by the user <NUM> is received through the microphone in the second electronic device <NUM>.

The first electronic device <NUM> may obtain the voice recognition result corresponding to the obtained first voice data at operation <NUM>. The voice recognition result may include at least one of, for example, text data, user's intention, and path rule.

The first electronic device <NUM> may transmit, to the second electronic device <NUM> which obtained the second voice data for the voice uttered by the user at the specific time, the voice recognition result to be used as the data for training the voice recognition model at operation <NUM>. For example, the first electronic device <NUM> may transmit the voice recognition result to be used as the data for learning to train at least one of an acoustic model and a language model included in the voice recognition model. At this time, the first electronic device <NUM> may further transmit the time information related to the voice to the second electronic device <NUM> along with the voice recognition result.

<FIG> is a flowchart of a second electronic device according to an embodiment of the disclosure.

Referring to <FIG>, the second electronic device <NUM> may be capable of communicating with the first electronic device <NUM>. In this case, the second electronic device <NUM> may exist in a situation where it is relatively difficult to recognize the user's voice in comparison with the first electronic device <NUM>. For example, the second electronic device <NUM> may be located at a second distance from the user, and the first electronic device <NUM> may be an electronic device <NUM> located at a first distance relatively closer to the user than the second distance. Alternatively, the ambient noise at the second electronic device <NUM> may be higher than the ambient noise at the first electronic device <NUM>. Alternatively, the performance of the microphone of the second electronic device <NUM> may be lower than that of the microphone of the first electronic device <NUM>.

In this situation, the second electronic device <NUM> may obtain second voice data regarding the voice uttered by the user at a specific time through the microphone provided in the second electronic device <NUM> at operation <NUM>. For example, the second electronic device <NUM> may receive a wake-up command for obtaining the second voice data from the first electronic device <NUM>. In response to the wake-up command, the second electronic device <NUM> may obtain second voice data regarding the voice uttered by the user at a specific time through the microphone provided in the second electronic device <NUM>.

The second electronic device <NUM> may obtain the voice recognition result corresponding to the first voice data from the first electronic device which obtained the first voice data for the voice uttered by the user at the specific time at operation <NUM>.

The second electronic device <NUM> may use the second voice data obtained in operation <NUM>, and the voice recognition result corresponding to the first voice data obtained in operation <NUM> as data for training the voice recognition model at operation <NUM>.

As an embodiment, the second electronic device <NUM> may receive time information related to the voice from the first electronic device <NUM> together with the voice recognition result. In this case, the second electronic device <NUM> may correct the obtained second voice data using time information related to the voice. The second electronic device <NUM> may use the corrected second voice data and a voice recognition result corresponding to the first voice data obtained in operation <NUM> as data for training the voice recognition model.

In an embodiment, the second electronic device <NUM> may obtain third voice data regarding the voice newly uttered by the user at a time different from the specific time when the user utters the voice in operation <NUM>. The second electronic device <NUM> may apply the obtained third voice data to the updated voice recognition model using the data for training in operation <NUM>. As a result of the application, the second electronic device <NUM> may obtain a voice recognition result corresponding to the third voice data.

In an embodiment, the second electronic device <NUM> may obtain third voice data about a voice uttered by the user and another user. In this case, the second electronic device <NUM>, using the first voice data, may obtain the predicted voice data regarding the user's voice which is expected to be obtained when the voice uttered by the user <NUM> is received through the microphone by the second electronic device <NUM>, from the third voice data. The second electronic device <NUM> may use the obtained predicted voice data and the voice recognition result corresponding to the first voice data as the data for training the voice recognition model.

<FIG> is a detailed block diagram of an electronic device in a network environment according to an embodiment of the disclosure.

Referring to <FIG>, an electronic device <NUM> may be any one of the first electronic device <NUM>, the second electronic device <NUM>, and the third electronic device <NUM> described in <FIG>, <FIG>, <FIG>, and <FIG>.

Referring to <FIG>, in a network environment <NUM>, the electronic device <NUM> may communicate with an electronic device <NUM> through a first network <NUM> (e.g., a near field range wireless communication network), or communicate with an electronic device <NUM> or a server <NUM> through a second network <NUM> (e.g., long distance wireless communication network). According to an embodiment, the electronic device <NUM> may communicate with the electronic device <NUM> through the server <NUM>. According to an embodiment, the electronic device <NUM> may include a processor <NUM>, a memory <NUM>, an input device <NUM>, a sound output device <NUM>, a display device <NUM>, an audio module <NUM>, and a communication module <NUM>. In some embodiments, at least one of the components (e.g., the display device <NUM>) may be omitted or one or more other components may be added to the electronic device <NUM>. In some embodiments, some of these components may be implemented in one integrated circuit.

The processor <NUM> may control the at least one another component (e.g., hardware or software component) of the electronic device <NUM> connected to the processor <NUM> by executing the software (e.g., the program <NUM>) and perform various data processing or operation. According to an embodiment, as at least a part of the data processing or operation, the processor <NUM> may load the command or data received from another component (e.g., the communication module <NUM>) to a volatile memory <NUM>, process command or data stored in the volatile memory <NUM>, and store the result data in a non-volatile memory <NUM>. According to one embodiment, the processor <NUM> may include a main processor <NUM> (e.g., a central processing unit or an application processor), and a secondary processor <NUM> (e.g., a graphics processing unit, an image signal processor, a sensor hub processor, or a communication processor) which may be operated together or independently. Additionally or alternatively, the secondary processor <NUM> may use less power than the main processor <NUM>, or may be set to be specialized to a designated function. The secondary processor <NUM> may be implemented separately from, or as a part of, the main processor <NUM>.

The secondary processor <NUM> may, for example, in place of the main processor <NUM> while the main processor <NUM> is in an inactive state (e.g., sleep) or along with the main processor <NUM> while the main processor <NUM> is in an active state (e.g., execution of an application) control a part of the functions or states related to at least one component (e.g., display device <NUM>, the communication module <NUM>) among the components of the electronic device <NUM>. According to one embodiment, the secondary processor <NUM> (e.g., an image signal processor or a communication processor) may be implemented as a part of a functionally related other components (e.g., communication module <NUM>). According to an embodiment, the secondary processor <NUM>, when a wake-up command is received from an external device, may convert the main processor <NUM> to an active state.

The memory <NUM> may store various data used by at least one component (e.g., processor <NUM>) of the electronic device <NUM>. The data may include, for example, software (e.g., program <NUM>) and input data or output data related with software instructions. The memory <NUM> may include the volatile memory <NUM> or non-volatile memory <NUM>.

The program <NUM> may be stored in the memory <NUM> as software, and include, for example, an operating system <NUM>, middleware <NUM>, or an application <NUM>.

The input device <NUM> may receive a command or data to be used for the components (e.g., processor <NUM>) of the electronic device <NUM> from the outside (e.g., user) of the electronic device <NUM>.

The sound output device <NUM> may output a sound signal to the outside of the electronic device <NUM>. The speaker may be used for general purposes, such as multimedia playback or recording playback, and the receiver may be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from, or as a part of, the speaker.

The display device <NUM> may visually provide information to an outside (e.g., user) of the electronic device <NUM>. The display device <NUM> may include, for example, a display, a hologram device, a projector, or a control circuit for controlling the device. According to an embodiment, the display device <NUM> may include a touch circuitry which is set to detect a touch or a sensor circuit (e.g., a pressure sensor) which is set to measure intensity of power generated by the touch.

The audio module <NUM> (e.g.: microphone) may convert sound into an electric signal, or convert an electric signal to sound, conversely. According to one embodiment, the audio module <NUM> may acquire sound through an input device <NUM>, or output sound through the sound output device <NUM>, or an external electronic device (e.g., electronic device <NUM>) (e.g., speaker or headphone) which is directly or wirelessly connected to the electronic device <NUM>.

A connection terminal <NUM> may include a connector through which the electronic device <NUM> may be physically connected to an external electronic device (e.g., the electronic device <NUM>). According to an embodiment, the connection terminal <NUM> may include, for example, a high definition multimedia interface (HDMI) connector, a universal serial bus (USB) connector, a secure digital (SD) card connector, or an audio connector (e.g., a headphone connector).

The communication module <NUM> may support establishment of direct (e.g.: wired) communication channel between the electronic device <NUM> and an external electronic device (e.g., electronic device <NUM>, electronic device <NUM>, or server <NUM>) or wireless communication channel, and communication through the established communication channels. The communication module <NUM> may include one or more communication processors which are operated independently of the processor <NUM> (e.g., application processor) and support direct (e.g., wired) communication or wireless communication. According to an embodiment, the communication module <NUM> may include a wireless communication module <NUM> (e.g., cellular communication module, near field wireless communication module, or global navigation satellite system (GNSS) communication module) or wired communication module <NUM> (e.g., local area network (LAN) communication module, or power line communication module). The corresponding communication module among these communication modules may communicate with an external electronic device via the first network <NUM> (e.g., Bluetooth, Wi-Fi direct or near field communication network such as infrared data association (IrDA)) or the second network <NUM> (e.g., telecommunication network such as cellular network, Internet, or computer network (e.g., LAN or WAN)). These types of communication modules may be incorporated into one component (e.g., a single chip) or implemented with a plurality of components (e.g., a plurality of chips) that are separate from each other.

At least a part of the components may be interconnected through the communication method (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) among peripheral devices and exchange a signal (e.g., command or data) from each other.

According to one embodiment, the command or data may be transmitted or received between the electronic device <NUM> and the external electronic device <NUM> via the server <NUM> connected to the second network <NUM>. Each of the electronic devices <NUM> and <NUM> may be devices which are the same or different types from the electronic device <NUM>. According to an embodiment, whole or a part of the operations executed by the electronic device <NUM> may be executed by one or more external devices among the external electronic devices <NUM>, <NUM>, or <NUM>. For example, when the electronic device <NUM> has to perform a function or service automatically, or in response to a request from a user or another device, the electronic device <NUM> may request one or more external electronic devices to perform at least a part of the function or the service instead of, or in addition to, performing the function or service by itself. The one or more external electronic devices that have received the request may execute at least a portion of the requested function or service, or an additional function or service associated with the request, and transmit the result of the execution to the electronic device <NUM>. The electronic device <NUM> may process the result as is or additionally, and provide the result as at least a portion of the response to the request. For this purpose, for example, cloud computing, distributed computing, or client-server computing technology may be used.

<FIG> is a view illustrating a system in a situation where there is a plurality of speakers according to an embodiment of the disclosure.

Referring to <FIG>, the first electronic device <NUM>, the second electronic device <NUM>, the third electronic device <NUM>, the first server <NUM>, and the second server <NUM> of <FIG> respectively correspond to the first electronic device <NUM>, the second electronic device <NUM>, the third electronic device <NUM>, the first server <NUM>, and the second server <NUM> of <FIG>, and <FIG>, and the redundant description will be omitted.

Referring to <FIG>, the first user <NUM>, a second user <NUM>, and a third user <NUM> are illustrated. In this case, the first user <NUM> may be disposed to be closer to the first electronic device <NUM>, and the second user <NUM> may be disposed to be closer to the second electronic device <NUM>, and the third user may be disposed to be closer to the third electronic device <NUM>.

In this case, the first user <NUM>, the second user <NUM>, and the third user <NUM> may utter voice together at a specific time (e.g., at a specific time or during the specific time). For example, at the specific time, the user <NUM> may utter the voice <NUM> "what time is it now?", the second user <NUM> may utter the voice <NUM> "the weather is good," and the third user <NUM> may utter voice <NUM>, "I'm hungry.

In this case, the microphones of each of the first electronic device <NUM>, the second electronic device <NUM>, and the third electronic device <NUM> may receive the voice signals uttered by each of the first user <NUM>, the second user <NUM>, and the third user <NUM>, or receive the voice signal uttered by another user together. In the embodiment of <FIG> and <FIG>, it is assumed that the microphone of the first electronic device <NUM> receives the voice signal uttered by the first user <NUM>, the microphone of the second electronic device <NUM> receives the voice signal uttered by the first user <NUM> and the second user <NUM>, and the microphone of the third electronic device <NUM> receives the voice signal uttered by the first user <NUM> and the third user <NUM>.

In the above situation, the first electronic device <NUM> located at a first distance (e.g., short distance) from the first user <NUM> may obtain the first voice data for the voice <NUM> uttered by the first user <NUM> at the specific time through the microphone provided in the first electronic device <NUM> (①). At this time, the second electronic device <NUM> located at the second distance (e.g., middle distance) from the first user <NUM> may obtain the second voice data in which not only the voice <NUM> uttered by the first user <NUM> at the specific time through the microphone provided in the second electronic device <NUM> but also voice <NUM> uttered by the second user <NUM> located closer to the second electronic device <NUM> at the specific time are included (②). In addition, the third electronic device <NUM> positioned at the third distance (e.g., long distance) from the user <NUM> may obtain third voice data in which not only the voice <NUM> uttered by the first user <NUM> at the specific time through the microphone provided in the third electronic device <NUM> but also voice <NUM> uttered by a third user <NUM> located closer to the third electronic device <NUM> at the specific time (③).

In <FIG>, when the first electronic device <NUM> obtains the first voice data, the first electronic device <NUM> may obtain the voice recognition result corresponding to the obtained first voice data. For example, the first electronic device <NUM> may apply the obtained first voice data to the voice recognition model provided in the first electronic device and obtain the voice recognition result. Alternatively, the first electronic device <NUM> may transmit the first voice data to the first server <NUM> which may communicate with the first electronic device <NUM> (④). The first electronic device <NUM> may obtain the voice recognition result corresponding to the first voice data from the first server <NUM> (⑤). At this time, the voice recognition result corresponding to the first voice data may mean the voice recognition result corresponding to the voice uttered by the first user. The various methods with which the first electronic device <NUM> obtains the voice recognition result from the first server <NUM> have been described in <FIG> and the redundant description will be omitted.

When the voice recognition result is obtained, the first electronic device <NUM> may transmit the voice recognition result corresponding to the voice <NUM> uttered by the first user <NUM>, and the first voice data regarding the voice <NUM> uttered by the first user <NUM> to the second electronic device <NUM> (⑥). In addition, the first electronic device <NUM> may transmit the voice recognition result corresponding to the voice <NUM> uttered by the first user <NUM> and the first voice data regarding the voice <NUM> uttered by the first user to the third electronic device <NUM> (⑦). In an embodiment, each of the voice recognition results corresponding to the voice <NUM> uttered by the first user <NUM> and the first voice data for the voice <NUM> uttered by the first user <NUM> may be transmitted to the second electronic device <NUM> or the third electronic device <NUM> with a time difference, respectively. For example, after the voice recognition result corresponding to the voice <NUM> uttered by the first user <NUM> is transmitted, the first voice data regarding the voice <NUM> uttered by the first user <NUM> may be transmitted, and after the first voice data for the voice <NUM> uttered by the first user <NUM> is transmitted, the voice recognition result corresponding to the voice <NUM> uttered by the first user <NUM> may be transmitted.

The second electronic device <NUM> may use the voice recognition result corresponding to the voice <NUM> uttered by the first user <NUM> as a part of the data for training. For this purpose, the second electronic device <NUM> may, by using the first voice data for the voice <NUM> uttered by the first user <NUM> uttered by the first electronic device <NUM> in operation ⑥, extract the predicted voice data for the voice of the first user <NUM> from the second voice data in which the voice <NUM> uttered by the first user <NUM> obtained by the second electronic device <NUM> in operation (<NUM>) and the voice <NUM> uttered by the second user <NUM> are combined. The predicted voice data may be the voice data for the voice <NUM> uttered by the first user <NUM>, from among the second voice data obtained through the microphone of the second electronic device <NUM>.

To be specific, the first voice data for the voice <NUM> uttered by the first user <NUM>, obtained by the first electronic device <NUM>, may be represented as A, and the second voice data, obtained by the second electronic device <NUM>, in which the voice <NUM> uttered by the first user <NUM> and the second user <NUM> are combined, may be represented as A' + B'.

At this time, the second voice data - the first voice data = (A '+ B') - A = B '+ residual A. Here, the second voice data - (second voice data-first voice data) = (A '+ B')-(B '+ residual A) = A'-residual A. Accordingly, A', which is voice data of the voice uttered by the first user <NUM> obtained by the second electronic device <NUM>, may be a value in which the residual A is added to the first voice data obtained from the first electronic device <NUM> by the second electronic device <NUM>.

The residual A may be a value of the utterance environment difference between the first electronic device <NUM> and the second electronic device <NUM>. For example, the residual A may be a difference value between the first voice data A for the first user <NUM> obtained by the first electronic device <NUM> in a situation where only the first user <NUM> utters, and the second voice data A' for the first user <NUM> obtained by the second electronic device <NUM>. Alternatively, in the non-utterance section, the residual A may be a difference value of the data for the environment signal obtained by the microphone provided in the first electronic device <NUM> and the data for the environment signal obtained by the microphone provided in the second electronic device <NUM>.

Through the above-described process, when the predicted voice data A' is obtained, the second electronic device <NUM> may form the predicted voice data and the voice recognition result corresponding to the first voice data received in operation ⑥ into a set, and use it as data for training. For example, when the voice recognition model is located in the second electronic device <NUM>, the second electronic device <NUM> may apply the {the predicted voice data generated in the second electronic device <NUM> and the voice recognition result obtained by the first electronic device <NUM>} to the voice recognition model as the data for training, and update the voice recognition model positioned in the second electronic device <NUM> (⑧).

In addition, the third electronic device <NUM> may use the voice recognition result corresponding to the voice <NUM> uttered by the first user <NUM> as part of the data for training. For this purpose, the third electronic device <NUM>, by using the first voice data for the voice <NUM> uttered by the first user <NUM> obtained by the first electronic device <NUM> in operation ⑦, may extract the predicted voice data for the voice of the first user <NUM> from the third voice data in which the voice <NUM> uttered by the first user <NUM> and the third user <NUM> obtained by the third electronic device <NUM> in operation ③ are combined. The predicted voice data may be the voice data for the voice <NUM> uttered by the first user <NUM> from among the third voice data obtained through the microphone of the third electronic device <NUM>. A method for extracting the predicted voice data has been described with respect to the second electronic device <NUM> and will not be further described.

The third electronic device <NUM> may form the predicted voice data and the voice recognition result corresponding to the first voice data received in operation ⑦ into a set and use it as data for training. For example, when the voice recognition model is positioned at the second server <NUM>, the third electronic device <NUM> may transmit {predicted voice data generated by the third electronic device <NUM>, voice recognition result obtained by the first electronic device <NUM>} as the data for training to the second server <NUM> in which the voice recognition model is positioned (⑨). The second server <NUM> may apply the received data for training to the voice recognition model and update the voice recognition model stored in the second server <NUM> (⑩).

<FIG> and <FIG> are views of a system considering an utterance environment according to an embodiment of the disclosure.

The first electronic device <NUM>, the second electronic device <NUM>, the third electronic device <NUM>, the first server <NUM>, and the second server <NUM> of <FIG> correspond to the first electronic device <NUM>, the second electronic device <NUM>, the third electronic device <NUM>, the first server <NUM>, and the second server <NUM> of <FIG> and <FIG>, respectively, and the redundant description will be omitted.

Referring to <FIG>, when the user <NUM> utters the voice <NUM>, the first electronic device <NUM> may obtain the first voice data for the voice <NUM> uttered by the user <NUM> at the specific time through h the microphone provided in the first electronic device <NUM> (①).

At this time, the second electronic device <NUM> positioned at the second distance (e.g., middle distance) from the user <NUM> may obtain the second voice data for the voice <NUM> uttered by the user <NUM> at the specific time through the microphone provided in the second electronic device <NUM> (②).

The second electronic device <NUM> may transmit the second voice data for the voice <NUM> uttered by the user <NUM> to the first electronic device <NUM> (②-<NUM>).

The first electronic device <NUM>, using the second voice data obtained from the second electronic device <NUM>, may obtain and store the first utterance environment difference between the first electronic device <NUM> and the second electronic device <NUM> (②-<NUM>).

To be specific, the first voice data for the voice <NUM> uttered by the user <NUM> obtained by the first electronic device <NUM> may be represented as A, and the second voice data for the voice <NUM> uttered by the user <NUM> obtained by the second electronic device <NUM> may be represented as A'. At this time, the second voice data ? first voice data = A-A' = residual A. The residual A may be stored as the first utterance environment difference value between the first electronic device <NUM> and the second electronic device <NUM>.

In addition, the third electronic device <NUM> positioned at the third distance (e.g., long distance) from the user <NUM> also may obtain the third voice data for the voice <NUM> uttered by the user <NUM> at the specific time through the microphone provided in the third electronic device <NUM> (③).

The third electronic device <NUM> may transmit the third voice data for the voice <NUM> uttered by the user <NUM> to the first electronic device <NUM> (③-<NUM>).

The first electronic device <NUM>, using the third voice data obtained from the third electronic device <NUM>, may obtain and store the second utterance environment difference value between the first electronic device <NUM> and the third electronic device <NUM> (③-<NUM>).

Referring to <FIG>, the user <NUM> may utter a voice <NUM> at a time different from the specific time at which the voice <NUM> is uttered. The first electronic device <NUM> may obtain fourth voice data of the voice <NUM> uttered by the user <NUM> through the provided microphone (④). The first electronic device <NUM> may obtain a voice recognition result corresponding to the obtained fourth voice data. For example, the first electronic device <NUM> may obtain a voice recognition result by applying a voice recognition result corresponding to the obtained fourth voice data to a voice recognition model provided in the first electronic device <NUM>. Alternatively, the first electronic device <NUM> may transmit the fourth voice data to the first server <NUM> capable of communicating with the first electronic device <NUM> (⑤). The first electronic device <NUM> may obtain a voice recognition result corresponding to the fourth voice data from the first server <NUM> (⑥).

In the meantime, the first electronic device <NUM>, by using the obtained fourth voice data and the first utterance environment difference value stored in the operation (②-<NUM>), may generate the first predicted voice data which is expected to be obtained, when the user receives the voice <NUM> uttered by the user <NUM> through the microphone in the second electronic device <NUM>. For example, the first electronic device <NUM> may generate first predicted voice data by adding the obtained fourth voice data and the first voice environment difference value. The first electronic device <NUM> may transmit a voice recognition result corresponding to the first predicted voice data and the fourth voice data acquired in operation ⑥ to the second electronic device <NUM> (⑦). The second electronic device <NUM> may update the voice recognition model by applying {the first predicted voice data generated in the first electronic device <NUM> and the voice recognition result obtained from the first electronic device <NUM>} as the data for training to the voice recognition model (⑧).

In addition, the first electronic device <NUM>, by using the obtained fourth voice data and the second utterance environment difference value stored in the operation (③-<NUM>), may generate second predicted voice data which is expected to be obtained when the voice <NUM> uttered by the user <NUM> is received via the microphone. For example, the first electronic device <NUM> may generate second predicted voice data by adding the obtained fourth voice data and the second voice environment difference value. The first electronic device <NUM> may transmit a voice recognition result corresponding to the second predicted voice data and the fourth voice data obtained in operation ⑥ to the third electronic device <NUM> (⑨). The third electronic device <NUM> may form {the second predicted voice data generated by the first electronic device <NUM>, the voice recognition result obtained from the first electronic device <NUM>} into a set, and use the data as data for learning. For example, when the voice recognition model is located in the second server <NUM>, the third electronic device <NUM> may transmit {the second predicted voice data generated in the first electronic device <NUM> and the voice recognition result obtained from the first electronic device <NUM>} as data for training to the second server <NUM> in which the voice recognition model is located (⑩). The second server <NUM>, by applying the received data for training to the voice recognition model, may update the voice recognition model stored in the second server <NUM> (⑪).

In one embodiment, the first electronic device <NUM> may transmit {the second predicted voice data generated in the first electronic device <NUM> and the voice recognition result obtained by the first electronic device <NUM>} directly to the second server <NUM>, without passing through the third electronic device <NUM>. The second server <NUM> may apply the received data for training to the voice recognition model and update the voice recognition model stored in the second server <NUM>.

In one embodiment, the above-described utterance environment difference value may be measured even in a situation where the user <NUM> does not utter a voice. For example, after purchasing the first electronic device <NUM>, the second electronic device <NUM>, and the third electronic device <NUM>, the user may activate the microphones of the electronic devices for a predetermined time. In this case, based on the difference value of the data in the environmental signal received by the microphones of the above-described electronic devices, the utterance environment difference value for generating the predicted voice data may be obtained.

The term "module" used in the disclosure may include units consisting of hardware, software, or firmware, and is used interchangeably with terms such as, for example, logic, logic blocks, parts, or circuits. A "module" may be an integrally constructed component or a minimum unit or part thereof that performs one or more functions. For example, according to an embodiment, the module may be configured as an application-specific integrated circuit (ASIC).

Meanwhile, various embodiments of the disclosure may be implemented in software (e.g., program <NUM>), including instructions stored on machine-readable storage media (e.g., memory <NUM>, internal memory <NUM> or external memory <NUM>) readable by a machine (e.g., first electronic device <NUM>, second electronic device <NUM>, third electronic device <NUM>, electronic device <NUM>). For example, a processor (e.g., processor <NUM>, processor <NUM>) of a device (e.g., first electronic device <NUM>, second electronic device <NUM>, third electronic device <NUM>) may call at least one instruction from among one or more instructions stored in the storage medium, and execute the called instruction. This enables that the apparatus to be operated to perform at least one instruction according to the at least one called instruction. Herein, the term "non-transitory" only denotes that a storage medium does not include a signal (e.g., an electromagnetic wave) but is tangible, and does not distinguish the case in which a data is semi-permanently stored in a storage medium from the case in which a data is temporarily stored in a storage medium.

According to an embodiment, the method according to various embodiments disclosed herein may be provided in a computer program product. A computer program product may be exchanged between a seller and a purchaser as a commodity. A computer program product may be distributed in the form of a machine-readable storage medium (e.g., compact disc read only memory (CD-ROM)) or distributed online through an application store (e.g. PlayStore ™) directly between two user devices (e.g., smartphones). In the case of on-line distribution, at least a portion of the computer program product may be stored temporarily or at least temporarily in a storage medium such as a manufacturer's server, a server of an application store, or a memory of a relay server.

Claim 1:
A method performed by a first electronic device (<NUM>) and a second electronic device (<NUM>), the method comprising:
obtaining, by the first electronic device (<NUM>), first voice data for a voice uttered by a user at a specific time through a microphone of the first electronic device (<NUM>);
obtaining, by the first electronic device (<NUM>), a first voice recognition result corresponding to the first voice data; and
transmitting, by the first electronic device (<NUM>), the first voice recognition result, to the second electronic device (<NUM>) which is configured to obtain second voice data for the voice uttered by the user at the specific time, the first voice recognition result being used to train a voice recognition model at the second electronic device (<NUM>),
obtaining, by the second electronic device (<NUM>), the second voice data for the voice uttered by the user at the specific time through a microphone of the second electronic device (<NUM>);
receiving, by the second electronic device (<NUM>), the first voice recognition result corresponding to the first voice data from the first electronic device (<NUM>) which is configured to obtain the first voice data for the voice uttered by the user at the specific time; and
using the second voice data and the first voice recognition result corresponding to the first voice data, as data for training the voice recognition model at the second electronic device (<NUM>).