Patent Description:
Recently, electronic devices providing a voice recognition function are increasing. An electronic device which has recognized a user utterance may execute a function mounted in the electronic device (e.g., an application installed in the electronic device) so as to not only provide a voice recognition service to a user but also provide a related service to various third-party service providers.

A personal electronic device which is to be secured provides a user registration procedure in order to provide a voice recognition service to a registered user only.

<CIT>, is about a device configured to receive speech input from a user. The speech input can include a command for accessing a restricted feature of the device. The speech input can be compared to a voiceprint (e.g., text-independent voiceprint) of the user's voice to authenticate the user to the device. Responsive to successful authentication of the user to the device, the user is allowed access to the restricted feature without the user having to perform additional authentication steps or speaking the command again. If the user is not successfully authenticated to the device, additional authentication steps can be request by the device (e.g., request a password).

Embodiments of the disclosure provide electronic device providing a voice recognition service providing a registered user with a voice recognition service, the accuracy of which is high, while simplifying a user registration procedure for voice recognition service provision and mitigating user inconveniences. For example, in order for an electronic device to simplify a voice input procedure for user registration and accurately verify a voice of a registered user, embodiments of the disclosure may train a speaker verification model by securing voice data of the user.

The solution is set out in the appended set of claims.

An electronic device and an operation method thereof according to various example embodiments of the disclosure can train a speaker verification model by effectively securing voice data of a registered user for a voice recognition service.

An electronic device and an operation method thereof according to various example embodiments of the disclosure can mitigate user inconveniences by simplifying a voice input procedure for user registration, and can update a speaker verification model by securing user voice data for verifying a registered user, whereby the accuracy of speaker verification can be increased.

<FIG> is a block diagram illustrating an example integrated intelligence system according to various embodiments. Referring to <FIG>, an integrated intelligence system <NUM> of an embodiment may include a user terminal <NUM>, an intelligence server <NUM>, and a service server <NUM>.

The user terminal <NUM> of an embodiment may be a terminal device (or an electronic device) which can be connected to the Internet, and may include, for example, and without limitation, a mobile phone, a smartphone, a personal digital assistant (PDA), a laptop computer, a television (TV), a home appliance, a wearable device, a head mounted device (HMD), a smart speaker, or the like.

According to an embodiment, the user terminal <NUM> may include a communication interface (e.g., including communication circuitry) <NUM>, a microphone <NUM>, a speaker <NUM>, a display <NUM>, a memory <NUM>, and/or a processor (e.g., including processing circuitry) <NUM>. The above-enumerated components may be operatively or electrically connected to each other.

The communication interface <NUM> of an embodiment may include various communication circuitry and be configured to be connected to an external device and transmit or receive data to or from the external device. The microphone <NUM> of an embodiment may receive a sound (e.g., a user utterance) and convert the sound into an electrical signal. The speaker <NUM> of an embodiment may output an electrical signal as a sound (e.g., a voice). The display <NUM> of an embodiment may be configured to display an image or video. The display <NUM> of an embodiment may also display a graphical user interface (GUI) of an executed app (or an application program).

The memory <NUM> of an embodiment may store a client module <NUM>, a software development kit (SDK) <NUM>, and multiple apps (e.g., applications) <NUM>. The client module <NUM> and the SDK <NUM> may configure a framework (e.g., a solution program) for performing a universal function. In addition, the client module <NUM> or the SDK <NUM> may configure a framework for processing a voice input.

The multiple apps <NUM> stored in the memory <NUM> of an embodiment may correspond to a program for performing a designated function. According to an embodiment, the multiple apps <NUM> may include a first app 155_1 and a second app 155_2. According to an embodiment, each of the multiple apps <NUM> may include multiple actions for performing a designated function. For example, the apps may include an alarm app, a message app, and/or a schedule app. According to an embodiment, the multiple apps <NUM> may be executed by the processor <NUM> and sequentially execute at least some of the multiple actions.

The processor <NUM> of an embodiment may include various processing circuitry and control an overall operation of the user terminal <NUM>. For example, the processor <NUM> may be electrically connected to the communication interface <NUM>, the microphone <NUM>, the speaker <NUM>, and the display <NUM> to perform a designated operation.

The processor <NUM> of an embodiment may also execute a program stored in the memory <NUM> to perform a designated function. For example, the processor <NUM> may execute at least one of the client module <NUM> or the SDK <NUM> to perform a subsequent operation for processing a voice input. For example, the processor <NUM> may control operations of the multiple apps <NUM> through the SDK <NUM>. A below operation described as an operation of the client module <NUM> or the SDK <NUM> may be an operation by the execution of the processor <NUM>.

The client module <NUM> of an embodiment may receive a voice input. For example, the client module <NUM> may receive a voice signal corresponding to a user utterance which is detected through the microphone <NUM>. The client module <NUM> may transmit the received voice input to the intelligence server <NUM>. The client module <NUM> may transmit state information of the user terminal <NUM> to the intelligence server <NUM>, together with the received voice input. The state information may be, for example, app execution state information.

The client module <NUM> of an embodiment may receive a result corresponding to the received voice input. For example, when the result corresponding to the received voice input can be calculated by the intelligence server <NUM>, the client module <NUM> may receive the result corresponding to the received voice input from the intelligence server <NUM>. The client module <NUM> may display the received result on the display <NUM>.

The client module <NUM> of an embodiment may receive a plan corresponding to the received voice input. The client module <NUM> may display, on the display <NUM>, the result of executing multiple actions of an app according to the plan. For example, the client module <NUM> may sequentially display the result of execution of the multiple actions on the display. In another example, the user terminal <NUM> may display only a part of the result (e.g., a result of the last operation) of execution of the multiple actions, on the display.

According to an embodiment, the client module <NUM> may receive a request for obtaining information necessary to obtain the result corresponding to a voice input, from the intelligence server <NUM>. According to an embodiment, in response to the request, the client module <NUM> may transmit the necessary information to the intelligence server <NUM>.

The client module <NUM> of an embodiment may transmit result information of executing multiple actions according to a plan, to the intelligence server <NUM>. Using the result information, the intelligence server <NUM> may identify that the received voice input is correctly processed.

The client module <NUM> of an embodiment may include a voice recognition module. According to an embodiment, the client module <NUM> may recognize a voice input of performing a restricted function, through the voice recognition module. For example, the client module <NUM> may perform an intelligence app for processing a voice input for performing a systematic operation through a designated input (e.g., Wake up!).

The intelligence server <NUM> of an embodiment may receive information related with a user voice input from the user terminal <NUM> through a communication network. According to an embodiment, the intelligence server <NUM> may change data related to the received voice input to text data. According to an embodiment, the intelligence server <NUM> may generate a plan for performing a task corresponding to the user voice input using the text data.

According to an embodiment, the plan may be generated by an artificial intelligence (AI) system. The artificial intelligence system may include a rule-based system, or may include a neural network-based system (e.g., a feedforward neural network (FNN) and a recurrent neural network (RNN)). The artificial intelligence system may be either a combination of the aforementioned systems or an artificial intelligence system different therefrom. According to an embodiment, the plan may be selected from a set of predefined plans, or may be generated in real time in response to a user request. For example, the artificial intelligence system may select at least one plan among multiple predefined plans.

The intelligence server <NUM> of an embodiment may transmit the result of the generated plan to the user terminal <NUM>, or transmit the generated plan to the user terminal <NUM>. According to an embodiment, the user terminal <NUM> may display the result of the plan on the display. According to an embodiment, the user terminal <NUM> may display the result of executing an operation according to the plan on the display.

The intelligence server <NUM> of an embodiment may include a front end <NUM>, a natural language platform <NUM>, a capsule database (DB) <NUM>, an execution engine <NUM>, an end user interface <NUM>, a management platform <NUM>, a big data platform <NUM>, and/or an analytic platform <NUM>.

The front end <NUM> of an embodiment may receive a voice input received from the user terminal <NUM>. The front end <NUM> may transmit a response corresponding to the voice input.

According to an embodiment, the natural language platform <NUM> may include an automatic speech recognition module (ASR module) <NUM>, a natural language understanding module (NLU module) <NUM>, a planner module <NUM>, a natural language generator module (NLG module) <NUM>, and/or a text-to-speech module (TTS module) <NUM>, each of which may include various processing circuitry and/or executable program instructions.

The automatic speech recognition module <NUM> of an embodiment may convert a voice input received from the user terminal <NUM> into text data. Using the text data of the voice input, the natural language understanding module <NUM> of an embodiment may identify a user's intention. For example, the natural language understanding module <NUM> may identify the user's intention by performing syntactic analysis or semantic analysis. The natural language understanding module <NUM> of an embodiment may identify a meaning of a word extracted from the voice input using a linguistic feature (e.g., syntactic factor) of a morpheme or phrase, and match the identified meaning of the word with the intention, so as to determine the user's intention.

The planner module <NUM> of an embodiment may generate a plan using the intention and the parameter determined by the natural language understanding module <NUM>. According to an embodiment, the planner module <NUM> may determine multiple domains necessary to perform a task, according to the determined intention. The planner module <NUM> may determine multiple actions included in each of the multiple domains which are determined according to on the intention. According to an embodiment, the planner module <NUM> may determine a parameter necessary to execute the determined multiple actions, or a result value output by the execution of the multiple actions. The parameter and the result value may be defined with a concept of a designated form (or class). Accordingly, the plan may include the multiple actions determined by the user's intention, and multiple concepts. The planner module <NUM> may determine relationships between the multiple actions and the multiple concepts operationally (or hierarchically). For example, the planner module <NUM> may determine, according to the multiple concepts, an execution sequence of the multiple actions that are determined according to the user's intention. In other words, the planner module <NUM> may determine the execution sequence of the multiple actions according to the parameter necessary for execution of the multiple actions and the result output by execution of the multiple actions. Accordingly, the planner module <NUM> may generate a plan including association information (e.g., ontology) between the multiple actions and the multiple concepts. The planner module <NUM> may generate the plan using information stored in a capsule database <NUM> in which a set of relationships between the concepts and the actions is stored.

The natural language generator module <NUM> of an embodiment may change designated information to a text form. The information changed to the text form may be in a form of a natural language speech. The text-to-speech conversion module <NUM> of an embodiment may change the information in the text form to information in a voice form.

According to an embodiment, a partial function or the whole function of the natural language platform <NUM> may be also implemented in the user terminal <NUM>.

The capsule database <NUM> may store information on relationships between multiple concepts and actions corresponding to multiple domains. A capsule of an embodiment may include multiple action objects (or operation information) and concept objects (or concept information) which are included in a plan. According to an embodiment, the capsule database <NUM> may store multiple capsules in a form of a concept action network (CAN). According to an embodiment, the multiple capsules may be stored in a function registry included in the capsule database <NUM>.

The capsule database <NUM> may include a strategy registry in which strategy information necessary to determine a plan corresponding to a voice input is stored. When there are multiple plans corresponding to the voice input, the strategy information may include reference information for determining one plan from the multiple plans. According to an embodiment, the capsule database <NUM> may include a follow up registry in which follow-up action information for proposing a follow-up action to a user in a designated condition is stored. The follow-up action may include, for example, a follow-up speech. According to an embodiment, the capsule database <NUM> may include a layout registry in which layout information of information output through the user terminal <NUM> is stored. According to an embodiment, the capsule database <NUM> may include a vocabulary registry in which vocabulary information included in capsule information is stored. According to an embodiment, the capsule database <NUM> may include a dialog registry in which information on a dialog (or an interaction) with a user is stored. The capsule database <NUM> may update the stored object using a developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating a vocabulary. The developer tool may include a strategy editor for generating and registering a strategy which determines a plan. The developer tool may include a dialog editor for generating a dialog with a user. The developer tool may include a follow up editor for activating a follow up goal and editing a follow up speech which provides a hint. The follow up goal may be determined according to a currently set goal, a user's preference, or an environmental condition. In an embodiment, the capsule database <NUM> may be also implemented in the user terminal <NUM>.

The execution engine <NUM> of an embodiment may obtain the result using the generated plan. The end user interface <NUM> may transmit the obtained result to the user terminal <NUM>. Accordingly, the user terminal <NUM> may receive the result and provide the received result to a user. The management platform <NUM> of an embodiment may manage information used in the intelligence server <NUM>. The big data platform <NUM> of an embodiment may collect a user's data. The analytic platform <NUM> of an embodiment may manage a quality of service (QoS) of the intelligence server <NUM>. For example, the analytic platform <NUM> may manage a component and a processing speed (or efficiency) of the intelligence server <NUM>.

The service server <NUM> of an embodiment may provide the user terminal <NUM> with a designated service (e.g., a food order or a hotel reservation). According to an embodiment, the service server <NUM> may be a server managed by a third party. The service server <NUM> of an embodiment may provide the intelligence server <NUM> with information for generating a plan corresponding to a received voice input. The provided information may be stored in the capsule database <NUM>. In addition, the service server <NUM> may provide the intelligence server <NUM> with information on the result of the plan.

In the above-described integrated intelligence system <NUM>, in response to a user input, the user terminal <NUM> may provide the user with various intelligence services. The user input may include, for example, an input through a physical button, a touch input, or a voice input.

In an embodiment, the user terminal <NUM> may provide a voice recognition service through an intelligence app (or a voice recognition app) stored therein. In this case, for example, the user terminal <NUM> may recognize a user utterance or a voice input received through the microphone, and provide the user with a service corresponding to the recognized voice input.

In an embodiment, the user terminal <NUM> may perform a designated operation independently, or together with the intelligence server and/or the service server according to a received voice input. For example, the user terminal <NUM> may execute an app corresponding to the received voice input, and perform a designated operation through the executed app.

In an embodiment, when the user terminal <NUM> provides a service together with the intelligence server <NUM> and/or the service server, the user terminal <NUM> may detect a user utterance using the microphone <NUM>, and generate a signal (or voice data) corresponding to the detected user utterance. The user terminal may transmit the voice data to the intelligence server <NUM> using the communication interface <NUM>.

As a response to a voice input received from the user terminal <NUM>, the intelligence server <NUM> according to an embodiment may generate a plan for performing a task corresponding to the voice input, or the result of performing an operation according to the plan. The plan may include, for example, multiple actions for performing a task corresponding to a user's voice input, and multiple concepts related to the multiple actions. The concepts may be obtained by defining a parameter input by execution of the multiple actions, or a result value output by the execution of the multiple actions. The plan may include association information between the multiple actions and the multiple concepts.

The user terminal <NUM> of an embodiment may receive the response using the communication interface <NUM>. The user terminal <NUM> may output a voice signal generated within the user terminal <NUM> to the outside using the speaker <NUM>, or output an image generated within the user terminal <NUM> to the outside using the display <NUM>.

<FIG> is a diagram illustrating an example form in which information on a relationship between a concept and an action is stored in a database according to various embodiments.

A capsule database (e.g., the capsule database <NUM>) of the intelligence server <NUM> may store a capsule in the form of a concept action network (CAN) <NUM>. The capsule database may store an action for processing a task corresponding to a user's voice input and a parameter necessary for the action, in the form of the concept action network (CAN).

The capsule database may store multiple capsules (capsule A <NUM> and capsule B <NUM>) corresponding to multiple domains (e.g., applications), respectively. According to an embodiment, one capsule (e.g., capsule A <NUM>) may correspond to one domain (e.g., a location (geo) and an application). In addition, one capsule may correspond to at least one service provider (or a content provider (e.g., CP <NUM><NUM> or CP <NUM><NUM>)) for performing a function for a domain related with the capsule. According to an embodiment, one capsule may include one or more actions <NUM> and one or more concepts <NUM> for performing a designated function.

The natural language platform <NUM> may generate a plan for performing a task corresponding to a received voice input, using a capsule stored in a capsule database. For example, the planner module <NUM> of the natural language platform may generate the plan using the capsule stored in the capsule database. For example, the planner module <NUM> may generate a plan <NUM> using actions <NUM> and <NUM> and concepts <NUM> and <NUM> of capsule A <NUM> and an action <NUM> and a concept <NUM> of capsule B <NUM>.

<FIG> is a diagram illustrating an example screen in which a voice input received through an intelligence app is processed by a user terminal according to various embodiments.

To process a user input through the intelligence server <NUM>, the user terminal <NUM> may execute the intelligence app.

According to an embodiment, in screen <NUM>, when recognizing a designated voice input (e.g., Wake up!) or receiving an input through a hardware key (e.g., a dedicated hardware key), the user terminal <NUM> may execute the intelligence app for processing the voice input. For example, the user terminal <NUM> may execute the intelligence app in a state in which a schedule app is executed. According to an embodiment, the user terminal <NUM> may display an object (e.g., an icon) <NUM> corresponding to the intelligence app on the display <NUM>. According to an embodiment, the user terminal <NUM> may receive a voice input by a user utterance. For example, the user terminal <NUM> may receive a voice input of "Tell me my schedule this week!". According to an embodiment, the user terminal <NUM> may display a user interface (UI) <NUM> (e.g., an input window) of the intelligence app, in which text data of the received voice input is displayed, on the display.

According to an embodiment, in screen <NUM>, the user terminal <NUM> may display a result corresponding to the received voice input on the display. For example, the user terminal <NUM> may receive a plan corresponding to the received user input, and display, on the display, "Your schedule this week" according to the plan.

According to an embodiment, the electronic device <NUM> may include a processor <NUM>, memory <NUM>, an input device <NUM>, a sound output device <NUM>, a display device <NUM>, an audio module <NUM>, a sensor module <NUM>, an interface <NUM>, a connecting terminal <NUM>, a haptic module <NUM>, a camera module <NUM>, a power management module <NUM>, a battery <NUM>, a communication module <NUM>, a subscriber identification module(SIM) <NUM>, or an antenna module <NUM>.

According to an embodiment, the processor <NUM> may include a main processor <NUM> (e.g., a central processing unit (CPU) or an application processor (AP)), and an auxiliary processor <NUM> (e.g., a graphics processing unit (GPU), a neural processing unit (NPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)) that is operable independently from, or in conjunction with, the main processor <NUM>.

The program 540may be stored in the memory <NUM> as software, and may include, for example, an operating system (OS) <NUM>, middleware <NUM>, or an application <NUM>.

The input device <NUM> may include, for example, a microphone, a mouse, a keyboard, a key (e.g., a button), or a digital pen (e.g., a stylus pen).

A corresponding one of these communication modules may communicate with the external electronic device via the first network <NUM> (e.g., a short-range communication network, such as Bluetooth™, wireless-fidelity (Wi-Fi) direct, or infrared data association (IrDA)) or the second network <NUM> (e.g., a long-range communication network, such as a cellular network, a <NUM> network, a next-generation communication network, the Internet, or a computer network (e.g., LAN or wide area network (WAN)).

According to an embodiment, the antenna module <NUM> may include an antenna including a radiating element including a conductive material or a conductive pattern formed in or on a substrate (e.g., PCB).

It is to be understood that if an element (e.g., a first element) is referred to, with or without the term "operatively" or "communicatively", as "coupled with," "coupled to," "connected with," or "connected to" another element (e.g., a second element), it includes that the element may be coupled with the other element directly (e.g., wiredly), wirelessly, or via a third element.

As used herein, the term "module" may include a unit implemented in hardware, software, or firmware, or any combination thereof and may interchangeably be used with other terms, for example, "logic," "logic block," "part," or "circuitry".

Wherein, the "non-transitory" storage medium is a tangible device, and may not include a signal (e.g., an electromagnetic wave), but this term does not differentiate between where data is semi-permanently stored in the storage medium and where the data is temporarily stored in the storage medium.

<FIG> is a block diagram illustrating an example configuration of an electronic device according to various embodiments. Referring to <FIG>, an electronic device <NUM> may include at least one processor (e.g., including various processing circuitry, e.g., an application processor <NUM> and an auxiliary processor <NUM>) <NUM>, a memory <NUM>, an input module (e.g., including input circuitry) <NUM>, a sound output module (e.g., including sound output circuitry) <NUM>, an audio module (e.g., including audio circuitry) <NUM>, a sensor module (e.g., including at least one sensor) <NUM>, and a camera module (e.g., including a camera) <NUM>. Components of the electronic device <NUM> illustrated in <FIG> correspond to a non-limiting example, and some of the illustrated components may be changed or deleted, or some components may be added. According to various embodiments, the electronic device <NUM> may include the electronic device <NUM> of <FIG>. According to various embodiments, the electronic device <NUM> may include the user terminal <NUM> of <FIG>. For example, the electronic device <NUM> may include some or all of the components of the electronic device <NUM> of <FIG> in addition to the illustrated components. For example, the electronic device <NUM> may include some or all of the components of the user terminal <NUM> of <FIG>. According to various embodiments, the electronic device <NUM> may include some of the components of the intelligence server <NUM> of <FIG>. According to various embodiments, the electronic device <NUM> may communicate with the intelligence server <NUM> of <FIG> and independently or jointly perform operations according to various embodiments. Hereinafter, the electronic device <NUM> of <FIG> will be described in detail with reference to <FIG>.

The application processor <NUM> (hereinafter, interchangeably referred to as a processor <NUM>) may include various processing circuitry and execute, for example, an application according to data stored in the memory <NUM> and may perform various data processing or computation. According to an embodiment, as at least part of the data processing or computation, the processor <NUM> may process a command or data received from another component (e.g., the sensor module <NUM> or the input module <NUM>).

According to an embodiment, the processor <NUM> may include a hardware structure specified for processing a speaker verification model and/or a voice recognition model for performing a voice service. For example, the processor <NUM> may include a voice service agent <NUM>, a speaker verification training module <NUM>, a first voice recognition module <NUM>, and a speaker verification module <NUM>. The voice recognition model and/or the speaker verification model may be generated through machine learning. Such learning may be performed, for example, by the electronic device <NUM> or via a separate server (e.g., the server <NUM> of <FIG>).

According to an embodiment, the voice service agent <NUM>, the speaker verification training module <NUM>, the first voice recognition module <NUM>, and the speaker verification module <NUM> may be implemented as software (e.g., the program <NUM>) including one or more instructions stored in a storage medium (e.g., internal memory <NUM> or external memory <NUM>) that is readable by a machine (e.g., the electronic device <NUM>). For example, a processor (e.g., the processor <NUM>) of the machine (e.g., the electronic device <NUM>) may allow performing at least one function of the modules by invoking at least one of the one or more instructions stored in the storage medium and operating the components of the machine according to the at least one invoked instruction.

According to an embodiment, the processor <NUM> may include at least a part of a hardware structure for implementing the voice service agent <NUM>, the speaker verification training module <NUM>, the first voice recognition module <NUM>, and the speaker verification module <NUM>, and allow performing at least one function of the modules by invoking at least one of the one or more stored instructions from the storage medium and operating the hardware structure according to the at least one invoked instruction.

The processor <NUM> may provide a voice-based service through the voice service agent <NUM>. According to an embodiment, the voice service agent <NUM> may provide a user with a response corresponding to a voice signal according to a user utterance received through the microphone of the input module <NUM>, through the sound output module <NUM>. According to an embodiment, at least a part of a series of operations of processing the voice signal and generating the response may be performed by the voice recognition module <NUM> or the voice service agent <NUM> of the electronic device <NUM>. According to an embodiment, at least a part of the series of operations of processing the voice signal and generating the response may be performed by the server <NUM>. For example, the server <NUM> may include the intelligence server <NUM> of <FIG> or may be implemented as a server separate from the intelligence server <NUM>. For example, a series of operations of processing the voice signal and generating the response may include an operation (action or action sequence) of converting a voice into text (automatic speech recognition (ASR), an operation of extracting an intent corresponding to text and related parameters and slots thereof, and an operation performed according to a plan corresponding to the intent.

The processor <NUM> may process a voice signal through the first voice recognition module <NUM> and convert the voice signal into text using a voice recognition algorithm. According to an embodiment, the first voice recognition module <NUM> may be implemented to recognize a limited number of words. According to an embodiment, a word which can be recognized by the first voice recognition module <NUM> may be pre-designated and stored in, for example, the memory <NUM>. According to an embodiment, the first voice recognition module <NUM> may recognize more words through the server <NUM>. According to an embodiment, the first voice recognition module <NUM> may store data required for a voice recognition operation in memory <NUM>. According to an embodiment, the first voice recognition module <NUM> may perform a hidden Markov model (HMM)-based voice recognition algorithm. For example, the first voice recognition module <NUM> may perform the voice recognition algorithm using information such as state initial probability, state transition probability, and/or observation probability required for the Markov model as a voice recognition model. According to an embodiment, the first voice recognition module <NUM> may use a neural network-based algorithm. For example, the first voice recognition module <NUM> may perform the voice recognition algorithm using information such as a layer, a node type and structure, a weight for each node, network connection information, an activation function required for the neural model as a voice recognition model. According to an embodiment, the first voice recognition module <NUM> may perform a recognition algorithm according to a combination of two types of voice recognition models. For example, the first voice recognition module <NUM> may use a hybrid model of the HMM model and the neural model.

The processor <NUM> may perform speaker verification of an input voice through the speaker verification module <NUM>. The speaker verification according to an embodiment may be speaker-dependent voice recognition. For example, the speaker verification may be performed using information on a voice feature of a registered user. For example, the speaker verification may be performed using the speaker verification model <NUM> stored in the memory <NUM>. For example, the speaker verification may be performed by comparing the information on the voice feature of the registered user, the information being trained by the speaker verification model <NUM>, with information on a feature of the input voice. For example, the speaker verification may be performed by a speaker verification score obtained by comparing the information on the voice feature of the registered user, the information being trained by the speaker verification model <NUM>, with the information on the feature of the input voice. For example, the information on the feature of the input voice may include various types of property information such as the frequency feature of a voice signal of the input voice, the duration or the volume of the voice, the number of phonemes, or the length of the phoneme.

According to an embodiment, the speaker verification module <NUM> may perform speaker verification with reference to additional information of the input voice. For example, when user verification such as biometric recognition is performed by the electronic device <NUM> within a configured time interval before or after the voice input, the input voice may be determined as a voice of the registered user.

According to an embodiment, the speaker verification training module <NUM> may generate the speaker verification model <NUM> and store the same in the memory <NUM> using feature information of the user voice input in the process in which a user registers the user's voice. For example, the speaker verification model <NUM> may be generated by learning a Gaussian mixture model - universal background model (GMM-UBM) according to a maximum likelihood linear regression (MLLR) scheme or a maximum a posteriori (MAP) scheme. For example, the speaker verification model <NUM> may be generated using a feature vector (I-vector) corresponding to the speaker. For example, the speaker verification model <NUM> may be generated using a statistics model such as a neural network model. For example, the speaker verification model <NUM> may be generated using a feature vector column of a voice signal, generated using a recognition algorithm such as dynamic time warping (DTW) and vector quantization (VQ). The speaker verification model <NUM> may include various recognition models in addition to the above-mentioned models, and the models may be generated using information which can represent a voice to be recognized. For example, the speaker verification model <NUM> may include property information of a voice. For example, the property information may include at least one of the frequency feature of a voice signal, the duration or the volume of the voice, the number of phonemes, or the length of the phoneme.

According to an embodiment, the speaker verification module <NUM> may analyze the input voice according to an utterance after user registration using the speaker verification model <NUM>. Accordingly, when the input voice is determined as a voice of a registered user, the speaker verification training module <NUM> may train and update the speaker verification model <NUM> using the input voice. For example, the speaker verification training module <NUM> may include an additionally input voice to the existing voice information and train the speaker verification model <NUM> using one or more of the above-described various schemes.

According to an embodiment, when user verification such as biometric recognition is performed by the electronic device <NUM> within a configured time interval before or after the voice input, with respect to feature information of the corresponding input voice, the processor <NUM> may train and update the speaker verification training model <NUM> by comparing feature information of a voice of a registered user with the feature information of the input voice and applying a weight according to similarity upon the result of the comparison.

The auxiliary processor <NUM> may be a processor having a relatively smaller power consumption amount than that of the application processor <NUM>. The auxiliary processor <NUM> may be configured to allow the electronic device <NUM> to perform a designated function while being continuously supplied with power even in an idle state. For example, the auxiliary processor <NUM> may control at least some of functions or states related to at least one component (e.g., the audio model <NUM>, the sensor module <NUM>, or the camera module <NUM>) among the components of the electronic device <NUM>, instead of the application processor <NUM> while the application processor <NUM> is in an inactive (e.g., sleep) state.

According to an embodiment, the auxiliary processor <NUM> may recognize the input voice through a second voice recognition module <NUM>. The second voice recognition module <NUM> included in the auxiliary processor <NUM> may perform voice recognition using a simple voice recognition algorithm which can be operated with low power. For example, the second voice recognition module <NUM> implemented by the auxiliary processor <NUM> that is a low power processor may be implemented to recognize a designated word (e.g., a wakeup word). In this case, the second voice recognition module <NUM> may be configured to use a voice recognition model for the designated word. For example, when a voice signal is input through the audio module <NUM> while the application processor <NUM> is in a sleep state and a pre-configured wakeup word (e.g., "Hi Bixby") is recognized through the second voice recognition module <NUM> in response to the input voice signal, the auxiliary processor <NUM> may transfer a wakeup signal which activates the processor <NUM>. For example, the auxiliary processor <NUM> may change the state of the processor <NUM> from the inactive state to the active state using the wakeup signal. For example, the processor <NUM> may activate the voice service agent <NUM> to provide a voice service function of the electronic device <NUM>. For example, the processor <NUM> may identify whether the wakeup word corresponds to the registered user's voice through the speaker verification module <NUM> before providing the voice service function of the electronic device <NUM>, and when the wakeup word does not correspond to the registered user's voice, the processor <NUM> may not activate the voice service agent <NUM> or may suspend providing the voice service function.

According to an embodiment, in a case in which the processor <NUM> is in an active state, when a user utterance occurs and the user utterance is received through a microphone (e.g., the microphone <NUM> of <FIG> or the input module <NUM> of <FIG> or <FIG>) and a voice signal is processed through the audio module <NUM> and is input, a pre-configured wakeup word may be recognized through the first voice recognition module <NUM> in response to the input voice signal. When the pre-configured wakeup word is recognized through the first voice recognition module <NUM>, the speaker recognition module <NUM> may identify whether the recognized wakeup word corresponds to the registered user's voice. For example, when the recognized wakeup word does not correspond to the registered user's voice, the processor <NUM> may not activate the voice service agent <NUM> or may suspend providing the voice service function. For example, when the recognized voice corresponds to the registered user's voice, the processor <NUM> may activate the voice service agent <NUM> to provide the voice service function. For example, when the recognized wakeup word corresponds to the registered user's voice, the processor <NUM> may train and update the speaker verification model <NUM> using the corresponding wakeup voice through the speaker verification training module <NUM>.

The configuration of the application processor <NUM> and the auxiliary processor <NUM> of <FIG> is illustrative, and the application processor <NUM> and the auxiliary processor <NUM> may, for example, be implemented as a single processor <NUM>.

The memory <NUM> may store various types of data used by at least one component (e.g., the processor <NUM>, the auxiliary processor <NUM>, or the sensor module <NUM>). Data may include, for example, software (e.g., the first app 155_1 and the second app 155_2 of <FIG> and the program <NUM> of <FIG>) and input data or output data of a command related thereto. The memory <NUM> may include the speaker verification model <NUM> for speaker verification according to various embodiments as described above.

The input module <NUM> may include various input circuitry and receive a command or data to be used for a component (e.g., the processor <NUM> or <NUM>) of the electronic device <NUM> from the outside (e.g., a user) of the electronic device <NUM>. The input module <NUM> may include, for example, a microphone or a touch input device (e.g., a touch panel or a touch screen).

The sound output module <NUM> may include various sound output circuitry and output a sound signal to the outside of the electronic device <NUM>.

The audio module <NUM> may include various audio circuitry and convert a sound into an electrical signal and vice versa. According to an embodiment, the audio module <NUM> may obtain the sound via the input module <NUM>, or output the sound via the sound output module <NUM> or an external electronic device (e.g., the electronic device <NUM> of <FIG>) (e.g., a speaker or a headphone) directly or wirelessly coupled with the electronic device <NUM>.

The sensor module <NUM> may include at least one sensor and detect an environmental state (e.g., a state of a user) external to the electronic device <NUM>, and then generate an electrical signal or a data value corresponding to the detected state. According to an embodiment, the sensor module <NUM> may include, for example, a biometric sensor such as a fingerprint sensor <NUM>-<NUM> or an iris sensor <NUM>-<NUM>. The fingerprint sensor <NUM>-<NUM> or the iris sensor <NUM>-<NUM> is an example of the biometric sensor, and the sensor module <NUM> may additionally include, for example, various biometric sensors such as a contactless vein sensor.

According to an embodiment, a part of at least one sensor, for example, the fingerprint sensor <NUM>-<NUM> may be included in the input module <NUM> of the electronic device <NUM>. According to an embodiment, the input module <NUM> may include a home button, a side button (e.g., a physical button or key or an input unit disposed in relation to execution of an intelligent agent function) of the electronic device <NUM>, and/or at least a predetermined area (e.g., an icon area) of a display (e.g., the display <NUM> of <FIG> or the display module <NUM> of <FIG>), and the fingerprint sensor may be included in the home button, the side button, or at least the predetermined area of the display <NUM>. A fingerprint recognition function may be performed when a user presses the home button, presses the side button assigned in relation to the execution of an intelligent agent function or a voice command function, or touches the predetermined area of the display <NUM>.

According to an embodiment, the iris sensor <NUM>-<NUM> may include a light emitting unit for outputting light having a designated wavelength band and a light receiving unit for receiving light having a designated wavelength band and recognizing an iris.

According to an embodiment, when user biometric verification is performed through a biometric sensor in the electronic device <NUM> within a configured time interval before or after the voice input through the audio module <NUM>, the processor <NUM> may determine that the input voice corresponds to a registered user's voice. For example, when the user biometric verification is performed through a biometric sensor such as the fingerprint sensor <NUM>-<NUM> or the iris sensor <NUM>-<NUM>, the processor <NUM> may determine that the input voice corresponds to a user's voice.

The camera module <NUM> may include at least one camera and capture a still image or moving images. According to an embodiment, the camera module <NUM> may be activated in a specific situation (e.g., a situation in which a wakeup word is received through the input module <NUM>) and operate as a biometric sensor by capturing an image (e.g., an image of a biometric feature or the face of a user).

<FIG> is a flowchart illustrating an example operation of an electronic device according to various embodiments.

According to an embodiment, at least one of operations <NUM> to <NUM> may be omitted, the sequence of some operations may change, or other operations may be added.

Referring to <FIG>, in operation <NUM>, one or more processors (e.g., the processor <NUM> of <FIG> or the processor <NUM>, <NUM>, or <NUM> of <FIG> and <FIG>) of an electronic device (e.g., the user terminal <NUM> of <FIG> or the electronic device <NUM> of <FIG> and <FIG>) may recognize a voice input (e.g., utterance) to the electronic device. For example, the processor may process a voice signal input to the electronic device and convert the voice signal to text using a voice recognition algorithm. For example, the processor may execute an application for a voice recognition service when the voice signal input to the electronic device corresponds to a designated wakeup word.

According to an embodiment, in operation <NUM>, the processor may verify a speaker of the input voice signal. For example, with respect to the voice signal, the processor may verify a speaker using a speaker verification model (e.g., the speaker verification model <NUM> of <FIG>) stored in a memory (e.g., the memory <NUM> of <FIG>). For example, the processor may perform speaker verification by comparing voice feature information of a registered user with feature information of the input voice.

For example, with respect to the voice signal, the processor may obtain a speaker verification score using the speaker verification model. For example, the speaker verification score may be obtained upon the speaker feature similarity according to the speaker verification model.

For example, the speaker feature similarity may correspond to a scale of the degree of similarity between a registered user's speaker feature according to the existing speaker verification model and a speaker feature of a newly input voice signal, and may be obtained according to a learning model.

For example, the speaker feature similarity may be obtained using a likelihood ratio test (LRT) in the GMM-UBM-type model, and may be obtained using the Euclidian distance or the cosine distance in a template matching-type model. For example, the speaker feature similarity may be obtained using the distance between speaker feature vectors in a neural network or HMM model. For example, the speaker feature similarity may be obtained using speaker feature conversion factor F according to each learning model using multiple learning models. For example, speaker feature conversion factor F according to each learning model may be a linear function or a non-linear function.

For example, when additional learning or updating is pre-configured for the speaker verification model for the registered user, the processor may allow a speaker verification operation to be performed for a recognized voice signal upon the recognition of a voice signal input to the electronic device.

For example, when the voice signal input to the electronic device is recognized as a designated wakeup word, the processor may allow the speaker verification operation to be performed for the recognized wakeup word.

According to an embodiment, in operation <NUM>, the processor may determine whether the speaker of the input voice signal is a registered user.

For example, the processor may determine that the speaker of the input voice signal corresponds to a registered user according to the result of the speaker verification in operation <NUM>. For example, the processor may determine that the speaker of the input voice signal corresponds to the registered user when the speaker verification score of the input voice signal is greater than a configured threshold value according to the result of the speaker verification in operation <NUM>.

According to an embodiment, according to whether other verification events have occurred, the processor may determine that the speaker of the input voice signal correspond to the registered user. For example, when other types of user verification are performed for the registered user within a designated time interval before or after (e.g., <NUM> seconds of a time interval before or after) an input time point of the input voice signal, the processor may determine that the speaker of the input voice signal corresponds to the registered user. For example, other types of user verification may include a type of verification according to biometric recognition including fingerprint recognition, iris recognition, vein recognition, or face recognition of a user.

According to an embodiment, the processor may determine whether the speaker corresponds to the registered user according to the result of the speaker verification for the input voice signal and whether other verification events have occurred.

According to an embodiment, the processor may terminate an application for the voice recognition service if the voice signal of the speaker does not correspond to the voice of the registered user (If "No" in operation <NUM>).

According to an embodiment, when the speaker is determined as the registered user, the processor may train and update the speaker verification model using the input voice signal in operation <NUM>. For example, the speaker verification model training may be performed for the existing speaker verification model according to the speaker feature similarity of the newly input voice. For example, the speaker verification model training may be performed by applying a weight based on speaker feature similarity of the newly input voice to the existing speaker verification model. For example, the weight based on the similarity may be obtained by applying a function which linearly or non-linearly increases as the similarity increases.

According to an embodiment, the weight based on the speaker feature similarity may have a value obtained by applying function g to the likelihood ratio test (LRT) in the GMM-UBM-type model, and may have a value obtained by applying function g to the Euclidian distance or the cosine distance in the template matching-type model. For example, the weight based on the speaker feature similarity may have a value obtained by applying function g to the distance between the speaker feature vectors in the neural network or HMM model. For example, the weight based on the speaker feature similarity may have a value obtained by applying function g to speaker feature conversion function F according to each learning model using multiple learning models.

According to an embodiment, representative speaker feature score S <NUM> may be obtained according to Equation <NUM> below by applying respective weights of speaker features of learning voices.

Here, F is a function of converting speaker features of learning voices, speaker features being multiplied by the weights, into the representative speaker feature, u; may correspond to an i-th learning voice speaker feature, and wi may correspond to a weight applied according to the i-th learning voice speaker feature.

According to an embodiment, the weight applied to each of the speaker features of the learning voices may be obtained according to Equation <NUM> below.

Here, g is a function of a correlation between the similarity and the weight, and si may correspond to the similarity between the registered speaker feature and the i-th learning voice speaker feature. A linear function or a non-linear function may be applied for function g.

According to an embodiment, after the similarity score for each learning voice speaker feature is obtained, representative similarity score S2 for all learning voice speaker features may be obtained according to Equation <NUM> below.

Here, F is a function of converting the similarity between speaker features of learning voices, speaker features being multiplied by the weights, into the representative speaker feature similarity, u; may correspond to the i-th learning voice speaker feature, uT may correspond to a current voice speaker feature to be learned, S(ui, uT) may correspond to the similarity between the currently learning voice speaker feature and the i-th learning voice speaker feature obtained by similarity calculation function S, and wi may correspond to a weight applied according to the i-th learning voice speaker feature.

Referring to <FIG>, in operation <NUM>, one or more processors (e.g., the processor <NUM> of <FIG> or the processor <NUM>, <NUM>, or <NUM> of <FIG> and <FIG>) of an electronic device (e.g., the user terminal <NUM> of <FIG> or the electronic device <NUM> of <FIG> and <FIG>) may recognize a voice input (e.g., utterance) to the electronic device. According to an embodiment, the voice input to the electronic device may be limited to a voice (e.g., a wakeup word) including a designated content that is pre-configured. According to an embodiment, the voice input of the electronic device may not be limited to a voice including a designated content that is pre-configured.

According to an embodiment, in operation <NUM>, the processor may determine whether other types of user verification have been performed within a designated time interval before or after (e.g., <NUM> seconds of a time interval before or after) an input time point of the input voice signal. For example, other types of user verification may include a type of verification according to biometric recognition including fingerprint recognition, iris recognition, or face recognition of a user.

According to an embodiment, the processor may terminate a speaker verification model training operation if it is determined that no other types of user verification are performed within the designated time interval before or after the input time point of the input voice signal in operation <NUM>.

According to an embodiment, it is determined that other types of user verification have been performed within the designated time interval before or after the input time point of the input voice signal, the processor may obtain a speaker verification score of the input voice signal and determine whether the obtained speaker verification score is greater than a designated threshold value in operation <NUM>. For example, different designated threshold values of the speaker verification score may be configured according to a learning model. For example, the speaker verification score may be obtained according to the above-described equations. For example, when the speaker verification score is configured to be a scale of <NUM> to <NUM>, the threshold value may be configured as a higher scale, for example, <NUM> or <NUM>, according to learning data and/or the feature of the learning model.

According to an embodiment, if the speaker verification score is greater than the threshold value, the processor may train and update the speaker verification model using the input voice signal in operation <NUM>. For example, the processor may extract a speaker feature of the input voice signal and update the speaker verification model by applying a weight obtained according to the above-described equations to the extracted speaker feature.

According to an embodiment, if the speaker verification score is not greater than the threshold value in operation <NUM>, the processor may terminate the speaker verification model training operation.

Referring to <FIG>, in operation <NUM>, one or more processors (e.g., the processor <NUM> of <FIG> or the processor <NUM>, <NUM>, or <NUM> of <FIG> and <FIG>) of an electronic device (e.g., the user terminal <NUM> of <FIG> or the electronic device <NUM> of <FIG> and <FIG>) may recognize a voice input (e.g., utterance) to the electronic device.

According to an embodiment, if other types of user verification have been performed within the designated time interval before or after the input time point of the input voice signal, the processor may obtain a speaker verification score of the input voice signal in operation <NUM> and may determine whether the obtained speaker verification score satisfies a designated condition in operation <NUM>. For example, the designated condition for the speaker verification score may be a condition in which the speaker verification score is greater than a minimum threshold value. If the speaker verification score does not satisfy the designated condition in operation <NUM>, the processor may terminate the operation.

According to an embodiment, if the speaker verification score satisfies the designated condition, the processor may obtain a weight to be applied to the input voice signal in operation <NUM>. For example, the weight may be obtained using a speaker verification model. For example, with respect to the existing speaker verification model, the weight may be obtained as a function according to the speaker feature similarity of a newly input voice. For example, a function for obtaining the weight may be a linear function or a non-linear function for the similarity according to the speaker verification model.

According to an embodiment, in operation <NUM>, the processor may train and update the speaker verification model by applying the obtained weight for the input voice signal. For example, the processor may extract a speaker feature of the input voice signal and may update the speaker verification model by applying the weight obtained according to the above-described equations to the extracted speaker feature.

According to an embodiment, if it is determined that no biometric verification is performed in operation <NUM>, the processor may request a user to perform biometric verification in operation <NUM>. For example, the processor may output guidance in a voice requesting to perform biometric verification, through a sound output module (e.g., the sound output module <NUM> of <FIG>) of the electronic device, or output the guidance in text and/or an image through a screen of a display (e.g., the display module <NUM> of <FIG>).

<FIG>, <FIG>, <FIG> and <FIG> are diagrams illustrating example user registration operations invoking a voice recognition service according to various embodiments.

Referring to <FIG>, when a user selects a menu item of user wakeup command registration for invoking a voice recognition service, one or more processors (e.g., the processor <NUM> of <FIG> or the processor <NUM>, <NUM>, or <NUM> of <FIG> and <FIG>) of an electronic device (e.g., the user terminal <NUM> of <FIG> or <FIG> or the electronic device <NUM> of <FIG> and <FIG>) may display guidance <NUM> for wakeup command registration on a display (e.g., the display <NUM> of <FIG> or the display module <NUM> of <FIG>). For example, the user may select a start icon <NUM> displayed on a display screen to start a user wakeup command registration operation. When the user selects a suspension icon <NUM>, the displayed registration menu may be released and the screen may switch to the previous screen or the home screen.

Referring to <FIG>, when the user starts the wakeup command registration operation, the processor may display guidance <NUM> for a wakeup command input, in text and/or an image, through a display.

Referring to <FIG>, when the user generates an utterance for the wakeup command input and the utterance is received through a microphone (e.g., the microphone <NUM> of <FIG> or the input module <NUM> of <FIG>) of the electronic device, the processor may output guidance <NUM> requesting the user to utter a received wakeup word (e.g., "Hi Bixby") a designated number of times.

Referring to <FIG>, when the user has completed the wakeup command registration operation, the processor may output guidance <NUM> inquiring of whether to perform additional wakeup command learning upon the selection of the user or automatically. For example, the user may select whether to perform additional wakeup command learning using a "Yes" or "No" button <NUM> or <NUM>.

According to an embodiment, when the user selects a function of using additional wakeup command learning, the processor may perform a registered user voice verification learning operation for updating the speaker verification model as described with reference to <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG> and <FIG> above.

<FIG> and <FIG> are diagrams illustrating examples of an operation of acquiring a voice of a registered voice of an electronic device according to various embodiments.

Referring to <FIG>, a user may utter a voice service wakeup command (e.g., "Hi Bixby") and input a voice of a designated instruction (e.g., "Order coffee") through a microphone (e.g., the microphone <NUM> of <FIG> or the input module <NUM> of <FIG>) of an electronic device (e.g., the user terminal <NUM> of <FIG> or <FIG> or the electronic device <NUM> of <FIG> and <FIG>). One or more processors (e.g., the processor <NUM> of <FIG> or the processor <NUM>, <NUM>, or <NUM> of <FIG> and <FIG>) may perform voice recognition on the user's wakeup command, and may activate a voice service agent (e.g., the voice service agent <NUM> of <FIG>) to analyze text of the instruction through voice recognition on the designated instruction of the user input. The processor may generate a plan for performing a task corresponding to the input designated instruction of the user (e.g., a plan for executing a payment app for ordering coffee).

Referring to <FIG>, for example, according to the text analysis of ordering coffee in the above-described example, the processor may identify a place (e.g., a café) where the user is currently located and execute an app for ordering coffee in the corresponding café, so as to allow the user to select one of coffee menu items. In addition, upon the selection of the user, the processor may perform a task according to a plan of executing a payment app and outputting guidance <NUM> for inputting biometric information (e.g., outputting a fingerprint icon for inducing input of fingerprint information). In another example, the processor may execute a coffee ordering app and a payment app, wherein the processor may output voice guidance (e.g., "Would you like to order iced Americano?") asking whether to order a type of coffee that the user usually orders. Accordingly, the processor may identify the user's decision and then execute a plan for outputting guidance <NUM> allowing the input of biometric information through the payment app.

According to an embodiment, when user biometric information is input through the payment app within a designated time interval after a wakeup word, "Hi Bixby", which is a voice service wakeup command of the user, is received, and the user is verified, the processor may perform a registered user voice verification learning operation for updating the speaker verification model using the received wakeup word as described with reference to <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG> and <FIG> above. For example, the designated time interval from a user utterance reception time point may be configured as a time taken until the user inputs biometric information through the payment app after the utterance time point.

According to an embodiment, when user biometric information is input through the payment app within the designated time interval after receiving the designated instruction (e.g., "Order coffee") that is the user's uttered voice, and the user is verified, the processor may perform the registered user voice verification learning operation for updating the speaker verification model using the received specific instruction as described with reference to <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG> and <FIG> above.

<FIG>, <FIG> and <FIG> are diagrams illustrating examples of an operation of acquiring a voice of a registered user of an electronic device according to various embodiments. Hereinafter, examples of identifying whether a user utterance corresponds to a registered user's voice by requesting user verification including biometric verification when receiving the user utterance will be described with reference to <FIG>, <FIG> and <FIG>.

Referring to <FIG>, the user may utter a voice service wakeup command (e.g., "Hi Bixby") through a microphone (e.g., the microphone <NUM> of <FIG> or the input module <NUM> of <FIG>) of an electronic device (e.g., the user terminal <NUM> of <FIG> or <FIG> or the electronic device <NUM> of <FIG> and <FIG>). For example, the electronic device may be in a standby state, wherein a main processor (e.g., the application processor <NUM> of <FIG> or <FIG>) is in an inactive (e.g., sleep) state. For example, when the electronic device is in the standby state and the main processor thereof is in the inactive state, a screen of the electronic device may be displayed as a screen (e.g., a clock screen) of a low power display (always on display (AOD) which can be operated at low power even in a state in which main power is off and the screen is off as shown in <FIG>.

According to an embodiment, as the voice service wakeup command of the user is received while the application processor is in a sleep state, a low power processor (e.g., the auxiliary processor <NUM> of <FIG> or <FIG>) may perform voice recognition on the received user utterance, and may transfer a wakeup signal so as to activate the application processor when the voice service wakeup command is recognized.

According to an embodiment, for example, before the wakeup signal is transferred to the application processor, the low power processor may induce user verification by outputting guidance <NUM> requesting to release the inactive (e.g., a lock screen mode) state of the electronic device as shown in <FIG> and/or outputting an icon <NUM> guiding verification through a fingerprint input of the user as shown in <FIG>. For example, the low power processor may obtain a speaker verification score of the received user utterance and induce user verification by outputting user verification guidance when the obtained score is equal to or less than a threshold value that is configured to have a relatively high value. For example, the low power processor may output the user verification guidance when the user does not perform user verification for releasing the lock within a designated time interval after the received user utterance.

According to an embodiment, when the user performs user verification in various ways such as biometric verification or using a password or a pattern for releasing the lock of the electronic device, the application processor may be activated according to the wakeup signal received from the low power processor.

According to an embodiment, as the voice service wakeup command of the user is received while the application processor is in the sleep state, the low power processor may perform voice recognition on the received user utterance and may transfer a wakeup signal so as to activate the application processor when the voice service wakeup command is recognized. Accordingly, for example, the application processor may induce user verification by outputting the icon <NUM> guiding verification through a fingerprint input of the user as shown in <FIG>.

According to an embodiment, when user verification is successfully done upon the user verification request output in response to the user utterance, the application processor may perform a registered user voice verification learning operation for updating the speaker verification model using the received user utterance voice as described with reference to <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG>, <FIG> and <FIG> above.

<FIG> is a diagram illustrating an example of an operation of acquiring a voice of a registered user of an electronic device according to various embodiments.

Referring to <FIG>, an electronic device (e.g., the user terminal <NUM> of <FIG> or <FIG> or the electronic device <NUM> of <FIG> or <FIG>) may be mounted in, for example, a vehicle <NUM>.

According to an embodiment, a user may utter a voice service wakeup command (e.g., "Hi Bixby") or a designated instruction through a microphone (e.g., the microphone <NUM> of <FIG> or the input module <NUM> of <FIG>).

According to an embodiment, one or more processors (e.g., the processor <NUM> of <FIG> or the processor <NUM>, <NUM>, or <NUM> of <FIG> and <FIG>) may perform voice recognition on the received wakeup command or the designated instruction, and activate a voice service agent (e.g., the voice service agent <NUM> of <FIG>), so as to analyze text including the instruction through voice recognition on the received wakeup command or the designated instruction.

According to an embodiment, when user face recognition is performed through the electronic device within a designated time interval before or after a time point at which the user utterance is received and the user is verified, the processor may determine that the corresponding utterance is made by a registered user. According to an embodiment, when it is determined that the user utterance is made by the registered user, the processor may perform a registered user voice verification learning operation for updating a speaker verification model using the received user utterance voice as described with reference to <FIG> above.

An electronic device (e.g., the user terminal <NUM> of <FIG> or <FIG> or the electronic device <NUM> of <FIG> and <FIG>) described in <FIG> may have a bar-type or a plate-type exterior, but is not limited thereto. For example, the illustrated electronic device may be a part of a rollable electronic device or a foldable electronic device. A "rollable electronic device" may refer to an electronic device, the at least a part of which can be wound or rolled, or can be received in a housing (not shown) since bending deformation can be applied to a display (e.g., the display <NUM> of <FIG>) thereof. According to the user's need, the display of the rollable electronic device may be unfolded or the wider space of the display may be exposed, whereby the user can use an expanded screen display area. A "foldable electronic device" may refer, for example, to an electronic device which has two different areas of a display, wherein two areas can be folded to face each other or be oriented opposite to each other.

Claim 1:
An electronic device (<NUM>, <NUM>) comprising:
a microphone (<NUM>, <NUM>);
at least one sensor (<NUM>);
a memory (<NUM>, <NUM>) storing a speaker verification model for verifying a voice of a registered user; and
at least one processor (<NUM>, <NUM>) operatively connected to the microphone, the at least one sensor, and the memory, wherein the at least one processor is configured to:
based on receiving an uttered voice through the microphone, identify whether user verification information is received through the at least one sensor within a designated time interval before or after a time point at which the uttered voice is received; and
based on completion of the user verification according to the user verification information, update the speaker verification model using the uttered voice,
wherein the at least one processor is configured to: compare voice feature information of the registered user with feature information of the uttered voice to obtain a designated weight according to similarity, and update the speaker verification model using the uttered voice to which the weight is applied.