Patent Description:
Text to speech (TTS) refers to a speech synthesis technology in which a machine synthesizes (or converts) a text into a human speech. Document <CIT>, discloses a method and associated apparatus for building a speech database for training a text - to - speech synthesis system.

To provide a speech of a style (for example, a pitch, an accent, a delivery speed, an intonation, a pronunciation, and a speaking habit) similar to a speech of a target speaker through a TTS service, a process of recording a speech spoken by the target speaker in advance and processing the recorded speech data is required. To implement a natural speech of a style similar to the target speaker for various texts, model learning through a method such as model adaptation and transfer learning based on a spoken speech of the target speaker for two hundred or more sentences (or a script of one hour or more) is required.

There is a difficulty in providing a personalized TTS service that provides a speech of a general user as the speech of the TTS service in that the target speaker should speak a very large number of sentences with an accurate pronunciation for a long time to register the speech of the target speaker as the speech of the TTS service as described above. Meanwhile, there is a method of obtaining a reference vector from the speech of the target speaker and inputting a text and the reference vector to a TTS model to obtain a synthesized sound having speech characteristics of the target speaker, to provide the personalized TTS service. In this case, there is an advantage that the model may not be trained (zero-shot/few-shot learned), but there is a disadvantage that it may not be known whether or not the reference vector has optimal performance (sound quality/prosody/pronouncement/speaker similarity, and the like).

The disclosure provides an electronic apparatus for providing a text to speech (TTS) service using a speech of a general user, and a controlling method thereof.

According to an embodiment of the disclosure, an electronic apparatus includes a microphone, a memory in which a TTS model and a plurality of evaluation texts are stored, and a processor configured to: obtain a reference vector of a user speech spoken by a user when the user speech is received through the microphone, generate a plurality of candidate reference vectors based on the reference vector, obtain a plurality of synthesized sounds by inputting the plurality of candidate reference vectors and the plurality of evaluation texts to the TTS model, identify at least one synthesized sound of the plurality of synthesized sounds based on a similarity between the plurality of synthesized sounds and the user speech and characteristics of the plurality of synthesized sounds, and store a reference vector of the at least one synthesized sound in the memory as a reference vector corresponding to the user for the TTS model.

According to another embodiment of the disclosure, a control method of an electronic apparatus including a memory in which a TTS model and a plurality of evaluation texts are stored includes obtaining a reference vector of a user speech spoken by a user when the user speech is received through a microphone, generating a plurality of candidate reference vectors based on the reference vector, obtaining a plurality of synthesized sounds by inputting the plurality of candidate reference vectors and the plurality of evaluation texts to the TTS model, identifying at least one synthesized sound of the plurality of synthesized sounds based on a similarity between the plurality of synthesized sounds and the user speech and characteristics of the plurality of synthesized sounds, and storing a reference vector of the at least one synthesized sound in the memory as a reference vector corresponding to the user for the TTS model.

According to the diverse embodiments of the disclosure, the electronic apparatus for providing a TTS service using a speech of a general user and a controlling method thereof may be provided.

In addition, according to an embodiment of the disclosure, a system may be configured to minimize the number of sentences requested to be spoken to register the voice of the TTS service and have optimal performance. In addition, according to an embodiment of the disclosure, the personalized TTS service may be provided through a voice of the user without retraining the TTS model for each user.

In describing the disclosure, when it is determined that a detailed description for the known functions or configurations related to the disclosure may unnecessarily obscure the gist of the disclosure, the detailed description therefor will be omitted. The invention is defined exclusively by the appended claims. Throughout the accompanying drawings, similar components will be denoted by similar reference numerals.

Expressions "first" or "second" used in the disclosure may indicate various components regardless of a sequence and/or importance of the components, will be used only to distinguish one component from the other components, and do not limit these components.

In the disclosure, an expression "A or B", "at least one of A and/or B", or "one or more of A and/or B", may include all possible combinations of items enumerated together. For example, "A or B", "at least one of A and B", or "at least one of A or B" may indicate all of <NUM>) a case where at least one A is included, <NUM>) a case where at least one B is included, or <NUM>) a case where both of at least one A and at least one B are included.

In the disclosure, singular forms include plural forms unless the context clearly indicates otherwise. It should be understood that terms "include" or "formed of" used in the specification specify the presence of features, numerals, steps, operations, components, parts, or combinations thereof mentioned in the specification, but do not preclude the presence or addition of one or more other features, numerals, steps, operations, components, parts, or combinations thereof.

When it is mentioned that any component (for example, a first component) is (operatively or communicatively) coupled to or is connected to another component (for example, a second component), it is to be understood that any component is directly coupled to another component or may be coupled to another component through the other component (for example, a third component). On the other hand, when it is mentioned that any component (for example, a first component) is "directly coupled" or "directly connected" to another component (for example, a second component), it is to be understood that the other component (for example, a third component) is not present between any component and another component.

An expression "configured (or set) to" used in the disclosure may be replaced by an expression "suitable for", "having the capacity to" "designed to", "adapted to", "made to", or "capable of" depending on a situation. A term "configured (or set) to" may not necessarily mean "specifically designed to" in hardware. Instead, in some situations, an expression "apparatus configured to" may mean that the apparatus may "do" together with other apparatuses or components. For example, a phrase "processor configured (or set) to perform A, B, and C" may mean a dedicated processor (for example, an embedded processor) for performing these operations or a generic-purpose processor (for example, a central processing unit (CPU) or an application processor) that may perform these operations by executing one or more software programs stored in a memory device.

<FIG> is view for describing an electronic apparatus according to an embodiment of the disclosure.

Referring to <FIG>, the electronic apparatus <NUM> according to an embodiment of the disclosure may be implemented as an interactive system.

Here, the electronic apparatus <NUM> may include at least one of a smartphone, a tablet personal computer (PC), a mobile phone, a video phone, an e-book reader, a desktop PC, a laptop PC, a netbook computer, a workstation, a server, a personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, a wearable device, or a robot. According to diverse embodiments, the wearable device may be implemented as at least one of an accessory-type wearable device (for example, a watch, a ring, a bracelet, an anklet, a necklace, a glasses, a contact lens, or a head-mounted-device (HMD)), a textile or clothing integral-type wearable device (for example, an electronic clothing) , a body attachment-type wearable device (for example, a skin pad or a tattoo), or a living body implantation-type wearable device (for example, an implantable circuit). However, this is only an example, and the electronic apparatus <NUM> is not limited thereto, and may be implemented as an electronic apparatus having various shapes and purposes.

The interactive system refers to a system capable of interacting with a user through a conversation, such as grasping a user's intention through a user speech and outputting a response corresponding to the user's intention.

As a specific embodiment, the electronic apparatus <NUM> may include an automatic speech recognition (ASR) module <NUM>, a natural language processing (NLP) module <NUM>, and a text to speech (TTS) module <NUM>. In addition, the electronic apparatus <NUM> may include a microphone <NUM> for receiving a user speech and an output interface <NUM> for outputting information responding to the user speech. For example, the output interface <NUM> may include a speaker for outputting a sound.

The ASR module <NUM> may convert a speech signal (that is, a user speech) received through the microphone <NUM> into a text (string) such as a word or phoneme sequence using a language model and an acoustic model. The language model may be a model assigning a probability to the word or phoneme sequence, and the acoustic model may be a model indicating a relationship between a speech signal and a text for the speech signal. These models may be configured based on a probability statistics or an artificial neural network.

The NLP module <NUM> may identify the meaning of words or sentences configuring a text corresponding to the user speech with respect to the text corresponding to the user speech using various analysis methods such as a morphological analysis, a syntactic analysis, and a semantic analysis for the text corresponding to the user speech, grasp a user' s intention based on the identified meaning, and obtain response information corresponding to the user's intention. In this case, the response information may be in the form of a text.

The TTS module <NUM> may convert a text into a speech signal and output the speech signal through the output interface <NUM> (for example, a speaker). That is, the response information obtained by the NLP module <NUM> may be converted from the form of the text to the form of the speech signal through the TTS module <NUM>.

Meanwhile, the electronic apparatus <NUM> according to an embodiment of the disclosure may provide a personalized TTS service. The personalized TTS service refers to a service that converts (or synthesizes) a text into a speech signal using a speech of an individual user (or another user) through the TTS module <NUM>. To this end, a process of registering the speech of the user in the electronic apparatus <NUM> in advance may be required. The electronic apparatus <NUM> according to the disclosure may minimize the number of sentences requested to be spoken by the user to register a voice of the TTS service. In addition, according to an embodiment of the disclosure, the personalized TTS service may be provided through a voice of the user without retraining a TTS model for each user. Specific contents for this will be described with reference to the accompanying drawings.

<FIG> and <FIG> are block diagrams for describing an operation for each component of the electronic apparatus according to an embodiment of the disclosure. <FIG> illustrates an operation when a reference vector satisfying an evaluation criterion of at least one evaluation text does not exist.

Referring to <FIG>, the electronic apparatus <NUM> according to an embodiment of the disclosure may include a microphone <NUM>, a memory <NUM>, and a processor <NUM>.

The microphone <NUM> may receive a user speech.

The memory <NUM> may store a plurality of evaluation texts. For example, the plurality of evaluation texts may be stored in an evaluation text database <NUM> in the memory <NUM>. A unit of the evaluation text may be one sentence, but this is only an example, and the unit of the evaluation text may be variously modified.

In addition, the memory <NUM> may store a reference vector of a user registered as a speaker of a TTS module <NUM>. The reference vector of the registered user may be stored in a reference vector storage module <NUM> in the memory <NUM> The reference vector of the registered user may indicate a reference vector most matched to a unique speech feature of the registered user.

The processor <NUM> may extract an optimal reference vector from a speech of the user and register the extracted optimal reference vector as a reference vector of the user, to synthesize a text into the speech of the user.

To this end, the processor <NUM> may execute instructionds included in each of a speaker encoder module <NUM>, a candidate reference vector generation module <NUM>, a TTS module <NUM>, and a synthesized sound evaluation module <NUM> to perform operations corresponding to each instruction. Here, the speaker encoder module <NUM>, the candidate reference vector generation module <NUM>, the TTS module <NUM>, and the synthesized sound evaluation module <NUM> may be stored in the memory <NUM> or be stored in a memory inside the processor <NUM>.

Specifically, when a user speech spoken by user A is received through the microphone <NUM>, the processor <NUM> may obtain a reference vector of the user speech from the user speech received through the speaker encoder module <NUM>.

For example, when a request (for example, in the form of a touch input, a speech command or the like of the user A) from the user A for user registration of the TTS module <NUM> is received, the processor <NUM> may provide a reference text (r) set to be spoken by the user A. Thereafter, when a user speech spoken by the user A is received through the microphone <NUM>, the processor <NUM> may obtain a reference vector from the user speech received through the speaker encoder module <NUM>. However, this is only an example, and when the user A speaks a natural language without providing the set reference text, it is also possible to recognize the natural language as a reference text.

Here, the reference vector (RV) (sr,A) is defined as a multi-dimensional (that, two-dimensional or more) vector (or vector column) of a speech feature appearing in a user speech (xr,A) of the user A (speaker A) who has spoken the reference text r. Each dimension (or column) of the reference vector may indicate speech features such as a prosody, a pronunciation, a frequency band, a speaker's age, and a speaker's gender. The reference text refers to a sentence (or a word or the like) spoken by the user, and may be assigned a domain (for example, a reading style, a conversation style, or a news style) according to a speaking method.

Referring to <FIG>, an analog acoustic signal received from the microphone <NUM> may be converted into a digital acoustic signal through an analog-to-digital converter (ADC). Here, the acoustic signal may include the user speech (xr,A) of the user A who has spoken the reference text (r). Meanwhile, the ADC may be implemented in a form in which it is embedded in the microphone <NUM> or the processor <NUM> or may be implemented as a separate device. That is, the ADC may be variously modified.

In this case, the processor <NUM> may identify an acoustic signal of a section corresponding to the user speech (xr,A) from the acoustic signal based on an energy level.

In addition, the processor <NUM> may divide the acoustic signal of the section corresponding to the user speech (xr,A) into in units of frames (for example, <NUM> and <NUM>) and apply a Fourier transform to each frame to calculate a spectrum. Here, the acoustic signal may indicate a waveform expressed in a time domain such as an amplitude (or a sound pressure) over time, and the spectrum may be a waveform expressed in a frequency domain such as an amplitude (or a sound pressure) according to a frequency. For example, the acoustic signal may indicate a waveform in a relationship where a horizontal axis is a time and a vertical axis is an amplitude, and the spectrum may indicate a waveform in a relationship where a horizontal axis is a frequency and a vertical axis is an amplitude. In this case, the spectrum may be a spectrum in a general frequency domain, or may be various types of spectra such as a mel spectrum obtained by applying a filter bank based on a mel scale indicating a relationship between frequencies sensitively perceived by a person to a spectrum and a spectrogram (SPG) having a relationship between a frequency axis and an amplitude axis. In addition, the spectrum may be a cepstrum or a mel cepstrum that may be transformed from the spectrum, and may include a pitch lag or a pitch correlation having pitch/harmonic information. However, this is only an example, and the spectrum may be various acoustic feature vectors having features of the speech.

In addition, the processor <NUM> may obtain the reference vector from the spectrum. As an example, the processor <NUM> may obtain the reference vector (sr,A) by inputting the mel spectrum to a speaker recognizer. However, this is only an example, and the processor <NUM> may obtain the reference vector (sr,A) using one of various algorithms (or neural networks) such as cepstrum, linear predictive coefficient (LPC), and filter bank energy, Wav2Vec, SincNet, and PASE. In this case, the obtained reference vector (sr,A) may be various types of vectors such as i-vector, d-vector, and x-vector.

Meanwhile, a specific method of obtaining the reference vector through the speaker encoder module <NUM> will be described later with reference to <FIG>.

In addition, the processor <NUM> may generate a plurality of candidate reference vectors ( <MAT>) based on the reference vector (sr,A) through the candidate reference vector generation module <NUM>.

Here, the plurality of candidate reference vectors may include one of a reference vector randomly selected based on the reference vector as a first embodiment, a reference vector generated based on the reference vector and a reference vector used for training the TTS module <NUM> as a second embodiment, and a reference vector generated by applying a masking vector to the reference vector as a third embodiment or a combination thereof. Specific contents will be described with reference to <FIG>.

<FIG> are views for describing a method of generating a candidate reference vector according to an embodiment of the disclosure. <FIG> represent reference vectors on planes <NUM>, <NUM>, and <NUM>, and mean that the closer the positions of the reference vectors on the planes <NUM>, <NUM>, and <NUM>, the more similar characteristics of the reference vectors.

Referring to <FIG>, as a first embodiment of the disclosure, a plurality of candidate reference vectors <NUM> (Sr,A) may include at least one reference vector randomly selected based on a reference vectors <NUM> (sr,A).

For example, the processor <NUM> may generate at least one candidate reference vector <NUM> ( <MAT>) by adding a noise to the reference vector <NUM> (sr,A) according to the following Equation <NUM>:
<MAT>.

Here, the noise may be a random value that follows a normal distribution, a uniform distribution, or various probability distributions, and may be a value for at least one of dimensions of the reference vector (sr,A).

In addition, the noise may have a value within a predetermined range. In this case, as illustrated in <FIG>, the candidate reference vectors <NUM> ( <MAT>) may be a vector existing within a predetermined radius based on the reference vectors <NUM> (sr,A).

Meanwhile, referring to <FIG>, as a second embodiment of the disclosure, a plurality of candidate reference vectors <NUM> (Sr,A) may include at least one reference vector generated based on a reference vector <NUM> (sr,A) and a reference vector used for training the TTS module <NUM>.

For example, the processor <NUM> may identify one reference vector <NUM> (for example, a reference vector sB of speaker B) having the closest distance from among the reference vectors of the plurality of speakers by comparing distances between the reference vector (sr,A) and reference vectors of a plurality of speakers (for example, the reference vector sB of the speaker B, a reference vector sC of speaker C, and the like) used for training the TTS module <NUM> with each other.

In addition, the processor <NUM> may generate at least one candidate reference vector <MAT> by applying interpolation to the reference vector (sr,A) and the reference vector <NUM> (for example, the reference vector sB of the speaker B) having the closest distance among the reference vectors used for training the TTS module <NUM> according to the following Equation <NUM>: <MAT>.

Here, wa and wb indicate arbitrary coefficients allowing the candidate reference vector ( <MAT>) to be positioned on a function (for example, a linear function or a quadratic function) passing through two reference vectors (sr,A and sB). For example, in a case of the linear function, wa may be <NUM> and wb may be <NUM>, wa may be <NUM> and wb may be <NUM>, or wa may be <NUM> and wb may be <NUM>.

Meanwhile, as a third embodiment of the disclosure, a plurality of candidate reference vectors (Sr,A) may include at least one reference vector generated by applying a masking vector to a reference vector (sr,A).

For example, the processor <NUM> may generate at least one candidate reference vector (sr,A) by applying a masking vector W to the reference vector (sr,A) according to the following Equation <NUM>: <MAT>.

Here, W is a masking vector, and may be configured as a matrix having <NUM> or <NUM>, or a value between <NUM> and <NUM>, or various values.

Meanwhile, the plurality of candidate reference vectors (Sr,A) according to an embodiment of the disclosure may be at least one combination of the first to third embodiments. That is, the plurality of candidate reference vectors (Sr,A) may include at least one of at least one reference vector (sr,A) randomly selected based on the reference vector (sr,A) , at least one reference vector generated based on the reference vector (sr,A) and a reference vector used for training the TTS module <NUM>, or at least one reference vector generated by applying the masking vector to the reference vector (sr,A). For example, a combination of the first embodiment and the second embodiment may appear as illustrated in <FIG>.

In addition, the processor <NUM> may obtain a plurality of synthesized sounds ( <MAT>) by inputting a plurality of candidate reference vectors ( <MAT>) and a plurality of evaluation texts (T = t<NUM>,. , tM) stored in the memory <NUM> to the TTS module <NUM>.

Specifically, the processor <NUM> may obtain a plurality of synthesized sounds ( <MAT>) generated for each of the plurality of evaluation texts (t<NUM>,. , tM) based on each of the plurality of candidate reference vectors ( <MAT>) by inputting the plurality of candidate reference vectors ( <MAT>) and the plurality of evaluation texts (T = t<NUM>,. , tM) to the TTS module <NUM>. Here, the generated synthesized sound ( <MAT>) is generated by synthesizing the candidate reference vector ( <MAT>) and the evaluation text (tm). In this case, when the number of candidate reference vectors is N and the number of evaluation texts is M, the number of synthesized sounds may be N x M.

Here, the plurality of evaluation texts (t<NUM>,. , tM) may include at least one evaluation text belonging to each of a plurality of domains (for example, a reading style, a conversation style, and a news style). That is, a domain may be assigned to each of the plurality of evaluation texts (t<NUM>,.

For example, the domain may include types such as a reading style, a conversation style, an interrogative sentence, and an exclamatory sentence according to a style of the text, and may include types such as a chatbot, natural language generation (NLG), a news, a dictionary, Wiki, music (a song title, a singer, and lyrics), home appliances (for example, an air conditioner), Bigdata (web crawl content), a fairy tale book, and a novel according to a content of the text. However, this is only an example, and the domain is not limited thereto, and may be variously modified.

As an embodiment, the TTS module <NUM> may include an acoustic model (AM) and a voice decoder (vocoder). Meanwhile, a detailed description of the TTS module <NUM> will be described later with reference to <FIG>.

The acoustic model may convert a text and a reference vector into an acoustic feature using at least one of various algorithms such as Tacotron, Tacotron <NUM>, Transformer, text2mel, and deep convolutional TTS (DCTTS). In this case, the generated acoustic feature may have a reference vector, that is, a feature (for example, a tone, a pitch, an intensity, and a pronunciation) of a corresponding speaker. Here, the acoustic feature may indicate a unique feature (for example, a tone, a pitch, an intensity, and a pronunciation) of a sound in a speech section (for example, a frame unit or a sentence unit). For example, the acoustic feature may be implemented as one of a spectrogram, a mel-spectrogram, a cepstrum, a pitch lag, a pitch correlation, a mel-frequency cepstrum modulation energy (MCME), a mel-frequency cepstral coefficient (MFCC), or the like, in which a waveform and a spectrum are combined with ach other, or a combination thereof.

The vocoder may generate synthesized sound by synthesizing the reference vector and the acoustic feature through various algorithms such as WaveNet, Parallel Wavenet, WaveGlow, WaveRNN, and LPCNet. For example, the vocoder may be a neural network-based artificial intelligence model learned to output a synthesized sound when the acoustic feature such as the mel spectrum and the reference vector are input.

In addition, the processor <NUM> may identify at least one of the plurality of synthesized sounds ( <MAT>) based on a similarity between the plurality of synthesized sounds ( <MAT>) and the user speech and characteristics of the plurality of synthesized sounds ( <MAT>) through the synthesized sound evaluation module <NUM>.

Specifically, the processor <NUM> may identify a candidate synthesized sound of which a similarity (that is, a speaker similarity) with the user speech (xr,A) among the plurality of synthesized sounds ( <MAT>) is a predetermined threshold value or more. Specific contents for this will be described with reference to <FIG> and <FIG>. In addition, the processor <NUM> may identify at least one of the candidate synthesized sounds based on at least one of a prosody, a pronunciation, or a sound quality of each of the candidate synthesized sounds. Specific contents for this will be described with reference to <FIG> and <FIG>.

<FIG> are views for describing a method of identifying a synthesized sound according to an embodiment of the disclosure.

Referring to <FIG>, as an embodiment, when each of the plurality of synthesized sounds ( <MAT>) is input to the speaker encoder module <NUM>, the processor <NUM> may obtain reference vectors ( <MAT>) for the plurality of synthesized sounds ( <MAT>) each output from the speaker encoder module <NUM>. In addition, the processor <NUM> may determine a similarity by comparing each of the reference vectors ( <MAT>) for the plurality of synthesized sounds ( <MAT>) with the reference vectors (sr,A) for the user speech (xr,A). Here, the reference vector (sr,A) for the user speech (xr,A) is output and obtained from the speaker encoder module <NUM> when the user speech (xr,A) is input to the speaker encoder module <NUM>.

Here, the similarity may be calculated through various methods such as a Euclidean distance and a cosine similarity method. In addition, the similarity between reference vectors may be regarded as speaker similarity between the synthesized sounds. That is, the processor <NUM> may identify a synthesized sound having a reference vector having a similarity of a predetermined threshold value or more among the reference vectors ( <MAT>) for the plurality of synthesized sounds ( <MAT>) as a candidate synthesized sound.

Meanwhile, the plurality of synthesized sounds ( <MAT>) may be classified into groups of units of the candidate reference vectors ( <MAT>) used to generate the plurality of synthesized sounds ( <MAT>). For example, synthesized sounds ( <MAT>) generated through a first candidate reference vector ( <MAT>) and first to M-th evaluation texts (t<NUM>,. , tM) may be classified into the same group. That is, synthesized sounds generated through one candidate reference vector and M evaluation texts may be classified into the same group.

In this case, the processor <NUM> may determine a deviation (or a distribution degree) of the reference vectors ( <MAT>) for the plurality of synthesized sounds ( <MAT>) in group units. In addition, the processor <NUM> may identify reference vectors of a group having the smallest deviation. In this case, the processor <NUM> may identify synthesized sounds synthesized through the reference vectors of the group having the smallest deviation as candidate synthesized sounds.

For example, a case where first to third groups are positioned as illustrated in a plane <NUM> of <FIG> when the synthesized sounds ( <MAT>) generated through the first candidate reference vector ( <MAT>) and the first to M-th evaluation texts (t<NUM>,. , tM) are classified into the first group ( <MAT>), synthesized sounds ( <MAT>) generated through a second candidate reference vector ( <MAT>) and the first to M-th evaluation texts (t<NUM>,. , tM) are classified into the second group ( <MAT>), and synthesized sounds ( <MAT>) generated through a third candidate reference vector ( <MAT>) and the first to M-th evaluation texts (t<NUM>,. , tM) are classified into the third group ( <MAT>) may be assumed. In this case, the processor <NUM> may identify a reference vector ( <MAT>) of the third group ( <MAT>) having the smallest deviation based on the reference vector (sr,A) for the user speech (xr,A). In this case, the processor <NUM> may identify synthesized sounds ( <MAT>) synthesized through the reference vector ( <MAT>) of the third group ( <MAT>) as candidate synthesized sounds.

Meanwhile, as illustrated in <FIG>, the processor <NUM> according to an embodiment of the disclosure may identify candidate synthesized sounds using a plurality of speaker encoder modules <NUM>-<NUM> to <NUM>-<NUM>. For example, first to third speaker encoder modules <NUM>-<NUM> to <NUM>-<NUM> may be modeled to output different types of reference vectors (for example, i-vector, d-vector, x-vector, and the like).

When the plurality of synthesized sounds ( <MAT>) and the user speech (xr,A) are input to the first speaker encoder module <NUM>-<NUM>, the processor <NUM> may obtain first reference vectors ( <MAT>) for the plurality of synthesized sounds ( <MAT>) output from the first speaker encoder module <NUM>-<NUM> and a first reference vector (sr,A) for the user speech (xr,A), and compare the first reference vectors ( <MAT>) and the first reference vector (sr,A) with each other to determine a first similarity.

In addition, when the plurality of synthesized sounds ( <MAT>) and the user speech (xr,A) are input to the second speaker encoder module <NUM>-<NUM>, the processor <NUM> may obtain second reference vectors ( <MAT>) for the plurality of synthesized sounds ( <MAT>) output from the second speaker encoder module <NUM>-<NUM> and a second reference vector (ir,A) for the user speech (xr,A) , and compare the second reference vectors ( <MAT>) and the second reference vector (ir,A) with each other to determine a second similarity.

In addition, when the plurality of synthesized sounds ( <MAT>) and the user speech (xr,A) are input to the third speaker encoder module <NUM>-<NUM>, the processor <NUM> may obtain third reference vectors ( <MAT>) for the plurality of synthesized sounds ( <MAT>) output from the third speaker encoder module <NUM>-<NUM> and a third reference vector (tr,A) for the user speech (xr,A), and compare the third reference vectors ( <MAT>) and the third reference vector (tr,A) with each other to determine a third similarity.

In addition, the processor <NUM> may identify a reference vector having the most excellent performance among the first to third similarities, and may identify synthesized sounds synthesized through the identified reference vector as candidate synthesized sounds. The reference vector having the most excellent performance may be a vector having the smallest deviation value or a vector exceeding a predetermined threshold value for each of the first to third similarities.

In addition, the processor <NUM> may identify at least one of the candidate synthesized sounds based on at least one of a prosody, a pronunciation, or a sound quality of each of the candidate synthesized sounds. That is, the processor <NUM> may identify the synthesized sound through one of a prosody, a pronunciation, and a sound quality of each of the candidate synthesized sounds, or a combination thereof.

Specifically, the processor <NUM> may calculate a prosody score, a pronunciation score, and a sound quality score of each of the candidate synthesized sounds, and identify at least one synthesized sound of which each of the prosody score, the pronunciation score, and the sound quality score is a predetermined threshold value or more among the candidate synthesized sounds.

As an example, the processor <NUM> may calculate the prosody score of each of the candidate synthesized sounds, and identify at least one synthesized sound of which the prosody score is a predetermined threshold value or more. For example, the processor <NUM> may calculate the prosody score by comparing a length, a speaking speed, and a pitch contour indicating a pitch over time for each phoneme set in the evaluation text tm with a length, a speaking speed, and a pitch contour for each phoneme of the candidate synthesized sound <MAT> corresponding to the evaluation text tm.

As an example, the processor <NUM> may calculate the pronunciation score of each of the candidate synthesized sounds, and identify at least one synthesized sound of which the pronunciation score is a predetermined threshold value or more.

As illustrated in (<NUM>) of <FIG>, the processor <NUM> according to an embodiment of the disclosure may input the synthesized sound <MAT> to the ASR module <NUM> to obtain a text, and compare the obtained text with the evaluation text tm corresponding to the synthesized sound <MAT> to calculate the pronunciation score. The ASR module <NUM> may analyze the speech using various algorithms and convert contents of the speech into a text format, as described above.

As illustrated in (<NUM>) of <FIG>, the processor <NUM> according to an embodiment of the disclosure may input the synthesized sound <MAT> to a forced alignment module <NUM> to identify boundaries between speech-phoneme and speech-word, and compare the identified boundaries with boundaries between speech-phoneme and speech-word in the evaluation text tm corresponding to the synthesized sound <MAT> to calculate a likelihood. At this time, the likelihood may be utilized as the pronunciation score.

As an example, the processor <NUM> may calculate the sound quality score of each of the candidate synthesized sounds, and identify at least one synthesized sound of which the sound quality score is a predetermined threshold value or more.

For example, the processor <NUM> may calculate the sound quality score for the synthesized sound <MAT> through various algorithms such as a signal-to-noise ratio (SNR), a harmonic-to-noise ratio (HNR), and sound quality evaluation through room characteristic estimation.

Meanwhile, the processor <NUM> may divide the plurality of candidate synthesized sounds according to a plurality of domains based on a domain to which each of the plurality of candidate synthesized sounds belongs (that is, a domain of an evaluation text used to generate the corresponding synthesized sound). The processor <NUM> may identify at least one synthesized sound for each domain based on at least one of a similarity, a prosody, a pronunciation, or a sound quality of each of one or more candidate synthesized sounds belonging to each domain.

Specifically, an embodiment in which the synthesized sound evaluation module <NUM> selects an optimal reference vector based on a similarity between the plurality of synthesized sounds ( <MAT>) and the user speech and characteristics of the plurality of synthesized sounds ( <MAT>) will be described with reference to <FIG>.

In tables of (<NUM>) to (<NUM>) of <FIG>, each row indicates an evaluation text (t<NUM>, t<NUM>, t<NUM>,. ) to which each domain is assigned, and each column indicates a candidate reference vector ( <MAT>). A combination of each row and column indicates scores (a speaker similarity, a prosody score, a pronunciation score, a sound quality score, and the like) of synthesized sounds generated according to a combination of the evaluation text and the candidate reference vector.

As an embodiment, as illustrated in (<NUM>) of <FIG>, when all of speaker similarities (that is, values in the same column) of a plurality of synthesized sounds in which one candidate reference vector and a plurality of evaluation texts are combined with each other are a predetermined value (for example, <NUM>) or more, it may be determined that the corresponding candidate reference vector satisfies an evaluation criterion for the speaker similarity, and when at least one of the speaker similarities is less than the predetermined value (for example, <NUM>), it may be determined that the corresponding candidate reference vector does not satisfy the evaluation criterion for the speaker similarity.

As an embodiment, as illustrated in (<NUM>) of <FIG>, when at least one of prosody scores of a plurality of synthesized sounds in which one candidate reference vector and a plurality of evaluation texts are combined with each other is a predetermined value (for example, <NUM>) or more, it may be determined that the corresponding candidate reference vector satisfies an evaluation criterion for the prosody score, and when all of the prosody scores of a plurality of synthesized sounds are less than the predetermined value (for example, <NUM>), it may be determined that the corresponding candidate reference vector does not satisfy the evaluation criterion for the prosody score.

As an embodiment, as illustrated in (<NUM>) of <FIG>, when at least one of pronunciation scores of a plurality of synthesized sounds in which one candidate reference vector and a plurality of evaluation texts are combined with each other is a predetermined value (for example, <NUM>) or more, it may be determined that the corresponding candidate reference vector satisfies an evaluation criterion for the pronunciation score, and when all of the pronunciation scores of a plurality of synthesized sounds are less than the predetermined value (for example, <NUM>), it may be determined that the corresponding candidate reference vector does not satisfy the evaluation criterion for the pronunciation score.

As an embodiment, as illustrated in (<NUM>) of <FIG>, when all of sound quality scores (that is, values in the same column) of a plurality of synthesized sounds in which one candidate reference vector and a plurality of evaluation texts are combined with each other are a predetermined value (for example, <NUM>) or more, it may be determined that the corresponding candidate reference vector satisfies an evaluation criterion for the sound quality score, and when at least one of the sound quality scores is less than the predetermined value (for example, <NUM>), it may be determined that the corresponding candidate reference vector does not satisfy the evaluation criterion for the sound quality score.

In addition, the processor <NUM> may store a reference vector ( <MAT>) of the identified at least one synthesized sound in the memory <NUM> as a reference vector ( <MAT>) corresponding to the user A for the TTS module <NUM>. That is, a reference vector satisfying the evaluation criterion among the plurality of candidate reference vectors may be registered as a reference vector of the user A and be stored in the reference vector storage module <NUM> of the memory <NUM>.

As described above, the electronic apparatus <NUM> according to an embodiment of the disclosure may obtain reference vectors optimized for various texts with only reference vectors obtained through a very small number of texts even though the user speaks the very small number of texts (for example, one to five texts) by using the fact that reference vectors of the same user have a distribution within a predetermined range. That is, unlike the related art, the electronic apparatus <NUM> may ensure good performance through synthesized sound evaluation, and may obtain a plurality of reference vectors from a spoken text even though the user speaks the text only once. Because the number of texts spoken to provide the personalized TTS service is very small, convenience of the user registering the personalized TTS service may be improved.

Meanwhile, the electronic apparatus <NUM> according to an embodiment of the disclosure may provide feedback to the user A when it is not enough to provide the personalized TTS service with only the user speech spoken by the user A in a process of registering the reference vector as a reference vector of the user A.

Taking <FIG> as an example, the reference vectors of the candidate synthesized sounds satisfying the speaker similarity of the predetermined value (for example, <NUM>) or more for all evaluation texts may be identified as <MAT>, the reference vectors of the candidate synthesized sounds satisfying the prosody score of the predetermined value (for example, <NUM>) or more for at least one evaluation text may be identified as <MAT>, the reference vectors of the candidate synthesized sounds satisfying the pronunciation score of the predetermined value (for example, <NUM>) or more for at least one evaluation text may be identified as <MAT>, and the candidate reference vectors of the candidate synthesized sounds satisfying the sound quality score of the predetermined value (for example, <NUM>) or more for all evaluation texts may be identified as <MAT>.

In this case, the processor <NUM> may store the reference vectors ( <MAT>) of the identified candidate synthesized sounds satisfying all evaluation criteria through the synthesized sound evaluation module <NUM> in the reference vector storage module <NUM> of the memory <NUM> as the reference vectors corresponding to the user A.

In addition, the processor <NUM> may divide the plurality of candidate synthesized sounds according to the plurality of domains based on the domain to which each of the plurality of candidate synthesized sounds belongs. Here, the candidate synthesized sounds ( <MAT>) may be generated through a combination of the reference vectors ( <MAT>) and the plurality of evaluation texts (t<NUM>, t<NUM>, t<NUM>,. ), and the domains to which the candidate synthesized sounds ( <MAT>, <MAT>) belong may be domains assigned to the evaluation texts (t<NUM>, t<NUM>, t<NUM>,. ) used to generate the candidate synthesized sounds.

In addition, the processor <NUM> may identify at least one synthesized sound for each domain based on at least one of the speaker similarity, the prosody, the pronunciation, or the sound quality of each of one or more candidate synthesized sounds belonging to each domain. In addition, the reference vectors of the identified at least one synthesized sound may be stored in the memory <NUM> according to the domain to which each evaluation text belongs.

Specifically, the processor <NUM> may determine whether or not a synthesized sound satisfying an evaluation criterion (for example, at least one of a speaker similarity, a prosody, a pronunciation, or a sound quality) exists for a specific domain.

For example, as illustrated in <FIG>, the processor <NUM> may identify a candidate synthesized sound generated according to a combination of an evaluation text t<NUM> and a reference vector <MAT> as a candidate synthesized sound of which a prosody score and a pronunciation score satisfy predetermined values. In addition, the processor <NUM> may identify a candidate synthesized sound generated according to a combination of an evaluation text t<NUM> and a reference vector <MAT> as a candidate synthesized sound of which a prosody score and a pronunciation score satisfy predetermined values. In this case, the processor <NUM> may evaluate (select) the reference vector <MAT> satisfying a prosody score and a pronunciation score of the evaluation text t<NUM> as a reference vector that may cover a domain of the evaluation text t<NUM>. In addition, the processor <NUM> may evaluate (select) the reference vector <MAT> satisfying a prosody score and a pronunciation score of the evaluation text t<NUM> as a reference vector that may cover a domain of the evaluation text t<NUM>.

Referring to <FIG>, the processor <NUM> may control the output interface <NUM> (see <FIG>) to output information requesting speaking of a sentence (r' ) belonging to the specific domain, when at least one synthesized sound satisfying the evaluation criterion (for example, at least one of the speaker similarity, the prosody, the pronunciation, or the sound quality) does not exist for the specific domain through the synthesized sound evaluation module <NUM>.

For example, as illustrated in <FIG>, the processor <NUM> may feed back a sentence (r') belonging to a domain assigned to an evaluation text t<NUM> to the user when a synthesized sound (or a reference vector) satisfying a prosody score and a pronunciation score of the evaluation text t<NUM> does not exist. Here, the fed-back sentence (r' ) may include a sentence, a word or the like inducing the user to speak a speech to cover a domain of the evaluation text t<NUM>. For example, if the evaluation text t<NUM> is a news domain, the fed-back sentence (r') may be t<NUM> or a news domain text.

For example, the processor <NUM> may determine at least one candidate synthesized sound belonging to a domain in which at least one synthesized sound does not exist among the plurality of domains, and determine characteristics of a synthesized sound of which a relatively low score of a prosody, a pronunciation, and a sound quality is calculated based on a prosody score, a pronunciation score, and a sound quality score calculated for the determined candidate synthesized sound. The processor <NUM> may output a speech requesting speaking of a sentence generated based on the determined characteristic through the speaker <NUM>.

As described above, the electronic apparatus <NUM> according to the disclosure may perform evaluation according to various evaluation criteria in a process of registering the speech of the user as the speech of the TTS module <NUM>. Therefore, a reference vector having optimal performance may be determined as the reference vector of the user. In addition, reference vectors that may cover various types of texts may be obtained by providing the feedback to the user when it is not sufficient to provide the personalized TTS service with only the user speech spoken by the user.

Meanwhile, the electronic apparatus <NUM> may synthesize a speech signal using the registered user speech after the user speech is registered in the TTS module <NUM>. This will be described in detail with reference to <FIG>.

<FIG> is a block diagram for describing an operation for each component of the electronic apparatus according to an embodiment of the disclosure. <FIG> illustrates a process of synthesizing a speech signal using a user speech after the user speech is registered in a TTS module <NUM>.

Referring to <FIG>, a case where an input data <NUM> (for example, a text t) is provided to the processor <NUM> is assumed. The input data <NUM> may be a text t obtained as a result of performing speech recognition on a subsequent user speech. Alternatively, the input data <NUM> may be a text t input through an input device (for example, a keyboard or the like).

For example, when a subsequent user speech of the user is received through the microphone <NUM>, the processor <NUM> may obtain a text t for a response to the subsequent user speech. In this case, the text t may be a text obtained through the ASR module <NUM> and the NLP module <NUM>.

In addition, the processor <NUM> may select a reference vector belonging to a domain of the text t among one or more reference vectors SA corresponding to the user A stored in the reference vector storage module <NUM> of the memory <NUM>, through a reference vector selection module <NUM>.

Here, when a plurality of reference vectors belonging to the domain of the text t are selected, the processor <NUM> may obtain a reference vector of a synthesized sound of which a score (for example a, prosody score or a pronunciation score) calculated based on characteristics of a synthesized sound synthesized with an evaluation text belonging to the domain of the text t is highest among the plurality of reference vectors. Here, the score calculated based on the characteristics of the synthesized speech synthesized with the evaluation text belonging to the domain of the text t may be stored in the memory <NUM> in a process of registering the user speech of the user A.

For example, assuming that a reference vector of a synthesized sound having the highest score among synthesized sounds synthesized with an evaluation text belonging to a reading style is <MAT> and a reference vector of a synthesized sound having the highest score among synthesized sounds synthesized with an evaluation text belonging to a conversation style is <MAT>, when the domain of the text t, which is the input data <NUM>, is the reading style, the reference vector <MAT> of the stored one or more reference vector SA corresponding to the user A may be selected as a reference vector (ŝt,A) belonging to the domain of the text t. Meanwhile, the processor <NUM> may also select ŝt,A having optimum performance for the given text t utilizing an arbitrary statistical model (DNN, HMM, GMM, or the like).

In addition, the processor <NUM> may input the text t, which is the input data <NUM>, and the selected reference vector (ŝt,A) to the TTS module <NUM> to obtain a speech generated for the text t based on the reference vector (ŝt,A).

In this case, the processor <NUM> may control a speaker <NUM> (see <FIG>) to output the obtained speech.

<FIG> is a view for describing a method of obtaining a reference vector according to an embodiment of the disclosure.

The speaker encoder module <NUM> may obtain a reference vector through the user speech. Here, the speaker encoder module <NUM> may include various types of modules such as a reference encoder, a global style token (GST), a variable autoencoder (VAE), an I-vector, and a neural network module.

As an embodiment, referring to <FIG>, the speaker encoder module <NUM> may include an acoustic feature extractor <NUM> and recurrent neural network (RNN) modules <NUM>-<NUM> to <NUM>-T.

The acoustic feature extractor <NUM> may extract an acoustic feature in units of frames. In this case, a dimension of the acoustic feature may be represented as (T x D). For example, when one frame is <NUM> and an <NUM>-dimensional acoustic feature is extracted, if a speech waveform of <NUM> seconds is input, T is <NUM> and D is <NUM>, such that an acoustic feature of (<NUM> x <NUM>) may be output. In general, the acoustic feature is fixed at the time of designing the TTS module <NUM>, and accordingly, D may have a fixed value regardless of a speech input.

The RNN modules <NUM>-<NUM> to <NUM>-T may output a vector of a fixed dimension regardless of T. For example, assuming that the reference vector is <NUM> dimensions, the RNN modules <NUM>-<NUM> to <NUM>-T may always output a vector of <NUM> dimensions regardless of T and D. The reference vector may be output in a state in which prosody or tone information (global information) rather than phoneme information (local information) included in the corresponding speech is compressed. In this case, the last state of the RNN modules <NUM>-<NUM> to <NUM>-T may be used as the reference vector of the disclosure.

<FIG> is a view for describing a TTS model according to an embodiment of the disclosure.

Referring to <FIG>, the TTS module <NUM> according to an embodiment of the disclosure may perform pre-processing on a text and a speech waveform through a language processor <NUM> and an acoustic feature extractor <NUM> to extract a phoneme and acoustic features, and learn a neural network-based acoustic model (AM) <NUM> and a vocoder <NUM> using the pre-processed phoneme and acoustic features as learning data.

Thereafter, the TTS module <NUM> may extract a phoneme from the text through the language processor <NUM>, input the extracted phoneme to the learned AM <NUM> to obtain an expected acoustic feature as an output, and input the obtained acoustic feature to the learned vocoder <NUM> to obtain a synthesized sound as an output.

However, the embodiment described above is only an example, and the disclosure is not limited thereto and may be variously modified.

<FIG> is a view for describing hardware components of the electronic apparatus according to an embodiment of the disclosure.

The microphone <NUM> is a component for receiving an analog acoustic signal. The microphone <NUM> may receive an acoustic signal including a user speech. The acoustic signal may indicate a sound wave having information such as a frequency and an amplitude.

The memory <NUM> is a component for storing an operating system (OS) for controlling a general operation of components of the electronic apparatus <NUM> and various data related to the components of the electronic apparatus <NUM>. The memory <NUM> may store information in various manners such as an electrical manner or a magnetic manner. The data stored in the memory <NUM> may be accessed by the processor <NUM>, and reading, writing, correction, deletion, update, and the like, of the data in the memory <NUM> may be performed by the processor <NUM>.

To this end, the memory <NUM> may be configured by hardware for temporarily or permanently storing data or information. For example, the memory <NUM> may be implemented as at least one hardware of a non-volatile memory, a volatile memory, a flash memory, a hard disk drive (HDD), a solid state drive (SDD), a random access memory (RAM), or a read only memory (ROM).

The processor <NUM> may be implemented as a general-purpose processor such as a central processing unit (CPU) or an application processor (AP), a graphics-dedicated processor such as a graphic processing unit (GPU) or a vision processing unit (VPU), or an artificial intelligence-dedicated processor such as a neural processing unit (NPU). In addition, the processor <NUM> may include a volatile memory for loading at least one instruction or module.

<FIG> is a view for describing additional hardware components of the electronic apparatus according to an embodiment of the disclosure.

Referring to <FIG>, the electronic apparatus <NUM> according to an embodiment of the disclosure may include at least one of an output interface <NUM>, an input interface <NUM>, a communication interface <NUM>, a sensor <NUM>, or a power supply <NUM>, in addition to the microphone <NUM>, the memory <NUM> and the processor <NUM>.

The output interface <NUM> is a component capable of outputting information. In addition, the output interface <NUM> may include at least one of a speaker <NUM> or a display <NUM>. The speaker <NUM> may directly output various alarms or audio messages as well as various audio data on which various processing works such as decoding, amplification, and noise filtering are performed by an audio processor (not illustrated). The display <NUM> may output information or data in a visual form. The display <NUM> may display an image frame on one area or all areas of a display that may be driven as pixels. To this end, the display <NUM> may be implemented as a liquid crystal display (LCD), an organic light emitting diodes (OLED) display, a micro LED display, a quantum dot LED (QLED) display, or the like. In addition, at least a part of the display <NUM> may be implemented in the form of a flexible display, and the flexible display may be bent, curved, or rolled without damage through a thin and flexible substrate like paper.

The input interface <NUM> may receive various user commands and transfer the received user commands to the processor <NUM>. That is, the processor <NUM> may recognize a user command input from the user through the input interface <NUM>. Here, the user command may be implemented in various manners such as a touch input (touch panel) of the user, or a key (keyboard) or button (physical button or mouse) input.

The communication interface <NUM> may transmit and receive various types of data by performing communication with various types of external apparatuses according to various types of communication manners. The communication interface <NUM> may include at least one of a Bluetooth module (Bluetooth manner), a Wi-Fi module (Wi-Fi manner), a wireless communication module (cellular manner such as <NUM>, <NUM>, or <NUM>), a near field communication (NFC) module (NFC manner), an infrared (IR) module (infrared manner), a Zigbee module (Zigbee manner), an ultrasonic module (ultrasonic manner), and the like, as a circuit performing various types of wireless communication, or an Ethernet module, a universal serial bus (USB) module, a high definition multimedia interface (HDMI), a displayport (DP), a D-subminiature (D-SUB), a digital visual interface (DVI), a thunderbolt, and components that perform wired communication.

The sensor <NUM> may be implemented as various sensors such as a camera, a proximity sensor, an illuminance sensor, a motion sensor, a time of flight (ToF) sensor, and a global positioning system (GPS) sensor. For example, the camera may divide light in pixel units, sense intensities of light for red (R), green (G), and blue (B) colors for each pixel, and convert the intensities of the light into electrical signals to obtain data representing a color, a shape, and a contrast of an object. In this case, a type of the data may be an image having R, G, and B color values for each of a plurality of pixels. The proximity sensor may sense the presence of a surrounding object and obtain data on whether or not the surrounding object exists or whether or not the surrounding object is approaching the electronic apparatus. The illuminance sensor may sense an amount of light (or a brightness) of the surrounding environment of the electronic apparatus <NUM> to obtain data on illuminance. The motion sensor may sense a moving distance, a moving direction, a gradient, and the like, of the electronic apparatus <NUM>. To this end, the motion sensor may be implemented by a combination of an acceleration sensor, a gyro sensor, a geomagnetic sensor, and the like. The ToF sensor may sense a time of flight from after various electromagnetic waves (for example, an ultrasonic wave, an infrared, a laser beam, and an ultra-wideband (UWB) wave) having a specific speed are emitted until the various electromagnetic waves return to an original position to obtain data on a distance to a target (or a position of the target). The GPS sensor may receive radio signals from a plurality of satellites, calculate distances to each satellite using transfer times of the received signals, and obtain data on a current position of the electronic apparatus <NUM> using triangulation on the calculated distances. However, the implementation of the sensor <NUM> described above is only an example, and the sensor <NUM> is not limited thereto, and may be implemented as various types of sensors.

The power supply <NUM> may supply power to the electronic apparatus <NUM>. For example, the power supply <NUM> may supply power to each component of the electronic apparatus <NUM> through an external commercial power supply or a battery.

<FIG> is a flowchart illustrating a controlling method of an electronic apparatus according to an embodiment of the disclosure.

Referring to <FIG>, the controlling method of an electronic apparatus <NUM> may include obtaining the reference vector of the user speech spoken by the user (S1010) when the user speech is received through the microphone <NUM>, generating the plurality of candidate reference vectors based on the reference vector (S1020), obtaining the plurality of synthesized sounds by inputting the plurality of candidate reference vectors and the plurality of evaluation texts to the TTS model (S1030), identifying at least one synthesized sound of the plurality of synthesized sounds based on the similarity between the plurality of synthesized sounds and the user speech and the characteristics of the plurality of synthesized sounds (S1040), and storing a reference vector of the at least one synthesized sound in the memory <NUM> as a reference vector corresponding to the user for the TTS model (S1050).

Specifically, in the controlling method of an electronic apparatus <NUM> according to the disclosure, when the user speech spoken by the user is received through the microphone <NUM>, the reference vector of the user speech may be obtained (S1010).

In addition, the plurality of candidate reference vectors may be generated based on the reference vector (S1020).

Here, the plurality of candidate reference vectors may include at least one reference vector randomly selected based on the reference vector, at least one reference vector generated based on the reference vector and the reference vector used for training the TTS model, and at least one reference vector generated by applying the masking vector to the reference vector.

In addition, the plurality of synthesized sounds may be obtained by inputting the plurality of candidate reference vectors and the plurality of evaluation texts to the TTS model (S1030).

As a specific embodiment, the plurality of synthesized sounds generated for each of the plurality of evaluation texts based on each of the plurality of candidate reference vectors may be obtained by inputting the plurality of candidate reference vectors and the plurality of evaluation texts to the TTS model.

In addition, at least one synthesized sound of the plurality of synthesized sounds may be identified based on the similarity between the plurality of synthesized sounds and the user speech and the characteristics of the plurality of synthesized sounds (S1040).

As a specific embodiment, a candidate synthesized sound of which a similarity with the user speech is a predetermined threshold value or more among the plurality of synthesized sounds may be identified. In addition, at least one synthesized sound of the candidate synthesized sounds may be identified based on at least one of the prosody, the pronunciation, or the sound quality of each of the candidate synthesized sounds.

Specifically, the prosody scores, the pronunciation scores, and the sound quality scores of each of the candidate synthesized sounds may be calculated. In addition, at least one synthesized sound of which each of the prosody score, the pronunciation score, and the sound quality score is a predetermined threshold value or more among the candidate synthesized sounds may be identified.

Meanwhile, the plurality of evaluation texts may include at least one evaluation text belonging to each of the plurality of domains.

In this case, in the identifying of the at least one synthesized sound, the plurality of candidate synthesized sounds may be divided according to the plurality of domains based on the domain to which each of the plurality of candidate synthesized sounds belongs. In addition, at least one synthesized sound for each domain may be identified based on at least one of the prosody, the pronunciation, or the sound quality of each of one or more candidate synthesized sounds belonging to each domain.

In addition, the reference vector of the at least one synthesized sound may be stored in the memory <NUM> as the reference vector corresponding to the user for the TTS model (S1050).

Meanwhile, the electronic apparatus <NUM> according to an embodiment according to the disclosure may further include the output interface <NUM> including at least one of the speaker <NUM> or the display <NUM>.

In this case, in the controlling method of the electronic apparatus <NUM>, the domain in which at least one synthesized sound does not exist among the plurality of domains may be determined. In addition, when the domain in which the synthesized sound does not exist is determined, the output interface <NUM> may be controlled to output information requesting speaking of a sentence belonging to the determined domain.

Specifically, at least one candidate synthesized sound belonging to the domain in which at least one synthesized sound does not exist among the plurality of domains may be determined. In addition, when the domain in which the synthesized sound does not exist is determined, the characteristics of the synthesized sound of which a relatively low score of the prosody, the pronunciation, and the sound quality is calculated may be determined based on the prosody score, the pronunciation score, and the sound quality score calculated for the determined candidate synthesized sound. In addition, the output interface <NUM> may be controlled to output information requesting speaking of a sentence generated based on the determined characteristics.

Meanwhile, the electronic apparatus <NUM> according to an embodiment of the disclosure may include the speaker <NUM>.

In this case, in the controlling method of the electronic apparatus <NUM>, when a subsequent user speech of the user is received through the microphone <NUM>, a text for a response to the subsequent user speech may be obtained.

In addition, the speech generated for the text based on the reference vector may be obtained by inputting the obtained text and one of one or more reference vectors corresponding to the user stored in the memory <NUM> to the TTS model.

To this end, the reference vector of the synthesized sound of which the score calculated based on the characteristics of the synthesized sound is highest among one or more reference vectors corresponding to the user stored in the memory <NUM> may be obtained.

In addition, the speaker <NUM> may be controlled to output the obtained speech.

According to the diverse embodiments of the disclosure as described above, the electronic apparatus for providing a TTS service using a speech of a general user and a controlling method thereof may be provided. In addition, according to an embodiment of the disclosure, the number of sentences requested to be spoken to register the voice of the TTS service may be minimized. In addition, according to an embodiment of the disclosure, the personalized TTS service may be provided through a voice of the user without retraining the TTS model for each user.

The diverse embodiments of the disclosure may be implemented by software including instructions stored in a machine-readable storage medium (for example, a computer-readable storage medium). A machine may be a device that invokes the stored instruction from the storage medium and may be operated depending on the invoked instruction, and may include the electronic apparatus (for example, the electronic apparatus <NUM>) according to the disclosed embodiments. In a case where a command is executed by the processor, the processor may directly perform a function corresponding to the command or other components may perform the function corresponding to the command under the control of the processor. The command may include codes created or executed by a compiler or an interpreter. The machine-readable storage medium may be provided in a form of a non-transitory storage medium. Here, the term 'non-transitory' means that the storage medium is tangible without including a signal, and does not distinguish whether data are semi-permanently or temporarily stored in the storage medium.

The methods according to the various embodiments may be included and provided in a computer program product. The computer program product may be traded as a product between a seller and a purchaser. The computer program product may be distributed in a form of a storage medium (for example, a compact disc read only memory (CD-ROM)) that may be read by the machine or online through an application store (for example, PlayStore™). In a case of the online distribution, at least portions of the computer program product may be at least temporarily stored in a storage medium such as a memory of a server of a manufacturer, a server of an application store, or a relay server or be temporarily created.

Claim 1:
An electronic apparatus (<NUM>) comprising:
a microphone (<NUM>);
a memory (<NUM>) in which a text to speech, TTS, model and a plurality of evaluation texts are stored; and
a processor (<NUM>) configured to:
obtain (S1010) a reference vector of a user speech spoken by a user when the user speech is received through the microphone, the apparatus characterised in that the processor (<NUM>) is further configured to
generate (S1020) a plurality of candidate reference vectors based on the reference vector,
obtain (S1030) a plurality of synthesized sounds by inputting the plurality of candidate reference vectors and the plurality of evaluation texts to the TTS model,
identify (S1040) at least one synthesized sound of the plurality of synthesized sounds based on a similarity between the plurality of synthesized sounds and the user speech and characteristics of the plurality of synthesized sounds, and
store (S1050) a reference vector of the at least one synthesized sound in the memory as a reference vector corresponding to the user for the TTS model.