Patent Description:
It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Embodiments of the present disclosure propose method and apparatus for generating speech through multi-speaker neural TTS synthesis. A text input may be received. Speaker latent space information of a target speaker may be provided through at least one speaker model. At least one acoustic feature may be predicted through an acoustic feature predictor based on the text input and the speaker latent space information. A speech waveform corresponding to the text input may be generated through a neural vocoder based on the at least one acoustic feature and the speaker latent space information.

It should be noted that the above one or more aspects comprise the features hereinafter fully described and particularly pointed out in the claims. The following description and the drawings set forth in detail certain illustrative features of the one or more aspects. These features are only indicative of the various ways in which the principles of various aspects may be employed, and this disclosure is intended to include all such aspects.

The disclosed aspects will hereinafter be described in connection with the appended drawings that are provided to illustrate and not to limit the disclosed aspects.

The present disclosure will now be discussed with reference to several example implementations. It is to be understood that these implementations are discussed only for enabling those skilled in the art to better understand and thus implement the embodiments of the present disclosure, rather than suggesting any limitations on the scope of the present disclosure.

Although a neural TTS system may generate natural speech with high fidelity, it needs a large amount of text-speech training data pairs due to its end-to-end model nature. Usually, a training corpus of around <NUM>+ hours of speech may be still not enough for training a good end-to-end neural TTS system. Herein, "corpus" may refer to a set of speeches with each speech being attached with a corresponding text, and thus a corpus may provide a plurality of text-speech data pairs. A challenge for the neural TTS system is its generalization ability. Degradation of naturalness on synthesizing an out-of-domain text often happens, especially for a long text with a rather complex context. Herein, "out-of-domain" text refers to a text input which is not involved in a training corpus, or for which no relevant text input is involved in the training corpus. The limits from generation model architecture in the neural TTS system may result in various out-of-domain errors, e.g., wrong pronunciation, strange prosody, repeating or skipping word/phoneme, etc. Although adding more training data is a brute force solution, such heavy data requirements cannot be satisfied by using a single speaker corpus which always provides limited text-speech data pairs. Moreover, it would also raise a very high cost bar to build a neural TTS system for speech customization which requires enough speech data for a target speaker.

In order to relief the dependency on data size by the neural TTS system, training data may be augmented by combining corpuses of multiple speakers into a multi-speaker corpus set. The multi-speaker corpus set may be used for training a multi-speaker neural TTS system. The multi-speaker neural TTS system may generate better speech than a single-speaker TTS system, and can be used for creating customized speech using a limited-size corpus. However, due to the limits of traditional model architecture of such multi-speaker neural TTS system, rich content and speaker information in the multi-speaker corpus set are not well modeled, and the generated speech still has unnatural and muffled problems. Moreover, speaker similarity is also low for a target speaker having only a small corpus. The overall performance of such system is still far from actual application requirements.

Embodiments of the present disclosure propose new approaches to building a multi-speaker neural TTS system with a well-designed multi-speaker corpus set. In some aspects, a high-quality multi-speaker corpus set may be prepared in consideration of content coverage, speaker variety, style variety, etc. For example, the corpus set may have wide content coverage in various knowledge domains, thus the multi-speaker neural TTS system may leverage content from different speakers in various domains and perform better in terms of generalization. For example, speakers in the corpus set may have a balanced distribution in terms of age, gender, accents, etc., which makes it much easier to create speech for a target speaker having only a small corpus. This may facilitate to create high-fidelity customized speech through the multi-speaker neural TTS system. The corpus set as discussed above will be helpful for the multi-speaker neural TTS system to generate close-to-human speech for out-of-domain text inputs, especially for long sentences with complex context, thus enriching premium voices.

In some aspects, the embodiments of the present disclosure propose new model architecture for the multi-speaker neural TTS system in order to make better use of the multi-speaker corpus set and improve speech generalization ability. The multi-speaker neural TTS system may be built with full utilization of latent space information of speakers in the corpus set. The multi-speaker neural TTS system may be further updated, e.g., retrained by a subset of training data pairs in the corpus set, adapted to a target speaker with a corpus of the target speaker, etc. For example, after the multi-speaker neural TTS system was trained by the corpus set, the multi-speaker neural TTS system may be further retrained or refined by a corpus of at least one speaker in the corpus set. For example, when it is to generate speech for a target speaker, e.g., stimulating voices of the target speaker to speak, the multi-speaker neural TTS system may be adapted to the target speaker through being updated or retrained by a corpus of the target speaker. Accordingly, the multi-speaker neural TTS system may generate high quality speech with high speaker similarity.

<FIG> illustrates an exemplary traditional neural TTS system <NUM>. The neural TTS system <NUM> may be configured for receiving a text input <NUM> and generating a speech waveform <NUM> corresponding to the text input <NUM>. The text input <NUM> may be a word, phrase, sentence, etc. It should be appreciated that although it is shown in <FIG> that the text input <NUM> is provided to the neural TTS system <NUM>, the text input <NUM> may also be firstly split into a sequence of elements, e.g., a phoneme sequence, a grapheme sequence, a character sequence, etc. through various existing techniques, e.g., Letter-to-Sound (LTS), etc., and then the sequence may be provided to the neural TTS system <NUM> as input. Thereinafter, a "text input" may also be broadly interpreted as a sequence of elements obtained from the text input, such as a phoneme sequence, a grapheme sequence, a character sequence, etc..

The neural TTS system <NUM> may comprise an acoustic feature predictor <NUM>. The acoustic feature predictor <NUM> may predict acoustic features <NUM> from the text input <NUM>. The acoustic features <NUM> may comprise various traditional TTS acoustic features, e.g., mel-spectrum, linear spectrum pairs (LSP), etc. The acoustic feature predictor <NUM> may be based on various model architectures, e.g., a sequence-to-sequence model architecture, etc. <FIG> shows an exemplary sequence-to-sequence acoustic feature predictor <NUM>, which may comprise an encoder <NUM>, an attention unit <NUM> and a decoder <NUM>.

The encoder <NUM> may convert information contained in the text input <NUM> into a space that is more robust and more suitable to learn alignment with acoustic features, e.g., converting the information in the text input <NUM> into text features in the space. The encoder <NUM> may be based on various network structures, e.g., a network structure comprising a combination of a plurality of convolutional neural network (CNN) layers and a plurality of recurrent neural network (RNN) layers, a network structure comprising a combination of <NUM>-D convolutional filters, highway networks and bi-directional RNN, and so on.

The attention unit <NUM> may implement an attention mechanism which acts as a bridge connecting the encoder <NUM> and the decoder <NUM>. For example, the attention mechanism may facilitate to make alignment between text features output by the encoder <NUM> and acoustic features. Various types of attention mechanism may be implemented by the attention unit <NUM>, e.g., soft attention, hard attention, location sensitive attention, Gaussian Mixture Model (GMM) attention, etc..

The decoder <NUM> may map the text features output by the encoder <NUM> to the acoustic features <NUM> under impacts by the attention mechanism in the attention unit <NUM>. The decoder <NUM> may be based on various network structures, e.g., a network structure comprising a combination of feed-forward layers, Long Short Term Memory (LSTM) layers and CNN layers, and so on.

The neural TTS system <NUM> may comprise a neural vocoder <NUM>. The neural vocoder <NUM> may generate the speech waveform <NUM> based on the predicted acoustic features <NUM> output by the acoustic feature predictor <NUM>. The neural vocoder <NUM> may be based on various network structures, e.g., a network structure which is based on neural generative models, and so on.

<FIG> illustrates an exemplary architecture of a multi-speaker neural TTS system <NUM> according to an embodiment. It should be appreciated that the architecture shown in <FIG> is only a diagram for the sake of explaining some concepts or aspects of the present disclosure, and the multi-speaker neural TTS system <NUM> may actually have various implementation approaches. The multi-speaker neural TTS system <NUM> may generate speech for a variety of speakers involved in a multi-speaker corpus set used for training the multi-speaker neural TTS system <NUM>, and may also generate speech for a new speaker. During generating speech, the multi-speaker neural TTS system <NUM> may consider discriminative information in a speaker latent space, e.g., speaker latent space information. During generating speech for a target speaker, the speaker latent space information may be used for controlling to generate speech with characteristics of the target speaker.

As shown in <FIG>, the multi-speaker neural TTS system <NUM> may be configured for receiving a text input <NUM>, and generating a speech waveform <NUM> corresponding to the text input <NUM> in a target speaker's voices. The multi-speaker neural TTS system <NUM> may comprise an acoustic feature predictor <NUM>, a neural vocoder <NUM> and a speaker model <NUM>.

The speaker model <NUM> may provide speaker latent space information <NUM> of a target speaker. The speaker latent space information <NUM> may be representations of speaker characteristics in the speaker latent space, e.g., a speaker embedding vector of the target speaker. The speaker latent space information <NUM> may be used as additional information, e.g., a condition, for the acoustic feature predictor <NUM> and the neural vocoder <NUM>. Accordingly, the speaker latent space information <NUM> may be considered during the processing by the acoustic feature predictor <NUM> and the neural vocoder <NUM>. It should be appreciated that although it is recited in some parts of the present disclosure that a speaker embedding vector is provided by a speaker model, the speaker embedding vector is merely an exemplary instance of speaker latent space information provided by the speaker model, and those operations or processes discussed in connection with the speaker embedding vector may also be applied for any other instances of the speaker latent space information in a similar way.

Basic functions and structures of the acoustic feature predictor <NUM> may be similar with those of the acoustic feature predictor <NUM> in <FIG>, except that it further takes the speaker latent space information <NUM> into consideration. In other words, the acoustic feature predictor <NUM> may predict acoustic features <NUM> based on the text input <NUM> and the speaker latent space information <NUM>. The acoustic feature predictor <NUM> may comprise an encoder <NUM>, an attention unit <NUM> and a decoder <NUM>. The speaker latent space information <NUM> may be combined with an output from the encoder <NUM>, and then passed to the attention unit <NUM>. The attention mechanism in the attention unit <NUM> may utilize the combination of the latent space information <NUM> and the output from the encoder <NUM> for impacting the processing by the decoder <NUM>. Accordingly, acoustic features output by the decoder <NUM> may be associated with the target speaker.

Basic functions and structures of the neural vocoder <NUM> is similar with those of the neural vocoder <NUM> in <FIG>, except that it further takes the speaker latent space information <NUM> into consideration. Although the predicted acoustic features <NUM> may have reflected speaker characteristics, it would be still helpful if the processing by the neural vocoder <NUM> may further consider speaker characteristics, since the neural vocoder <NUM> is trained by the multi-speaker corpus set and is actually a speaker adaptive neural vocoder. The neural vocoder <NUM> may generate the speech waveform <NUM> based on the acoustic features <NUM> and the speaker latent space information <NUM>. In an implementation, the neural vocoder <NUM> may generate the speech waveform <NUM> sample by sample, wherein a collection of the samples forms the speech waveform <NUM>.

Through utilizing the speaker latent space information <NUM> as a condition for the acoustic feature predictor <NUM> and the neural vocoder <NUM>, the multi-speaker neural TTS system <NUM> may generate highly natural output speech which sounds very similar to the target speaker.

<FIG> illustrates exemplary implementations of a speaker model <NUM> according to an embodiment. The speaker model <NUM> may correspond to the speaker model <NUM> in <FIG>. The speaker model <NUM> may be implemented in various approaches.

In an implementation, the speaker model <NUM> may be implemented through a speaker embedding selector <NUM>. The speaker embedding selector <NUM> may obtain identity information <NUM> of a target speaker which may be any types of information capable of identifying the target speaker from other speakers, e.g., a random or assigned number for the target speaker, the name of the target speaker, description information of the target speaker, etc., and the identity information is briefly denoted as "target speaker ID" hereinafter. The speaker embedding selector <NUM> may try to retrieve a speaker embedding vector corresponding to the target speaker ID <NUM> from a speaker embedding vector database <NUM>. The speaker embedding vector database <NUM> may comprise a plurality of speaker embedding vectors corresponding to a plurality of speakers respectively. The speaker embedding vector database <NUM> may be established through collecting speaker embedding vectors of those speakers in a multi-speaker corpus set during training the multi-speaker neural TTS system, or collecting speaker embedding vectors of previous target speakers during applying the multi-speaker neural TTS system.

In an implementation, the speaker model <NUM> may be implemented through a speaker encoder <NUM>. The speaker encoder <NUM> may generate a speaker embedding vector corresponding to the target speaker based on a corpus <NUM> of the target speaker. For example, the corpus <NUM> of the target speaker may be obtained, which includes a plurality of speech waveforms of the target speaker. Acoustic features may be extracted from the speech waveforms in the corpus <NUM> through various traditional techniques, and provided to the speaker encoder <NUM>. The speaker encoder <NUM> may generate the speaker embedding vector corresponding to the target speaker based on the acoustic features of the target speaker. The speaker encoder <NUM> may be implemented by various techniques. For example, the speaker encoder may be a neural network for generating an embedding vector based on acoustic features.

<FIG> illustrates an exemplary implementation of a speaker encoder <NUM> according to an embodiment. The speaker encoder <NUM> may correspond to the speaker encoder <NUM> in <FIG>. As shown in <FIG>, the speaker encoder <NUM> may be based on a neural network which is used for generating a speaker embedding vector <NUM> based on acoustic features <NUM>. The speaker encoder <NUM> may sequentially comprise a plurality of convolutional layers <NUM>, average pooling <NUM>, a plurality of fully connected (FC) layers <NUM> and an affine projection <NUM>. The speaker embedding vector <NUM> may be formed by L2-normalization of projection output. The speaker encoder <NUM> may be trained by a corpus set of multiple speakers, and is designed for text-independent speaker recognition. Thus, the speaker encoder <NUM> may, independently from contents, provide better estimation of speaker embedding vectors.

<FIG> illustrates an exemplary implementation <NUM> of a multi-speaker neural TTS system according to an embodiment. The implementation <NUM> shows an exemplary structure of the multi-speaker neural TTS system <NUM> in <FIG>, wherein the speaker model <NUM> in <FIG> is implemented as a single speaker model <NUM>. The speaker model <NUM> may provide a speaker embedding vector <NUM> of a target speaker. The speaker embedding vector <NUM> may be provided to an acoustic feature predictor <NUM> and a neural vocoder <NUM> respectively. The acoustic feature predictor <NUM> may receive a text input <NUM> and predict acoustic features <NUM> based on the text input <NUM> and the speaker embedding vector <NUM>. The neural vocoder <NUM> may generate a speech waveform <NUM> corresponding to the text input <NUM> based on the acoustic features <NUM> and the speaker embedding vector <NUM>.

<FIG> illustrates an exemplary implementation <NUM> of a multi-speaker neural TTS system according to an embodiment. The implementation <NUM> shows an exemplary structure of the multi-speaker neural TTS system <NUM> in <FIG>, wherein the speaker model <NUM> in <FIG> is implemented as two different speaker models <NUM> and <NUM>. The speaker model <NUM> and the speaker model <NUM> are established for providing respective speaker embedding vectors of the target speaker to an acoustic feature predictor <NUM> and a neural vocoder <NUM> respectively.

The speaker model <NUM> may provide a speaker embedding vector <NUM> of the target speaker to the acoustic feature predictor <NUM>. The acoustic feature predictor <NUM> may receive a text input <NUM> and predict acoustic features <NUM> based on the text input <NUM> and the speaker embedding vector <NUM>.

The speaker model <NUM> may provide a speaker embedding vector <NUM> of the target speaker to the neural vocoder <NUM>. The neural vocoder <NUM> may generate a speech waveform <NUM> corresponding to the text input <NUM> based on the acoustic features <NUM> and the speaker embedding vector <NUM>.

<FIG> illustrates an exemplary implementation of an acoustic feature predictor <NUM> according to an embodiment. The acoustic feature predictor <NUM> may correspond to the acoustic feature predictor <NUM> in <FIG>, the acoustic feature predictor <NUM> in <FIG>, or the acoustic feature predictor <NUM> in <FIG>. The acoustic feature predictor <NUM> may comprise an encoder <NUM>, an attention unit <NUM> and a decoder <NUM>.

A text input <NUM> may be provided to the encoder <NUM> which may correspond to the encoder <NUM> in <FIG>. A text embedding unit <NUM> in the encoder <NUM> may convert the text input <NUM> into a text embedding vector, and the text embedding vector may be further processed through a plurality of convolutional layers <NUM> and a bi-directional LSTM (BLSTM) <NUM> in the encoder <NUM>. The encoder <NUM> may output text features corresponding to the text input <NUM>, which are further combined with a speaker embedding vector <NUM>. For example, a concatenating unit <NUM> may be used for providing a combination of the speaker embedding vector <NUM> and the text features, wherein the speaker embedding vector <NUM> may correspond to the speaker latent space information <NUM> in <FIG>, the speaker embedding vector <NUM> in <FIG>, or the speaker embedding vector <NUM> in <FIG>.

The combination of the speaker embedding vector <NUM> and the text features may be provided to the attention unit <NUM> which may correspond to the attention unit <NUM> in <FIG>. An attention mechanism implemented in the attention unit <NUM> may utilize the combination of the speaker embedding vector <NUM> and the text features to impact the processing by the decoder <NUM>, wherein the decoder <NUM> may correspond to the decoder <NUM> in <FIG>.

The decoder <NUM> may comprise a pre-net <NUM> consisted of feed-forward layers, a uni-directional LSTM (ULSTM) <NUM>, a linear projection <NUM> and a post-net <NUM> consisted of convolutional layers. The ULSTM <NUM> may receive an input from the pre-net <NUM> and provide its output to the linear projection <NUM>, and meanwhile the processing by the ULSTM <NUM> is impacted by the attention unit <NUM>. The linear projection <NUM> may provide its output to the pre-net <NUM> and the post-net <NUM> respectively. Finally, an output from the post-net <NUM> and the output from the linear projection <NUM> may be combined so as to produce acoustic features <NUM>. The acoustic features <NUM> may correspond to the acoustic features <NUM> in <FIG>, the acoustic features <NUM> in <FIG>, or the acoustic features <NUM> in <FIG>. In an implementation, the linear projection <NUM> may also be used for generating stop tokens.

It should be appreciated that the structure of the acoustic feature predictor <NUM> in <FIG> is exemplary, and depending on specific application designs and requirements, the acoustic feature predictor <NUM> may be implemented in any other approaches.

<FIG> illustrates an exemplary implementation of a neural vocoder <NUM> according to an embodiment. The neural vocoder <NUM> may correspond to the neural vocoder <NUM> in <FIG>, the neural vocoder <NUM> in <FIG>, and the neural vocoder <NUM> in <FIG>.

As discussed above, speaker characteristics may be further considered by the neural vocoder, such that the neural vocoder may get more information of the target speaker in a speaker latent space. Since a speaker embedding vector which reflects speaker characteristics may have different dimensions and different value ranges from acoustic features, the speaker embedding vector and the acoustic features may be firstly transformed into the same dimension with a similar dynamic range of value through, e.g., neural networks.

As shown in <FIG>, a speaker embedding vector <NUM> may be input to a neural network <NUM>. The speaker embedding vector <NUM> may correspond to the speaker latent space information <NUM> in <FIG>, the speaker embedding vector <NUM> in <FIG>, or the speaker embedding vector <NUM> in <FIG>. The neural network <NUM> may be based on various structures, e.g., a <NUM>×<NUM> convolutional layer <NUM>. Through the neural network <NUM>, a transformed speaker embedding vector may be obtained.

Moreover, acoustic features <NUM>, which may correspond to the acoustic features <NUM> in <FIG>, may be input to a neural network <NUM>. The neural network <NUM> may be based on various structures, e.g., a Quasi-Recurrent Neural Network (QRNN) <NUM> followed by a <NUM>×<NUM> convolutional layer <NUM>. Through the neural network <NUM>, transformed acoustic features may be obtained, which may have the same dimension and similar dynamic range of value as compared with the transformed speaker embedding vector.

The transformed acoustic features and the transformed speaker embedding vector may be combined together, and further provided to the neural vocoder <NUM>. The neural vocoder <NUM> may be based on a neural generative model, and may generate a speech waveform <NUM> based on the combination of the transformed acoustic features and the transformed speaker embedding vector.

In an implementation, the neural vocoder <NUM> may comprise a plurality of dilated convolutional layers <NUM> which are grouped into a certain number of cycles. The plurality of dilated convolutional layers <NUM> may take the combination of the transformed acoustic features and the transformed speaker embedding vector as a condition. Skip connection <NUM> may be performed on outputs by the plurality of dilated convolutional layers <NUM>. The neural vocoder <NUM> may further sequentially comprise Rectified Linear Unit (ReLU) <NUM>, <NUM>×<NUM> convolutional layer <NUM>, ReLU <NUM>, <NUM>×<NUM> convolutional layer <NUM>, a plurality of feed-forward layers <NUM> and a MoL unit <NUM>. Through the above structure, the neural vocoder <NUM> may generate speech samples in a sequential way and finally output the speech waveform <NUM>.

It should be appreciated that the structure of the neural vocoder <NUM> in <FIG> is exemplary, and depending on specific application designs and requirements, the neural vocoder <NUM> may be implemented in any other approaches.

<FIG> illustrates an exemplary process for training a multi-speaker neural TTS system according to an embodiment.

In order to train a multi-speaker neural TTS system <NUM>, a multi-speaker corpus set <NUM> for training may be prepared. The corpus set <NUM> may comprise a plurality of corpuses of a plurality of speakers, e.g., corpus <NUM> of speaker <NUM>, corpus <NUM> of speaker <NUM>, etc. The corpus set is prepared in consideration of content coverage, speaker variety, style variety, etc..

In an aspect, the corpus set <NUM> has wide content coverage in various knowledge domains. Herein, "content" may refer to information expressed by speech waveforms in a corpus, and content coverage may be assessed with different linguistic contexts, e.g., phoneme, triphone, syllable, etc. In order to have a wide content coverage, speech waveforms in the corpus set <NUM> may come from various speech sources, e.g., news reporting, lectures, movie dubbings, daily conversations, etc. Accordingly, the collection of corpuses of different speakers in the corpus set <NUM> may provide rich content or linguistic coverage.

In an aspect, speakers in the corpus set <NUM> may have a balanced distribution in terms of age, gender, accents, etc. For example, the speakers may cover different age ranges, e.g., middle-aged people, old people, young people, etc. The speakers may cover different genders, e.g., male and female. The speakers may have different accents, e.g., American accent, British accent, Australian accent, etc. The speaker variety in the corpus set <NUM> may help to capture characteristics of different kinds of speakers and generate expressive speeches.

In an aspect, various styles of speech may be included in the corpus set <NUM>. Herein, "style of speech" may refer to expression manner of a speaker, e.g., telling a story, giving a lecture, daily chatting, etc. The styles of speech may be also associated with requirements of products or users.

Moreover, the corpus set <NUM> may also be prepared in consideration of any other aspects for improving its variety and richness.

The training corpus set <NUM> prepared as discussed above may facilitate to enable the multi-speaker neural TTS system <NUM> to leverage content from different speakers, generate close-to-human speech for out-of-domain text inputs, enrich premium voices, create new speeches for a target speaker having only a small corpus, etc..

It is shown in <FIG> that the multi-speaker neural TTS system <NUM> may comprise at least one speaker model <NUM>, an acoustic feature predictor <NUM> and a neural vocoder <NUM>. The architecture of the multi-speaker neural TTS system <NUM> may be the same as the multi-speaker neural TTS system <NUM> in <FIG>, and may be specifically implemented in either the structure in <FIG> or the structure in <FIG>. In other words, if the at least one speaker model <NUM> is a single speaker model, it may connect to both the acoustic feature predictor <NUM> and the neural vocoder <NUM>. While if the at least one speaker model <NUM> comprises two separate speaker models, one speaker model may connect to the acoustic feature predictor <NUM>, and another speaker model may connect to the neural vocoder <NUM>.

Training data for any one or any combination of the at least one speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may be obtained based on the speech waveforms in the corpus set <NUM>. For example, various derived information may be obtained from the speech waveforms, e.g., text information obtained through applying any existing speech recognition techniques, acoustic features obtained through applying any existing acoustic feature extracting techniques, speaker embedding vectors obtained through applying any existing speaker recognition techniques, etc. The derived information together with the speech waveforms in the corpus set <NUM> may form various training data for any one or any combination of the at least one speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM>. For example, training data consisted of pairs of text input and acoustic features may be used for training the acoustic feature predictor <NUM>, training data consisted of pairs of acoustic features and speech waveform may be used for training the neural vocoder <NUM>, training data consisted of collections of text input, acoustic features and speech waveform may be used for training the at least one speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> jointly, and so on.

In the case that the at least one speaker model <NUM> is a single speaker model, the speaker model, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may be trained separately based on the corpus set <NUM>. Alternatively, the speaker model, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may also be trained jointly based on the corpus set <NUM>.

In the case that the at least one speaker model <NUM> comprises two speaker models, the acoustic feature predictor <NUM> and its corresponding speaker model may be trained jointly based on the corpus set <NUM>, and the neural vocoder <NUM> and its corresponding speaker model may be trained jointly based on the corpus set <NUM>. Alternatively, all of the two speaker models, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may also be trained jointly based on the corpus set <NUM>.

<FIG> illustrates an exemplary process for updating a multi-speaker neural TTS system according to an embodiment. In some implementations, after a multi-speaker neural TTS system has been trained, the multi-speaker neural TTS system may be further updated, e.g., retrained, by a subset of training data pairs in a multi-speaker corpus set which has been used for training the multi-speaker neural TTS system. The subset for updating may be one or more corpuses of one or more speakers in the corpus set. Through this way, the multi-speaker neural TTS system may be further refined by the corpus set.

In <FIG>, a multi-speaker neural TTS system <NUM> may be updated by a corpus set <NUM> which has been used for training the multi-speaker neural TTS system <NUM>. The multi-speaker neural TTS system <NUM> may correspond to the multi-speaker neural TTS system <NUM> in <FIG>, and the corpus set <NUM> may correspond to the corpus set <NUM> in <FIG>.

A corpus m of a speaker in existing in the corpus set <NUM> may be extracted from the corpus set <NUM>. Speech waveforms in the corpus in may be used for forming training data that is for further updating the multi-speaker neural TTS system <NUM>. The training data may be formed from the corpus in in a similar approach as discussed above in connection with <FIG>, and at least one speaker model <NUM>, an acoustic feature predictor <NUM> and a neural vocoder <NUM> in the multi-speaker neural TTS system <NUM> may be updated in a similar way with the training procedure of the multi-speaker neural TTS system <NUM> as discussed above in connection with <FIG>. For example, any one or any combination of the at least one speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may be retrained based on the training data.

Through the updating process in <FIG>, the generalization ability of the multi-speaker neural TTS system <NUM> may be further improved. It should be noted that the speaker in may also be deemed as a target speaker. Moreover, although it is shown in <FIG> that only one corpus m is extracted from the corpus set, a different number of corpuses in the corpus set may be used for updating or retraining the multi-speaker neural TTS system.

<FIG> illustrates an exemplary process for updating a multi-speaker neural TTS system according to an embodiment. In some implementations, when deploying or applying a multi-speaker neural TTS system, e.g., when it is to generate speech waveforms for a target speaker through the multi-speaker neural TTS system, the multi-speaker neural TTS system may be updated by a corpus of the target speaker, so as to adapt the multi-speaker neural TTS system to the target speaker.

In <FIG>, a multi-speaker neural TTS system <NUM> may be used for generating speech for a target speaker, e.g., a new speaker <NUM> which is not existed in a corpus set used for training the multi-speaker neural TTS system <NUM>. The multi-speaker neural TTS system <NUM> may correspond to the multi-speaker neural TTS system <NUM> in <FIG> or the multi-speaker neural TTS system <NUM> in <FIG>.

A corpus <NUM> of the new speaker <NUM> may be obtained and used for forming training data that is for further updating the multi-speaker neural TTS system <NUM>. The training data may be formed from the corpus <NUM> of the new speaker <NUM> in a similar approach as discussed above in connection with <FIG>, and at least one speaker model <NUM>, an acoustic feature predictor <NUM> and a neural vocoder <NUM> in the multi-speaker neural TTS system <NUM> may be updated in a similar way with the training procedure of the multi-speaker neural TTS system <NUM> as discussed above in connection with <FIG>. For example, any one or any combination of the at least one speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may be updated or retrained based on the training data.

Through the updating process in <FIG>, the multi-speaker neural TTS system <NUM> may be better adapted to the new speaker, and accordingly generate high quality speech waveforms with high speaker similarity, e.g., better stimulating voices of the new speaker to speak.

<FIG> illustrates an exemplary processing flow <NUM> for generating a speech waveform according to an embodiment. The processing flow <NUM> may be performed by a multi-speaker neural TTS system for generating a speech waveform corresponding to a text input in a target speaker's voice.

A text input <NUM> may be received. At <NUM>, the processing flow <NUM> may further determine whether a target speaker <NUM> is a new speaker.

If it is determined that the target speaker <NUM> is not a new speaker, e.g., the target speaker <NUM> is an existing speaker in a corpus set used for training the multi-speaker neural TTS system, a speaker embedding vector corresponding to the target speaker may be selected at <NUM> from a speaker embedding vector database through, e.g., a speaker embedding selector in a speaker model in the multi-speaker neural TTS system.

If it is determined that the target speaker <NUM> is a new speaker, e.g., the target speaker <NUM> is not existed in the corpus set, a corpus of the target speaker <NUM> may be obtained at <NUM>.

At <NUM>, the corpus of the target speaker <NUM> may be used for updating the multi-speaker neural TTS system according to, e.g., the updating process in <FIG>.

At <NUM>, a speaker embedding vector of the target speaker <NUM> may be generated through, e.g., a speaker encoder in the speaker model in the multi-speaker neural TTS system.

At <NUM>, acoustic features may be predicted through, e.g., an acoustic feature predictor in the multi-speaker neural TTS system based on the text input <NUM> and the speaker embedding vector provided by the step <NUM> or <NUM>.

At <NUM>, a speech waveform corresponding to the text input <NUM> may be generated through, e.g., a neural vocoder in the multi-speaker neural TTS system based on the acoustic features and the speaker embedding vector.

It should be appreciated that depending on the specific structure of the multi-speaker neural TTS system, the predicting step at <NUM> and the generating step at <NUM> may utilize the same speaker embedding vector or different speaker embedding vectors. For example, if a single speaker model connects to both the acoustic feature predictor and the neural vocoder, the speaker embedding vector utilized by the predicting step at <NUM> and the generating step at <NUM> may be the same. While if two different speaker models connect to the acoustic feature predictor and the neural vocoder respectively, the speaker embedding vector utilized by the predicting step at <NUM> may be different from that utilized by the generating step at <NUM>.

Moreover, it should be appreciated that, depending on specific designs and requirements, any steps and step orders in the processing flow <NUM> may be adjusted, omitted or replaced according to the embodiments of the present disclosure. Any additional steps may also be added into the processing flow <NUM>. For example, if it is determined at <NUM> that the target speaker is not a new speaker, e.g., the target speaker <NUM> is an existing speaker in the training corpus set, the multi-speaker neural TTS system may also be updated based on a corpus of the target speaker in the corpus set according to, e.g., the updating process in <FIG>.

<FIG> illustrates an exemplary architecture of a multi-speaker neural TTS system <NUM> according to an embodiment.

The multi-speaker neural TTS system <NUM> may comprise a speaker information extractor <NUM>, an acoustic feature predictor <NUM> and a neural vocoder <NUM>, wherein the speaker information extractor <NUM> may comprise or implement at least one speaker model <NUM>. The speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may correspond to the speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> in <FIG> respectively.

In some implementations, the speaker information extractor <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may be implemented in a hardware level, e.g., implemented by respective hardware units. Herein, a hardware unit may refer to one or more processors, one or more chips, one or more central processing units (CPU), one or more graphic processing unit (GPU), etc., or any combination thereof. In some implementations, the speaker information extractor <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may also be implemented in a software level, e.g., implemented by computer programs, processor instructions, etc. The multi-speaker neural TTS system <NUM> is not limited to any specific implementation approaches.

The speaker information extractor <NUM> may be configured for providing speaker latent space information of a target speaker through the at least one speaker model <NUM>.

The acoustic feature predictor <NUM> may be configured for predicting at least one acoustic feature based on a text input and the speaker latent space information.

The neural vocoder <NUM> may be configured for generating a speech waveform corresponding to the text input based on the at least one acoustic feature and the speaker latent space information.

In an implementation, the at least one speaker model <NUM> may comprise: a first speaker model, configured for providing first speaker latent space information in the speaker latent space information; and a second speaker model, configured for providing second speaker latent space information in the speaker latent space information. The acoustic feature predictor <NUM> may be configured for predicting the at least one acoustic feature based on the text input and the first speaker latent space information. The neural vocoder <NUM> may be configured for generating the speech waveform based on the at least one acoustic feature and the second speaker latent space information.

In an implementation, at least one of the at least one speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> was pre-trained separately based on a plurality of corpuses of a plurality of speakers in a corpus set, and/or any two or more of the at least one speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> were pre-trained jointly based on the plurality of corpuses of the plurality of speakers.

In an implementation, if the at least one speaker model <NUM> comprises a first speaker model and a second speaker model, the first speaker model and the acoustic feature predictor were pre-trained jointly based on a plurality of corpuses of a plurality of speakers in a corpus set, and/or the second speaker model and the neural vocoder were pre-trained jointly based on the plurality of corpuses of the plurality of speakers.

In an implementation, at least one of the at least one speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may be updated separately based on a corpus of the target speaker, and/or any two or more of the at least one speaker model <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may be updated jointly based on the corpus of the target speaker.

In an implementation, in the case that the at least one speaker model <NUM> comprises a first speaker model and a second speaker model, the first speaker model and the acoustic feature predictor may be updated jointly based on a corpus of the target speaker, and/or the second speaker model and the neural vocoder may be updated jointly based on the corpus of the target speaker.

Moreover, the speaker information extractor <NUM>, the acoustic feature predictor <NUM> and the neural vocoder <NUM> may also be configured for performing any other processes or operations according to the embodiments of the present disclosure as mentioned above.

<FIG> illustrates a flowchart of an exemplary method <NUM> for generating speech through multi-speaker neural TTS synthesis according to an embodiment.

At <NUM>, speaker latent space information of a target speaker may be provided through at least one speaker model.

At <NUM>, at least one acoustic feature may be predicted through an acoustic feature predictor based on the text input and the speaker latent space information.

At <NUM>, a speech waveform corresponding to the text input may be generated through a neural vocoder based on the at least one acoustic feature and the speaker latent space information.

In an implementation, the at least one speaker model may comprise a single speaker model.

In an implementation, the at least one speaker model may comprise a first speaker model and a second speaker model. The providing may comprise: providing first speaker latent space information through the first speaker model; and providing second speaker latent space information through the second speaker model. The predicting may comprise: predicting the at least one acoustic feature based on the text input and the first speaker latent space information. The generating may comprise: generating the speech waveform based on the at least one acoustic feature and the second speaker latent space information.

In an implementation, the providing may comprise: generating a speaker embedding vector of the target speaker based on a corpus of the target speaker; or selecting a speaker embedding vector of the target speaker from a speaker embedding vector database.

In an implementation, the method <NUM> may further comprise: generating at least one transformed acoustic feature based on the at least one acoustic feature through a first neural network; and generating transformed speaker latent space information based on the speaker latent space information through a second neural network. The generating the speech waveform may comprise: generating the speech waveform based on a combination of the at least one transformed acoustic feature and the transformed speaker latent space information.

In an implementation, the method <NUM> may further comprise: updating at least one of the at least one speaker model, the acoustic feature predictor and the neural vocoder separately based on a corpus of the target speaker; and/or updating any two or more of the at least one speaker model, the acoustic feature predictor and the neural vocoder jointly based on the corpus of the target speaker.

In an implementation, if the at least one speaker model comprises a first speaker model and a second speaker model, the method <NUM> may further comprise: updating the first speaker model and the acoustic feature predictor jointly based on a corpus of the target speaker; and/or updating the second speaker model and the neural vocoder jointly based on the corpus of the target speaker.

In an implementation, at least one of the at least one speaker model, the acoustic feature predictor and the neural vocoder was pre-trained separately based on a plurality of corpuses of a plurality of speakers, and/or any two or more of the at least one speaker model, the acoustic feature predictor and the neural vocoder were pre-trained jointly based on the plurality of corpuses of the plurality of speakers.

In an implementation, in the case that the at least one speaker model comprises a first speaker model and a second speaker model, the first speaker model and the acoustic feature predictor were pre-trained jointly based on a plurality of corpuses of a plurality of speakers, and/or the second speaker model and the neural vocoder were pre-trained jointly based on the plurality of corpuses of the plurality of speakers.

In an implementation, the plurality of corpuses are prepared based on at least one of content coverage, speaker variety and style variety.

It should be appreciated that the method <NUM> may further comprise any steps/processes for generating speech through multi-speaker neural TTS synthesis according to the embodiments of the present disclosure as mentioned above.

<FIG> illustrates an exemplary apparatus <NUM> for generating speech through multi-speaker neural TTS synthesis according to an embodiment.

The apparatus <NUM> may comprise: a text input receiving module <NUM>, for receiving a text input; a speaker latent space information providing module <NUM>, for providing, through at least one speaker model, speaker latent space information of a target speaker; an acoustic feature predicting module <NUM>, for predicting, through an acoustic feature predictor, at least one acoustic feature based on the text input and the speaker latent space information; and a speech waveform generating module <NUM>, for generating, through a neural vocoder, a speech waveform corresponding to the text input based on the at least one acoustic feature and the speaker latent space information.

Moreover, the apparatus <NUM> may also comprise any other modules configured for generating speech through multi-speaker neural TTS synthesis according to the embodiments of the present disclosure as mentioned above.

The apparatus <NUM> may comprise at least one processor <NUM> and a memory <NUM> storing computer-executable instructions. When executing the computer-executable instructions, the at least one processor <NUM> may: receive a text input; provide, through at least one speaker model, speaker latent space information of a target speaker; predict, through an acoustic feature predictor, at least one acoustic feature based on the text input and the speaker latent space information; and generate, through a neural vocoder, a speech waveform corresponding to the text input based on the at least one acoustic feature and the speaker latent space information.

The at least one processor <NUM> may be further configured for performing any operations of the methods for generating speech through multi-speaker neural TTS synthesis according to the embodiments of the present disclosure as mentioned above.

The embodiments of the present disclosure may be embodied in a non-transitory computer-readable medium. The non-transitory computer-readable medium may comprise instructions that, when executed, cause one or more processors to perform any operations of the methods for generating speech through multi-speaker neural TTS synthesis according to the embodiments of the present disclosure as mentioned above.

It should be appreciated that all the operations in the methods described above are merely exemplary, and the present disclosure is not limited to any operations in the methods or sequence orders of these operations.

It should also be appreciated that all the modules in the apparatuses described above may be implemented in various approaches. These modules may be implemented as hardware, software, or a combination thereof. Moreover, any of these modules may be further functionally divided into sub-modules or combined together.

Processors have been described in connection with various apparatuses and methods. These processors may be implemented using electronic hardware, computer software, or any combination thereof. Whether such processors are implemented as hardware or software will depend upon the particular application and overall design constraints imposed on the system. By way of example, a processor, any portion of a processor, or any combination of processors presented in the present disclosure may be implemented with a microprocessor, microcontroller, digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a state machine, gated logic, discrete hardware circuits, and other suitable processing components configured to perform the various functions described throughout the present disclosure. The functionality of a processor, any portion of a processor, or any combination of processors presented in the present disclosure may be implemented with software being executed by a microprocessor, microcontroller, DSP, or other suitable platform.

Software shall be construed broadly to mean instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules, applications, software applications, software packages, routines, subroutines, objects, threads of execution, procedures, functions, etc. The software may reside on a computer-readable medium. A computer-readable medium may include, by way of example, memory such as a magnetic storage device (e.g., hard disk, floppy disk, magnetic strip), an optical disk, a smart card, a flash memory device, random access memory (RAM), read only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), a register, or a removable disk. Although memory is shown separate from the processors in the various aspects presented throughout the present disclosure, the memory may be internal to the processors, e.g., cache or register.

Claim 1:
A method (<NUM>) for generating speech through multi-speaker neural text-to-speech "TTS" synthesis, comprising:
receiving (<NUM>) a text input;
providing (<NUM>), through at least one speaker model, speaker latent space information of a target speaker;
predicting (<NUM>), through an acoustic feature predictor, at least one acoustic feature based on the text input and the speaker latent space information;
generating (<NUM>), through a neural vocoder, a speech waveform corresponding to the text input based on the at least one acoustic feature and the speaker latent space information;
generating at least one transformed acoustic feature based on the at least one acoustic feature through a first neural network; and
generating transformed speaker latent space information based on the speaker latent space information through a second neural network, and
wherein the generating the speech waveform comprises: generating the speech waveform based on a combination of the at least one transformed acoustic feature and the transformed speaker latent space information.