Patent Description:
<NPL> describes an end-to-end method for transforming audio from one style to another. For the case of speech, by conditioning on speaker identities, a single model is trained to transform words spoken by multiple people into multiple target voices. For the case of music, musical instruments are specified. Architecturally, the method is a fully-differentiable sequence-to-sequence model based on convolutional and hierarchical recurrent neural networks. It is designed to capture long-term acoustic dependencies, requires minimal post-processing, and produces realistic audio transforms.

<NPL> describes a technique for augmenting neural text-to-speech (TTS) with low-dimensional trainable speaker embeddings to generate different voices from a single model. Deep Voice <NUM> is based on a similar pipeline with Deep Voice <NUM>, but constructed with higher performance building blocks and demonstrates a significant audio quality improvement over Deep Voice <NUM>. Deep Voice <NUM> introduces a post-processing neural vocoder, and demonstrates a significant audio quality improvement.

<NPL> describes that emergent technologies in the fields of audio speech synthesis and video facial manipulation have the potential to drastically impact our societal patterns of multimedia consumption. This research project seeks to examine four emerging software programs - Face2Face, FakeApp, Adobe VoCo and Lyrebird - that are designed to facilitate the synthesis of speech and manipulate facial features in videos.

It is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

Embodiments of the present disclosure propose methods and apparatuses for training an acoustic model. The acoustic model may be for implementing cross-speaker style transfer and comprise at least a style encoder.

In some embodiments, training data may be obtained, the training data comprising a text, a speaker identity (ID), a style ID and acoustic features corresponding to a reference audio. A reference embedding vector may be generated, through the style encoder, based on the acoustic features. Adversarial training may be performed to the reference embedding vector with at least the style ID and the speaker ID, to remove speaker information and retain style information. A style embedding vector may be generated, through the style encoder, based at least on the reference embedding vector being performed the adversarial training. Predicted acoustic features may be generated based at least on a state sequence corresponding to the text, a speaker embedding vector corresponding to the speaker ID, and the style embedding vector.

In some other embodiments, training data may be obtained, the training data at least comprising a first text, a first speaker ID, and a second text, a second speaker ID and style reference acoustic features corresponding to a style reference audio. First transfer acoustic features may be generated, through the acoustic model, based at least on the first text, the first speaker ID, and a first transfer style embedding vector, wherein the first transfer style embedding vector is generated by the style encoder based on the style reference acoustic features. Second transfer acoustic features may be generated, through a duplicate of the acoustic model, based at least on the second text, the second speaker ID and a second transfer style embedding vector, wherein the second transfer style embedding vector is generated by a duplicate of the style encoder based on the first transfer acoustic features. Cyclic reconstruction loss may be calculated with the style reference acoustic features and the second transfer acoustic features.

It should be noted that the above one or more aspects comprise the features hereinafter fully described and particularly pointed out in the claims. The following description and the drawings set forth in detail certain illustrative features of the one or more aspects. These features are only indicative of the various ways in which the principles of various aspects may be employed.

The disclosed aspects will hereinafter be described in connection with the appended drawings that are provided to illustrate and not to limit the disclosed aspects.

The present disclosure will now be discussed with reference to several example implementations It is to be understood that these implementations are discussed only for enabling those skilled in the art to better understand and thus implement the embodiments of the present disclosure, rather than suggesting any limitations on the scope of the present disclosure.

A conventional TTS system may include an acoustic model and a vocoder. The acoustic model may predict acoustic features, e.g., mel-spectrum sequence, based on a text input. The vocoder may convert the predicted acoustic features into a speech waveform. Generally, the acoustic model will determine speech characteristics in terms of prosody, timbre, etc. The acoustic model may be speaker-dependent, e.g., trained with speech data of a target speaker. The trained TTS system may convert a text input into speech having similar timbre, prosody, etc. with the target speaker. In some cases, it may be desirable to synthesize speech in a specific speaking style, e.g., in an approach of newscaster, reading, storytelling, happy emotion, sad emotion, etc. Herein, "style" refers to the approach of uttering or speaking, which may be characterized by, e.g., prosody, timbre change, etc..

A straightforward way is to collect audio data of a target speaker in a target style, and train the TTS system with these audio data. The trained TTS system may perform speech synthesis in the target speaker's voice and in the target style.

Another way is to perform style transfer in speech synthesis. A style embedding vector corresponding to a target style may be obtained and introduced into the TTS system, so as to guide the synthesized speech to the target style. The style transfer may include single-speaker style transfer and cross-speaker style transfer.

In the single-speaker style transfer, audio data of a target speaker in a plurality of styles may be collected for training the TTS system. The trained TTS system may perform speech synthesis in the target speaker's voice and in different target styles.

In the cross-speaker style transfer, audio data of a plurality of speakers in a plurality of styles may be collected for training the TTS system. The trained TTS system may perform speech synthesis in any target speaker's voice and in any target style. This will significantly enhance the style imposing capability of the TTS system. Style embedding vector is a key influencing factor in the cross-speaker style transfer. In one aspect, techniques such as Global Style Token (GST), etc. have been proposed to extract a style embedding vector. However, these technologies cannot guarantee sufficient accuracy and robustness. In another aspect, since the style embedding vector is learned from collected multi-speaker multi-style audio data during training, it likely contains speaker information or content information, which will reduce the quality of synthesized speech in terms of prosody, timbre, etc. In yet another aspect, during the training of the TTS system, a text input, a speaker identity and an audio, which act as training data, are usually paired, e.g., the audio is spoken by the speaker and content spoken by the speaker is the text input. Therefore, in the synthesis stage or the stage of applying the TTS system, when it is desired to synthesize speech for a certain target text in a voice of speaker A, if an audio or acoustic features of speaker B for another text different from the target text is used as style reference, the quality of synthesized speech will be low. This is because paired training data is used during training, and such unpaired situation has not been considered. Although it is proposed in some existing TTS systems that unpaired inputs may be used during training, wherein an unpaired input may refer to that, e.g., an input audio is for a text different from a text input, since unpaired prediction results generated for the unpaired inputs usually do not have ground truth labels or effective constraints, it may be still unable to train a high-quality TTS system effectively.

Embodiments of the present disclosure propose a scheme for effectively training an acoustic model in a TTS system, so as to predict high-quality acoustic features. In particular, a style encoder in the acoustic model may be well trained to facilitate to implement cross-speaker style transfer. TTS including this acoustic model will be able to implement style transfer speech synthesis with higher quality.

In some embodiments of the present disclosure, it is proposed to apply adversarial training to the style encoder during the training of the acoustic model, so as to improve the quality of style embedding vectors.

An adversarial training mechanism such as Domain Adversarial Training (DAT) may be adopted for retaining as much pure style information as possible in style embedding vectors generated by the style encoder, and for removing as much speaker information, content information, etc., as possible from the style embedding vectors. When performing cross-speaker style transfer speech synthesis, it is expected that the timbre of a synthesized speech is the timbre of a target speaker. Through the DAT, a style embedding vector may be prevented from containing information of a reference speaker in a style reference audio, e.g., timbre information of the reference speaker, etc., thereby preventing the timbre of a synthesized speech from being undesirably changed, e.g., becoming a mixture of the timbres of the target speaker and the reference speaker. Accordingly, audio fidelity of synthesized speech may be improved. In other words, a speaking style may be effectively transferred to the target speaker, and meanwhile, a synthesized speech may have a timbre and audio fidelity similar with the target speaker's own voice. In an implementation, in the DAT, a style classifier and a speaker classifier which connects to a gradient reversal layer may be applied for retaining style information and removing speaker information in a style embedding vector.

The style encoder may adopt, e.g., a Variational Auto Encoder (VAE), a Gaussian Mixture Variational Auto Encoder (GMVAE), etc. As compared with the GST, the VAE is more suitable for speech generation and has better performance. Through the VAE, a latent variable having Gaussian distribution may be inferred from a style reference audio in a variational manner, and the Gaussian distribution of the latent variable may be further used for obtaining a style embedding vector, wherein the latent variable may be regarded as a simplified inherent factor that leads to a relevant speaking style. The GMVAE is an extension of the VAE. Through adopting the GMVAE and multi-style audio data in the training, a set of Gaussian distributions may be learned, which represent a Gaussian mixture distribution of latent variables that lead to each speaking style. The latent variables obtained through the VAE or the GMVAE have Gaussian distribution or Gaussian mixture distribution respectively, which are in low dimensions, and retain more prosody-related information and contain, e.g., less content information, speaker information, etc. A style embedding vector may correspond to a prior distribution or a posterior distribution of a latent variable having Gaussian distribution or Gaussian mixture distribution. In particular, a prior distribution of a latent variable is a good and robust representation of a speaking style, therefore higher quality and more stable style transfer may be implemented through adopting a prior distribution to obtain a style embedding vector. In one aspect, the prior distribution may be speaker-independent, e.g., one style has a global prior distribution. In another aspect, the prior distribution may also be speaker-dependent, e.g., each speaker's style has a corresponding prior distribution. When it is desired to transfer a style of a specific reference speaker to a target speaker, it would be advantageous to rely on a prior distribution of the speaker. Through training, a prior distribution learned for each style and/or each reference speaker may be a good and robust representation of style embedding. Moreover, since a prior distribution of each speaking style is more representative and content-independent for the speaking style, optionally, in the case of adopting these prior distributions to obtain a style embedding vector of each style, there is no need to input a target style reference audio in the synthesis stage, thereby having higher quality and stability.

A speaker look-up table (LUT) may be used for obtaining a speaker embedding vector. The resulting speaker embedding vector is more robust in controlling the speaker identity of a synthesized speech.

Training data obtained from multi-speaker multi-style audio may be adopted. These training data may be in a supervised form, e.g., attached with style labels, speaker labels, etc. These labels may be used in the DAT for calculating a gradient back-propagation factor, etc..

In other embodiments of the present disclosure, it is proposed to adopt a combination of paired input and unpaired input and adopt a cyclic training mechanism for the acoustic model, during the training of the acoustic model.

On the input side, there are two sets of input, i.e., paired input and unpaired input. The paired input includes, e.g., a first text and a paired audio corresponding to the first text, wherein the paired audio may be an audio in which a first speaker says the first text in a first style, and the first speaker is a target speaker of speech synthesis. The unpaired input includes, e.g., the first text and an unpaired audio that does not correspond to the first text, wherein the unpaired audio may be an audio in which a second speaker says a second text in a second style, and the second style may be a target style of the style transfer. Through adopting paired input and unpaired input in the training data, it may avoid quality degradation in a situation of taking unpaired input in the synthesis stage, which is due to the fact of always being in a paired situation during the training. Therefore, it may facilitate to implement high-quality cross-speaker style transfer.

On the output side, there are two outputs, i.e., paired output and unpaired output, and the unpaired output may also be referred to as a transfer output. The paired output is predicted acoustic features when the first speaker says the first text in the first style. The unpaired output is predicted acoustic features when the first speaker says the first text in the second style. The unpaired output may achieve cross-speaker style transfer.

For the paired output, acoustic features of the paired audio may be used as a ground truth label for calculating loss metrics, e.g., reconstruction loss. In order to obtain a ground truth label for the transfer output during the training, a cyclic training mechanism may be introduced to the above basic acoustic model, to provide a good loss metric for unpaired output to ensure quality. For example, a cyclic training framework may be formed with a basic acoustic model and a duplicate of the basic acoustic model. The duplicate of the basic acoustic model has the same or similar architecture, parameters, etc. as the basic acoustic model. The unpaired output by the basic acoustic model may be further input to the duplicate of the basic acoustic model, as a reference for the style transfer performed by the duplicate of the basic acoustic model. The duplicate of the basic acoustic model may generate a second unpaired output for the second text, which is predicted acoustic features when the second speaker says the second text in the second style. For the second unpaired output, acoustic features of the unpaired audio may be used as a ground truth label for calculating loss metrics, e.g., cyclic reconstruction loss.

Moreover, any other loss metrics may also be considered during the cyclic training process, e.g., style loss, Generative Adversarial Network (GAN) loss, etc. Moreover, the above cyclic training mechanism is not limited by whether the training data has style labels. Moreover, in the case of adopting the above cyclic training mechanism, specific implementations of the style encoder are not subject to any limitations, which may be a VAE, a GMVAE or any other encoder capable of generating style embedding vectors.

It should be understood that the term "embedding vector" herein may broadly refer to a representation of information in the latent space, which may also be referred to as embedding, latent representation, latent space representation, latent space information representation, etc., and is not limited to adopt a data form of vector, but also covers any other data form, e.g., sequence, matrix, etc..

<FIG> illustrates an exemplary conventional style transfer TTS system <NUM>.

The TTS system <NUM> may be configured for receiving a text <NUM>, and generating a speech waveform <NUM> corresponding to the text <NUM>. The text <NUM> may comprise word, phrase, sentence, passage, etc. It should be understood that although the text <NUM> is shown as being provided to the TTS system <NUM> in <FIG>, the text <NUM> may be first divided into a sequence of elements, e.g., a phoneme sequence, a grapheme sequence, a character sequence, etc., and this sequence is then provided to the TTS system <NUM> as input. Herein, the input "text" may broadly refer to words, phrases, sentences, etc. included in the text, or a sequence of elements obtained from the text, e.g., a phoneme sequence, a grapheme sequence, a character sequence, etc..

The TTS system <NUM> may include an acoustic model <NUM>. The acoustic model <NUM> may predict or generate acoustic features <NUM> according to the text <NUM>. The acoustic features <NUM> may include various TTS acoustic features, e.g., mel-spectrum, linear spectrum pair (LSP), etc. The acoustic model <NUM> may be based on various model architectures, e.g., sequence-to-sequence model architecture, etc. <FIG> illustrates an exemplary sequence-to-sequence acoustic model <NUM>, which may include a text encoder <NUM>, an attention module <NUM>, and a decoder <NUM>.

The text encoder <NUM> may convert information contained in the text <NUM> into a space that is more robust and more suitable for learning alignment with acoustic features. For example, the text encoder <NUM> may convert the information in the text <NUM> into a state sequence in the space, which may also be referred to as a text encoder state sequence. Each state in the state sequence corresponds to a phoneme, a grapheme, or a character in the text <NUM>.

The attention module <NUM> may apply an attention mechanism. The attention mechanism establishes a connection between the text encoder <NUM> and the decoder <NUM>, to facilitate to align between text features output by the text encoder <NUM> and the acoustic features. For example, a connection between each decoding step and a text encoder state may be established, and the connection may indicate each decoding step should correspond to which text encoder state with what weight. The attention module <NUM> may take the text encoder state sequence and an output of the previous step by the decoder as input, and generate a context vector that represents a weight with which the next decoding step shall align with each text encoder state.

The decoder <NUM> may map a state sequence output by the encoder <NUM> to the acoustic features <NUM> under the influence of the attention mechanism in the attention module <NUM>. In each decoding step, the decoder <NUM> may take a context vector output by the attention module <NUM> and an output of the previous step by the decoder as input, and output acoustic features of one or more frames, e.g., mel-spectrum.

In the case of utilizing the TTS system <NUM> to generate speech based on a target style, the state sequence output by the text encoder <NUM> may be combined with a style embedding vector <NUM> corresponding to the target style prepared in advance, to extend the text encoder state sequence. The extended text encoder state sequence may be provided to the attention module <NUM> for subsequent speech synthesis.

The TTS system <NUM> may include a vocoder <NUM>. The vocoder <NUM> may generate the speech waveform <NUM> based on the acoustic features <NUM> predicted by the acoustic model <NUM>.

As described above, due to the limitations by system architecture, model design or training approach, the style embedding vector adopted in the conventional TTS system may be unable to characterize a speaking style very well, thus limiting the quality of cross-speaker style transfer speech synthesis. The embodiments of the present disclosure propose a novel training approach for a style encoder, so that the trained style encoder may generate a style embedding vector that is beneficial to achieve high-quality cross-speaker style transfer, thereby enabling an acoustic model to predict acoustic features that are beneficial to achieve high-quality cross-speaker style transfer.

<FIG> illustrates an exemplary operating process <NUM> of an acoustic model in a synthesis stage according to an embodiment. Herein, the synthesis stage may refer to a stage in which a trained TTS system is applied for speech synthesis after the TTS system is trained. The acoustic model in <FIG> is applied for generating corresponding acoustic features for an input target text through cross-speaker style transfer.

The acoustic model may comprise basic components, e.g., a text encoder <NUM>, an attention module <NUM>, a decoder <NUM>, etc. Moreover, the acoustic model may further include components, e.g., an extending module <NUM>, a speaker LUT <NUM>, a style encoder <NUM> trained according to the embodiments of the present disclosure, etc..

Input to the acoustic model may comprise, e.g., a target text <NUM>, a target speaker ID <NUM>, a target style reference audio <NUM>, etc. The acoustic model aims to generate acoustic features corresponding to the target text <NUM>. The target speaker ID <NUM> is an identification of a target speaker, wherein the acoustic model aims to generate acoustic features in the target speaker's voice. The target speaker ID may be any identification used for indexing the target speaker, e.g., character, number, etc. The target style reference audio <NUM> is used as a reference for performing cross-speaker style transfer, which may be, e.g., an audio spoken by a speaker different from the target speaker for a text different from the target text <NUM>. The style of the target style reference audio <NUM> may be referred to as a target style, and the acoustic model aims to generate acoustic features in the target style.

The text encoder <NUM> may encode the target text <NUM> into a corresponding state sequence.

The speaker LUT <NUM> may generate a corresponding speaker embedding vector <NUM> based on the target speaker ID <NUM>. For example, a plurality of speaker embedding vectors that characterize different target speakers may be predetermined, and mapping relationship between the plurality of target speaker IDs and the plurality of speaker embedding vectors may be established through a look up table. When the target speaker ID <NUM> is input, the speaker embedding vector <NUM> corresponding to this ID may be retrieved with the mapping relationship in the speaker LUT <NUM>. By using the speaker LUT <NUM>, the TTS system may be enabled to become a multi-speaker TTS system, i.e., speech may be synthesized with voices of different speakers. It should be understood that in the case of a single-speaker TTS system, i.e., when the TTS system is used for synthesizing speech with a specific target speaker's voice, the processing of adopting the speaker LUT to obtain a speaker embedding vector may also be omitted.

The style encoder <NUM> is a generative encoder, which may be obtained through an adversarial training mechanism or a cyclic training mechanism according to the embodiments of the present disclosure. The style encoder <NUM> may be used for extracting style information from an audio, e.g., generating a style embedding vector <NUM> based at least on the target style reference audio <NUM>. In an implementation, the style encoder <NUM> may first extract acoustic features <NUM> from the target style reference audio <NUM>, and then generate the style embedding vector <NUM> based on the acoustic features <NUM>. It should be understood that, herein, the processing of generating a style embedding vector based on an audio by a style encoder may broadly refer to generating the style embedding vector directly based on the audio or based on acoustic features of the audio.

In an implementation, the style encoder <NUM> may be based on the VAE. In this case, the style encoder <NUM> may determine a posterior distribution of a latent variable having Gaussian distribution based on the acoustic features <NUM>, and generate the style embedding vector <NUM>, e.g., by sampling on the posterior distribution, etc..

In an implementation, the style encoder <NUM> may be based on the GMVAE. In this case, the style encoder <NUM> may determine a posterior distribution of a latent variable having Gaussian mixture distribution based on the acoustic features <NUM> and a target style ID <NUM>, and generate the style embedding vector <NUM>, e.g., by sampling on the posterior distribution, etc. The target style ID may be any identification used for indexing a target style, e.g., character, number, etc. It should be understood that although <FIG> shows that the optional target style ID <NUM> is input to the acoustic model, the GMVAE-based style encoder <NUM> may also operate without directly receiving a target style ID. For example, the style encoder <NUM> may infer a corresponding target style based at least on the acoustic features <NUM> of the target style reference audio <NUM>, and use the inferred target style along with the acoustic features <NUM> for generating the style embedding vector <NUM>.

The extending module <NUM> may extend the state sequence output by the text encoder <NUM> with the speaker embedding vector <NUM> and the style embedding vector <NUM>. For example, the speaker embedding vector <NUM> and the style embedding vector <NUM> may be concatenated to the state sequence, or the speaker embedding vector <NUM> and the style embedding vector <NUM> may be superimposed on the state sequence. Through the processing by the extending module <NUM>, the speaker embedding vector <NUM> and the style embedding vector <NUM> may be introduced into the generating process of acoustic features, so that the acoustic model may generate acoustic features based at least on the target text, the speaker embedding vector, and the style embedding vector.

The extended text encoder state sequence is provided to the attention module <NUM>. The decoder <NUM> will predict or generate the final acoustic features <NUM> under the influence of the attention module <NUM>. The acoustic features <NUM> may then be used by a vocoder of the TTS system for generating a corresponding speech waveform.

The speech synthesized by the TTS system including the acoustic model shown in <FIG> will have the target speaker's voice, have the target speaking style, and take the target text as speech content. Since the style encoder <NUM> may generate the high-quality style embedding vector <NUM> for cross-speaker style transfer, the TTS system may also generate high-quality synthesized speech accordingly.

<FIG> illustrates an exemplary operating process <NUM> of an acoustic model in a synthesis stage according to an embodiment. The acoustic model in <FIG> has a substantially similar architecture with the acoustic model in <FIG>.

Input to the acoustic model in <FIG> may include, e.g., a target text <NUM>, a target speaker ID <NUM>, a target style ID <NUM>, an optional reference speaker ID <NUM>, etc..

A text encoder <NUM> may encode the target text <NUM> into a corresponding state sequence.

A speaker LUT <NUM> may generate a corresponding speaker embedding vector <NUM> based on the target speaker ID <NUM>.

The style encoder <NUM> is an encoder that adopts at least the LUT technique, which may be obtained through the adversarial training mechanism according to the embodiments of the present disclosure. The style encoder <NUM> may be based on the GMVAE. The style encoder <NUM> may determine a prior distribution of a latent variable having Gaussian mixture distribution based on the target style ID <NUM> and the optional reference speaker ID <NUM> and by adopting at least the LUT technique, and generate a style embedding vector <NUM>, e.g., by sampling on the prior distribution or calculating a mean value on the prior distribution.

The style encoder <NUM> may be speaker-dependent or speaker-independent, which depends on whether the same style may be shared among different speakers or needs to be distinguished among different speakers. For example, for a certain style, if different speakers have the same or similar speaking approaches in this style, a speaker-independent style encoder may be used for generating a global style embedding vector for this style. For a certain style, if different speakers have different speaking approaches in this style, a speaker-dependent style encoder may be used for generating different style embedding vectors for different speakers for this style, i.e., characterization of this style considers at least the style itself and speakers. In this case, a style embedding vector may not only include information that characterizes prosody, but also include information that characterizes, e.g., timbre change. Although timbre information reflecting a speaker's voice may be removed from the style embedding vector as much as possible in the embodiments of the present disclosure, the timbre change information may be retained to reflect a specific speaking approach of a specific speaker in the style.

In an implementation, the style encoder <NUM> may be speaker-independent, so that the style embedding vector <NUM> may be determined only based on the target style ID <NUM>. For example, the style encoder <NUM> may first determine a style intermediate representation vector corresponding to the target style ID <NUM> with a style intermediate representation LUT. The style intermediate representation vector is an intermediate parameter generated during the acquisition of the final style embedding vector, which includes lower-level style information as compared with a style embedding vector. Then, the style encoder <NUM> may determine a prior distribution of a latent variable based on the style intermediate representation vector, and generate the style embedding vector <NUM> by sampling or averaging the prior distribution. The style intermediate representation LUT may be created during the training stage, which includes mapping relationship between multiple style IDs and multiple style intermediate representation vectors.

In another implementation, the style encoder <NUM> may be speaker-dependent, so that the style embedding vector <NUM> may be determined based on both the target style ID <NUM> and the reference speaker ID <NUM>. The reference speaker ID may be any identification used for indexing different speakers associated with a certain target style, e.g., character, number, etc. For example, the style encoder <NUM> may first determine a style intermediate representation vector corresponding to the target style ID <NUM> with a style intermediate representation LUT, and determine a speaker intermediate representation vector corresponding to the reference speaker ID <NUM> with a speaker intermediate representation LUT. The speaker intermediate representation vector may characterize a speaker, but it only includes lower-level speaker information as compared with a speaker embedding vector. Then, the style encoder <NUM> may determine a prior distribution of a latent variable based on the style intermediate representation vector and the speaker intermediate representation vector, and generate the style embedding vector <NUM> by sampling or averaging the prior distribution. The speaker intermediate representation LUT may also be created during the training stage, which includes mapping relationship between multiple speaker IDs and multiple speaker intermediate representation vectors.

It should be understood that although it is discussed above that the style encoder <NUM> may determine the prior distribution based on the target style ID and the optional reference speaker ID, sample or average the prior distribution, and generate the style embedding vector in the synthesis stage, the style encoder <NUM> may also operate in different approaches. In one approach, a prior distribution LUT may be created during the training stage, which includes mapping relationship between multiple prior distributions generated during the training and corresponding target style IDs and possible speaker IDs. Therefore, in the synthesis stage, the style encoder may directly retrieve a corresponding prior distribution from the prior distribution LUT based on a target style ID and an optional reference speaker ID. Then, the prior distribution may be sampled or averaged to generate a style embedding vector. In another approach, a prior distribution mean value LUT may be created during the training stage, which includes mapping relationship between mean values of multiple prior distributions generated during the training and corresponding target style IDs and possible speaker IDs. Therefore, in the synthesis stage, the style encoder may directly retrieve a mean value of a corresponding prior distribution from the prior distribution mean value LUT based on a target style ID and an optional reference speaker ID. Then, this mean value may be used for forming a style embedding vector. In another approach, a style embedding vector LUT may be created during the training stage, which includes mapping relationship between multiple style embedding vectors generated during the training and corresponding target style IDs and possible speaker IDs. Therefore, in the synthesis stage, the style encoder may directly retrieve a corresponding style embedding vector from the style embedding vector LUT based on a target style ID and an optional reference speaker ID.

The extending module <NUM> may extend the state sequence output by the text encoder <NUM> with the speaker embedding vector <NUM> and the style embedding vector <NUM>. The extended text encoder state sequence is provided to an attention module <NUM>. A decoder <NUM> will predict or generate the final acoustic features <NUM> under the influence of the attention module <NUM>. The acoustic features <NUM> may then be used by a vocoder of the TTS system for generating a corresponding speech waveform.

Different from <FIG> in which a target style reference audio is required to be input for specifying a target style, the process <NUM> in <FIG> only requires the inputting of a target style ID and an optional reference speaker ID for specifying a target style, and thus the style encoder may output a style embedding vector with higher stability and robustness.

<FIG> illustrates an exemplary process <NUM> for training an acoustic model according to an embodiment. The process <NUM> may be for training, e.g., the acoustic model in <FIG>, the acoustic model in <FIG>, etc. In the case of performing the process <NUM> for training the acoustic model, a style encoder in the acoustic model may be, e.g., a VAE, a GMVAE, etc., and may be obtained through an adversarial training mechanism.

Training data may be obtained first. Each piece of training data may comprise various types of information extracted from a reference audio. For example, <FIG> shows that a text <NUM>, a speaker ID <NUM>, a style ID <NUM>, acoustic features <NUM>, etc. corresponding to an exemplary reference audio are extracted from the reference audio. The text <NUM> is speech content in the reference audio. The speaker ID <NUM> is an identification of a speaker of the reference audio. The style ID <NUM> is an identification of a style adopted by the reference audio. The acoustic features <NUM> are extracted from the reference audio.

A text encoder <NUM> is trained for encoding the text <NUM> into a state sequence. A speaker LUT <NUM> may be used for generating a speaker embedding vector <NUM> based on the speaker ID <NUM>. A style encoder <NUM> may be trained based on, e.g., speaker ID, style ID, acoustic features <NUM>, etc., and output a style embedding vector <NUM> corresponding to the style of the reference audio. An extending module <NUM> may extend the state sequence output by the text encoder <NUM> with the speaker embedding vector <NUM> and the style embedding vector <NUM>. An attention module <NUM> may generate a context vector based at least on the extended state sequence. Optionally, the attention module <NUM> may generate a context vector based on the extended state sequence and an output of the previous step of a decoder. A decoder <NUM> may predict acoustic features <NUM> based at least on the context vector. Optionally, the decoder <NUM> may predict acoustic features based on the context vector and an output of the previous step of the decoder.

According to the process <NUM>, the style encoder <NUM> may be obtained through an adversarial training mechanism such as DAT. For example, an adversarial training module <NUM> may be used for implementing the adversarial training mechanism. During the generating of the style embedding vector <NUM> by the style encoder <NUM>, a reference embedding vector <NUM> may be obtained as an intermediate parameter. For example, the style encoder <NUM> may comprise a reference encoder formed by a convolutional neural network (CNN), a long short-term memory (LSTM) network, etc., which is used for generating the reference embedding vector <NUM> based on the acoustic features <NUM>. The reference embedding vector <NUM> generally has a high dimension and is designed for obtaining as much information as possible from the acoustic features <NUM>. Adversarial training may be performed on the reference embedding vector <NUM> in order to remove speaker information and retain style information. The style encoder <NUM> may further generate the style embedding vector <NUM> based on the reference embedding vector <NUM> being performed the adversarial training. For example, the style encoder <NUM> may include a full connection (FC) layer. The full connection layer may generate the style embedding vector <NUM> based on the reference embedding vector <NUM> being performed the adversarial training and the style ID <NUM>, or may generate the style embedding vector <NUM> based on the reference embedding vector <NUM> being performed the adversarial training, the style ID <NUM> and the speaker ID <NUM>. Compared with the reference embedding vector <NUM>, the style embedding vector <NUM> has a low dimension, and captures higher-level information about, e.g., speaking style.

In an implementation, the adversarial training module <NUM> may implement DAT with at least a speaker classifier <NUM> and a style classifier <NUM>. The speaker classifier <NUM> may generate a speaker classification result, e.g., prediction of probability of different speakers, based on input features, e.g., a reference embedding vector. The style classifier <NUM> may generate a style classification result, e.g., prediction of probability of different speaking style, based on input features e.g., a reference embedding vector. In one aspect, gradient reversal processing may be first performed on the reference embedding vector <NUM> through a gradient reversal layer at <NUM>, and then the speaker classifier <NUM> may generate a speaker classification result for the reference embedding vector being performed the gradient reversal processing. In another aspect, the style classifier <NUM> may generate a style classification result for the reference embedding vector <NUM>. The adversarial training module <NUM> may calculate a gradient back-propagation factor through a loss function. The loss function is based at least on a comparison result between the style classification result and the style ID <NUM> and a comparison result between the speaker classification result and the speaker ID <NUM>. In one aspect, the optimizing process that is based on the loss function may cause the speaker classification result predicted by the speaker classifier <NUM> for the input features to approximate the speaker ID <NUM>. Since the gradient reversal processing is performed on the reference embedding vector <NUM> before the speaker classifier <NUM>, the optimizing process is actually performed toward reducing information contained in the reference embedding vector <NUM> that helps the speaker classifier <NUM> to output a correct classification result, thereby achieving the removal of speaker information. In another aspect, the optimizing process that is based on the loss function may cause the style classification result predicted by the style classifier <NUM> for the input features to approximate the style ID <NUM>. The more accurate the classification result from the style classifier <NUM> is, the more information about style the reference embedding vector <NUM> includes, thereby achieving the retaining of style information.

The reference embedding vector <NUM> being performed the adversarial training will retain as much style information as possible, and remove as much speaker information as possible. Therefore, the style embedding vector <NUM> which is further generated based on the reference embedding vector <NUM> will also retain as much style information as possible and remove as much speaker information as possible The style embedding vector <NUM> may lead to subsequent high-quality acoustic features <NUM> and further high-quality synthesized speech.

Through the training by the process <NUM>, two types of acoustic models may be obtained, e.g., the generative acoustic model as shown in <FIG> and the acoustic model adopting at least the LUT technique as shown in <FIG>.

It should be understood that the training of the acoustic model in <FIG> may be deemed as a part of the training of the entire TTS system. For example, when training a TTS system including an acoustic model and a vocoder, the training process in <FIG> may be applied to the acoustic model in the TTS system.

As described above, the style encoder may adopt, e.g., VAE, GMVAE, etc. Therefore, in the training process <NUM> in <FIG>, the style embedding vector <NUM> may correspond to a prior distribution or a posterior distribution of a latent variable having Gaussian distribution or Gaussian mixture distribution. Further training details in the case that the style encoder adopts VAE or GMVAE will be discussed hereinafter in conjunction with <FIG>.

<FIG> illustrates an exemplary data flow <NUM> within a style encoder in a training stage according to an embodiment. The data flow <NUM> may be used for further illustrating the training mechanism when the style encoder <NUM> in <FIG> adopts the VAE.

As shown in <FIG>, input used for training the style encoder may comprise acoustic features <NUM>. The acoustic features <NUM> may be further provided to a reference encoder <NUM>.

The reference encoder <NUM> may encode the acoustic features <NUM> into a reference embedding vector <NUM>. In an embodiment, the reference encoder <NUM> may comprise, e.g., CNN, LSTM, etc. The reference embedding vector <NUM> may be passed to a full connection layer <NUM>, for determining characterization parameters of a Gaussian distribution of a latent variable z. For example, the full connection layer <NUM> may comprise two full connection layers for generating a mean value and a variance of the latent variable z respectively. The style embedding vector <NUM> may be obtained through, e.g., sampling the determined Gaussian distribution. The distribution determined by the full connection layer <NUM> may be deemed as a posterior distribution q of the latent variable z.

Based on the example of the data flow <NUM>, after the training is completed, the style encoder may generate a style embedding vector based on input acoustic features of a target style reference audio.

<FIG> illustrates an exemplary data flow <NUM> within a style encoder in a training stage according to an embodiment. The data flow <NUM> may be used for further illustrating the training mechanism when the style encoder <NUM> in <FIG> adopts the GMVAE.

As shown in <FIG>, input used for training the style encoder may comprise acoustic features <NUM>, a style ID <NUM>, an optional speaker ID <NUM>, etc. corresponding to a reference audio. When the training does not adopt the speaker ID <NUM>, the style encoder may be deemed as a speaker-independent style encoder. When the training adopts the speaker ID <NUM>, the style encoder may be deemed as a speaker-dependent style encoder.

The acoustic features <NUM> may be provided to a reference encoder <NUM>. Similar to the reference encoder <NUM> in <FIG>, the reference encoder <NUM> may encode the acoustic features <NUM> into a reference embedding vector <NUM>.

The style ID <NUM> may be provided to a style intermediate representation LUT <NUM> in order to output a corresponding style intermediate representation vector.

The reference embedding vector <NUM> and the style intermediate representation vector may be passed to a full connection layer <NUM>, for determining characterization parameters of a Gaussian mixture distribution of a latent variable z. For example, the full connection layer <NUM> may comprise two full connection layers for generating a mean value and a variance of the latent variable z respectively. A style embedding vector <NUM> may be obtained through sampling the determined Gaussian mixture distribution. The distribution determined by the full connection layer <NUM> may be deemed as a posterior distribution q of the latent variable z.

When the training input includes the speaker ID <NUM>, the speaker ID <NUM> may be provided to a speaker intermediate representation LUT <NUM> in order to output a corresponding speaker intermediate representation vector.

The style intermediate representation vector output by the style intermediate representation LUT <NUM> and the possible speaker intermediate representation vector output by the speaker intermediate representation LUT <NUM> may be passed to a full connection layer <NUM>, for determining characterization parameters of a Gaussian mixture distribution of a latent variable z. The distribution determined by the full connection layer <NUM> may be deemed as a prior distribution p of the latent variable z. It should be understood that, through using a plurality of training data for training, a plurality of prior distributions <NUM> may be finally obtained, wherein each prior distribution corresponds to a speaking style. Through sampling or averaging a prior distribution, a style embedding vector corresponding to the prior distribution may be obtained.

Based on the example of the data flow <NUM>, after the training is completed, the style encoder will have, e.g., an operating mode similar with the generative acoustic model shown in <FIG>, an operating mode similar with the acoustic model adopting at least the LUT technique shown in <FIG>, etc..

It should be understood that in <FIG>, depending on whether the style encoder adopts the VAE or the GMVAE, there exists corresponding computational constraints between a prior distribution p and a posterior distribution q of a latent variable z. Some details about the VAE and the GMVAE will be further discussed below.

The conventional VAE constructs a relationship between an unobservable continuous random latent variable z and an observable data set x. qΦ(z|x) is introduced as an approximation to the true posterior densitypθ(z|x)which is intractable. Following the variational principle, logpθ(x), as an optimization target, may be represented as: <MAT> wherein x is a data sample (e.g., acoustic features), z is a latent variable, a prior distribution pθ (z) over z is Gaussian distribution, and <IMG>(θ, Φ; x) is a variational lower boundary to be optimized. KL[qΦ(z|x)∥pθ(z)] may correspond to KL loss, and <MAT> may correspond to reconstruction loss.

When applying VAE to a TTS for style-related modeling, the training target of pure TTS and VAE may be merged as: <MAT> wherein Loss is the total loss, and the conditional reconstruction likelihood pθ(x|z) in Equation (<NUM>) is modified to depend on both the latent variable z and an input text t, i.e., pθ(x|z, t). Optionally, the stop token loss lstop of the pure TTS may also be included in the total loss.

The distribution of the latent variable z may be influenced by a style distribution variable corresponding to a speaking style and an optional speaker distribution variable corresponding to a speaker. The influence to the latent variable z by the speaking style will be discussed below by taking the GMVAE as an example.

In the GMVAE, the latent variable z is parameterized by a Gaussian mixture model. The main target to maximize is: <MAT> wherein x is a data sample, t is an input text, z is a latent variable with Gaussian mixture distribution, and mean value and variance of z are parameterized at least with a style distribution variable y corresponding to a speaking style.

When the model training includes the adversarial training shown in <FIG>, the total loss may be represented as: <MAT> wherein <IMG> is a variational lower boundary of the GMVAE-based TTS, as shown in Equation (<NUM>), Lstyle and Lspk are losses of a style classifier and a speaker classifier calculated by using, e.g., cross-entropy, respectively, and lstop is a stop token loss in the TTS calculated by using, e.g., cross-entropy.

It should be understood that the above parts only present examples of determining latent variable distributions in the VAE and the GMVAE, and these examples may be modified and supplemented in any approaches according to specific application requirements. For example, any of the above Equations (<NUM>) to (<NUM>) may be modified, so as to introduce a style distribution variable and/or a speaker distribution variable to influence the distribution of the latent variable z. For example, an introduction of the style distribution variable y is exemplarily presented in Equation (<NUM>), and a speaker distribution variable corresponding to a reference speaker may also be introduced into any of the above equations in a similar manner.

According to the embodiments of the present disclosure, a combination of paired input and unpaired input may be adopted during the training of an acoustic model, and a cyclic training mechanism may be adopted for the acoustic model to solve the problem of lack of ground truth labels in transfer outputs.

<FIG> illustrates an exemplary process <NUM> for training an acoustic model according to an embodiment. The process <NUM> may be for training, e.g., the acoustic model in <FIG>. In the process <NUM>, a cyclic training framework may be formed with an acoustic model <NUM>, which is a basic model, and a duplicate <NUM> of the acoustic model, and a style encoder and an acoustic model with higher-performance may be obtained at least through a cyclic training mechanism.

In <FIG>, the acoustic model <NUM> to be trained may comprise a text encoder <NUM>, an attention module <NUM>, a decoder <NUM>, an extending module <NUM>, a speaker LUT <NUM>, a style encoder <NUM>, etc. For the purpose of training, an additional style encoder <NUM> is also provided in <FIG>, however, it should be understood that after the acoustic model has been trained, the style encoder <NUM> may be omitted. The duplicate <NUM> of the acoustic model has the same or similar architecture, parameters, etc. as the acoustic model <NUM>. A text encoder <NUM>', an attention module <NUM>', a decoder <NUM>', an extending module <NUM>', a speaker LUT <NUM>', a style encoder <NUM>' and a style encoder <NUM>' in the duplicate <NUM> of the acoustic model may correspond to the text encoder <NUM>, the attention module <NUM>, the decoder <NUM>, the extending module <NUM>, the speaker LUT <NUM>, the style encoder <NUM> and the style encoder <NUM> in the acoustic model <NUM>, respectively. It should be understood that the text encoders, the attention modules, the decoders, the extending modules, the speaker LUT, the style encoders, etc. in <FIG> have similar functions with the corresponding components in <FIG>.

Training data may be obtained first. Each piece of training data may comprise various types of information extracted from a speaker reference audio and a style reference audio. The speaker reference audio is an audio from a target speaker of style transfer speech synthesis. The style reference audio is an audio with a target style of the style transfer speech synthesis. For example, <FIG> shows a text m <NUM>, a speaker A ID <NUM>, speaker reference acoustic features <NUM>, etc., extracted from an exemplary speaker reference audio <NUM>. The speaker reference audio <NUM> may be denoted as [spk_A, sty_a, m], wherein spk_A denotes a speaker A of the audio, sty_a denotes a style a of the audio, and m denotes the text m corresponding to the audio. The speaker reference acoustic features <NUM> refer to acoustic features extracted from the speaker reference audio <NUM>. <FIG> further shows a text n <NUM>, a speaker B ID <NUM>, style reference acoustic features <NUM>, etc., extracted from an exemplary style reference audio <NUM>. The style reference audio <NUM> may be denoted as [spk_B, sty_b, n], wherein spk_B denotes a speaker B of the audio, sty_b denotes a style b of the audio, and n denotes the text n corresponding to the audio. The style reference acoustic features <NUM> refer to acoustic features extracted from the style reference audio <NUM>.

The text m <NUM> and the speaker reference audio <NUM>, or the text m <NUM> and the speaker reference acoustic features <NUM> extracted from the speaker reference audio <NUM>, may be used as a paired input to the acoustic model <NUM>, for predicting a paired output. For example, the text encoder <NUM> may encode the text m <NUM> into a state sequence corresponding to the text m. The speaker LUT <NUM> may generate a speaker embedding vector <NUM> corresponding to the speaker A based on the speaker A ID <NUM>. The style encoder <NUM> may generate a speaker style embedding vector <NUM> corresponding to the style a based at least on the speaker reference acoustic features <NUM>. The extending module <NUM> may extend the state sequence of the text m output by the text encoder <NUM> with the speaker embedding vector <NUM> and the speaker style embedding vector <NUM>. The decoder <NUM> may predict first paired acoustic features <NUM> at least under the influence of the attention module <NUM>. The first paired acoustic features <NUM> adopt the speaker A's voice, adopt the style a, and are directed to the text m, and thus may be denoted as [spk_A, sty_a, m]. The first paired acoustic features <NUM> are a paired output by the acoustic model <NUM>. It may be seen that, through the acoustic model <NUM>, the first paired acoustic features <NUM> may be generated based at least on the text m <NUM>, the speaker A ID <NUM>, and the speaker style embedding vector <NUM> corresponding to the style a.

The text m <NUM> and the style reference audio <NUM>, or the text m <NUM> and the style reference acoustic features <NUM> extracted from the style reference audio <NUM>, may be used as an unpaired input to the acoustic model <NUM>, for predicting an unpaired output. The style encoder <NUM> may generate a transfer style embedding vector <NUM> corresponding to the style b based at least on the style reference acoustic features <NUM>. The extending module <NUM> may use the speaker embedding vector <NUM> and the transfer style embedding vector <NUM> for extending the state sequence of the text m output by the text encoder <NUM>. The decoder <NUM> may predict first transfer acoustic features <NUM> at least under the influence of the attention module <NUM>. The first transfer acoustic features <NUM> adopt the speaker A's voice, adopt the style b, and are directed to the text m, and thus may be denoted as [spk_A, sty_b, m]. The first transfer acoustic features <NUM> are an unpaired output by the acoustic model <NUM>. It may be seen that, through the acoustic model <NUM>, the first transfer acoustic features <NUM> may be generated based at least on the text m <NUM>, the speaker A ID <NUM>, and the transfer style embedding vector <NUM> corresponding to the style b.

The speaker reference acoustic features <NUM> corresponding to the speaker reference audio <NUM> in the training data may be used as a ground truth label for the first paired acoustic features <NUM>, so that the speaker reference acoustic features <NUM> and the first paired acoustic features <NUM> may be used for calculating loss metrics, e.g., reconstruction loss, etc. However, there is no ground truth label for the first transfer acoustic features <NUM> in the training data, and thus, loss metrics for the first transfer acoustic features <NUM> cannot be calculated effectively. In view of this situation, the process <NUM> further introduces the duplicate <NUM> of the acoustic model to solve the problem of difficulty in calculating loss metrics for the transfer output.

The text n <NUM> and the style reference audio <NUM>, or the text n <NUM> and the style reference acoustic features <NUM> extracted from the style reference audio <NUM>, may be used as a paired input to the duplicate <NUM> of the acoustic model, for predicting a paired output. For example, the text encoder <NUM>' may encode the text n <NUM> into a state sequence corresponding to the text n. The speaker LUT <NUM>' may generate a speaker embedding vector <NUM> corresponding to the speaker B based on the speaker B ID <NUM>. The style encoder <NUM>' may generate a speaker style embedding vector <NUM> corresponding to the style b based at least on the style reference acoustic features <NUM>. The extending module <NUM>' may extend the state sequence of the text n output by the text encoder <NUM>' with the speaker embedding vector <NUM> and the speaker style embedding vector <NUM>. The decoder <NUM>' may predict second paired acoustic features <NUM> at least under the influence by the attention module <NUM>'. The second paired acoustic features <NUM> adopt the speaker B's voice, adopt the style b, and are directed to the text n, and thus may be denoted as [spk_B, sty_b, n]. The second paired acoustic features <NUM> are a paired output by the duplicate <NUM> of the acoustic model. It may be seen that, through the duplicate <NUM> of the acoustic model, the second paired acoustic features <NUM> may be generated based at least on the text n <NUM>, the speaker B ID <NUM>, and the speaker style embedding vector <NUM> corresponding to the style b.

The text n <NUM> and the first transfer acoustic features <NUM> may be used as an unpaired input to the duplicate <NUM> of the acoustic model, for predicting an unpaired output. The style encoder <NUM>' may generate a transfer style embedding vector <NUM> corresponding to the style b based at least on the first transfer acoustic features <NUM>. The extending module <NUM>' may use the speaker embedding vector <NUM> and the transfer style embedding vector <NUM> for extending the state sequence of the text n output by the text encoder <NUM>'. The decoder <NUM>' may predict second transfer acoustic features <NUM> at least under the influence of the attention module <NUM>'. The second transfer acoustic features <NUM> adopt the speaker B's voice, adopt the style b, and are directed to the text n, and thus may be denoted as [spk_B, sty_b, n]. The second transfer acoustic features <NUM> are an unpaired output by the duplicate <NUM> of the acoustic model. It may be seen that, through the duplicate <NUM> of the acoustic model, the second transfer acoustic features <NUM> may be generated based at least on the text n <NUM>, the speaker B ID <NUM>, and the transfer style embedding vector <NUM> corresponding to the style b.

The style reference acoustic features <NUM> of the style reference audio <NUM> may be used as a ground truth label for the second paired acoustic features <NUM>, and thus the style reference acoustic features <NUM> and the second paired acoustic features <NUM> may be used for calculating loss metrics, e.g., reconstruction loss, etc. Moreover, the style reference acoustic features <NUM> of the style reference audio <NUM> in the training data may be used as a ground truth label for the second transfer acoustic features <NUM>, and thus the style reference acoustic features <NUM> and the second transfer acoustic features <NUM> may be used for calculating loss metrics, e.g., cyclic reconstruction loss <NUM>. The cyclic reconstruction loss <NUM> is a reconstruction loss calculated according to the cyclic training process in <FIG>.

Through training the acoustic model according to the process <NUM>, since both paired inputs and unpaired inputs are adopted during the training, even if there are unpaired inputs in the synthesis stage, high-quality cross-speaker style transfer may still be achieved. Moreover, since the cyclic training process determines ground truth labels for transfer outputs, which may be used for calculating loss metrics, the performance of the trained acoustic model may be greatly enhanced.

It should be understood that the loss metrics considered in the process <NUM> are not limited to the above-mentioned reconstruction loss and cyclic reconstruction loss, and any other loss metrics may also be considered. Moreover, the above cyclic training mechanism is not limited by whether the training data has style labels, i.e., it is not required to label styles in the training data. Moreover, the specific implementation of the style encoder in <FIG> is not limited in any approaches, and it may be a VAE, a GMVAE or any other encoder that can be used for generating a style embedding vector. Moreover, the adversarial training process in <FIG> may also be combined into the process <NUM> in <FIG>. For example, the adversarial training mechanism implemented by the adversarial training module <NUM> in <FIG> is further applied to the style encoder in <FIG>.

<FIG> illustrates a flowchart of an exemplary method <NUM> for training an acoustic model according to an embodiment. The acoustic model may be for implementing cross-speaker style transfer and comprise at least a style encoder. The method <NUM> may be based at least on, e.g., the exemplary training processes discussed in <FIG>.

At <NUM>, training data may be obtained, the training data comprising a text, a speaker ID, a style ID and acoustic features corresponding to a reference audio.

At <NUM>, a reference embedding vector may be generated, through the style encoder, based on the acoustic features.

At <NUM>, adversarial training may be performed to the reference embedding vector with at least the style ID and the speaker ID, to remove speaker information and retain style information.

At <NUM>, a style embedding vector may be generated, through the style encoder, based at least on the reference embedding vector being performed the adversarial training.

At <NUM>, predicted acoustic features may be generated based at least on a state sequence corresponding to the text, a speaker embedding vector corresponding to the speaker ID, and the style embedding vector.

In an implementation, the generating a reference embedding vector may comprise: generating the reference embedding vector based on the acoustic features through a CNN and a LSTM network in the style encoder.

In an implementation, the performing adversarial training may comprise: generating, through a style classifier, a style classification result for the reference embedding vector; performing gradient reversal processing to the reference embedding vector; generating, through a speaker classifier, a speaker classification result for the reference embedding vector being performed the gradient reversal processing; and calculating a gradient back-propagation factor through a loss function, the loss function being based at least on a comparison result between the style classification result and the style ID and a comparison result between the speaker classification result and the speaker ID.

In an implementation, the adversarial training may be performed by a DAT module.

In an implementation, the generating a style embedding vector may comprise: generating, through a full connection layer in the style encoder, the style embedding vector based at least on the reference embedding vector being performed the adversarial training, or based at least on the reference embedding vector being performed the adversarial training and the style ID.

Moreover, the generating a style embedding vector may comprise: generating, through a second full connection layer in the style encoder, the style embedding vector based at least on the style ID, or based at least on the style ID and the speaker ID.

In an implementation, the style encoder may be a VAE or a GMVAE.

In an implementation, the style embedding vector may correspond to a prior distribution or a posterior distribution of a latent variable having Gaussian distribution or Gaussian mixture distribution.

In an implementation, the method <NUM> may further comprise: obtaining a plurality of style embedding vectors corresponding to a plurality of style IDs respectively, or obtaining a plurality of style embedding vectors corresponding to a plurality of combinations of style ID and speaker ID respectively, through training the acoustic model with a plurality of training data.

In an implementation, the method <NUM> may further comprise: encoding the text into the state sequence through a text encoder in the acoustic model; and generating the speaker embedding vector through a speaker LUT in the acoustic model. The generating predicted acoustic features may comprise: extending the state sequence with the speaker embedding vector and the style embedding vector; generating, through an attention module in the acoustic model, a context vector based at least on the extended state sequence; and generating, through a decoder in the acoustic model, the predicted acoustic features based at least on the context vector.

In an implementation, the method <NUM> may further comprise, during applying the acoustic model: receiving an input, the input comprising a target text, a target speaker ID, and a target style reference audio and/or a target style ID; generating, through the style encoder, a style embedding vector based at least on acoustic features of the target style reference audio and/or the target style ID; and generating acoustic features based at least on the target text, the target speaker ID and the style embedding vector.

Moreover, the input may further comprise a reference speaker ID. The generating a style embedding vector may be further based on the reference speaker ID.

In an implementation, the method <NUM> may further comprise, during applying the acoustic model: receiving an input, the input comprising a target text, a target speaker ID, and a target style ID; selecting, through the style encoder, a style embedding vector from a plurality of predetermined candidate style embedding vectors based at least on the target style ID; and generating acoustic features based at least on the target text, the target speaker ID and the style embedding vector.

Moreover, the input may further comprise a reference speaker ID. The selecting a style embedding vector may be further based on the reference speaker ID.

In an implementation, the acoustic features may be mel-spectrum extracted from the reference audio.

It should be understood that the method <NUM> may further comprise any step/process for training an acoustic model according to the embodiments of the present disclosure described above.

<FIG> illustrates a flowchart of an exemplary method <NUM> for training an acoustic model according to an embodiment. The acoustic model may be for implementing cross-speaker style transfer and comprise at least a style encoder. The method <NUM> may be based at least on, e.g., the exemplary training process discussed in <FIG>.

At <NUM>, training data may be obtained, the training data at least comprising a first text, a first speaker ID, and a second text, a second speaker ID and style reference acoustic features corresponding to a style reference audio.

At <NUM>, first transfer acoustic features may be generated, through the acoustic model, based at least on the first text, the first speaker ID and a first transfer style embedding vector, wherein the first transfer style embedding vector is generated by the style encoder based on the style reference acoustic features.

At <NUM>, second transfer acoustic features may be generated, through a duplicate of the acoustic model, based at least on the second text, the second speaker ID and a second transfer style embedding vector, wherein the second transfer style embedding vector is generated by a duplicate of the style encoder based on the first transfer acoustic features.

At <NUM>, cyclic reconstruction loss may be calculated with the style reference acoustic features and the second transfer acoustic features.

In an implementation, the first text and the first speaker ID may correspond to a speaker reference audio, and the training data may further comprise speaker reference acoustic features corresponding to the speaker reference audio.

In the foregoing implementation, the method <NUM> may further comprise: generating, through the acoustic model, first paired acoustic features based at least on the first text, the first speaker ID and a first speaker style embedding vector, wherein the first speaker style embedding vector is generated by an additional style encoder based on the speaker reference acoustic features; and calculating reconstruction loss with the speaker reference acoustic features and the first paired acoustic features. Further, the first text and the style reference acoustic features may be an unpaired input to the acoustic model, and the first text and the speaker reference acoustic features may be a paired input to the acoustic model.

In the foregoing implementation, the method <NUM> may further comprise: generating, through the duplicate of the acoustic model, second paired acoustic features based at least on the second text, the second speaker ID and a second speaker style embedding vector, wherein the second speaker style embedding vector is generated by a duplicate of the additional style encoder based on the style reference acoustic features; and calculating reconstruction loss with the style reference acoustic features and the second paired acoustic features. Further, the second text and the first transfer acoustic features may be an unpaired input to the duplicate of the acoustic model, and the second text and the style reference acoustic features may be a paired input to the duplicate of the acoustic model.

In an implementation, the style encoder may be obtained through an adversarial training for removing speaker information and retaining style information.

In an implementation, the style reference acoustic features may be a ground truth label for calculating the cyclic reconstruction loss.

In an implementation, the method <NUM> may further comprise, during applying the acoustic model: receiving an input comprising a target text, a target speaker ID and a target style reference audio, the target style reference audio corresponding to a text different from the target text and/or a speaker ID different from the target speaker ID; generating, through the style encoder, a style embedding vector based on the target style reference audio; and generating acoustic features based at least on the target text, the target speaker ID and the style embedding vector.

<FIG> illustrates an exemplary apparatus <NUM> for training an acoustic model according to an embodiment. The acoustic model may be for implementing cross-speaker style transfer and comprise at least a style encoder.

The apparatus <NUM> may comprise: a training data obtaining module <NUM>, for obtaining training data, the training data comprising a text, a speaker ID, a style ID and acoustic features corresponding to a reference audio; a reference embedding vector generating module <NUM>, for generating, through the style encoder, a reference embedding vector based on the acoustic features; an adversarial training performing module <NUM>, for performing adversarial training to the reference embedding vector with at least the style ID and the speaker ID, to remove speaker information and retain style information; a style embedding vector generating module <NUM>, for generating, through the style encoder, a style embedding vector based at least on the reference embedding vector being performed the adversarial training; and an acoustic feature generating module <NUM>, for generating predicted acoustic features based at least on a state sequence corresponding to the text, a speaker embedding vector corresponding to the speaker ID, and the style embedding vector.

In an implementation, the adversarial training performing module <NUM> may be for: generating, through a style classifier, a style classification result for the reference embedding vector; performing gradient reversal processing to the reference embedding vector; generating, through a speaker classifier, a speaker classification result for the reference embedding vector being performed the gradient reversal processing; and calculating a gradient back-propagation factor through a loss function, the loss function being based at least on a comparison result between the style classification result and the style ID and a comparison result between the speaker classification result and the speaker ID.

In an implementation, the style embedding vector generating module <NUM> may be for: generating, through a full connection layer in the style encoder, the style embedding vector based at least on the reference embedding vector being performed the adversarial training, or based at least on the reference embedding vector being performed the adversarial training and the style ID.

In an implementation, the style embedding vector generating module <NUM> may be for: generating, through a second full connection layer in the style encoder, the style embedding vector based at least on the style ID, or based at least on the style ID and the speaker ID.

Moreover, the apparatus <NUM> may further comprise any other module that performs the steps of the methods for training an acoustic model (e.g., the method <NUM> in <FIG>, etc.) according to the embodiments of the present disclosure described above.

The apparatus <NUM> may comprise: a training data obtaining module <NUM>, for obtaining training data, the training data at least comprising a first text, a first speaker ID, and a second text, a second speaker ID and style reference acoustic features corresponding to a style reference audio; a first transfer acoustic features generating module <NUM>, for generating, through the acoustic model, first transfer acoustic features based at least on the first text, the first speaker ID and a first transfer style embedding vector, wherein the first transfer style embedding vector is generated by the style encoder based on the style reference acoustic features; a second transfer acoustic features generating module <NUM>, for generating, through a duplicate of the acoustic model, second transfer acoustic features based at least on the second text, the second speaker ID and a second transfer style embedding vector, wherein the second transfer style embedding vector is generated by a duplicate of the style encoder based on the first transfer acoustic features; and a cyclic reconstruction loss calculating module <NUM>, for calculating cyclic reconstruction loss with the style reference acoustic features and the second transfer acoustic features.

In the foregoing implementation, the apparatus <NUM> may further comprise: a first paired acoustic features generating module, for generating, through the acoustic model, first paired acoustic features based at least on the first text, the first speaker ID and a first speaker style embedding vector, wherein the first speaker style embedding vector is generated by an additional style encoder based on the speaker reference acoustic features; and a reconstruction loss calculating module, for calculating reconstruction loss with the speaker reference acoustic features and the first paired acoustic features. Further, the first text and the style reference acoustic features may be an unpaired input to the acoustic model, and the first text and the speaker reference acoustic features may be a paired input to the acoustic model.

In the foregoing implementation, the apparatus <NUM> may further comprise: a second paired acoustic features generating module, for generating, through the duplicate of the acoustic model, second paired acoustic features based at least on the second text, the second speaker ID and a second speaker style embedding vector, wherein the second speaker style embedding vector is generated by a duplicate of the additional style encoder based on the style reference acoustic features; and a reconstruction loss calculating module, for calculating reconstruction loss with the style reference acoustic features and the second paired acoustic features. Further, the second text and the first transfer acoustic features may be an unpaired input to the duplicate of the acoustic model, and the second text and the style reference acoustic features may be a paired input to the duplicate of the acoustic model. Further, the style encoder may be a VAE or a GMVAE. Further, the style encoder may be obtained through an adversarial training for removing speaker information and retaining style information. Further, the style reference acoustic features may be a ground truth label for calculating the cyclic reconstruction loss.

The apparatus <NUM> may comprise: at least one processor <NUM>; and a memory <NUM> storing computer-executable instructions that, when executed, cause the at least one processor <NUM> to perform any step/process of the methods for training an acoustic model (e.g., the method <NUM> in <FIG>, the method <NUM> in <FIG>, etc.) according to the embodiments of the present disclosure described above.

The embodiments of the present disclosure may be embodied in a non-transitory computer-readable medium. The non-transitory computer-readable medium may comprise instructions that, when executed, cause one or more processors to perform any operations of the methods for training an acoustic model according to the embodiments of the present disclosure described above.

It should be understood that all the operations in the methods described above are merely exemplary, and the present disclosure is not limited to any operations in the methods or sequence orders of these operations.

It should also be understood that all the modules in the apparatuses described above may be implemented in various approaches. These modules may be implemented as hardware, software, or a combination thereof. Moreover, any of these modules may be further functionally divided into sub-modules or combined together.

Processors are described in connection with various apparatus and methods. These processors may be implemented using electronic hardware, computer software, or any combination thereof. Whether these processors are implemented as hardware or software will depend on the specific application and the overall design constraints imposed on the system. By way of example, a processor, any portion of a processor, or any combination of processors presented in this disclosure may be implemented as a microprocessor, a micro-controller, a digital signal processor (DSP), a field programmable gate array (FPGA), a programmable logic device (PLD), state machine, gate logic, discrete hardware circuitry, and other suitable processing components configured to perform the various functions described in this disclosure. The functions of a processor, any portion of a processor, or any combination of processors presented in this disclosure may be implemented as software executed by a microprocessor, a micro-controller, a DSP, or other suitable platforms.

Software should be considered broadly to represent instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, applications, software applications, software packages, routines, subroutines, objects, running threads, processes, functions, etc. Software may reside on computer readable medium. Computer readable medium may include, e.g., a memory, which may be, e.g., a magnetic storage device (e.g., a hard disk, a floppy disk, a magnetic strip), an optical disk, a smart card, a flash memory device, a random access memory (RAM), a read only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), a register, or a removable disk. Although a memory is shown as being separate from the processor in various aspects presented in this disclosure, a memory may also be internal to the processor (e.g., a cache or a register).

Claim 1:
A method (<NUM>) for training an acoustic model, the acoustic model being for implementing cross-speaker style transfer and comprising at least a style encoder, the method comprising:
obtaining (<NUM>) training data, the training data comprising a text, a speaker identity (ID), a style ID and acoustic features corresponding to a reference audio;
generating (<NUM>), through the style encoder, a reference embedding vector based on the acoustic features;
performing (<NUM>) adversarial training to the reference embedding vector with at least the style ID and the speaker ID, to remove speaker information and retain style information;
generating (<NUM>), through the style encoder, a style embedding vector based at least on the reference embedding vector being performed the adversarial training; and
generating (<NUM>) predicted acoustic features based at least on a state sequence corresponding to the text, a speaker embedding vector corresponding to the speaker ID, and the style embedding vector.