Patent Description:
Speech synthesis systems use text-to-speech (TTS) models to generate speech from textual input. The generated/synthesized speech should accurately convey the message (intelligibility) while sounding like human speech (naturalness) with an intended prosody (expressiveness). Speech synthesis systems have evolved from concatenative and parametric synthesis models to models with neural networks. This evolution has significantly improved the naturalness of synthesized speech, but TTS models today are still unable to fully preserve the input text fed into the TTS model to generate the synthesized speech. That is, most TTS models, even neural network TTS models, translate the input text into a phoneme representation without preserving the corresponding graphemes. Due to the lack of graphemes, TTS models cannot take advantage of the relationship between phonemes and graphemes to further improve the functionality of TTS models.

<NPL>", describes a semi-supervised training framework for the end-to-end text-to-speech (TTS) model, Tacotron. It describes allowing Tacotron to utilize textual and acoustic knowledge contained in large, publicly-available text and speech corpora. These external data are unpaired and potentially noisy. They embed each word in the input text into word vectors and condition the Tacotron encoder on them. They use an unpaired speech corpus to pre-train the Tacotron decoder in the acoustic domain. They fine-tune the model using available paired data. They demonstrate that the proposed framework enables Tacotron to generate intelligible speech using less than half an hour of paired training data.

<NPL>", describes a method for combining multiple types of linguistic information in a single encoder, named representation mixing, enabling flexible choice between character, phoneme, or mixed representations during inference.

One aspect of the disclosure provides a computer-implemented method for implementing phenome and grapheme tokens for neural text-to-speech synthesis as defined by claim <NUM>. The computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations including receiving, at an encoder of a speech synthesis model, a text input including a sequence of words represented as an input encoder embedding. The input encoder embedding includes a plurality of tokens, with the plurality of tokens including a first set of grapheme tokens representing the text input as respective graphemes and a second set of phoneme tokens representing the text input as respective phonemes. The operations also include, for each respective phoneme token of the second set of phoneme tokens: identifying a respective word of the sequence of words corresponding to the respective phoneme token and determining a respective grapheme token representing the respective word of the sequence of words corresponding to the respective phoneme token. The operations also include generating, by the encoder, an output encoder embedding based on a relationship between each respective phoneme token and the corresponding grapheme token determined to represent a same respective word as the respective phoneme token.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, identifying, by the encoder, the respective word of the sequence of words corresponding to the respective phoneme token includes identifying a position in the respective word corresponding to the respective phoneme token and determining, by the encoder, the respective grapheme token representing the respective word of the sequence of words corresponding to the respective phoneme token includes determining the respective grapheme token representing the respective word of the sequence of words corresponding to the respective phoneme token at the position in the respective word corresponding to the respective phoneme token.

In some examples, each token of the plurality of tokens of the input encoder embedding represents a combination of one of a grapheme token embedding or a phoneme token embedding, a segment embedding, a word position embedding, and/or a position embedding. In these examples, identifying the respective word of the sequence of words corresponding to the respective phoneme token may include identifying the respective word of the sequence of words corresponding to the respective phoneme token based on a respective word position embedding associated with the respective phoneme token. Here, determining the respective grapheme token representing the respective word of sequence of words corresponding to the respective phoneme token may include determining that the respective grapheme token includes a corresponding word position embedding that matches the respective word position embedding of the respective phoneme token.

In some implementations, the speech synthesis model includes an attention mechanism in communication with the encoder. The speech synthesis model may include a duration-based upsampler in communication with the encoder. Further, the plurality of tokens of the input encoder embedding may include a special token identifying a language of the input text.

According to the invention, the operations also include pre-training the encoder of the speech synthesis model by: feeding the encoder a plurality of training examples, each training example represented as a sequence of training grapheme tokens corresponding to a training sequence of words and a sequence of training phoneme tokens corresponding to the same training sequence of words; masking a training phoneme token from the sequence of training phoneme tokens for a respective word from the training sequence of words; and masking a training grapheme token from the sequence of training phoneme tokens for the respective word from the training sequence of words.

In some implementations, the speech synthesis model includes a multilingual speech synthesis model and the operations further include pre-training the encoder of the speech synthesis model using a classification objective to predict a classification token of the plurality of tokens of the input encoder embedding, the classification token including a language identifier. In other implementations, the speech synthesis model includes a multilingual speech synthesis model and the output encoder embedding includes a sequence of encoder tokens, wherein each encoder token includes language information about the input text.

In still other implementations, the speech synthesis model includes a multi-accent speech synthesis model and the operations further include pre-training the encoder of the speech synthesis model using a classification objective to predict a classification token of the plurality of tokens of the input encoder embedding, wherein the classification token includes an accent identifier.

Another aspect of the disclosure provides a system for implementing phenome and grapheme tokens for neural text-to-speech synthesis as defined in claim <NUM>. The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that when executed on the data processing hardware causes the data processing hardware to perform operations including receiving, at an encoder of a speech synthesis model, a text input including a sequence of words represented as an input encoder embedding. The input encoder embedding includes a plurality of tokens, with the plurality of tokens including a first set of grapheme tokens representing the text input as respective graphemes and a second set of phoneme tokens representing the text input as respective phonemes. The operations also include, for each respective phoneme token of the second set of phoneme tokens: identifying a respective word of the sequence of words corresponding to the respective phoneme token and determining a respective grapheme token representing the respective word of the sequence of words corresponding to the respective phoneme token. The operations also include generating, by the encoder, an output encoder embedding based on a relationship between each respective phoneme token and the corresponding grapheme token determined to represent a same respective word as the respective phoneme token.

This aspect may include one or more of the following optional features. In some implementations, identifying, by the encoder, the respective word of the sequence of words corresponding to the respective phoneme token includes identifying a position in the respective word corresponding to the respective phoneme token and determining, by the encoder, the respective grapheme token representing the respective word of the sequence of words corresponding to the respective phoneme token includes determining the respective grapheme token representing the respective word of the sequence of words corresponding to the respective phoneme token at the position in the respective word corresponding to the respective phoneme token.

According to the invention , the also include pre-training the encoder of the speech synthesis model by: feeding the encoder a plurality of training examples, each training example represented as a sequence of training grapheme tokens corresponding to a training sequence of words and a sequence of training phoneme tokens corresponding to the same training sequence of words; masking a training phoneme token from the sequence of training phoneme tokens for a respective word from the training sequence of words; and masking a training grapheme token from the sequence of training phoneme tokens for the respective word from the training sequence of words.

Fast-paced development of neural end-to-end text-to-speech (TTS) synthesis has enabled the generation of speech to approach human levels of naturalness. The neural network of these TTS systems generally includes an encoder that encodes an input text representation into hidden states and a decoder that decodes spectrogram frames or waveform samples from the hidden states. These TTS systems may then use either an attention or a duration-based upsampler to connect the encoder to the decoder.

During development of these TTS systems, the form of the input to the neural network that represents the text has evolved. For example, early TTS systems would receive purely characters of the input text as the input to the neural network. Yet over time, it was discovered that TTS systems may achieve better stability if, instead of purely characters, the input to the neural network was phonemes (i.e., how a text representation is pronounced). Unfortunately, phoneme-based TTS models are not without their setbacks. For instance, one obvious issue with phoneme-based models occurs when a pronunciation of two words is the same, but these two words actually have different meaning; that is, a homophone. To illustrate, the sentence "To cancel the payment, press one; or to continue, two, " is an example of a homophone, which may be frequently used by conversational AI agents for call centers. In the phoneme representation of this sentence, the trailing ". , two, " can easily be confused with ". , too, " since "too" is a word that occurs more frequently in regular English. That is, a predictive system is more likely to output the word "too" even though "two" is the intention. In this example, different prosodies are expected in natural speech at the comma position in the two patterns. A moderate pause is expected at the comma in the case of ". , two" while often there is no pause for the pattern ". " When a TTS model is phoneme-based, the phoneme inputs to the neural network for the textual representation of this example sentence lack the written context of the sentence to ensure a consistently accurate output; an output where the synthesized speech has a pause after the comma to enable the synthesized speech to sound like the input sentence.

To overcome such setbacks with phoneme-based models, it would therefore be advantageous to have the input to the neural network of the TTS model include both phonemes and graphemes. That is, the input is a representation of the pronunciation of the text for the input (i.e., a phoneme) as well as a representation of how the text is written (i.e., a grapheme). Ideally, the inclusion of graphemes would reduce or eliminate issues stemming from pronunciation ambiguity (e.g., homophones). Yet, producing an input for a neural network that includes both phonemes and graphemes is not as straightforward as it may sound. One complexity is that a phoneme and a grapheme may represent the same content, but at varying lengths. For example, a word in a sentence may have two subwords or graphemes, but only a single phoneme or, in a converse example, a word in a sentence may have one subword or grapheme (e.g., the subword is the same as the word) and two phonemes. Therefore, a first sequence of phonemes representing a sentence would have inherent alignment issues with a second sequence of graphemes that represent the same sentence.

Even though this problem exists, some approaches have tried to combine phoneme and grapheme representations for the input. Instead of proposing a solution to this alignment issue, these approaches tend to generally avoid the issue. For example, one approach is to combine the phoneme and grapheme representations at the word level by concatenating grapheme-based embeddings with phoneme embeddings. Since these approaches do not handle the alignment challenges between phonemes and grapheme-based tokens (i.e., at the sub-word level), these approaches do not fully exploit the phoneme-grapheme relationship. Meaning that, these approaches limit the potential that the incorporation of graphemes may offer the TTS model (e.g., in terms of accuracy).

To address the issues with previous approaches, the current technique described herein accounts for the alignment (or misalignment) between phonemes and graphemes. More particularly, the approach represents the alignment relationship between phoneme tokens and grapheme tokens (i.e., at the sub-word or tokenized level) over the entirety of the text represented as the input to the neural network (i.e., the input to the encoder of the neural network). This approach is similar to an encoder for a Bidirectional Encoder Representations from Transformers (BERT) model (e.g., described in Devlin, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, available at https://arxiv. org/pdf/<NUM>. pdf, and incorporated herein by reference). In a traditional BERT model, the encoder received inputs corresponding to multiple sentences (i.e., segments) that are identified by a segment identifier (ID). The input to the BERT represented the multiple sentences as a sum of a phoneme-based token embedding, a segment embedding, and a position embedding. Although similar in some respects, the current technique may be considered an augmented BERT or PnG BERT because this technique includes phonemes and graphemes for the token embedding. To account for the graphemes at the token-level, the augmented BERT not only includes the token embedding, the segment embedding, and the position embedding, of the traditional BERT, but also includes a word-position embedding that provides word-level alignment between phonemes and graphemes. Therefore, the input to the augmented BERT is a representation of four types of embeddings while the traditional BERT only included three types of embeddings without any representation for the graphemes.

Since the augmented BERT approach represents the phoneme-grapheme relationship as an input to the encoder of the neural network, this augmented BERT encoder (also referred to as an augmented encoder) may be used in different types of TTS models. That is, since neural networks are typically encoder-decoder structures with either attention or duration-based upsampling, the augmented encoder may replace encoders in other encoder-decoder structures for TTS models. This means that the augmented encoder is compatible with both attention-based neural networks and duration-based neural networks without any significant modification. Therefore, functionally speaking, the augmented encoder may be used in monolingual TTS models, locale TTS models, multilingual TTS models, and/or multi-accent TTS models.

Another reason that the current technique builds on the BERT model is that BERT is a model architecture that uses pre-training to improve its natural language processing. Generally speaking, that pre-training is self-supervised pre-training on a large text corpora using objectives, such as a language model (LM) or a masked-language model (MLM). For a traditional BERT, the pre-training was performed only on graphemes (at the sub-word level) or at the sentence level. Pre-training is traditionally not done on phonemes. Since the augmented BERT is able to model a relationship between the phonemes and graphemes, this relationship may also be translated to pre-training such that the augmented BERT may pre-train on both phonemes and graphemes.

Referring to <FIG>, in some implementations, a speech environment <NUM> includes a user <NUM> communicating a spoken utterance <NUM> to a speech-enabled device <NUM> (also referred to as a device <NUM> or a user device <NUM>). The user <NUM> (i.e., speaker of the utterance <NUM>) may speak the utterance <NUM> as a query or a command to solicit a response from the device <NUM> or to have the device <NUM> execute a task specified by the query. The device <NUM> is configured to capture sounds from one or more users <NUM> within the speech environment <NUM>. Here, the audio sounds may refer to a spoken utterance <NUM> by the user <NUM> that functions as an audible query, a command for the device <NUM>, or an audible communication captured by the device <NUM>. Speech-enabled systems of the device <NUM> or associated with the device <NUM> (e.g., a digital assistant interface) may field the query for the command by answering the query and/or causing the command to be performed.

Here, the device <NUM> captures audio data <NUM> corresponding to the spoken utterance <NUM> by the user <NUM>. The device <NUM> may correspond to any computing device associated with the user <NUM> and capable of receiving audio data <NUM>. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smart watches), music player, casting devices, smart appliances (e.g., smart televisions) and internet of things (IoT) devices, remote controls, smart speakers, etc. The device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations related to speech and/or text processing. In some examples, the device <NUM> includes one or more applications (i.e., software applications) where each application may utilize one or more speech processing systems <NUM>, <NUM>, <NUM> associated with device <NUM> to perform various functions within the application. For instance, the device <NUM> includes an assistant application configured to communicate synthesized playback audio <NUM> (also referred to as synthesized speech <NUM>) to the user <NUM> to assist the user <NUM> with various tasks.

The device <NUM> further includes an audio subsystem with an audio capturing device (e.g., a microphone) <NUM> for capturing and converting audio data <NUM> within the speech environment <NUM> into electrical signals and a speech output device (e.g., a speaker) <NUM> for communicating an audible audio signal (e.g., a synthesized playback signal <NUM> from the device <NUM>). While the device <NUM> implements a single audio capturing device <NUM> in the example shown, the device <NUM> may implement an array of audio capturing devices <NUM> without departing from the scope of the present disclosure, whereby one or more audio capturing devices <NUM> in the array may not physically reside on the device <NUM>, but be in communication with the audio subsystem (e.g., peripherals of the device <NUM>). For example, the device <NUM> may correspond to a vehicle infotainment system that leverages an array of microphones positioned throughout the vehicle.

Furthermore, the device <NUM> is configured to communicate via a network <NUM> with a remote system <NUM>. The remote system <NUM> may include remote resources <NUM>, such as remote data processing hardware <NUM> (e.g., remote servers or CPUs) and/or remote memory hardware <NUM> (e.g., remote databases or other storage hardware). The device <NUM> may utilize the remote resources <NUM> to perform various functionality related to speech processing and/or synthesized playback communication. For instance, the device <NUM> is configured to perform speech recognition using a speech recognition system <NUM> and/or conversion of text-to-speech using a TTS system <NUM> (e.g., using the TTS model <NUM>). These systems <NUM>, <NUM>, <NUM> may reside on the device <NUM> (referred to as on-device systems) or reside remotely (e.g., reside on the remote system <NUM>), but in communication with the device <NUM>. In some examples, some of these systems <NUM>, <NUM>, <NUM> reside locally or on-device while others reside remotely. In other words, any of these systems <NUM>, <NUM>, <NUM> may be local or remote in any combination. For instance, when a system <NUM>, <NUM>, <NUM> is rather large in size or processing requirements, the system <NUM>, <NUM>, <NUM> may reside in the remote system <NUM>. Yet when the device <NUM> may support the size or the processing requirements of one or more systems <NUM>, <NUM>, <NUM>, the one or more systems <NUM>, <NUM>, <NUM> may reside on the device <NUM> using the data processing hardware <NUM> and/or the memory hardware <NUM>. Optionally, the one or more of the systems <NUM>, <NUM>, <NUM> may reside on both locally/on-device and remotely. For instance, one or more of the systems <NUM>, <NUM>, <NUM> may default to execute on the remote system <NUM> when a connection to the network <NUM> between the device <NUM> and remote system <NUM> is available, but when the connection is lost or the network <NUM> is unavailable, the systems <NUM>, <NUM>, <NUM> instead execute locally on the device <NUM>.

A speech recognition system <NUM> receives audio data <NUM> as an input and transcribes that audio signal into a transcription <NUM> as an output. Generally speaking, by converting the audio data <NUM> into a transcription <NUM>, the speech recognition system <NUM> allows the device <NUM> to recognize when a spoken utterance <NUM> from the user <NUM> corresponds to a query, a command, or some other form of audio communication. The transcription <NUM> refers to a sequence of text that the device <NUM> may then use to generate a response to the query or the command. For instance, if the user <NUM> asks the device <NUM> the question of "what will the weather be like today," the device <NUM> passes the audio data <NUM> corresponding to the question "what will the weather be like today" to the speech recognition system <NUM>. The speech recognized system <NUM> converts the audio data <NUM> into a transcript that includes the text of "what will the weather be like today?" The device <NUM> may then determine a response to the query using the text or portions of the text. For instance, in order to determine the weather for the current day (i.e., today), the device <NUM> passes the text (e.g., "what will the weather be like today?") or identifying portions of the text (e.g., "weather" and "today") to a search engine. The search engine may then return one or more search results that the device <NUM> interprets to generate a response for the user <NUM>.

In some implementations, the device <NUM> or a system associated with the device <NUM> identifies text <NUM> (also referred to as a sequence of text <NUM> or input text <NUM>) that the device <NUM> will communicate to the user <NUM> as a response to a query of the spoken utterance <NUM>. The device <NUM> may then use the TTS system <NUM> to convert the text <NUM> into corresponding synthesized playback audio <NUM> for the device <NUM> to communicate to the user <NUM> (e.g., audibly communicate to the user <NUM>) as the response to the query of the spoken utterance <NUM>. In other words, the TTS system <NUM> receives, as input, text <NUM> and converts the text <NUM> to an output of synthesized playback audio <NUM> (e.g., through a series of neural networks) where the synthesized playback audio <NUM> is an audio signal defining an audible rendition of the text <NUM>. For example, the playback audio <NUM> is a verbalization or a narration of the input text <NUM>. In some examples, the input text <NUM> refers to a sequence of text or characters in a particular natural language (e.g., English, Spanish, or French). The sequence of characters can include letters, numbers, punctuation marks, and/or other special characters. When the TTS system <NUM> generates the playback audio <NUM>, the playback audio <NUM> is synthesized speech in that it approximates how a human would verbalize the sequence of characters defining the input text <NUM>.

The TTS system <NUM> (or other speech synthesis system) includes a TTS model <NUM> (e.g., the TTS model <NUM> of <FIG>) that utilizes a deep neural network to generate the synthesized playback audio <NUM>. The TTS model <NUM> processes embeddings that are encoded representations of speech features (e.g., features of the input text <NUM>) to generate audio waveforms (e.g., time-domain audio waveforms that define an audio signal's amplitude over time). Once generated, the TTS system <NUM> communicates the synthesized playback audio <NUM> to the device <NUM> to allow the device <NUM> to output the synthesized playback audio <NUM>. For instance, the device <NUM> outputs the synthesized playback audio <NUM> of "today is sunny" at a speaker <NUM> of the device <NUM>. Here, the TTS model <NUM> of the TTS system <NUM> is configured to control the speech-related attributes of the synthesized speech <NUM>. In other words, the TTS model <NUM> is configured to simulate the voice of a human speaker in terms of naturalness while also being able to generate diverse synthesize speech by modeling fine-grained latent features. Although <FIG> depicts an example of a TTS system <NUM> in the context of an assistant application, the TTS system <NUM> (e.g., using the TTS model <NUM>) is applicable in other text-to-speech scenarios, such as, for example, navigation or reading documents.

Referring to <FIG>, the TTS model <NUM> includes an augmented encoder <NUM>, an adapter <NUM>, and a decoder <NUM>. The augmented encoder <NUM> receives text <NUM> as an input that is converted to an input embedding <NUM> for a transformer <NUM> to encode into a context vector Vc. Generally speaking, encoder-decoder architecture uses adapters <NUM> because the lengths of the input sequence and the output sequence for the decoder <NUM> are different. Therefore, the adapter <NUM> functions as a mechanism that addresses how the input sequence corresponds to the output sequence. For instance, how many audio frames correspond to a token of a context vector Vc. As shown in <FIG>, there are two options of adapters <NUM>, 204a-b. For the first option, a first adapter 204a uses attention (generally referred to as an attention-based TTS model) to provide the context vector Vc in a compatible form for the decoder <NUM> to decode into synthesized speech <NUM> (e.g., a spectrogram or waveform). For the second option, a second adapter 204b performs duration-based upsampling (referred to as a duration-based TTS model) instead of attention on the context vector Vc to provide the context vector Vc in a compatible form for the decoder <NUM> to decode into synthesized speech <NUM>.

To form the input embedding <NUM>, the TTS model <NUM> functions similarly to the BERT model as previously described in that the input embedding <NUM> is a combination of embeddings E that represent the input text <NUM>. In some implementations, the input embedding <NUM> corresponds to a sequence of words (e.g., a sentence or multiple sentences) represented as a plurality of tokens <NUM>, 212a-n or sequence of tokens <NUM>. The plurality of tokens <NUM> include a first set of phoneme tokens <NUM>, 212P<NUM>-n representing the text input <NUM> and a second set of graphemes tokens <NUM>, <NUM><NUM>-n also representing the text input <NUM>. That is, both the first set and the second set of tokens <NUM> represent the same text input <NUM>. Therefore, if the input text <NUM> is the sentence "My dog is cute," the first set of phoneme tokens 212P represents the sentence "my dog is cute" as phoneme tokens 212P and the second set of grapheme tokens <NUM> represents that same sentence "my dog is cute" as grapheme tokens <NUM>. Here, the phoneme tokens 212P may refer to International Phonetic Alphabet (IPA) phonemes while the grapheme tokes <NUM> may correspond to subword units. Similar to the original BERT model, the plurality of tokens <NUM> may also include special tokens shown as a CLS token <NUM>, 212CLS and a SEP token <NUM>, 212SEP. The CLS token 212CLS is a special token that may be prepended to the first segment (i.e., sentence) or leading segment for the tokens <NUM> while the SEP token 212SEP functions as a separator appended to each segment to indicate where one segment ends and another segment begins. For example, when the input text <NUM> includes two sentences represented as two segments, the sequence of tokens <NUM> would include an SEP token 212SEP separating the two segments (e.g., as shown in <FIG>). In some configurations, all the tokens representing the input text <NUM> share the same identifier (ID) space for purposes of embedding lookup and masked language modeling (MLM) classification.

In some examples, unlike the original BERT which uses a combination of three types of embeddings (e.g., a token embedding, a segment embedding, and a position embedding) to form the input embedding, the augmented encoder <NUM> forms the input embedding <NUM> from four types of embeddings E. For example, the augmented encoder <NUM> uses four types of embeddings E rather than three types of embeddings E to provide the alignment context for the augmented encoder <NUM> to incorporate graphemes in addition to phonemes. Referring to <FIG>, the input embedding <NUM> represents the sum of a position embedding E, Ep, a word position embedding E, Ewp, a segment embedding E, Es, and a token embedding E, Et. Here, the word position embedding Ewp differs from the position embedding Ep in that the position embedding is an overall index of position for the plurality of tokens <NUM>. Whereas, the word position embedding Ewp represents a position at a sub-word level (i.e., where in a word of a segment a token occurs). By including a word position embedding Ewp, the input embedding <NUM> is able to represent, for example, that a second phoneme in a sentence corresponds to a first word in a second wordpiece (i.e., word position) of the first word while also representing that the second grapheme for that same sentence corresponds to a second word in the sentence. Meaning that, the aligment between a phoneme and a grapheme is represented in the input embedding <NUM>.

<FIG> illustrates a vertical dotted box around each individual token <NUM> of the input embedding <NUM> (i.e., a tokenized step of the input embedding <NUM>) to depict that each token <NUM> of the input embedding <NUM> is the combination of a position embedding E, Ep, a word position embedding E, Ewp, a segment embedding E, Es, and a token embedding E, Et. Here, the position embedding Ep refers to an overall or global location for a token step with respect to the input embedding <NUM> (e.g., the entire current input embedding <NUM>). In <FIG>, there are nine positions or tokenized steps starting from an index of zero. The word position embedding Ewp refers to which word that a token <NUM> corresponds to in a sequence of words from the input text <NUM>. <FIG> has an input text <NUM> of three words. The segment embedding Es identifies which phrase or sentence corresponds to a token <NUM>. With the augmented encoder <NUM> having an input corresponding to graphemes and phonemes, the input text <NUM> includes at least two segments for each single segment of input text <NUM>. In other words, each segment of the input text <NUM> is repeated twice, once as a phoneme segment (shown in <FIG> as segment A) and once as a grapheme segment (shown in <FIG> as segment B). Accordingly, if the input text <NUM> was two or four sentences, meaning two or four segments for a traditional BERT, the input embedding <NUM> for the augmented encoder <NUM> would be four segments or eight segments respectively (i.e., each sentence represented once as graphemes and once as phonemes). Additionally, the tokens embeddings Et forming the input embedding <NUM> correspond to sub-unit representations of the words of a segment (or sentence). As previously stated, the phoneme tokens may be individual IPA phonemes while the grapheme tokens are wordpiece subwords.

With continued reference to <FIG>, each token <NUM> of the input embedding <NUM> is shown in a vertical dotted box to illustrate that a respective token <NUM> is a combination of multiple embeddings E. The first token 212a of the input embedding <NUM> is a combination of a first token embedding Et, Et1 of the CLS token, a first segment embedding Es, EsA, a first word position embedding Ewp, Ewp0, and a first position embedding Ep, Ep0. The second token 212b corresponding to the first phoneme token 212p1 in the set of phoneme tokens 212P for the input embedding <NUM> is a combination of a second token embedding Et, Et2 of the phoneme token P1, the first segment embedding Es, EsA, a second word position embedding Ewp, Ewp1, and a second position embedding Ep, Ep1. The third token 212c corresponding to the second phoneme token 212p2 in the set of phoneme tokens 212P for the input embedding <NUM> is a combination of a third token embedding Et, Et3 of the phoneme token P2, the first segment embedding Es, EsA, a second word position embedding Ewp, Ewp1, and a third position embedding Ep, Ep2. The fourth token 212d corresponding to the nth phoneme token 212pn in the set of phoneme tokens 212P for the input embedding <NUM> is a combination of a nth token embedding Et, Et4 of the phoneme token Pn, the first segment embedding Es, EsA, a third word position embedding Ewp, Ewp2, and a fourth position embedding Ep, Ep3. The fifth token 212e corresponding to the SEP token 212SEP for the input embedding <NUM> is a combination of a fifth token embedding Et, Et5 of the SEP token SEP, the first segment embedding Es, EsA, a third word position embedding Ewp, Ewp3, and a fifth position embedding Ep, Ep4. The sixth token 212f corresponding to the first grapheme token 212G1 of the second set of grapheme tokens <NUM> for the input embedding <NUM> is a combination of a sixth token embedding Et, Et6 of the grapheme token G1, a second segment embedding Es, EsB, the first word position embedding Ewp, Ewp1, and a sixth position embedding Ep, Ep4. The seventh token <NUM> corresponding to the second grapheme token 212G2 of the second set of grapheme tokens <NUM> for the input embedding <NUM> is a combination of a seventh token embedding Et, Et7 of the grapheme token G2, the second segment embedding Es, EsB, the second word position embedding Ewp, Ewp2, and a seventh position embedding Ep, Ep7. The eighth token <NUM> corresponding to the nth grapheme token 212Gn of the second set of grapheme tokens <NUM> for the input embedding <NUM> is a combination of a eighth token embedding Et, Et7 of the grapheme token Gn, the second segment embedding Es, EsB, the second word position embedding Ewp, Ewp2, and an eighth position embedding Ep, Ep8. The ninth token 212i corresponding to a second SEP token 212SEP (e.g., indicating the end of the second segment) for the input embedding <NUM> is a combination of a ninth token embedding Et, Et9 of the CEP token, the second segment embedding Es, EsB, the third word position embedding Ewp, Ewp3, and a ninth position embedding Ep, Ep9. Here, since segment A and segment B are for the same sentence, embeddings E that represent the first word position (e.g., the first word position embedding Ewp1) for a phoneme token 212P of the input embedding <NUM> are the same as embeddings E that represent the first word position (e.g., the first word position embedding Ewp1) for a grapheme token <NUM>; indicating that the grapheme and phoneme token <NUM> occur at the same word-piece level.

The transformer <NUM> of the augmented encoder <NUM> receives the input encoder embedding <NUM> and generates the context vector Vc as an output of the augmented encoder <NUM> (also referred to as an output encoder embedding (Vc). Much like the input encoder embedding <NUM>, the output encoder embedding or context vector Vc may also be a sequence of output tokens (e.g., shown as V1-V9) based on the input tokens <NUM>. Referring to <FIG>, the transformer <NUM> is configured to receive the input encoder embedding <NUM> and to generate the output encoder embedding Vc based on a relationship between the phoneme tokens 212P and the grapheme tokens <NUM> of the input encoder embedding <NUM>. For each phoneme token 212P of the set of phoneme tokens 212P from the input encoder embedding <NUM>, the transformer <NUM> identifies a respective word of the text input <NUM> corresponding to the respective phoneme token 212P. The transformer <NUM> may identify the respective word corresponding to the respective phoneme token 212P based on the word position embedding Ewp for the respective phoneme. For instance, for the first phoneme 212p of the input encoder embedding <NUM>, the word position embedding Ewp is an embedding that identifies that the first phoneme token 212P is in the first word of segment A (i.e., the phoneme segment). Once the transformer <NUM> identifies the respective word corresponding to the respective phoneme token 212P, the transformer <NUM> determines which grapheme token <NUM> also belongs with that respective word. In this example, the first phoneme token 212P corresponds to the first word of the input text <NUM> and the first grapheme token <NUM> also corresponds to the first word of the input text <NUM> (e.g., depicted as the first word (WP = <NUM>) in segment B (S = B)). Similar to the phoneme token 212P, the transformer <NUM> may determine that the grapheme token <NUM> corresponds to the word position of the first phoneme token 212P using the word position embedding E, Ewp for the set of grapheme tokens <NUM>. For example, the word position embedding E, Ewp1 corresponding to the first phoneme token 212P1 matches or is the same as the word position embedding E, Ewp1 corresponding to the first grapheme token 212G1. By identifying this relationship between a phoneme token 212P and a grapheme token 212P, the transformer <NUM> represents this relationship as part of the context vector Vc. The transformer <NUM> may then repeat this process for each input token <NUM>.

In some configurations, the transformer <NUM> generates each token V of the context vector Vc by representing a particular input token <NUM> as its relationship to all other input tokens <NUM>. For example, <FIG> shows that for the current token step of the input token 212P1, the output token V2 of the transformer <NUM> accounts for the first phoneme token's relationship to all other input tokens <NUM>. In some implementations, the transformer <NUM> may be configured to weight or to qualitatively represent the strength of the current token step (e.g., the first phoneme token 212P) to each of the other input tokens <NUM> such that each other input token <NUM> has some influence on the output token V2. Here, since the first phoneme token 212P1 has a strong relationship (e.g., a shared embedding E at the word-position level) with the first grapheme token 212G1, the transformer <NUM> may generate the output token V2 with some representation of this relationship.

Generating each token V of the context vector Vc as a representation of a particular input token's relationship to all other input tokens <NUM> may be beneficial for different types of TTS models. That is, often there are some TTS situations that have historically needed specialized TTS models or TTS models with a particular architecture to accommodate for their particular TTS tasks. This has generally been true with multi-lingual TTS models or multi-accent TTS models. A multi-lingual TTS model refers to a TTS model that may accommodate for input text <NUM> from various languages and, accordingly, generate synthesized speech <NUM> in these various languages. This is in contrast to monolingual (or single language) TTS models. Multi-lingual TTS models tend to be problematic because these TTS models traditionally employed phoneme-based TTS models. Phonemes present a challenge for multi-lingual models because phonemes may have a large overlap across different languages. Therefore, with this overlap that leads to language ambiguity, traditional multi-lingual models had to incorporate an additional language input for the model to be effective. But using an additional language input may have some difficulty when code-switching occurs. Code-switching refers to when a single sentence includes multiple languages. In a code-switching situation, a language input combined with a text input may be unable to identify the multiple languages or which particular tokens correspond to which languages. Stated differently, it is difficult to get an accurate language identifier for each token in a sentence. However, the augmented encoder <NUM> may resolve these issues because each token V of the context vector Vc is a representation of a particular input token's relationship to all other input tokens <NUM>. Therefore, if one of the input tokens <NUM> for the input encoder embedding <NUM> includes a language identifier, that language information will be carried to the respective token V of the output encoder embedding Vc. For instance, <FIG> illustrates one of the specialty tokens 212CLS includes a language identifier <NUM> that is then translated to the second token V2 of the context vector Vc. Although <FIG> depicts the language identifier <NUM> associated with a specialty token 212CLS, the input encoder embedding <NUM> may be configured such that other input tokens <NUM> include a language identifier <NUM>. Additionally or alternatively, for a multi-accent TTS model, the identifier <NUM> may instead be a locale or accent identifier <NUM>.

Besides multi-lingual TTS models, the augmented encoder <NUM> may also be incorporated into multi-accent TTS models. Multi-accent TTS models have also historically had some challenges. For example, multi-accent TTS models, like multi-lingual TTS models tend to use phoneme-based TTS models. This may be attributed to the fact that alignment issues between phonemes and graphemes for the encoder input had not been resolved. Using the current techniques of the augmented encoder <NUM>, however, now multi-accent TTS models can leverage the use of graphemes to identify locale or accents for the input text <NUM>. With these alignment improvements to represent the relationship between phonemes and graphemes, the augmented encoder <NUM> may be used in various TTS model, such as monolingual TTS models, single-locale TTS models, multi-lingual TTS models, multi-accent TTS models, attention-based TTS, and duration-based TTS models without much, if any significant modifications. Thus, the augmented encoder <NUM> may replace encoders used in TTS systems without disrupting the other architecture of these TTS systems.

Referring to <FIG>, much like the traditional BERT, the augmented encoder <NUM> undergoes a pre-training process <NUM>. Only here, the pre-training process <NUM> of the augmented encoder <NUM> uses both phonemes and graphemes unlike the traditional BERT whose pre-training process uses only graphemes. During the pre-training process, the augment encoder <NUM> pre-trains on a plain text corpus <NUM>. In some implementations, when training on the plain text corpus <NUM>, the pre-training process <NUM> obtains the phonemes using a grapheme to phoneme (G2P) conversion system and obtains the graphemes using a subword text tokenizer. In some examples, the pre-training process <NUM> may vary depending on the type of TTS model <NUM> that will incorporate the augmented encoder <NUM>. For instance, when pre-training the augmented encoder <NUM> for monolingual or single-locale TTS models, the pre-training process <NUM> only uses a single objective <NUM>, or masked language modeling (MLM) objective <NUM>, 320a. Generally speaking, when training with a MLM objective <NUM>, 320a, some percentage of the input tokens <NUM>, 312a-n are masked at random in order to predict the masked input tokens <NUM>. This masked prediction allows for both right context and left context (i.e., bidirectional context). Yet with the augmented encoder <NUM>, the input content <NUM> of each training example <NUM>, which corresponds to a sequence of words (i.e., a sentence or sentences), is represented twice - once as a sequence of training phoneme tokens <NUM>, 312b-d and once as a sequence of training grapheme tokens <NUM>, 312f-h. Here, if the MLM objective 320a randomly applied masking to these training token <NUM>, a training token <NUM> masked in the training phoneme tokens 312b-d may have its counterpart present (i.e., not masked) in the training grapheme tokens 312f-h; biasing the token prediction process of pre-training. For this reason, the pre-training process <NUM> with the MLM objective 320a instead applies a random masking at the word-level, but makes sure it is consistent between the phoneme and grapheme segments. That is, if the phoneme(s) of a word are masked, then the corresponding graphemes are masked as well. For example, <FIG> depicts the two masked phonemes <NUM>, <NUM>p1,<NUM> for the first word of segment A and the one masked grapheme <NUM>, <NUM>G for the corresponding first word of segment B (i.e., the complementary grapheme). Alternatively, the MLM objective 320a may be implemented using other masking strategies during the pre-training process <NUM>. For instance, the original BERT masking may be applied, but with an increased masking ratio, or the pre-training process <NUM> applies the masking in a P2G and G2P-like manner where all tokens in one segment (e.g., the grapheme segment - segment B) are masked out while all tokens in the other segment are kept (e.g., the phoneme segment - segment A).

For other TTS models <NUM>, such as multilingual TTS models and multi-accent TTS models, the pre-training process <NUM> additionally uses a classification objective <NUM>, 320b. Here, the pre-training process <NUM> using the classification objective 320b trains the output of the augmented encoder <NUM> to predict a locale or language. For instance, as described previously, a special token such as the CLS token may carry a language or locale identifier <NUM>. Here, when pre-training, the augmented encoded <NUM> learns to predict the language or locale and to indicate the language or locale as an identifier associated with a token, such as the CLS token. In some examples, language classification may be easier relative to accent classification. In these example, the classification loss for the classification objective 320b may use a lower weight (e.g., <NUM>).

Since the pre-training process <NUM> pre-trains the augmented encoder <NUM> of the TTS model <NUM> to learn how to predict input encoder embeddings <NUM>, a fine-tuning process then trains the augmented encoder <NUM> incorporated into the full TTS model <NUM> for a particular task. The fine-tuning process initializes the weights from the pre-trained model and then these weights proceed to be further tuned during the TTS model training. In some examples, the fine-tuning freezes the embeddings and lower layers of the transformer <NUM> while fine-tuning the higher layers of the transformer <NUM> in order to prevent degradation and promote the generalization of the trained TTS model <NUM>. Degradation may otherwise occur because the fine-tuning process uses a smaller training data set. The fine-tuning process may pass only the hidden states from a final transformer layer on the phoneme tokens 212P downstream to TTS components (e.g., the adapter <NUM> and the decoder <NUM>). Even though these hidden states only refer to phoneme positions, these hidden states still carry information from the graphemes as well as the language (or locale) based on the way the augmented encoder <NUM> was trained. In some configurations, the fine-tuning process turns off the MLM objective 320a, but yet keeps the classification objective 320b active for multilingual or multi-accent TTS models. For instance, by keeping the classification objective 320b active during the fine-tuning process, the language information may be maintained in encoded representations.

<FIG> is a flow chart of an example arrangement of operations for a method <NUM> of generating an output encoder embedding Vc using both phonemes and graphemes. At operation <NUM>, the method <NUM> receives, at an encoder <NUM> of a speech synthesis model <NUM>, a text input <NUM> includes a sequence of words represented as an input encoder embedding <NUM>. The input encoder embedding <NUM> includes a plurality of tokens <NUM> where the plurality of tokens <NUM> includes a first set of grapheme tokens <NUM>, <NUM> representing the text input <NUM> as respective graphemes and a second set of phoneme tokens <NUM>, 212P representing the text input <NUM> as respective phonemes. The method <NUM> performs operations <NUM> and <NUM> for each respective phoneme token 212P of the second set of phoneme tokens 212P. At operations <NUM>, the method <NUM> identifies a respective word of the sequence of words corresponding to the respective phoneme token 212P. At operation <NUM>, the method <NUM> determines a respective grapheme token <NUM> representing the respective word of the sequence of words corresponding to the respective phoneme token 212P. At operation <NUM>, the method <NUM> generates an output encoder embedding Vc based on a relationship between each respective phoneme token 212P and the corresponding grapheme token <NUM> determined to represent a same respective word as the respective phoneme token 212P.

<FIG> is schematic view of an example computing device <NUM> that may be used to implement the systems (e.g., the TTS system <NUM>, the TTS model <NUM>, and/or the augmented encoder <NUM>) and methods (e.g., the method <NUM>) described in this document.

The computing device <NUM> includes a processor <NUM> (e.g., data processing hardware), memory <NUM> (e.g., memory hardware), a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM>. Also, multiple computing devices <NUM> may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multiprocessor system).

Claim 1:
A computer-implemented method (<NUM>) when executed by data processing hardware (<NUM>) causes the data processing hardware (<NUM>) to perform operations comprising:
pre-training an encoder (<NUM>) of a speech synthesis model (<NUM>) by:
feeding the encoder (<NUM>) a plurality of training examples (<NUM>), each training example (<NUM>) represented as a sequence of training grapheme token (<NUM>) embeddings corresponding to a training sequence of words and a sequence of training phoneme tokens (212P) corresponding to the same training sequence of words;
masking a training phoneme token (212P) from the sequence of training phoneme tokens (212P) for a respective word from the training sequence of words; and
masking a training grapheme token (<NUM>) embedding from the sequence of training phoneme tokens (212P) for the respective word from the training sequence of words;
receiving, at the pre-trained encoder (<NUM>) of the speech synthesis model (<NUM>), a text input (<NUM>) comprising a sequence of words represented as an input encoder embedding (<NUM>), the input encoder embedding (<NUM>) comprising a plurality of tokens (<NUM>), the plurality of tokens (<NUM>) comprising a first set of grapheme token (<NUM>) embeddings representing the text input (<NUM>) as respective graphemes and a second set of phoneme tokens (212P) representing the text input (<NUM>) as respective phonemes;
for each respective phoneme token (212P) of the second set of phoneme tokens (212P):
identifying, by a transformer (<NUM>) of the pre-trained encoder (<NUM>), a respective word of the sequence of words corresponding to the respective phoneme token (212P); and
determining, by the transformer (<NUM>) of the pre-trained encoder (<NUM>), a respective grapheme token (<NUM>) embedding representing the respective word of the sequence of words corresponding to the respective phoneme token (212P); and
generating, by the transformer (<NUM>) of the pre-trained encoder (<NUM>), an output encoder embedding (Vc) based on the correspondence between each respective phoneme token (212P) and the respective grapheme token (<NUM>) embedding determined to represent a same respective word as the respective phoneme token (212P);
decoding, by a decoder (<NUM>), the output encoder embedding (Vc) into synthesized speech (<NUM>).