Patent Description:
<CIT> describes methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for presenting notifications in an enterprise system. In one aspect, a method include actions of obtaining a template that defines (i) trigger criteria for presenting a notification type and (ii) content rules for determining content to include in a notification of the notification type. Additional actions include accessing enterprise resources of an enterprise, the enterprise resources including data describing entities related to the enterprise and relationships among the entities. Further actions include, accessing user information specific to a user and determining that the trigger criteria is satisfied by the enterprise resources and the user information. Additional actions include generating a particular notification of the notification type based at least on the content rules and providing the particular notification to the user. Users frequently interact with voice-enabled devices, such as smart phones, smart watches, and smart speakers, through digital assistants. These digital assistants provide a dialog with the users to enable the users to complete tasks and obtain answers to questions they have all through natural, conversational interactions. Ideally, during a dialog between a user and the digital assistant, the user should be able to communicate as if the user were talking to another person, via spoken queries directed toward their voice-enabled device running the digital assistant. The digital assistant will provide these spoken queries to an automated speech recognizer (ASR) system to process and recognize the spoken request so that an action can be performed. Additionally, the digital assistant will also employ a text-to-speech (TTS) system to convert textual representations of responses to the query into synthesized speech for audible output from the user's voice-enabled device. Often, there is overlap in the vocabulary between spoken queries and corresponding TTS responses during a digital assistant dialog, whereby a user pronunciation of a word in a spoken query is different than a TTS pronunciation of the same word present in a digital assistant response to the query when audibly output as synthesized speech.

One aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations for selecting which one of a user pronunciation of a particular word or a text-to-speech pronunciation of the particular word is more reliable for use in text-to-speech audio. The operations include receiving a user pronunciation of a particular word present in a query spoken by a user. The operations also include receiving a text-to-speech (TTS) pronunciation of the same particular word that is present in a TTS input. The TTS input includes a textual representation of a response to the query and the TTS pronunciation of the particular word is different than the user pronunciation of the particular word. The operations also include obtaining user pronunciation-related features associated with the user pronunciation of the particular word and obtaining TTS pronunciation-related features associated with the TTS pronunciation of the particular word. The operations also include generating, as output from a pronunciation decision model configured to receive the user pronunciation-related features and the TTS pronunciation-related features as input, a pronunciation decision selecting the one of the user pronunciation of the particular word or the TTS pronunciation of the particular word that is associated with a highest confidence for use in TTS audio. The operations also include providing, for audible output from a user device associated with the user, the TTS audio that includes a synthesized speech representation of the response to the query using the one of the user pronunciation for the particular word or the TTS pronunciation for the particular word that was selected by the pronunciation decision output from the pronunciation decision model.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the operations further include receiving audio data that corresponds to the query spoken by the user and processing, using an automated speech recognizer (ASR), the audio data to generate a transcription of the query. In these implementations, receiving the user pronunciation of the particular word includes at least one of: extracting the user pronunciation of the particular word from an intermittent state of the ASR while using the ASR to process the audio data; extracting a user acoustic representation of the particular word from the audio data that conveys the user pronunciation of the particular word; or processing the audio data to generate a user phoneme representation that conveys the user pronunciation of the particular word. The user pronunciation-related features associated with the user pronunciation of the particular word may also include one or more confidence features associated with the ASR recognizing the particular word in the audio data.

In some examples, the user pronunciation-related features associated with the user pronunciation of the particular word include at least one of a geographical area of the user when the query was spoken by the user, linguistic demographic information associated with the user, or a frequency of using the user pronunciation when pronouncing the particular word in previous queries spoken by the user and/or other users. Receiving the TTS pronunciation of the particular word may include: receiving, as input to a TTS system, the TTS input that includes a textual representation of the response to the query; generating, as output from the TTS system, an initial sample of TTS audio that includes an initial synthesized speech representation of the response to the query; and extracting a TTS acoustic representation of the particular word from the initial sample of the TTS audio, the TTS acoustic representation conveys the TTS pronunciation of the particular word.

Optionally, receiving the TTS pronunciation of the particular word may include processing the textual representation of the response to the query to generate a TTS phoneme representation that conveys the TTS pronunciation of the particular word. In some examples, the TTS pronunciation-related features associated with the TTS pronunciation of the particular word includes at least one of a verified preferred pronunciation for the particular word, an unverified pronunciation for the particular word estimated using pronunciation mining form one or more auxiliary information sources, a pronunciation variant feature that indicates whether any other variants for pronouncing the particular word exists, or a pronunciation complexity feature that indicates a likelihood of user mispronunciation of the particular word.

In some implementations, after generating the pronunciation decision selecting the one of the user pronunciation of the particular word or the TTS pronunciation of the particular word, the operations further include receiving explicit feedback from the user that indicates which one of the user pronunciation of the particular word or the TTS pronunciation of the particular word the user prefers for pronouncing the particular word in subsequent TTS outputs and updating the pronunciation decision model based on the explicit feedback from the user. Here, when the explicit feedback from the user indicates that the user prefers the user pronunciation of the particular word, the operations further include updating a TTS system to use the user pronunciation of the particular word when generating TTS audio that includes the particular word. In some examples, after providing the TTS audio for audible output from the user device, the operations further include receiving audio data that corresponds to a subsequent query spoken by the user or another user that includes the particular word, determining implicit user feedback that indicates whether or not the user or the other user pronounced the particular word in the subsequent query the same as the one of the user pronunciation for the particular word or the TTS pronunciation for the particular word that was selected by the pronunciation decision, and updating the pronunciation decision model based on the implicit user feedback.

Another aspect of the disclosure provides a system that includes data processing hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations for selecting which one of a user pronunciation of a particular word or a text-to-speech pronunciation of the particular word is more reliable for use in text-to-speech audio. The operations include receiving a user pronunciation of a particular word present in a query spoken by a user. The operations also include receiving a text-to-speech (TTS) pronunciation of the same particular word that is present in a TTS input. The TTS input includes a textual representation of a response to the query and the TTS pronunciation of the particular word is different than the user pronunciation of the particular word. The operations also include obtaining user pronunciation-related features associated with the user pronunciation of the particular word and obtaining TTS pronunciation-related features associated with the TTS pronunciation of the particular word. The operations also include generating, as output from a pronunciation decision model configured to receive the user pronunciation-related features and the TTS pronunciation-related features as input, a pronunciation decision selecting the one of the user pronunciation of the particular word or the TTS pronunciation of the particular word that is associated with a highest confidence for use in TTS audio. The operations also include providing, for audible output from a user device associated with the user, the TTS audio that includes a synthesized speech representation of the response to the query using the one of the user pronunciation for the particular word or the TTS pronunciation for the particular word that was selected by the pronunciation decision output from the pronunciation decision model.

This aspect may include one or more of the following optional features. In some implementations, the operations further include receiving audio data that corresponds to the query spoken by the user and processing, using an automated speech recognizer (ASR), the audio data to generate a transcription of the query. In these implementations, receiving the user pronunciation of the particular word includes at least one of: extracting the user pronunciation of the particular word from an intermittent state of the ASR while using the ASR to process the audio data; extracting a user acoustic representation of the particular word from the audio data that conveys the user pronunciation of the particular word; or processing the audio data to generate a user phoneme representation that conveys the user pronunciation of the particular word. The user pronunciation-related features associated with the user pronunciation of the particular word may include one or more confidence features associated with the ASR recognizing the particular word in the audio data.

In some implementations, after generating the pronunciation decision selecting the one of the user pronunciation of the particular word or the TTS pronunciation of the particular word, the operations further include receiving explicit feedback from the user that indicates which one of the user pronunciation of the particular word or the TTS pronunciation of the particular word the user prefers for pronouncing the particular word in subsequent TTS outputs and updating the pronunciation decision model based on the explicit feedback from the user. Here, when the explicitly feedback from the user indicates that the user prefers the user pronunciation of the particular word, the operations further comprise updating a TTS system to use the user pronunciation of the particular word when generating TTS audio that includes the particular word. In some examples, after providing the TTS audio for audible output from the user device, the operations further include receiving audio data that corresponds to a subsequent query spoken by the user or another user that includes the particular word, determining implicit user feedback that indicates whether or not the user or the other user pronounced the particular word in the subsequent query the same as the one of the user pronunciation for the particular word or the TTS pronunciation for the particular word that was selected by the pronunciation decision, and updating the pronunciation decision model based on the implicit user feedback. According to another aspect, there is provided a computer-readable medium as set forth in claim <NUM>. According to a further aspect, there is provided a computer program as set forth in claim <NUM>.

During a dialog between a user and a digital assistant, such as when a user issues a spoken query for the digital assistant to perform an action and the digital assistant audibly outputs a text-to-speech (TTS) response related to the query, there is often overlap in vocabulary of the spoken query and the TTS response. For instance, the user may speak a voice search query, "Where is Bexar?" in which corresponding audio data is passed to an automated speech recognizer (ASR) for conversion into a corresponding transcript for interpretation by a natural language understanding (NLU) module and provided as a search query to a search engine to retrieve a result. Using the retrieved result, the digital assistant may generate a TTS input that includes a textual representation of a response to the query. Here, the textual representation of the response states, "Bexar is located in Texas", which is provided in the TTS input to the TTS system for generating corresponding TTS audio that includes synthesized speech for audible output from a device associated with the user.

In some instances, a user pronounces a word in the spoken query differently than how the TTS system will pronounce the same word present in a textual representation of the response to the query provided as the TTS input. The difference in competing pronunciations may be attributed to a variety of reasons, such as lack of normalization between the ASR system and the TTS system which are generally trained separately and often out of sync in terms of updates, rare words/terms (e.g., contact names, emerging words, etc.) that the TTS system has not been trained to pronounce, words that a user is unknowingly mispronouncing in which the ASR system is robust for accurately recognizing and the TTS system pronounces correctly, and words that a user is purposely mispronouncing to spoof the ASR system to name a few.

Implementations herein are directed toward identifying when a user pronunciation of a term spoken by a user in a query is different than a TTS pronunciation of the same term present in a response to the query, and using a pronunciation decision model to determine which one of the user pronunciation or the TTS pronunciation to use for pronouncing the word in TTS audio that conveys the response to the user as synthesized speech. As described in greater detail below, the pronunciation decision model may be trained and continuously updated to select between competing pronunciations of a same word based on TTS pronunciation-related features and/or user pronunciation-related features. That is, based on user pronunciation-related features associated with a user pronunciation of a particular word and TTS pronunciation-related features associated with a TTS pronunciation of the particular word, the pronunciation decision model may estimate a confidence in the user pronunciation of the particular word and a confidence in the TTS pronunciation of the same particular word to determine which one of the user pronunciation or the TTS pronunciation is more reliable.

Referring to <FIG>, in some implementations, a speech environment <NUM> includes a user <NUM> speaking a query <NUM> to a speech-enabled device <NUM> (also referred to as a device <NUM> or a user device <NUM>). Specifically, the user <NUM> is having a dialog with a digital assistant <NUM> (also referred to as a 'digital assistant interface') executing on the user device <NUM> where the query <NUM> spoken by the user <NUM> requests the digital assistant <NUM> to perform an operation. The user <NUM> (i.e., speaker of the query <NUM>) may speak the query <NUM> to solicit a response from the digital assistant <NUM> or to have the digital assistant <NUM> execute a task specified by the query <NUM>. The device <NUM> is configured to capture sounds from one or more users <NUM> within the speech environment <NUM>. Speech-enabled systems of the device <NUM> or associated with the device <NUM> (e.g., the digital assistant <NUM>) may field the query <NUM>, perform one or more operations specified by the query <NUM>, and provide a response to the query as synthesized speech <NUM> for audible output from the device <NUM>.

Here, the device <NUM> captures audio data <NUM> corresponding to the spoken query <NUM> by the user <NUM>. The device <NUM> may correspond to any computing device associated with the user <NUM> and capable of receiving audio data <NUM>. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, e-book readers, etc.), computers, wearable devices (e.g., smart watches), music player, casting devices, smart appliances (e.g., smart televisions) and internet of things (IoT) devices, remote controls, smart speakers, etc. The device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and storing instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations related to speech and/or text processing. In some examples, the device <NUM> includes one or more applications (i.e., software applications) where each application may utilize one or more speech processing systems/models <NUM>, <NUM>, <NUM> associated with the device <NUM> to perform various functions within the application. For instance, the device <NUM> executes the digital assistant <NUM> configured to communicate synthesized playback audio <NUM> (also referred to as synthesized speech <NUM>) to the user <NUM> to converse with the user <NUM> and assist with the performance of various tasks.

The device <NUM> further includes an audio subsystem with an audio capturing device (e.g., a microphone) <NUM> for capturing and converting audio data <NUM> within the speech environment <NUM> into electrical signals and a speech output device (e.g., a speaker) <NUM> for communicating an audible audio signal (e.g., a synthesized playback signal <NUM> from the device <NUM>). While the device <NUM> implements a single audio capturing device <NUM> in the example shown, the device <NUM> may implement an array of audio capturing devices <NUM> without departing from the scope of the present disclosure, whereby one or more audio capturing devices <NUM> in the array may not physically reside on the device <NUM>, but be in communication with the audio subsystem (e.g., peripherals of the device <NUM>). For example, the device <NUM> may correspond to a vehicle infotainment system that leverages an array of microphones positioned throughout the vehicle. Similarly, the speech output device <NUM> may include one or more speakers either residing on the device <NUM>, in communication therewith, or a combination where one or more speakers reside on the device <NUM> and one or more other speakers are physically removed from the device <NUM> but in communication with the device <NUM>.

Furthermore, the device <NUM> may be configured to communicate via a network <NUM> with a remote system <NUM>. The remote system <NUM> may include remote resources <NUM>, such as remote data processing hardware <NUM> (e.g., remote servers or CPUs) and/or remote memory hardware <NUM> (e.g., remote databases or other storage hardware). The device <NUM> may utilize the remote resources <NUM> to perform various functionality related to speech processing and/or synthesized playback communication. For instance, the device <NUM> is configured to perform speech recognition using an automated speech recognition (ASR) system <NUM> and/or conversion of text to speech using a text-to-speech (TTS) system <NUM>. Additionally, a pronunciation decision model <NUM> may generate a pronunciation decision <NUM> selecting between a user pronunciation <NUM> or a TTS pronunciation <NUM> for a particular word that is present in both the query <NUM> spoken by the user <NUM> and a TTS input <NUM> corresponding to a textual representation of a response to the query <NUM>. The TTS system <NUM> produces TTS audio <NUM> including a synthesized speech representation of the response to the query <NUM>, whereby the TTS audio <NUM> pronounces the particular word using the one of the user pronunciation <NUM> or the TTS pronunciation <NUM> that was selected by the pronunciation decision <NUM> output from the pronunciation decision model <NUM>.

These systems/models <NUM>, <NUM>, <NUM> may reside on the device <NUM> (referred to as on-device systems) or reside remotely (e.g., reside on the remote system <NUM>), but in communication with the device <NUM>. In some examples, some of these systems <NUM>, <NUM>, <NUM> reside locally or on-device while others reside remotely. In other words, any of these systems <NUM>, <NUM>, <NUM> may be local, remote, or both in any combination. For instance, when a system <NUM>, <NUM>, <NUM> is rather large in size or processing requirements, the system <NUM>, <NUM>, <NUM> may reside in the remote system <NUM>. Yet when the device <NUM> may support the size or the processing requirements of one or more systems <NUM>, <NUM>, <NUM>, the one or more systems <NUM>, <NUM>, <NUM> may reside on the device <NUM> using the data processing hardware <NUM> and/or the memory hardware <NUM>. Optionally, the one or more of the systems <NUM>, <NUM>, <NUM> may reside on both locally/on-device and remotely. For instance, one or more of the systems <NUM>, <NUM>, <NUM> may default to execute on the remote system <NUM> when a connection to the network <NUM> between the device <NUM> and remote system <NUM> is available, but when the connection is lost or the network <NUM> is unavailable, the systems <NUM>, <NUM>, <NUM> instead execute locally on the device <NUM>.

In the example shown, the ASR system <NUM> receives, as input, audio data <NUM> corresponding to the query <NUM> and processes the audio data <NUM> to generate, as output, a transcription <NUM> of the query <NUM>. The ASR system <NUM> may include natural language understanding (NLU) functionality to perform query interpretation (e.g., semantic analysis) on the transcription <NUM>. The transcription <NUM> includes a graphemic representation represented by a sequence of text that the digital assistant <NUM> may then use to generate the response to the query <NUM>, and more particularly, generate the TTS input <NUM> corresponding to the textual representation of the response to the query <NUM>. For instance, continuing with the example from earlier, the user <NUM> may speak the query "Where is Bexar?" which is captured and converted into corresponding audio data <NUM> for processing by the ASR system <NUM> to generate the corresponding transcription <NUM>. While the canonical pronunciation for the word Bexar is "BAY-∂r" in the context of Texas, the user <NUM> unknowingly mispronounced the word Bexar as "Bak-sar" based on its English spelling by enunciating the "X". Notably, the ASR system <NUM> is robust for accurately recognizing the user mispronunciation of the word Bexar as the city in Texas.

After generating the transcription <NUM> (and performing any NLU functionality), the digital assistant <NUM> may then determine the response to the query <NUM> using the transcription <NUM>. For instance, in order to determine the location for a city name (e.g., Bexar), the digital assistant <NUM> may pass the transcription <NUM> ("Where is Bexar?) or a search string that includes identifying portions of the transcription (e.g., "where" and "Bexar") to a search engine. The search engine may then return one or more search results that the digital assistant <NUM> interprets to generate the TTS input <NUM> that includes the textual representation of the response to the query <NUM>. Notably, the word "Bexar" is present in both the transcription <NUM> of the query <NUM> and the TTS input <NUM>, whereby the transcription <NUM> and the textual representation of the TTS input <NUM> both share a same graphemic representation of the word "Bexar". The textual representation may include a sequence of graphemes/characters in a particular natural language. The sequence of graphemes/characters can include letters, numbers, punctuation marks, and/or other special characters.

The TTS system <NUM> may convert the TTS input <NUM> into corresponding TTS audio <NUM> (e.g., synthesized speech) that the device <NUM> will audibly output for communicating the response to the query <NUM> to the user <NUM>. Prior to audible output of the TTS audio <NUM>, the TTS system <NUM> and/or the pronunciation decision model <NUM> may identify that the user pronunciation <NUM> of a particular word present in the query <NUM> is different than the TTS pronunciation <NUM> of the same particular word that is present in the TTS input <NUM>. In the example shown, the user pronunciation <NUM> of the word Bexar is "Bak-sar" while the TTS pronunciation <NUM> of the word Bexar is "BAY-∂r". In some implementations, the pronunciation decision model <NUM> obtains user pronunciation-related features (UPFs) <NUM> associated with the user pronunciation <NUM> of the particular word and TTS pronunciation-related features (TTSPFs) <NUM> associated with the TTS pronunciation <NUM> of the particular word. While a multitude of different UPFs <NUM> and TTSPFs <NUM> may be obtained and form a basis for generating the pronunciation decision <NUM> as described in greater detail below with reference to <FIG>, the pronunciation decision model <NUM> depicted in the present example of <FIG> obtains UPFs <NUM> that include user statistics indicating that users frequently mispronounce Bexar, as well as TTSPFs <NUM> indicating that Bexar in the context of Texas is widely pronounced as "BAY-∂r". Accordingly, the pronunciation decision model <NUM> may estimate a confidence for the user pronunciation <NUM> of Bexar that is lower than a confidence estimated for the TTS pronunciation <NUM> of Bexar due to the fact that users commonly mispronounce Bexar and the TTS pronunciation <NUM> is widely adopted when spoken in the context of Texas. Notably, the TTS pronunciation <NUM> of "BAY- ∂r" for the word Bexar present in the textual representation of the TTS response is not verified, but instead learned by the TTS system <NUM> through pronunciation mining from one or more auxiliary information sources. For instance, the auxiliary information sources may include audio and/or video web sources that may be mined for pronunciation of various terms within various contexts for use by the TTS system <NUM>.

When the TTS system <NUM> generates the TTS audio <NUM> from the TTS input <NUM>, the TTS output <NUM> includes synthesized speech that approximates how a human would pronounce words formed by the sequence of graphemes/characters defining the TTS input <NUM> including the textual representation of the response to the query <NUM>. Continuing the example, the pronunciation decision <NUM> selecting the TTS pronunciation <NUM> of the particular word Bexar results in maintaining the use of the TTS pronunciation <NUM> in the TTS audio <NUM> and providing the TTS audio <NUM> for audible output from the user device <NUM>. Thus, the TTS audio <NUM> is audibly output as a synthesized speech representation of the response to the query <NUM> and uses the TTS pronunciation <NUM> of "BAY- ∂r" for the textual representation of the word Bexar that is present in the TTS input <NUM>.

In some examples, after providing the TTS audio <NUM> for audible output using the selected one of the user pronunciation <NUM> or the TTS pronunciation <NUM> for a particular word, the pronunciation decision model <NUM> and/or TTS system <NUM> prompts the user <NUM> to provide explicit feedback indicating if the user <NUM> agrees with the pronunciation decision <NUM> or instead prefers the other one of the user pronunciation <NUM> or the TTS pronunciation <NUM> that was not selected in the pronunciation decision <NUM>. More specifically, the explicit feedback indicating a preferred pronunciation for a particular word may be used as a TTSPF <NUM> obtained by the pronunciation decision model <NUM> when making a subsequent pronunciation decision <NUM> for competing user and TTS pronunciations of the particular word. The user <NUM> may be prompted to provide explicit feedback when the model <NUM> estimates confidences for the user and TTS pronunciations <NUM>, <NUM> that are close (e.g., within a threshold confidence range) and/or when the confidences fail to satisfy a confidence threshold. The explicit feedback may be used to update the pronunciation decision model <NUM> and/or the TTS system <NUM>. Prior to updating the pronunciation decision model <NUM> and/or TTS system <NUM>, additional verification steps may be carried out to verify that the preferred pronunciation for the particular word is a reasonable variant and/or verify that the explicit feedback is not an adversarial user correction.

In an alternative implementation of the example above, if the TTSPFs <NUM> indicated that the user pronunciation <NUM> of "Bak-sar" for the word Bexar was previously specified by the user <NUM> as a preferred pronunciation, then the pronunciation decision model <NUM> would likely estimate a higher confidence for the user pronunciation <NUM> than the TTS pronunciation <NUM>, thereby resulting in the TTS system <NUM> modifying the TTS audio <NUM> to adopt the user pronunciation <NUM>. In this implementation, if the TTS system <NUM> were a local/custom TTS system <NUM> specific to the user, the TTSPF <NUM> indicating the user pronunciation <NUM> of "B∂k-sar" for the word Bexar as a preferred pronunciation may result in retraining the TTS system <NUM> so that the TTS system <NUM> initially generates TTS audio <NUM> by pronouncing the word Bexar as "Bak-sar" when present in the TTS input <NUM>.

The TTS system <NUM> may include any type of TTS system capable of converting input text (e.g., TTS input) into a corresponding synthetic speech representation as TTS audio <NUM>. For instance, the TTS system <NUM> may employ a parametric TTS model or a TTS model <NUM> (TTS model of <FIG>) that utilizes a deep neural network (e.g., an attention-based Tacotron network) to generate the TTS audio <NUM> for audible output as synthesized speech. In some implementations, the TTS model processes embeddings that are encoded representations of speech features (e.g., features of the TTS input <NUM>) to generate audio waveforms (e.g., time-domain audio waveforms that define an audio signal's amplitude over time) representing TTS audio <NUM>. Once generated, the TTS system <NUM> communicates the TTS audio <NUM> to the device <NUM> to allow the device <NUM> to output the TTS audio <NUM> as a synthesized speech representation of the response to the query <NUM>. For instance, the device <NUM> audibly outputs, from the one or more speakers <NUM>, the TTS audio <NUM> of "Bexar is in Texas" using the TTS pronunciation <NUM> of B∂k sar selected by the pronunciation decision <NUM> output from the model <NUM>. Here, the TTS model <NUM> of the TTS system <NUM> is configured to control the speech-related attributes of the synthesized speech <NUM>. In other words, the TTS model <NUM> is configured to simulate the voice of a human speaker in terms of naturalness while also being able to generate diverse synthesized speech by modeling fine-grained latent features. Although <FIG> depicts an example of a TTS system <NUM> in the context of an application for a digital assistant <NUM>, the TTS system <NUM> (e.g., using the TTS model <NUM>) is applicable in other text-to-speech scenarios, such as, for example, voice search, navigation or reading documents.

In some implementations, when the TTS model <NUM> associated with the TTS system <NUM> resides on the user device <NUM> but is trained as a global model shared across users in a geographical area, federation learning techniques are employed to update/retrain the model <NUM>. For instance, the user device <NUM> may retrain/update parameters of an on-device version of the TTS model <NUM> executing locally and then share the updated parameters or training losses with a server for updating the global TTS model <NUM>. In doing so, the global TTS model <NUM> may be retrained/updated using the parameter updates/training losses received from multiple users without requiring the users across the user population to share their audio data, query content, or other information that users may not want to share.

Referring to <FIG>, in some examples, the TTS model <NUM> includes an encoder-decoder network architecture having an encoder <NUM> and a decoder <NUM>. In some implementations, the encoder-decoder <NUM>, <NUM> structure corresponds to the sequence-to-sequence recurrent neural network (RNN) of Tacotron <NUM> (e.g., described in<NPL>). In some configurations, the encoder <NUM> is configured to receive, as input, the TTS input <NUM> or an embedding corresponding to the TTS input <NUM> (e.g., character embeddings) and generate, as output, a context vector Vc for each Mel frequency spectrogram that the decoder <NUM> will later generate. The context vector Vc may be a fixed length and generally define features that appear in particular positions corresponding to the sequence of characters forming textual representation of the TTS input <NUM>. In some configurations, the TTS input <NUM> includes grapheme sequence that is first converted into a corresponding phoneme sequence (e.g., via a normalization engine such as a grapheme-to-phoneme model) prior to being input into the encoder <NUM>.

The encoder <NUM> may include one or more convolutional layers followed by a bidirectional long short-term memory (LTSM) layer. The neurons in each convolution a layer may receive input from a small subset of neurons in a previous layer. In this respect, neuron connectivity allows the convolutional layers to learn filters that activate when particular hidden features appear in positions in the sequence of characters corresponding to textual representation of the TTS input <NUM>. In some implementations, the filter in each convolutional layer may span a series of characters (e.g., four, five, or six characters). Each convolutional layer may be followed by batch normalization and rectified linear units (RELUs). When the encoder <NUM> includes one or more convolutional layers, a bidirectional LSTM layer may follow these convolutional layers. Here, the bidirectional LSTM is configured to process the hidden features generated by the final convolutional layer in order to generate a sequential feature representation of the sequence of characters corresponding to the TTS input <NUM>. The sequential feature representation may include a sequence of feature vectors.

In some implementations, the encoder <NUM> also includes an attention network configured to receive a sequential feature representation from the encoder <NUM> and to process the sequential feature representation to generate the context vector Vc for each decoder output step. That is, the attention network can generate a fixed length context vector Vc for each frame of a Mel frequency spectrogram that a decoder <NUM> will later generate. A frame refers to a unit of the Mel frequency spectrogram that is based on a small portion of the input signal (e.g., a <NUM> millisecond sample). The architecture of the attention network may vary depending on the particular TTS system <NUM>. Some examples of attention networks include additive attention networks, location sensitive attention networks, Gaussian Mixture Model (GMM) attention networks (e.g., to improve generalization to long utterances), forward attention networks, stepwise monotonic attention networks, or dynamic convolution attention networks. With an attention network, the TTS model <NUM> may be able to generate an output sequence (e.g., a sequence of output log-mel spectrogram frames) based on additional inputs (e.g., with speech embeddings e) that receive particular attention weights in order to generate the context vector Vc.

The decoder <NUM> is configured as a neural network (e.g., an autoregressive recurrent neural network) to generate an output audio signal As (e.g., an output sequence mel-frequency spectrograms) of expressive speech that includes the intended speech-related attributes (e.g., the TTS pronunciation for each word present in the TTS input <NUM>, the intended prosody and/or speech characteristics). For instance, based on the context vector Vc, the decoder <NUM> predicts a representation of a speech signal (e.g., a mel frame or spectrogram frame) from the encoded representation generated by the encoder <NUM>. That is, the decoder <NUM> is configured to receive, as input, one or more context vectors Vc and may generate, for each context vector Vc, a corresponding frame of a mel-frequency spectrogram where a mel-frequency spectrogram is a frequency-domain representation of sound. In some examples, the decoder <NUM> includes an architecture similar to Tacotron <NUM>. In other words, the decoder <NUM> may include an architecture having a pre-net, a Long Short-Term Memory (LSTM) subnetwork, a linear projection, and a convolutional post-net.

In some configurations, the TTS model <NUM> also includes a speech synthesizer <NUM> (also referred to as a synthesizer <NUM>). The synthesizer <NUM> can be any network that is configured to receive a Mel frequency spectrogram and to generate output samples of TTS audio <NUM> based on the Mel frequency spectrogram as synthesized speech. In some other implementations, the synthesizer <NUM> includes a vocoder. For instance, the speech synthesizer <NUM> may include a WaveRNN vocoder (e.g., as described by "<NPL>). Here, the WaveRNN vocoder may generate <NUM>-bit signals sampled at <NUM> conditioned on spectrograms predicted by the TTS model <NUM>. In some other implementations, the synthesizer <NUM> is a trainable spectrogram to waveform inverter. After the synthesizer <NUM> generates the waveform, an audio subsystem can generate the TTS audio <NUM> using a waveform and provide the TTS audio <NUM> for audible playback (e.g., on the device <NUM>) as synthesized speech, or provide the generated waveform to another system to allow the other system to generate and play back the TTS audio <NUM>. Generally speaking, the synthesizer <NUM> has little to no impact on resulting pronunciation, prosody and/or style of the synthesized speech <NUM>, and in practice, only impacts audio fidelity of the TTS audio <NUM> as the synthesizer <NUM> converts a representation of a speech signal (e.g., a mel frame or spectrogram frame output by the decoder <NUM>) into a waveform.

<FIG> shows a schematic view of an example pronunciation decision model <NUM> generating a pronunciation decision <NUM> that selects which one of a user pronunciation <NUM> of a particular word or a different TTS pronunciation <NUM> of the same particular word is more reliable for use in TTS audio <NUM>. The pronunciation decision model <NUM> may execute on the user device <NUM>, the remote system <NUM>, or a combination thereof. In some implementations, the pronunciation decision model <NUM> is user-specific in that the pronunciation decision model <NUM> is customized for making pronunciation decisions <NUM> specific to the user <NUM> based on learnings from dialogs/interactions between the user <NUM> and the digital assistant <NUM>. In other implementations, the pronunciation decision model <NUM> is shared by multiple users and instantaneously learns to make pronunciation decisions <NUM> based on dialogs all the users <NUM> have with their digital assistants <NUM>.

The pronunciation decision model <NUM> may receive the user pronunciation <NUM> for the particular word using various techniques. For instance, the user pronunciation <NUM> of the particular word may be extracted from an intermittent state of the ASR system <NUM> while the ASR system <NUM> was processing the audio data <NUM> corresponding to the query <NUM> spoken by the user. In another implementation, the model <NUM> receives the user pronunciation <NUM> of the particular word by extracting a user acoustic representation of the particular word from the audio data. Here, the user acoustic representation conveys the user pronunciation <NUM> for the particular word. In yet another implementation, the audio data <NUM> corresponding to the query <NUM> is processed to generate a user phoneme representation that conveys the user pronunciation of the particular word. The presented disclosure is not limited to any specific technique for obtaining user pronunciation of words.

The pronunciation decision model <NUM> may receive the TTS pronunciation <NUM> for the particular word using various techniques. In one example, the pronunciation decision model <NUM> receives a phoneme representation that conveys the TTS pronunciation <NUM> of the particular word. For instance, the TTS system <NUM> may include a grapheme-to-phoneme model (e.g., normalization engine of <FIG>) that converts a grapheme representation of the TTS input <NUM> into a corresponding phoneme representation. In another example, a TTS acoustic representation (e.g., a waveform) of the particular word is extracted from an initial sample of TTS audio generated by the TTS system <NUM> (but not audibly output). Here, the acoustic representation conveys the TTS pronunciation <NUM> of the particular word. The TTS acoustic representation could also be extracted from the audio signal As (e.g., the output sequence mel-frequency spectrograms) generated as output from the decoder <NUM> of the TTS model <NUM> of <FIG>. In yet another example, the TTS pronunciation <NUM> is extracted from an intermittent state of the TTS system <NUM> while processing the textual representation of the TTS input <NUM> for conversion into speech. For instance, the extracted intermittent state of the TTS system <NUM> may include a sequence context vectors Vc output from the encoder <NUM> of the TTS model <NUM> of <FIG>.

For a particular word present in both a query <NUM> spoken by the user <NUM> and a TTS input <NUM> including a textual representation of a response to the query <NUM>, the TTS system <NUM> or the pronunciation decision model <NUM> may identify when the user pronunciation <NUM> and the TTS pronunciation <NUM> of the particular word are different from one another, and thus competing as to which one will ultimately be used for pronouncing the particular word in the TTS audio <NUM> provided for audible output as synthesized speech. The pronunciation decision model <NUM> is trained to generate the pronunciation decision <NUM> based on one or more user pronunciation-related features (UPFs) <NUM> and/or one or more TTS pronunciation-related features (TTSPFs) <NUM> associated with the particular word. Here, the pronunciation decision <NUM> selects the one of the user pronunciation <NUM> or the TTS pronunciation <NUM> associated with a highest confidence to use for pronouncing the particular word in the TTS audio <NUM> conveying the response to the query <NUM>. In some examples, the pronunciation decision model <NUM> includes a user pronunciation confidence estimator <NUM> and a TTS pronunciation estimator <NUM>. In these examples, the user pronunciation confidence estimator <NUM> estimates a user pronunciation confidence <NUM> for the user pronunciation <NUM> of the particular word that indicates a likelihood that the user pronunciation <NUM> is preferred. On the other hand, the TTS pronunciation estimator <NUM> estimates a TTS pronunciation confidence <NUM> for the TTS pronunciation <NUM> of the particular word that indicates a likelihood that the TTS pronunciation <NUM> is preferred. In some implementations, the pronunciation decision model <NUM> generates the pronunciation decision <NUM> as a log-likely ratio between the user pronunciation confidence <NUM> and the TTS pronunciation confidence <NUM>.

The UPFs <NUM> associated with the user pronunciation <NUM> of the particular word may include a multitude of different features conveying information helpful for the pronunciation decision model <NUM> in ascertaining the reliability (or unreliability) associated with the pronunciation of the particular word when spoken by the user <NUM> in the query <NUM>. For instance, the transcription <NUM> of the query <NUM> may convey the word type of the particular word and the context of the particular word in view of other words recognized in the transcription. Additionally, linguistic demographic information associated with the user <NUM> may convey a proficiency of the user in a particular language. For instance, a user <NUM> that speaks American English as a native language may be prone to mispronouncing a name of an Italian restaurant, whereas a bilingual speaker of English and Italian is less likely to mispronounce Italian names/words. The UPFs may provide any type of demographic information, such as age and gender, associated with the user <NUM>. The geographical area in which the user <NUM> is located when the query <NUM> was spoken may serve as a UPF <NUM> associated with the user pronunciation <NUM> of the particular word. For example, in a query <NUM> for navigation instructions to a street address with an uncommon name, the TTS system <NUM> may be inclined to mispronounce the uncommon name since the presence of the uncommon name may be scarce or non-existent in training examples used to train the TTS system <NUM>. Accordingly, UPFs <NUM> specifying that the user's geographical area is proximate to the requested street address with the uncommon name may serve as a strong indicator that the user pronunciation <NUM> of the uncommon name may be more reliable than the TTS pronunciation <NUM> of the uncommon name. The user may explicitly grant access to the user's geographical area and may deny access to the geographical area at any time. In this scenario, coupled with the geographical area of the user <NUM> provided by the UPF <NUM>, a TTSPF <NUM> indicating that the location/dialect for which the TTS system <NUM> is trained may be outside the geographical area associated with the requested street address, thereby further bolstering the user pronunciation confidence <NUM> and/or reducing the TTS pronunciation confidence <NUM>.

Moreover, historical information providing a frequency of using the user pronunciation <NUM> when pronouncing the particular word in previous queries spoken by the user and/or other users may serve as UPFs <NUM> provided as input to the pronunciation decision model <NUM>. For example, multiple users across a geographical area frequently using a same user pronunciation <NUM> for a particular word may indicate emergent pronunciations and/or untracked dialectical variations for which the TTS system <NUM> and/or ASR system <NUM> have not been trained. In addition to improving the confidence pronunciation decision <NUM> generated by the model <NUM>, the knowledge of emergent pronunciations and/or untracked dialectical variations in pronouncing a particular word may be used for updating the TTS system <NUM> to learn to produce TTS audio <NUM> that reflects the user pronunciation <NUM> of the particular word.

In some examples, establishing a threshold frequency of the user <NUM> using the user pronunciation <NUM> of the particular word in previous queries <NUM> serves as implicit user feedback indicating the user pronunciation <NUM> as a preferred pronunciation of the particular word. For example, in one or more previous dialog sessions where the decision model <NUM> selected the TTS pronunciation <NUM> over the different user pronunciation <NUM> for pronouncing the particular word in TTS audio <NUM>, subsequent queries <NUM> where the user <NUM> and/or other users continued to use the user pronunciation <NUM> for pronouncing the particular word may indicate that the user <NUM> and/or other users prefer the user pronunciation <NUM> over the TTS pronunciation <NUM>. This implicit feedback may be further used to train/update the TTS system <NUM>.

In some additional implementations, the UPFs <NUM> associated with the user pronunciation <NUM> of the particular word include one or more ASR confidence features indicating a likelihood that the recognition of the particular word by the ASR system <NUM> in the audio data <NUM> is correct. The pronunciation decision model <NUM> may obtain the ASR confidence features directly from the ASR system <NUM> used for generating the transcription <NUM> of the query <NUM>. The ASR confidence features may include, without limitation, likelihood/posterior scores associated with recognizing the particular word, a confidence score (obtained by the ASR system <NUM> or an auxiliary confidence model) indicating the likelihood that the particular word was recognized correctly, and any recognition confusability associated with recognizing the word. Here, recognition confusability may be ascertained from word lattice posteriors of candidate recognition results for the transcription <NUM> and/or entropy of posterior distribution of possible recognition results of a neural network model of the ASR system <NUM>.

In one example, a previous misrecognition of the particular word by the ASR system <NUM> when pronounced using the TTS pronunciation <NUM> followed by a subsequent successful recognition of the particular word that instead pronounced the word using the user pronunciation <NUM> may indicate that the user pronunciation <NUM> is not reliable because it is likely that the user <NUM> only chose the user pronunciation <NUM> to spoof the ASR system <NUM> and avoid misrecognition errors. Such knowledge may be used to update the ASR system <NUM> to become robust in recognizing the TTS pronunciation <NUM> of the particular word. Additionally or alternatively, knowledge that a previous misrecognition of the particular word by the ASR <NUM> when pronounced using the user pronunciation <NUM> followed by a subsequent recognition of the particular word that was instead pronounced using the TTS pronunciation <NUM> may be used to update the ASR system <NUM> to become robust in recognizing the user pronunciation <NUM>.

Similar to the UPFs <NUM>, the TTSPFs <NUM> associated with the TTS pronunciation <NUM> of the particular word may include a multitude of different features conveying information helpful for the pronunciation decision model <NUM> in ascertaining the reliability (or unreliability) for using the TTS pronunciation <NUM> to pronounce the particular word present in the TTS input <NUM> received by the TTS system <NUM>. For instance, the TTS input <NUM> including the textual representation of the response to the query <NUM> may convey the word type of the particular word and the context of the particular word in view of other words present in the TTS input <NUM>. Additionally, the geographical area/dialect for which the TTS system <NUM> is trained may be provided as a TTSPF <NUM> to the pronunciation decision model <NUM>.

In some examples, the TTSPFs <NUM> indicate whether the TTS pronunciation <NUM> includes a verified preferred pronunciation for the particular word. A TTS pronunciation <NUM> that is verified as a preferred pronunciation may serve as a strong indicator for increasing the TTS pronunciation confidence <NUM> estimated by the TTS pronunciation confidence estimator <NUM> for the TTS pronunciation <NUM> and decreasing the user pronunciation confidence <NUM> estimated by the user pronunciation confidence estimator <NUM> for the user pronunciation <NUM>. A preferred pronunciation for a particular word may be verified manually during training of the TTS system <NUM> by training the TTS system <NUM> on training sample pairs that map audio of preferred pronunciations of particular words with corresponding graphemic representations. For instance, when a user <NUM> who does not speak Swedish as a native language is traveling abroad in Sweden and speaks a query <NUM> for directions to a city in Sweden but mispronounces the city name, TTSPFs <NUM> indicating that the TTS system <NUM> is trained for Sweden and the TTS pronunciation <NUM> of the city name is verified will increase confidence that the TTS pronunciation <NUM> is more reliable than the user pronunciation <NUM>.

Preferred pronunciations may also include user-verified preferred pronunciations in which the user <NUM> provides preferred pronunciations for words specified by the user and configures the TTS system <NUM> to use the preferred pronunciations for pronouncing the words in TTS audio <NUM>. This scenario is common for pronouncing custom names or contact names that may have unique pronunciations that the TTS system <NUM> would otherwise pronounce differently based solely on graphemic representations of the those names. Similarly, the ASR system <NUM> may have difficulty in accurately recognizing these contact names due to their unique pronunciations. To provide a preferred pronunciation for a particular word (e.g., a name or other proper noun), the user may speak the word using the preferred pronunciation and the TTS system <NUM> may map the preferred pronunciation to a corresponding graphemic representation so that the TTS system <NUM> pronounces the word using the preferred pronunciation when the corresponding graphemic representation is present in a TTS input <NUM>.

In other examples, the user <NUM> provides a preferred pronunciation for a particular word via explicit feedback during a previous dialog session with the digital assistant <NUM>. In these examples, after audible output of TTS audio <NUM> conveying a response to a query in which a particular word is pronounced differently than how the word was pronounced in the query spoken by the user <NUM>, the digital assistant <NUM> may prompt the user <NUM> to provide explicit feedback to indicate which pronunciation the user prefers for pronouncing the word in subsequent TTS audio <NUM>. In another scenario, without being prompted, the user <NUM> may provide explicit feedback by blurting out a response such as, "You didn't say it right" or "You mean [using the preferred pronunciation]?", if the user <NUM> is not satisfied with the pronunciation of a particular word in TTS audio <NUM>. While a TTSPF <NUM> indicating that the TTS pronunciation <NUM> of a particular word includes a user-verified preferred pronunciation seems to conflict the notion of why the user would consciously choose to use a different user pronunciation <NUM> of the particular word when speaking a query <NUM>, the user <NUM> may be intentionally mispronouncing, or at least pronouncing the particular word inconsistent with the pronunciation preferred by the user <NUM>, as an effort to spoof the ASR system <NUM> and avoid misrecognition errors.

In some implementations, the TTSPFs <NUM> indicate whether the TTS pronunciation <NUM> includes an unverified pronunciation for the particular word. An unverified pronunciation may increase the confidence in the reliability of the TTS pronunciation, but to a lesser extent than a verified pronunciation. The TTS pronunciation <NUM> for a particular word may be unverified when the pronunciation is learned/estimated by the TTS system <NUM> via pronunciation mining from one or more auxiliary information sources. For instance, in the example above, the TTS system <NUM> may learn the TTS pronunciation <NUM> for the word/term Bexar as an unverified pronunciation estimated using pronunciation mining from audio and/or video information sources that correctly use the canonical pronunciation "BAY-ar" in the context of Texas.

Additionally or alternatively, other TTSPFs <NUM> input to the pronunciation decision model <NUM> for making pronunciation decisions <NUM> include, without limitation, pronunciation variant and pronunciation complexity features. A pronunciation variant feature may indicate whether any other variants for pronouncing the particular word exist. These variants in pronunciation of a particular word may be learned by the pronunciation decision model <NUM> over time. Different pronunciations may be mapped to different contexts. For instance, a vast majority of speakers may use a particular pronunciation of a particular word while speakers in a specific geographical region may exclusively pronounce the same word differently. On the other hand, a pronunciation complexity feature may indicate a likelihood of user mispronunciation of the particular word. The TTS system <NUM> may provide a pronunciation complexity feature indicating the strong likelihood of user mispronunciation by identifying specific phoneme sequences/representations that may be perceived as difficult for users to pronounce. TTSPFs <NUM> may further indicate mis-pronunciation statistics indicating a frequency in which a user population mispronounces the particular word in queries.

<FIG> provides a schematic view <NUM> of an example dialog between a user <NUM> and a digital assistant <NUM> executing on a user device <NUM> in which a user pronunciation <NUM> for a particular word present in a query <NUM> is different than a TTS pronunciation <NUM> of the same word present in a TTS input <NUM> conveying a response to the query <NUM>. In the example shown, the user <NUM> speaks the query <NUM>, "Play songs by A$AP Rocky," directed toward the digital assistant <NUM>. In this example, the user pronunciation <NUM> of the word "A$AP" is A-S-A-P and the ASR system <NUM> is robust for correctly recognizing the user pronunciation <NUM> as "A$AP". Here, the ASR system <NUM> generates a transcript <NUM> of the query <NUM> and applies NLU functionality to interpret the query <NUM> as a request for the digital assistant <NUM> to have a music player application audibly output music songs by the artist named A$AP Rocky. The digital assistant <NUM> also generates a response to the query <NUM> confirming that the digital assistant <NUM> understands the query <NUM> and is in progress of fulfilling the query <NUM>. Here, the digital assistant <NUM> generates a textual representation of the response to the query <NUM>, "Playing latest songs by A$AP Rocky", which is provided as a TTS input <NUM> to the TTS system <NUM> for conversion into TTS audio <NUM>.

However, in this scenario, the TTS system <NUM> does not know how pronounce the word "A$AP" present in the TTS input <NUM>, and therefore no TTS pronunciation <NUM> exists for the word "A$AP". Based on UPFs <NUM> that include one or more ASR confidence features indicating there is a strong confidence for recognizing the word A$AP pronounced using the user pronunciation <NUM>, the pronunciation decision model <NUM> may conclude that the user pronunciation confidence <NUM> is high and therefore generate a pronunciation decision <NUM> that selects the user pronunciation <NUM> of A-S-A-P for pronouncing the word A$AP in the TTS audio <NUM>. In the example shown, the TTS audio <NUM> is audibly output from the user device <NUM> and includes a synthesized speech representation of the response to the query <NUM> that uses the user pronunciation <NUM> to pronounce the word "A$AP".

If the pronunciation decision <NUM> is associated with a confidence that does not meet a certain threshold, the model <NUM> and/or TTS system <NUM> may determine there may be some doubt for using the user pronunciation <NUM> and therefore prompt the user <NUM> to indicate whether or not the agrees with the pronunciation decision <NUM> or prefers a different pronunciation. The user's response to the prompt may include explicit feedback as described above. The prompt may be an audible prompt in which the TTS system <NUM> outputs synthesized speech asking the user <NUM> if he or she agrees with the pronunciation decision or prefers a different pronunciation. The prompt may additionally or alternatively include a visual prompt that display a notification on a screen in communication with the user device <NUM>. Here, the user could provide a user input indication selecting a graphical element indicating the preferred pronunciation, or at a minimum, indicating satisfaction or dissatisfaction with the pronunciation decision <NUM>.

When the pronunciation decision <NUM> selects the user pronunciation <NUM> as more reliable than the TTS pronunciation <NUM> (or no TTS pronunciation <NUM> is available), scenarios may exist when the TTS system <NUM> is unable to produce TTS audio <NUM>, or the TTS audio <NUM> would otherwise include a synthesis quality not meeting quality standards, using the user pronunciation <NUM> for pronouncing the particular word. This scenario may occur when the user pronunciation <NUM> of the particular word is spoken in an accent/dialect different than the accent/dialect for which the TTS system <NUM> (and associated TTS model <NUM>) was trained. The TTS system <NUM> may employ a variety of different techniques for producing the TTS audio <NUM> using the user pronunciation <NUM> on-the-fly. In some examples, the TTS system <NUM> uses an acoustic representation of the particular word extracted from the audio data <NUM> corresponding to the query <NUM> spoken by the user. In these examples, the acoustic representation of the particular word extracted from the audio data <NUM> may be inserted into the TTS audio <NUM>. On the other hand, the TTS system <NUM> may further extract a phonemic representation from the acoustic representation of the particular word and use the phonemic representation to produce the TTS audio <NUM> with the user pronunciation <NUM> of the particular word.

In additional examples, the TTS system <NUM> obtains a latent representation derived from a portion of the audio data <NUM> corresponding to the spoken query <NUM> that includes the user pronunciation <NUM> of the particular word. In these examples, the latent representation guides the TTS system <NUM> (and associated TTS model) to produce the TTS audio <NUM> that pronounces the particular word using the user pronunciation <NUM>. In yet another example, voice conversion techniques are applied to the portion of the audio data <NUM> corresponding to the spoken query <NUM> that includes the user pronunciation <NUM> to produce TTS audio <NUM> that uses the user pronunciation <NUM> to pronounce the particular word in the synthesized voice.

<FIG> is a flowchart of an example arrangement of operations for a method <NUM> of selecting one of a user pronunciation <NUM> or a different TTS pronunciation <NUM> of a word that is present in both a spoken query <NUM> and a TTS input <NUM> that includes a textual representation of a response to the query <NUM>. At operation <NUM>, the method <NUM> includes receiving a user pronunciation <NUM> of a particular word present in a query <NUM> spoken by a user <NUM>. At operation <NUM>, the method <NUM> includes receiving a TTS pronunciation <NUM> of the same particular word that is present in a TTS input <NUM>. The TTS input <NUM> includes a textual representation of a response to the query <NUM> and the TTS pronunciation <NUM> of the particular word is different than the user pronunciation <NUM> of the particular word. At operation <NUM>, the method <NUM> includes obtaining user pronunciation-related features <NUM> associated with the user pronunciation <NUM> of the particular word. At operation <NUM>, the method <NUM> includes obtaining TTS pronunciation-related features <NUM> associated with the TTS pronunciation <NUM> of the particular word. At operation <NUM>, the method <NUM> includes generating, as output from a pronunciation decision model <NUM> configured to receive the user pronunciation-related features <NUM> and the TTS pronunciation-related features <NUM> as input, a pronunciation decision <NUM> selecting the one of the user pronunciation <NUM> of the particular word or the TTS pronunciation <NUM> of the particular word that is associated with a highest confidence for use in TTS audio <NUM>. At operation <NUM>, the method <NUM> includes providing, for audible output from a user device <NUM> associated with the user <NUM>, the TTS audio <NUM> that includes a synthesized speech representation of the response to the query <NUM> using the one of the user pronunciation <NUM> for the particular word or the TTS pronunciation <NUM> for the particular word that was selected by the pronunciation decision <NUM> output from the pronunciation decision model <NUM>.

<FIG> is schematic view of an example computing device <NUM> (e.g., system <NUM>) that may be used to implement the systems and methods described in this document.

The computing device <NUM> includes a processor <NUM> (e.g., memory hardware <NUM>), memory <NUM> (e.g., memory hardware <NUM>), a storage device <NUM>, a high-speed interface/controller <NUM> connecting to the memory <NUM> and high-speed expansion ports <NUM>, and a low speed interface/controller <NUM> connecting to a low speed bus <NUM> and a storage device <NUM>. The processor <NUM> (i.e., the data processing hardware <NUM> of the user device <NUM> or the data processing hardware <NUM> of the remote system <NUM>) can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display <NUM> coupled to high speed interface <NUM>. The memory <NUM> and the storage device <NUM> may include the memory hardware <NUM> of the user device <NUM> or the memory hardware <NUM> of the remote system <NUM>).

Claim 1:
A computer-implemented method (<NUM>) when executed on data processing hardware (<NUM>) causes the data processing hardware (<NUM>) to perform operations comprising:
receiving a user pronunciation (<NUM>) of a particular word present in a query (<NUM>) spoken by a user;
receiving a text-to-speech, TTS, pronunciation (<NUM>) of the same particular word that is present in a TTS input (<NUM>), the TTS input (<NUM>) comprising a textual representation of a response to the query (<NUM>), and the TTS pronunciation (<NUM>) of the particular word is different than the user pronunciation (<NUM>) of the particular word;
obtaining user pronunciation-related features (<NUM>) associated with the user pronunciation (<NUM>) of the particular word;
obtaining TTS pronunciation-related features (<NUM>) associated with the TTS pronunciation (<NUM>) of the particular word;
generating, as output from a pronunciation decision model (<NUM>) configured to receive the user pronunciation-related features (<NUM>) and the TTS pronunciation-related features (<NUM>) as input, a pronunciation decision (<NUM>) selecting the one of the user pronunciation (<NUM>) of the particular word or the TTS pronunciation (<NUM>) of the particular word that is associated with a highest confidence for use in TTS audio (<NUM>); and
providing, for audible output from a user device (<NUM>) associated with the user, the TTS audio (<NUM>) comprising a synthesized speech representation of the response to the query (<NUM>) using the one of the user pronunciation (<NUM>) for the particular word or the TTS pronunciation (<NUM>) for the particular word that was selected by the pronunciation decision (<NUM>) output from the pronunciation decision model (<NUM>).