Patent Description:
This specification generally relates to speech processing.

Speech processing is the study of speech signals and the processing methods of signals. The signals are usually processed in a digital representation, so speech processing can be regarded as a special case of digital signal processing, applied to speech signals. Aspects of speech processing includes the acquisition, manipulation, storage, transfer and output of speech signals.

Speech synthesizers typically require a transcription as an input. A speech synthesizer receives a transcription and outputs audio data of a synthesized utterance of the transcription. In order to convert a user's speech to a synthesized utterance, an automated speech recognizer would have to perform automated speech recognition on the audio data of the user's speech to generate a transcription of the user's speech. The speech synthesizer would then generate a synthesized utterance of the transcription of the user's speech.

This technique of performing automated speech recognition and speech synthesis may be taxing on a computing system. It would be beneficial to have a process that is capable of converting speech audio received from a user to speech audio in a voice other than the voice of the user without the need to perform automated speech recognition on the user's speech. The discussion below describes a process of using a model trained using machine learning to convert speech audio in the voice of a speaker to speech audio in a different voice without performing speech recognition. The model receives the speech audio spoken by the speaker and converts the speech audio to a mathematical representation. The model converts the mathematical representation to speech audio in a different voice without performing speech recognition on the speech audio spoken by the speaker. Such a process is for example also described in <NPL>.

In some implementations, a speech synthesis system is able to convert first audio data including an utterance in a first voice into second audio data that includes the same utterance in a second voice. The conversion can be done by acting directly on samples or features of the first audio data, without converting the audio to an intermediate representation (e.g., text, phones, etc.). The system can use a sequence-to-sequence to normalize arbitrary speech, potentially including background noise, and generate the same content in the voice of a single predefined target speaker. The source speech can be from any speaker or accent, and may contain complex prosodic patterns, imperfections, and background noise, all of which are removed through the normalization process as the first audio data is converted into clean second audio data with a fixed accent and consistent articulation and prosody. In other words, the system may be used to project away all non-linguistic information, including speaker characteristics, and to retain only what is been said, not who, how, or where it is said.

This type of normalization has multiple potential applications. Fully normalizing any voice to a single speaker with clean audio could significantly simplify speech recognition models, which could be reduced to supporting a single speaker. Removing the identity of the speaker might be useful when logging sensitive and private speech data, allowing users to transmit only converted speech to servers erased of 'acoustic' identity. Reducing all accents into a single voice with a predefined accent may also alleviate biases and discrimination while maintaining a natural human voice as opposed to acoustically masked audio, for example, for phone interviews or recorded candidate talks given to hiring committees. Another application would be to facilitate the understanding of speech content of accents that are foreign to the listener, i.e. improving intelligibility of heavily accented speech.

According to an innovative aspect of the subject matter described in this application, a method for end to end speech conversion is defined by claim <NUM>.

These and other implementations can each optionally include one or more of the following features. The actions further include receiving, by the computing device, data indicating that a bot that is configured to conduct conversations with a given human is not configured to generate a response to a third utterance received from a human; and, based on receiving the data indicating that the bot is not configured to generate the response to the third utterance received from the human, transmitting, by the computing device and to a human operator, a request to respond to the third utterance received from the human. The action of receiving the first audio data of the first utterance of the one or more first terms spoken by the user includes receiving the first audio data of the first utterance of the one or more first terms spoken by the human operator in response to the third utterance. The action of receiving the first audio data of the first utterance of the one or more first terms spoken by the user includes receiving the first audio data of the first utterance of the one or more first terms spoken by the user while answering a telephone call.

The actions further include receiving audio data of collection of utterances; obtaining a transcription of each utterance in the collection of utterances; providing the transcriptions of each utterance as an input to a text to speech model; receiving, for each transcription of each utterance, audio data of an additional collection of utterances in a synthesized voice; and training the model using the audio data of the collection of utterances and the audio data of an additional collection of utterances in a synthesized voice. The actions further include receiving, by the computing device, third audio data of a third utterance of one or more third terms spoken by an additional user; providing, by the computing device, the third audio data as an input to the model; in response to providing the third audio data as an input to the model, receiving, by the computing device, fourth audio data of a fourth utterance of the one or more third terms spoken in the synthesized voice; and providing, for output by the computing device, the fourth audio data of the fourth utterance of the one or more third terms spoken in the synthesized voice. The actions further include bypassing, by the computing device, obtaining a transcription of the first utterance. The model is configured to adjust a time period between each of the one or more first given terms. The model is configured to adjust a speaking time of each of the one or more first given terms.

Other implementations of this aspect include corresponding systems, apparatus, and computer programs recorded on computer storage devices, each configured to perform the operations of the methods.

Particular implementations of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages. A computing system is able to receive audio data of an utterance spoken in a voice of a user and output audio data of a synthesized utterance spoken in a synthesized voice without the added overhead of performing automated speech recognition on the received audio data.

<FIG> illustrates an example system <NUM> that converts speech audio <NUM> received from a user <NUM> to synthesized speech audio <NUM> without performing speech recognition. Briefly, and as described in more detail below, the user <NUM>, who speaks with a British accent speaks the utterance <NUM> in the vicinity of the computing device <NUM>. The computing device <NUM> transmits the audio data <NUM> of the utterance <NUM> to the speech to speech conversion server <NUM>. The speech to speech conversion server <NUM> converts the audio data <NUM> of the utterance <NUM> to audio data <NUM> of a synthesized utterance <NUM>. The speech to speech conversion server <NUM> transmits the audio data <NUM> of the synthesized utterance <NUM> to computing device <NUM>, and the computing device <NUM> outputs the synthesized utterance <NUM>. In some implementations, the functionality of the end to end speech conversion server <NUM> is built into the computing device <NUM> or the computing device <NUM> or both.

In more detail, the user <NUM> and the user <NUM> are speaking with each other through computing device <NUM> and computing device <NUM>. The user <NUM> and the user <NUM> may be speaking over a telephone call or another type of voice communication protocol, for example, voice over internet protocol. While the user <NUM> and the user <NUM> may speak the same language, it may be difficult for user <NUM> to understand user <NUM> because user <NUM> has a thick accent. In this example, the user <NUM> may be British, and the user <NUM> may have an easier time understanding an American accent <NUM> than the British accent <NUM> of user <NUM>.

To address this problem, the computing device <NUM> may provide audio data <NUM> of the utterance <NUM> and other utterances spoken by the user <NUM> to the speech to speech conversion server <NUM>. The speech to speech conversion server <NUM> may be configured to convert audio data of an utterance spoken by a user who may speak with an accent into audio data of a synthesized utterance that may have a different accent. To accomplish this conversion, a server would typically be configured to perform speech recognition on the audio data of the utterance spoken by the user with the accent. The speech recognizer may be configured to recognize speech spoken in the accent of the user or may be configured to recognize speech spoken in any accent. The server would then provide the transcription to a speech synthesizer that would generate audio data of synthesized speech with a different accent. The speech to speech conversion server <NUM> operates in a different manner.

The speech to speech conversion server <NUM> receives the audio data <NUM> of the utterance <NUM> from the computing device <NUM> and provides the audio data <NUM> of the utterance <NUM> to the model <NUM>. The speech to speech conversion server <NUM> trains the model <NUM> to convert the audio data <NUM> of the utterance <NUM> spoken in a British accent <NUM> to audio data <NUM> of the synthesized utterance <NUM> in an American accent <NUM>. The speech to speech conversion server <NUM> does not use a speech recognizer <NUM> to perform this conversion. The speech recognizer <NUM> may remain inactive during the conversion process. Instead, the model <NUM> provides the audio data <NUM> of the utterance <NUM> to an encoder <NUM>. The encoder <NUM> may be configured to convert the audio data <NUM> of the utterance <NUM> to an internal representation, such as a series of vectors. For example, as the encoder <NUM> receives the audio data <NUM> of the utterance <NUM>, the encoder <NUM> may process five frames of audio and convert those five frames of audio to ten vectors. The vectors are not a transcription of the frames of audio data <NUM>, but rather a mathematical representation of the frames of the audio data <NUM>. The model <NUM> provides the series of vectors to the spectrogram decoder <NUM>. The spectrogram decoder <NUM> may be configured to generate audio data of a synthesized utterance based on the vectors received from the encoder <NUM>. For example, the spectrogram decoder <NUM> may receive the ten vectors from the encoder <NUM> that represent the five frames of audio. The spectrogram decoder <NUM> generates five frames of audio data <NUM> of the synthesized utterance <NUM> that includes the same words or parts of words as the five frames of audio data, but with a different voice than the user <NUM>.

The speech to speech conversion server <NUM> provides the audio data <NUM> of the synthesized utterance <NUM> to the computing device <NUM>. In the example shown in <FIG>, the speech to speech conversion server <NUM> provides audio data <NUM> of the synthesized utterance of "Can I make an appointment for tomorrow?" The synthesized utterance <NUM> may have an American accent <NUM>. In some implementations, the synthesized utterance <NUM> may have the same accent of the user <NUM> and a different voice than the user <NUM>. The voice of the synthesized utterance <NUM> may be such that the user <NUM> or another user would be unable to identify the user <NUM> as the speaker of the utterance <NUM>. In some implementations, the cadence of the synthesized utterance <NUM> may be different than the cadence of utterance <NUM>. The speech to speech conversion server <NUM> may adjust the cadence of the synthesized utterance <NUM> to increase the likelihood that the user <NUM> would be able to understand the synthesized utterance <NUM>.

The computing device <NUM> receives the audio data <NUM> of the synthesized utterance <NUM> and outputs the audio data <NUM> through a speaker or other audio output device. In some implementations, the speech to speech conversion server <NUM> continuously generates portions of the synthesized utterance <NUM> as the user <NUM> speaks the corresponding portions of the utterance <NUM>. For example, the speech to speech conversion server <NUM> may generate one second of the synthesized utterance <NUM> after receiving one second of the utterance <NUM>. By continuously generating portions of the synthesized utterance <NUM>, the conversation between user <NUM> and <NUM> may be more naturally paced. In some implementations, the speech to speech conversion server <NUM> may determine when the user <NUM> has stopped speaking. After determining that the user <NUM> has stopped speaking, the speech to speech conversion server <NUM> converts the audio data <NUM> of the utterance <NUM> to the audio data <NUM> of the synthesized utterance <NUM>.

The speech to speech conversion server <NUM> includes various components to generate the training data and to train the model <NUM>. The speech to speech conversion server <NUM> includes transcriptions of utterances <NUM> and audio data of the utterances <NUM>. The utterances may be utterances spoken by different users with different types of accents. In some implementations, the transcriptions of utterances <NUM> are generated by an automated speech recognizer. The speaker of each utterance may verify the accuracy of the transcription before the transcription is stored in the transcriptions of utterances <NUM> and the audio data is stored in the audio data of the utterances <NUM>. In some implementations, the transcriptions of utterances <NUM> are generated by one or more people.

The speech to speech conversion server <NUM> provides the transcriptions of utterances <NUM> to a speech synthesizer <NUM>. The speech synthesizer is configured to generate audio data of synthesized utterances <NUM> of the transcriptions <NUM>. The speech synthesizer may be configured to generate the audio data of synthesized utterances <NUM> in a single voice. The voice may have a particular accent such as an American accent or a British accent. The audio data of synthesized utterances <NUM> may be free of any background noise or other audio artifacts.

The speech to speech conversion server <NUM> provides the audio data of synthesized utterances <NUM> and the audio data of utterances <NUM> to the model trainer <NUM>. The model trainer <NUM> trains the model <NUM> using machine learning techniques. The model trainer <NUM> trains the model <NUM> to receive audio data similar to the audio data of utterances <NUM> and output audio data similar to the audio data of synthesized utterances <NUM> without performing speech recognition on the received audio data. The model trainer <NUM> trains the model <NUM> to output utterances in the same voice as the synthesized utterances from the speech synthesizer <NUM> even when the model <NUM> receives different inputs that include audio data of different utterances in different voices.

In some implementations, the speech to speech conversion server <NUM> may use audio data of utterances <NUM> that include different audio characteristics. This may result in a model <NUM> that is configured to handle an input of audio data that has those different characteristics. In some implementations, the speech to speech conversion server <NUM> may add audio characteristics to the audio data of utterances <NUM> so that the model trainer <NUM> trains the model <NUM> to handle audio characteristics similar to the added audio characteristics.

For example, the speech to speech conversion server <NUM> may add varying levels of noise to the audio data of utterances <NUM>. The varying levels of noise can include different types of noise such as stationary noise and/or non-stationary noise. The stationary noise can include varying levels of road noise, varying levels of background speech noise similar to a cocktail party or restaurant, varying levels of fan noise, and/or any other similar type of noise. The non-stationary noise can include varying levels of television noise, varying levels of wind gust noise, varying levels of background music noise, and/or any other similar type of noise. The speech to speech conversion server <NUM> may add different levels and different types of noise to audio data of the same utterance. This may result in multiple audio data samples that match a same transcription with each audio data sample including the same underlying utterance audio data with different levels and different types of noise added. With the addition of noise, the model <NUM> may be better configured to process received audio data that include background noise in addition to the audio data of the utterance.

As another example, the speech to speech conversion server <NUM> may process audio data of utterances <NUM> from users who may have speech impediments that cause the use to speak with disfluencies, such as those users with amyotrophic lateral sclerosis. The model trainer <NUM> may train the model <NUM> using audio data of utterances from users with speech impediments and transcriptions of the utterances so that the model <NUM> is able to receive audio data of an utterance spoken by a user with a speech impediment and output audio data of an utterance with a more consistent cadence that may be easier for another user to understand.

As another example, the speech to speech conversion server <NUM> may be configured to translate an utterance into a different language without transcribing the utterance. In this instance, the audio data of the utterances <NUM> may include utterances spoken in a first language, such as English. The transcriptions of the utterances <NUM> may include a transcription of the translation of the utterance in a second language, such as a Spanish translation. The speech synthesizer <NUM> may be configured to generate synthesized speech in the second language, such as synthesized Spanish utterances. The model trainer <NUM> trains the model <NUM> using the audio data of the utterances <NUM> in the first language and the audio data of the synthesized utterances <NUM> in the second language using machine learning. The resulting model <NUM> is configured to receive audio data of an utterance in a first language, e.g., English, and output audio data of a synthesized utterances in a second language, e.g., Spanish without transcribing the received utterance.

<FIG> illustrates an example system <NUM> that converts speech audio <NUM> received from an operator <NUM> who is interrupting a conversation between an automated agent <NUM> and a user <NUM> to speech audio <NUM> that imitates the automated agent <NUM>. Briefly, and as described in more detail below, the user <NUM> is conducting a conversation with the automated agent <NUM>. During the conversation, the user <NUM> speaks an utterance <NUM> to which the automated agent <NUM> is unable to respond. The operator <NUM> receives an indication that the automated agent <NUM> is unable to respond to the utterance <NUM>. The operator <NUM> provides an utterance <NUM> to respond to the utterance <NUM>. The speech to speech conversion server <NUM> converts the speech audio <NUM> of the utterance <NUM> to the voice of the automated agent <NUM> so that the user <NUM> is under the impression that the user <NUM> is still conversing with the same party. In some implementations, the functionality of the automated agent <NUM> is built into the computing device <NUM> or the computing device <NUM> or both. In some implementations, the functionality of the end to end speech conversion server <NUM> is built into the computing device <NUM> or the computing device <NUM> or both.

In more detail and in stage A, the user <NUM> initiates a telephone conversation with the automated agent <NUM>. The computing device <NUM> connects with the automated agent <NUM>. The user speaks utterance <NUM> and asks the automated agent, "Can I reserve a table for two. " The automated agent <NUM> may imitate a person such that the user <NUM> is unable to distinguish the automated agent <NUM> from an actual person. In some implementations, the automated agent <NUM> may initiate the telephone conversation with the user <NUM>. In some implementations, the conversation between the user <NUM> and the automated agent <NUM> may be a communication channel other than a telephone call such as a VOIP call or other type of voice communication.

In stage B, the computing device <NUM> detects the utterance <NUM> through a microphone or another input device and processes the audio data of the utterance <NUM> using an audio subsystem. The audio subsystem may include the microphone, an analog to digital converter, a buffer, and various other audio filters. The microphone may be configured to detect sounds in the surrounding area such as speech, e.g., the utterance <NUM>. The analog to digital converter may be configured to sample the audio data detected by the microphone. The buffer may store the sampled audio data for processing by the computing device <NUM> or for transmission by the computing device <NUM>. In some implementations, the audio subsystem may be continuously active or may be active during times when the computing device <NUM> is expecting to receive audio such as during a telephone call. In this case, the microphone may be detect audio in response to the initiation of the telephone call with the automated agent <NUM>. The analog to digital converter may be constantly sampling the detected audio data during the telephone call. The buffer may store the latest sampled audio data such as the last ten seconds of sound. The computing device <NUM> may provide the sampled and filtered audio data <NUM> of the utterance <NUM> to the automated agent <NUM>.

The automated agent <NUM> receives the audio data <NUM> of the utterance <NUM> and determines an appropriate response. The automated agent <NUM> may apply a series of rules, decision trees, neural network, and/or another decision process to determine an appropriate response. The automated agent <NUM> may generate a transcription of an appropriate response and provide the transcription to a speech synthesizer. In stage C, the speech synthesizer may generate audio data <NUM> that represents the utterance <NUM>, "For tonight?" as a response to "Can I reserve a table for two?" While a speech synthesizer may generate the utterance <NUM>, the user <NUM> may be unable to determine that the user <NUM> is speaking to a computer.

In stage D, the computing device <NUM> receives the audio data <NUM> of the utterance <NUM>. The computing device <NUM> outputs the audio data <NUM> through a speaker or other type of audio output device. The user <NUM> hears the utterance <NUM> in the synthesized voice <NUM>.

In stage E, the user <NUM> responds to the utterance <NUM> by speaking utterance <NUM>, "I have to watch the game tonight. Did you know that Smith is playing?" The computing device <NUM> detects the utterance <NUM> and processes the utterance <NUM> using the audio subsystem. In stage F, the computing device <NUM> provides the audio data <NUM> of the utterance <NUM> to the automated agent <NUM>.

The automated agent <NUM> receives the audio data <NUM> of the utterance <NUM>. The automated agent <NUM> processes the audio data <NUM> of the utterance <NUM> in a similar fashion to processing the audio data <NUM> of the utterance <NUM>. The automated agent <NUM> may apply a series of rules, decision trees, neural network, and/or another decision process to determine an appropriate response to the utterance <NUM>. In this instance, the automated agent <NUM> is unable to determine an appropriate response. The automated agent <NUM> may be unable to determine an appropriate response to a user utterance in instances where the user utterance is off topic from the conversation.

In order to continue the conversation, the automated agent <NUM> may notify an operator <NUM> who is standing by to assist when the automated agent <NUM> or other automated agents are unable to generate an appropriate response to a user utterance <NUM>. In stage G, the automated agent <NUM> generates a summary <NUM> of the conversation between the automated agent <NUM> and the user <NUM>. The summary <NUM> may consist of a transcript of the conversation up to the point where the automated agent <NUM> was unable to generate appropriate response to a user utterance <NUM>. Alternatively or additionally, the summary <NUM> may consist of a description of any tasks accomplished as a result of the conversation or agreements made between the automated agent <NUM> and the user <NUM>. In the example of <FIG>, the summary <NUM> consists of a transcript of the conversation between the automated agent <NUM> and the user <NUM>. The automated agent <NUM> may also include the status <NUM> of the conversation. The status <NUM> may describe the task that the automated agent <NUM> was unable to perform. The status <NUM> may indicate that the automated agent <NUM> was unable to perform speech recognition on the audio data <NUM> of the utterance <NUM>. In that case, the summary <NUM> would include the audio data <NUM> of the utterance <NUM>. The status <NUM> may indicate that the automated agent was unable to generate a response to the utterance <NUM>.

In stage H, the computing device <NUM> of the operator <NUM> receives the summary <NUM> and the status <NUM>. The operator <NUM> reviews the summary <NUM> and the status <NUM>. The operator <NUM> speaks utterance <NUM> as a response to the utterance <NUM> of the user <NUM> and an attempt to steer the conversation back to the original topic or to a topic that that automated agent <NUM> is likely to understand. The computing device <NUM> detects the utterance <NUM> and processes the utterance <NUM> using an audio subsystem in a similar fashion to the computing device <NUM> processing utterance <NUM> and utterance <NUM>. In the example shown in <FIG>, the user <NUM> speaks," I didn't. It should be exciting. What day did you want the reservation?".

In stage I, the computing device <NUM> transmits the audio data <NUM> of the utterance <NUM> to the speech to speech conversion server <NUM>. The speech to speech conversion server <NUM> may be similar to the speech to speech conversion server <NUM> in that the speech to speech conversion server <NUM> is configured to receive audio data of an utterance spoken in a first voice and output audio data of an utterance that includes the same words and terms spoken in a second, different voice without performing speech recognition on the audio data of the utterance spoken in the first voice. The speech to speech conversion server <NUM> may be configured to generate audio data of utterances in the same synthesized voice <NUM> as the automated agent <NUM>.

In stage J, the speech to speech conversion server <NUM> converts the audio data of utterance <NUM> to the audio data <NUM> of utterance <NUM>. The speech to speech conversion server <NUM> transmits the audio data <NUM> of utterance <NUM> to the computing device <NUM>. In some implementations, the speech to speech conversion server <NUM> transmits the audio data <NUM> of utterance <NUM> to the computing device <NUM>. The computing device <NUM> then transmits the audio data <NUM> of utterance <NUM> to the automated agent <NUM>. The automated agent transmits the audio data <NUM> of utterance <NUM> to the computing device <NUM>. In some implementations, the speech to speech conversion server <NUM> transmits the audio data <NUM> of utterance <NUM> to the automated agent <NUM>. The automated agent transmits the audio data <NUM> of utterance <NUM> to the computing device <NUM>. In some implementations, the computing device <NUM> provides a transcription of the utterance <NUM> to the automated agent <NUM> so the automated agent remains aware of the content of the conversation with the user <NUM>. The automated agent <NUM> may use the transcription of the utterance <NUM> to update the model and/or rules used by the automated agent <NUM> to generate responses.

In stage K, the computing device <NUM> outputs the utterance <NUM> through a speaker or another type of audio output device. The user <NUM> hears the utterance <NUM> and because the utterance <NUM> is in the same synthesized voice <NUM> as the utterance <NUM>, the user <NUM> is unaware that another party is participating in the conversation. The user <NUM> may respond to the utterance <NUM> by speaking a new utterance. The operator <NUM> may continue to monitor the conversation to ensure that the automated agent <NUM> is able to seamlessly take over the conversation. If necessary, the operator <NUM> can continue to speak with the user <NUM> through the speech to speech conversion server <NUM> for the remainder of the conversation or for a portion of the remaining conversation. During the conversation, the user <NUM> may be under the impression that the user <NUM> is speaking to the same real person.

<FIG> illustrates an example system <NUM> that converts speech audio <NUM> received from a callee <NUM> screening a call to speech audio <NUM> that prevents the caller <NUM> from determining that the callee <NUM> answered the call. Briefly, and as described in more detail below, the caller <NUM> places a telephone call to the callee <NUM>. The callee <NUM> may be unsure about taking the call, but instead of letting the call go to voicemail, the callee <NUM> answers and screens the call. The callee <NUM> may use a call screening feature of the computing device <NUM> that accesses a speech to speech conversion server <NUM>. The speech to speech conversion server <NUM> converts the audio data <NUM> of the utterance <NUM> spoken in the callee's voice to audio data <NUM> of the synthesized utterance <NUM> spoken in a general voice. The caller <NUM> answers the screening question unaware that the callee <NUM>, and perhaps an actual person, answered the call. In some implementations, the functionality of the end to end speech conversion server <NUM> is built into the computing device <NUM> or the computing device <NUM> or both.

In more detail and in stage A, the caller <NUM>, Alice, initiates a telephone call with the callee <NUM>, Bob. The computing device <NUM> indicates that the computing device <NUM> is receiving an incoming call by outputting a notification <NUM>. Instead of a telephone call, the caller <NUM> may initiate a voice communication over an alternate type of communication channel such as VOIP or similar type of voice communication. The caller <NUM> may initiate the telephone call from computing device <NUM>. The computing device <NUM> of the callee <NUM> indicates that the callee <NUM> is receiving a telephone call. The computing device <NUM> may give the callee <NUM> the option of answering the telephone call directly, ignoring the telephone call, sending the telephone call to voicemail, or initiating call screening.

In stage B, the callee <NUM> initiates the call screening option. Upon selection of the call screening option, the computing device <NUM> initiates communication with the speech to speech conversion server <NUM>. The computing device <NUM> indicates that the computing device <NUM> will send the speech to speech conversion server <NUM> audio data for conversion to another voice.

In stage C, the callee <NUM> speaks utterance <NUM>. The computing device <NUM> detects the utterance <NUM> through a microphone or another type of audio input device and processes the audio data using an audio subsystem. The audio subsystem may include the microphone, an analog to digital converter, a buffer, and various other audio filters. The microphone may be configured to detect sounds in the surrounding area such as speech, e.g., the utterance <NUM>. The analog to digital converter may be configured to sample the audio data detected by the microphone. The buffer may store the sampled audio data for processing by the computing device <NUM> or for transmission by the computing device <NUM>. In some implementations, the audio subsystem may be continuously active or may be active during times when the computing device <NUM> is expecting to receive audio such as during a telephone call. In this case, the microphone may be detect audio in response to the initiation of the call screening option. The analog to digital converter may be constantly sampling the detected audio data during the telephone call. The buffer may store the latest sampled audio data such as the last ten seconds of sound. The computing device <NUM> may provide the sampled and filtered audio data <NUM> of the utterance <NUM> to the speech to speech conversion server <NUM> in stage D.

The speech to speech conversion server <NUM> receives the audio data <NUM> of the utterance <NUM> spoken by the callee <NUM> from the computing device <NUM>. In some implementations, the computing device <NUM> provides instructions to the speech to speech conversion server <NUM> to convert the audio data <NUM> of the utterance <NUM> spoken by the callee <NUM> to an utterance spoken in a different voice. In some implementations, the computing device <NUM> provides instructions for where the speech to speech conversion server <NUM> should send the audio data <NUM> of the synthesized utterance <NUM> spoken in the different voice. For example, the computing device <NUM> may provide a phone number or a device identifier for computing device <NUM> and instructions to transmit the audio data <NUM> of the synthesized utterance <NUM> spoken in the different voice. In some implementations, the computing device <NUM> may provide instructions to the speech to speech conversion server <NUM> to transmit the audio data <NUM> of the synthesized utterance <NUM> spoken in the different voice back to the computing device <NUM> so that the computing device can transmit the audio data <NUM> of the synthesized utterance <NUM> spoken in the different voice to the computing device <NUM>.

In stage E, the speech to speech conversion server <NUM> generates the audio data <NUM> of the synthesized utterance <NUM> spoken in the different voice that the voice of the callee <NUM>. The speech to speech conversion server <NUM> may be similar to the speech to speech conversion server <NUM> in that the speech to speech conversion server <NUM> is configured to receive audio data of an utterance spoken in a first voice and output audio data of an utterance that includes the same words and terms spoken in a second, different voice without performing speech recognition on the audio data of the utterance spoken in the first voice. In this example, the speech to speech conversion server <NUM> receives audio data <NUM> of utterance <NUM> spoken in the voice of the callee <NUM>. The speech to speech conversion server <NUM> provides the audio data <NUM> of utterance <NUM> spoken in the voice of the callee <NUM> to a model that generates, without performing speech recognition on the audio data <NUM>, the audio data <NUM> of the utterance <NUM> spoken in a general voice that sounds like an actual person and does not sound like the callee <NUM>. The speech to speech conversion server <NUM> provides the audio data <NUM> of the utterance <NUM> to the computing device <NUM>. In some implementations, the speech to speech conversion server <NUM> provides the audio data <NUM> of the utterance <NUM> to the computing device <NUM> and the computing device <NUM> provides the audio data <NUM> of the utterance <NUM> to the computing device <NUM>.

In stage F, the computing device <NUM> outputs the audio data <NUM> of the utterance <NUM> through a speaker or other audio output device of the computing device <NUM>. The utterance <NUM> is not in the voice of the callee <NUM> but rather in a different general voice that sounds like an actual person and does not sound like the callee <NUM>. In the example of <FIG>, the caller <NUM> hears, "Please state your name and the purpose of your call" in a voice that does not sound like the callee <NUM>. The caller <NUM> may be under the impression that the caller <NUM> is conversing with a secretary or assistant of the callee <NUM>.

In stage G, the caller <NUM> responds to utterance <NUM> by speaking utterance <NUM>. The utterance <NUM> is detected by a microphone or other audio input device of the computing device <NUM>. An audio subsystem of the computing device <NUM> processes the utterance <NUM>. In the example of <FIG>, the caller <NUM> says, "This is Alice. I'm calling to schedule a meeting with Bob.

In stage H, the computing device <NUM> transmits the audio data <NUM> of the utterance <NUM> to the computing device <NUM>. The call screening feature of this example may work in one direction. In other words, with the call screening feature disguises the voice of the callee <NUM> who activated the call screening feature. The voice of the caller <NUM> remains unchanged.

In stage I, the computing device <NUM> outputs the audio data <NUM> of the utterance <NUM> through a speaker or other audio output device of the computing device <NUM>. The utterance <NUM> is in the voice of the caller <NUM>. The callee <NUM> hears, "This is Alice. I'm calling to schedule a meeting with Bob" in the voice of the caller <NUM>.

In stage J and with call screening still active, the callee <NUM> speaks utterance <NUM>. The utterance <NUM> is detected by the microphone or other audio input device of the computing device <NUM>. The audio subsystem of the computing device <NUM> processes the utterance <NUM>. In the example of <FIG>, the callee <NUM> says, "One moment.

In stage K and with call screening still active, the computing device <NUM> transmits the audio data <NUM> of the utterance <NUM> to the speech to speech conversion server <NUM>. The speech to speech conversion server <NUM> provides the audio data <NUM> of the utterance <NUM> to the same model as in stage D. The model generates the audio data <NUM> of the utterance <NUM> in a voice other than the voice of the callee <NUM>. In some implementations, the voice of the utterance <NUM> is the same as the voice of the utterance <NUM>. The speech to speech conversion server <NUM> generates the audio data <NUM> of the utterance <NUM> without performing speech recognition on the audio data <NUM>.

In stage L, the speech to speech conversion server <NUM> provides the audio data <NUM> of the utterance <NUM> to the computing device <NUM>. In some implementations, the speech to speech conversion server <NUM> provides the audio data <NUM> of the utterance <NUM> to the computing device <NUM> and the computing device <NUM> provides the audio data <NUM> of the utterance <NUM> to the computing device <NUM>.

In stage M, the computing device <NUM> outputs the audio data <NUM> of the utterance <NUM> through the speaker or other audio output device of the computing device <NUM>. The utterance <NUM> is not in the voice of the callee <NUM> but rather in the same general voice as the utterance <NUM> or another voice that sounds like an actual person. In the example of <FIG>, the caller <NUM> hears, "One moment" in a voice that does not sound like the callee <NUM>. The caller <NUM> may continue to be under the impression that the caller <NUM> is conversing with a secretary or assistant of the callee <NUM>.

In stage N, the callee <NUM> speaks utterance <NUM>. The utterance <NUM> is detected by the microphone or other audio input device of the computing device <NUM>. The audio subsystem of the computing device <NUM> processes the utterance <NUM>. In the example of <FIG>, the callee <NUM> says, "Hi Alice. This is Bob. " Before speaking the utterance <NUM>, the callee <NUM> may deactivate the call screening mode of the computing device <NUM>. The callee <NUM> may deactivate the call screening mode anytime during execution of stages K, L, or M and before stage N. By deactivating call screening mode, the computing device <NUM> returns to transmitting audio data of utterances spoken by the callee <NUM> to the computing device <NUM> instead of transmitting audio data of utterances spoken by the callee <NUM> to the speech to speech conversion sever <NUM>. In some implementations, the computing device <NUM> provides an indication to the speech to speech conversion sever <NUM> that the computing device <NUM> will not transmit audio data of subsequently received utterances to the speech to speech conversion sever <NUM> for converting to a different voice.

In stage O and with call screening inactive, the computing device <NUM> transmits the audio data <NUM> to the computing device <NUM>. This audio data transmission may be similar to an audio data transmission that happens during a typical voice conversation between two users using computing devices similar to computing device <NUM> and computing device <NUM>.

In stage P, the computing device <NUM> outputs the utterance <NUM> through the speaker or other audio output device of the computing device <NUM>. In the example of <FIG>, the computing device <NUM> outputs the utterance <NUM>, "Hi Alice. This is Bob. " The voice of the utterance <NUM> is the voice of the callee <NUM>. The caller <NUM> is likely under the impression that the person who screened the call transferred the call to the callee <NUM> and that the caller <NUM> was not speaking with the callee <NUM> during the entire call.

<FIG> is a flowchart of an example process <NUM> for converting speech audio received from a user to synthesized speech audio without performing speech recognition. In general, the process <NUM> receives audio data of an utterance that is spoken by a user. The process <NUM> converts the audio data of the utterance to audio data of another utterance in a different voice by applying the audio data of the utterance to a model. The different voice is a synthesized voice that sounds like an actual person. A person listening to the other utterance in the different voice may not realize that the original user spoke the utterance before conversion to the different voice. The process <NUM> generates the audio data of the other utterance in the different voice without performing speech recognition on the received audio data. The process <NUM> outputs the audio data of the other utterance in the different voice. The process <NUM> will be described as being performed by a computer system comprising one or more computers, for example, the system <NUM> of <FIG>, the system <NUM> of <FIG>, or the system <NUM> of <FIG>.

The system receives first audio data of a first utterance of one or more first terms spoken by a user (<NUM>). The user may speak in the typical voice of the user. In some implementations, the user speaks the first utterance while answering a telephone call. In some implementations, the user may activate a call screening feature of the system before answering the telephone call.

The system provides the first audio data as an input to a model that is configured to receive first given audio data of a first given utterance of one or more first given terms spoken in a first voice and output second given audio data of a second given utterance of the one or more first given terms spoken in a synthesized voice without performing speech recognition on the first given audio data (<NUM>). The model may use an encoder to encode the first audio data into a series of vectors that represent the audio data. The vectors may be different than a transcription of the first audio data. The model may use a decoder to generate the outputted audio data. The decoder may be configured to convert the vectors to synthesized speech in a voice that is different than the voice of the user. In some implementations, the model bypasses transcribing the first audio data of the first utterance.

The system, in response to providing the first audio data as an input to the model, receives second audio data of a second utterance of the one or more first terms spoken in the synthesized voice (<NUM>). In some implementations, the speaking time of each of the one or more first terms in the first utterance may be different than the speaking time of each of the one or more first terms in the second utterance. According to the invention, the time periods between each of the one or more first terms in the first utterance are different than the time periods between each of the one or more terms in the second utterance.

The system provides, for output, the second audio data of the second utterance of the one or more first terms spoken in the synthesized voice (<NUM>). The system may output the second audio data to a speaker or other audio output device. Another user may hear the second utterance and may be unaware that original user spoke the first utterance. The second utterance may sound like the voice of an actual person even though the system generated the audio data of the second utterance using the model. In some implementations, the synthesized voice may have gender-neutral qualities such that a listener may not be able to determine whether the speaker is a male or a female. The pitch of a gender-neutral synthesized voice may be an average of the pitch for a female synthesized voice and the pitch for a male synthesized voice.

In some implementations, the system may receive an utterance from a different user. The system may apply the audio data of the utterance from the different user to the model. The model may output audio data of a synthesized utterance in the same synthesized voice. In other words, the model may be configured to convert audio data of utterances spoken by different people to utterances in the same synthesized voice.

In some implementations, the system may train the model using a collection of utterances received by the system and by other systems. The system obtains a transcription of each utterance in the collection of utterances. The system may generate the transcriptions using automated speech recognition or by manual transcription. The system provides each transcription to a speech synthesizer, or text to speech model, that generates the synthesized utterances in a synthesized voice. The system trains the model using machine learning and the collection of utterances and the corresponding synthesized utterances. The trained model is configured generate a synthesized utterance in the same synthesized voice based on receiving an utterance spoken by a user. The trained model does not use speech recognition to generate the synthesized utterance.

In some implementations, the system may be part of an automated agent, or bot, that is configured to conduct voice conversations with a user. The user may be under the impression that instead of speaking to a computer, the user is speaking with a live person. The automated agent may not be able to generate an appropriate response to every utterance that the automated agent may receive from the user. In this instance, an operator may be standing by to jump in for the automated agent to generate a response to a user utterance so that the conversation can continue. The system may assist in disguising the voice of the operator so that the user is under the impression that the user is still speaking to the same person. The system may convert the voice of the operator to the voice of the automated agent so that the user hears the same voice even when the operator generates the response instead of the automated agent.

In more detail, this document describes an end-to-end speech-to-speech model that maps an input spectrogram directly to another spectrogram, without any intermediate discrete representation. The network is composed of an encoder, a spectrogram decoder, and a phoneme decoder, followed by a vocoder to synthesize a time-domain waveform. This model can be trained to normalize speech from any speaker even for speech that includes accents, emotions, complex prosodic patterns, imperfections, and background noise, into the voice of a clean single predefined target speaker with a fixed accent and consistent articulation and prosody. This document describes the impact of this approach on speech recognition performance. Moreover, this document demonstrates that the same architecture can be trained on a speech separation task. In some implementations, the end-to-end speech-to-speech model can translate Spanish speech into synthesized English speech.

Encoder-decoder models with attention may be used in modeling a variety of complex sequence-to-sequence problems. These models may be used for speech and natural language processing, such as machine translation, speech recognition, and combined speech translation. The models may also be used in end-to-end Text-To-Speech (TTS) synthesis and Automatic Speech Recognition (ASR), using a single neural network that directly generates the target sequences, given virtually raw inputs.

This document describes combining state of the art speech recognition and synthesis models to build a direct end-to-end speech-to-speech sequence transducer which generates a speech spectrogram as a function of a different input spectrogram, without depending on an intermediate discrete representation. The model may first be applied to voice normalization and speech separation tasks. This model can be used to directly translate one language to another, for example, from Spanish speech into English speech.

In some implementations, a unified sequence-to-sequence model may normalize arbitrary speech, potentially including background noise, and generate the same content in the voice of a single predefined target speaker. The source speech can be from any speaker or accent, contain complex prosodic patterns, imperfections, and background noise, all of which are converted into a clean signal with a fixed accent and consistent articulation and prosody. The task is to project away all non-linguistic information, including speaker characteristics, and to retain only what is been said, not who, how, or where it is said.

Such a normalization system has multiple potential applications. Fully normalizing any voice to a single speaker with clean audio could simplify ASR models, which could be reduced to supporting a single speaker. Removing the identity of the speaker might be useful when logging sensitive and private speech data, allowing users to transmit only converted speech to servers erased of "acoustic" identity. Reducing all accents into a single voice with a predefined accent may also alleviate biases and discrimination while maintaining a natural human voice as opposed to acoustically masked audio, for example, for phone interviews or recorded candidate talks given to hiring committees. Another application would be to facilitate the understanding of speech content of accents that are foreign to the listener, e.g., improving intelligibility of heavily accented speech.

In some implementations, voice conversion may include using mapping code books, neural networks, dynamic frequency warping, and Gaussian mixture models. These techniques may modify only the input speaker's voice. In some implementations, voice conversion may include accent conversion. The models described in this document may normalize all speakers to a single voice and accent, as well as normalize prosody and use an end-to-end neural architecture that directly generates a target signal. In some implementations, voice conversion may be a filtering and/or transformation based approach.

The end-to-end sequence-to-sequence model architecture takes an input source speech and generates/synthesizes target speech as output. In some implementations, the only training requirement of such a model is a parallel corpus of paired input-output speech utterances.

As shown in <FIG>, the network is composed of an encoder and a decoder with attention, followed by a vocoder to synthesize a time-domain waveform. The encoder converts a sequence of acoustic frames into a hidden feature representation which the decoder consumes to predict a spectrogram. In some implementations, the core architecture of this model includes an attention-based end-to-end ASR model and/or end-to-end TTS models.

The base encoder configuration may be similar to other encoders with some variations discussed below. From an example input speech signal sampled at <NUM>, the encoder may extract <NUM>-dimensional log-mel spectrogram acoustic feature frames over a range of <NUM>-<NUM>, calculated using a Hann window, <NUM> frame length, <NUM> frame shift, and <NUM>-point Short-Time Fourier Transform (STFT).

In this example, the input features are passed into a stack of two convolutional layers with ReLU activations, each consisting of <NUM> kernels shapes <NUM> x <NUM> in time x frequency, and strided by <NUM> x <NUM>, downsampling in time by a total factor of <NUM>, decreasing the computation in the following layers. Batch normalization is applied after each layer.

The resulting downsampled sequence is passed into a bidirectional convolutional LSTM (CLSTM) layer using a <NUM> x <NUM> filter, e.g., convolving only across the frequency axis within each time step. Finally, this is passed into a stack of three bidirectional LSTM layers of size <NUM> in each direction, interleaved with a <NUM>-dimension linear projection, followed by batch normalization and ReLU activation, to compute the final <NUM>-dim encoder representation.

In some implementations, the decoder's targets are <NUM> dimensional STFT magnitudes, computed with the same framing as the input features, <NUM>-point STFT.

The system uses the decoder network comprised of an autoregressive RNN to predict the output spectrogram from the encoded input sequence one frame at a time. The prediction from the previous decoder time step is first passed through a small pre-net containing two fully connected layers of <NUM> ReLU units, which may help to learn attention. The pre-net output and attention context vector may be concatenated and passed through a stack of two unidirectional LSTM layers with <NUM> units. The concatenation of the LSTM output and the attention context vector is then projected through a linear transform to produce a prediction of the target spectrogram frame. Finally, these predictions are passed through <NUM>-layer convolutional post-net which predicts a residual to add to the initial prediction. Each post-net layer has <NUM> filters shaped <NUM> x <NUM> followed by batch normalization and tanh activation.

To synthesize an audio signal from the predicted magnitude spectrogram, the system uses the Griffin-Lim algorithm to estimate a phase consistent with the predicted magnitude, followed by an inverse STFT. In some implementations, neural vocoders such as WaveNet may produce improved synthesis quality. In some implementations, WaveNet could replace Griffin-Lim.

In some implementations, the system may be configured to generate speech sounds instead of arbitrary audio. Jointly training the encoder network to simultaneously learn a high level representation of the underlying language serves to bias the spectrogram decoder predictions toward a representation of the same underlying speech content. An auxiliary ASR decoder may be added to predict the (grapheme or phoneme) transcript of the output speech, conditioned on the encoder latent representation. Such a multitask trained encoder can be thought of as learning a latent representation of the input that maintains information about the underlying transcript, e.g., one that is closer to the latent representation learned within a TTS sequence-to-sequence network.

In some implementations, the decoder input is created by concatenating a <NUM>-dimensional embedding for the grapheme emitted at the previous time step, and a <NUM>-dimensional attention context vector. This is passed into a <NUM> unit LSTM layer. Finally, the concatenation of the attention context and LSTM output is passed into a softmax layer which predicts the probability of emitting each grapheme in the output vocabulary.

The speech-to-speech model may be used to convert speech from an arbitrary speaker to use the voice of a predefined canonical speaker. As discussed above, the system may require a parallel corpus of utterances spanning a variety of speakers and recording conditions, each mapped to speech from a canonical speaker. Since it may be impractical to have a single speaker record hours of utterances in clean acoustic environment, a TTS system may be used to generate training targets from a large hand or machine transcribed corpus of speech. Essentially, this reduces the task to reproducing any input speech in the voice of a single-speaker TTS system. There are multiple advantages of using a TTS system to generate this parallel corpus: (<NUM>) the audio is spoken with a single predefined speaker and accent using a standard language; (<NUM>) without any background noise; (<NUM>) using high quality pronunciations with no disfluencies; and (<NUM>) synthesizing large amounts of data as needed for scaling to large corpora.

This document describes end-to-end speech-to-speech model that converts an input spectrogram directly to another spectrogram, without any intermediate symbolic representation. The model be trained to normalize any utterance from any speaker to a single predefined speaker's voice, preserving the linguistic content and projecting away non-linguistic content. In some implementations, the same model can be trained to successfully identify, separate, and reconstruct the loudest speaker in a mixture of overlapping speech, which improves ASR performance. In some instances, the model may translate speech in one language directly to speech in another language.

For a task where preserving speaker identify is the goal, one might leverage TTS synthesis to introduce variation in the target speaker identity, e.g. to match the original speaker, and control the speaking style or prosody directly. Such technologies could be used to synthesize training targets for a normalization model which maintains speaker identity, but enforces neutral prosody, or vice-versa, one which normalizes speaker identity, but maintains the prosody of the input speech.

<FIG> shows an example of a computing device <NUM> and a mobile computing device <NUM> that can be used to implement the techniques described here. The mobile computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

The computing device <NUM> includes a processor <NUM>, a memory <NUM>, a storage device <NUM>, a high-speed interface <NUM> connecting to the memory <NUM> and multiple high-speed expansion ports <NUM>, and a low-speed interface <NUM> connecting to a low-speed expansion port <NUM> and the storage device <NUM>. Each of the processor <NUM>, the memory <NUM>, the storage device <NUM>, the high-speed interface <NUM>, the high-speed expansion ports <NUM>, and the low-speed interface <NUM>, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a GUI on an external input/output device, such as a display <NUM> coupled to the high-speed interface <NUM>. Also, multiple computing devices may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The high-speed interface <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low-speed interface <NUM> manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface <NUM> is coupled to the memory <NUM>, the display <NUM> (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports <NUM>, which may accept various expansion cards (not shown). In the implementation, the low-speed interface <NUM> is coupled to the storage device <NUM> and the low-speed expansion port <NUM>. The low-speed expansion port <NUM>, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The memory <NUM> stores information within the mobile computing device <NUM>. An expansion memory <NUM> may also be provided and connected to the mobile computing device <NUM> through an expansion interface <NUM>, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory <NUM> may provide extra storage space for the mobile computing device <NUM>, or may also store applications or other information for the mobile computing device <NUM>. Specifically, the expansion memory <NUM> may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory <NUM> may be provide as a security module for the mobile computing device <NUM>, and may be programmed with instructions that permit secure use of the mobile computing device <NUM>.

The memory may include, for example, flash memory and/or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier. that the instructions, when executed by one or more processing devices (for example, processor <NUM>), perform one or more methods, such as those described above. The instructions can also be stored by one or more storage devices, such as one or more computer- or machine-readable mediums (for example, the memory <NUM>, the expansion memory <NUM>, or memory on the processor <NUM>). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver <NUM> or the external interface <NUM>.

The mobile computing device <NUM> may communicate wirelessly through the communication interface <NUM>, which may include digital signal processing circuitry where necessary. The communication interface <NUM> may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver <NUM> using a radio-frequency. In addition, a GPS (Global Positioning System) receiver module <NUM> may provide additional navigation- and location-related wireless data to the mobile computing device <NUM>, which may be used as appropriate by applications running on the mobile computing device <NUM>.

Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet. In some implementations, the systems and techniques described here can be implemented on an embedded system where speech recognition and other processing is performed directly on the device.

Claim 1:
A computer-implemented method comprising:
receiving, by a computing device, first audio data of a first utterance of one or more first terms spoken by a user;
providing, by the computing device, the first audio data as an input to a model that is configured to receive first given audio data of a first given utterance of one or more first given terms spoken in a first voice and output second given audio data of a second given utterance of the one or more first given terms spoken in a synthesized voice without performing speech recognition on the first given audio data,
wherein the model is configured to adjust a time period between each of the one or more first given terms;
in response to providing the first audio data as an input to the model, receiving, by the computing device, second audio data of a second utterance of the one or more first terms spoken in the synthesized voice,
wherein the time periods between each of the one or more first terms in the first utterance are different than the time periods between each of the one or more terms in the second utterance; and
providing, for output by the computing device, the second audio data of the second utterance of the one or more first terms spoken in the synthesized voice.