Patent Description:
Speech-to-speech translation (S2ST) is highly beneficial for breaking down communication barriers between people not sharing a common language. Conventionally, S2ST systems are composed of a cascade of three components: automatic speech recognition (ASR); text-to-text machine translation (MT), and text-to-speech (TTS) synthesis. Recently, advancements in direct speech-to-text translation (ST) have outperformed the cascade of ASR and MT, thereby making a two component cascade of ST and TTS as S2ST feasible. An example implementation thereof may e.g. be found in the conference paper of the Interspeech <NUM>, by <NPL>.

One aspect of the disclosure provides a direct speech-to-speech translation (S2ST) model as defined by independent claim <NUM>. The S2ST model includes an encoder configured to receive an input speech representation that corresponds to an utterance spoken by a source speaker in a first language and encode the input speech representation into a hidden feature representation. The S2ST model also includes an attention module configured to generate a context vector that attends to the hidden representation encoded by the encoder. The S2ST model also includes a decoder configured to receive the context vector generated by the attention module and predict a phoneme representation that corresponds to a translation of the utterance in a second different language. The S2ST model also includes a synthesizer configured to receive the context vector and the phoneme representation and generate a translated synthesized speech representation corresponding to a translation of the utterance spoken in the different second language.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the encoder includes a stack of conformer blocks. In other implementations, the encoder includes a stack of one of transformer blocks or lightweight convolutional blocks. In some examples, the synthesizer includes a duration model network configured to predict a duration of each phoneme in a sequence of phonemes represented by the phoneme representation. In these examples, the synthesizer may be configured to generate the translated synthesized speech representation by upsampling the sequence of phonemes based on the predicted duration of each phoneme. The translated synthesized speech representation may be configured to a speaking style/prosody of the source speaker.

In some implementations, the S2ST model is trained on pairs of parallel source language and target language utterances each including a voice spoken in the source utterance. In these implementations, at least one of the source language utterance or the target language utterance includes speech synthesized by a text-to-speech model trained to generate synthesized speech in the voice of the source utterance. In some examples, the S2ST module further includes a vocoder configured to receive the translated synthesized speech representation and synthesize the translated synthesized speech representation into an audible output of the translated synthesized speech representation. Optionally, the phoneme representation may include a probability distribution of possible phonemes in a phoneme sequence corresponding to the translated synthesized speech representation.

Another aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations for direct speech-to-speech translation. The operations include receiving, as input to a direct speech-to-speech translation (S2ST) model, an input speech representation that corresponds to an utterance spoken by a source speaker in a first language. The operations also include encoding the input speech representation into a hidden feature representation by an encoder of the S2ST model. The operation also include generating, by a decoder of the S2ST model, a context vector that attends to hidden feature representation encoded by the encoder. The operations also include receiving the context vector generated by the attention module at a decoder of the S2ST model. The operations also include predicting, by the decoder, a phoneme representation that corresponds to a translation of the utterance in a second different language. The operations also include receiving the context vector and the phoneme representation at a synthesizer of the S2ST model. The operations also include generating, by the synthesizer, a translated synthesize speech representation that corresponds to the translation of the utterance spoken in the different second language.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, the encoder includes a stack of conformer blocks. In other implementations, the encoder includes a stack of one of transformer blocks or lightweight convolutional blocks. In some examples, the synthesizer includes a duration model network configured to predict a duration of each phoneme in a sequence of phonemes represented by the phoneme representation. In these examples, generating the translated synthesized speech representation may include upsampling the sequence of phonemes based on the predicted duration of each phoneme.

The translated synthesize speech representation may be configured to a speaking style/prosody of the source speaker. In some implementations, the S2ST model is trained on pairs of parallel source language and target language utterances each including a voice spoken in the source utterance. In these implementations, at least one of the source language utterance or the target language utterance may include speech synthesized by a text-to-speech model trained to generate synthesized speech in the voice of the source utterance. In some examples, the operations further include receiving the translated synthesized speech representation at a vocoder of the S2ST model and synthesizing, by the vocoder, the translated synthesized speech representation into an audible output of the translated synthesized speech representation. Optionally, the phoneme representation may include a probability distribution of possible phonemes in a phoneme sequence corresponding to the translated synthesized speech representation.

Speech-to-speech translation (S2ST) is highly beneficial for breaking down communication barriers between people not sharing a common language. Conventionally, S2ST systems are composed of a cascade of three components: automatic speech recognition (ASR); text-to-text machine translation (MT), and text-to-speech (TTS) synthesis. Recently, advancements in direct speech-to-text translation (ST) have outperformed the cascade of ASR and MT, thereby making a two component cascade of ST and TTS as S2ST feasible.

Direct S2ST includes directly translating speech in one language to speech in another language. Stated differently, direct S2ST systems/models are configured to convert an input audio waveform or spectrograms corresponding to speech spoken in a first language by a human speaker directly into an output audio waveform or spectrograms corresponding to synthesized speech in a second language different than the first language without converting the input audio waveform into an intermediate representation (e.g., text or phonemes). As will become apparent, direct S2ST models, as well as techniques for training direct S2ST models, will enable a user to speak in his/her native language, and be understood by, both other humans and speech interfaces (e.g., digital assistants) by enabling recognition and/or reproduction of the user's speech as synthesized audio in a different language. A recent direct S2ST model underperformed cascaded S2ST systems in terms of translation quality, while also suffering from robustness issues of the output synthesized speech in terms of babbling and long pauses. These issues are attributed to the use of an attention-based approach for synthesizing speech.

Implementations herein are directed toward a robust direct S2ST model that is trained end-to-end, outperforms existing direct S2ST systems, and is comparable to cascaded systems in terms of translation quality, speech naturalness, and speech robustness. Notably, compared to cascaded systems, direct S2ST systems/models have the potential of: retaining paralinguistic and non-linguistic information during the translation, such as speaker's voice and prosody; working on languages without written form; reducing computational requirements and inference latency; avoiding error compounding across sub-systems; and providing ease in the handling contents that do not need to be translated, such as names and other proper nouns. Implementations herein are further directed toward a voice retaining technique in S2ST that does not rely on any explicit speaker embedding or identifier. Specifically, the trained S2ST model is trained to only retain a voice of the source speaker that is provided in the input speech without the ability to generate speech in a voice different from the source speaker. Notably, the ability to retain the source speaker's voice is useful for production environments by proactively mitigating misuse for creating spoofing audio artifacts.

<FIG> shows a speech conversion environment <NUM> that employs a direct speech-to-speech translation (S2ST) model <NUM> configured to translate input utterances spoken by a source speaker in a first language directly into corresponding output utterances in a different second language, and vice versa. As will become apparent, the direct S2ST model <NUM> is trained end-to-end. Notably, the direct S2ST model <NUM> is distinct from a cascaded S2ST system that employs a separate combination of an automated speech recognizer (ASR) component, a text-to-text machine translation (MT) component, and text-to-speech (TTS) synthesis component, or other cascaded S2ST systems that employ a cascade of a direct speech-to-text translation (ST) component followed by a TTS component.

In the example shown, the direct S2ST model <NUM> is configured to convert input audio data <NUM> corresponding to an utterance <NUM> spoken in a first/source language (e.g., Spanish) by a source speaker <NUM> into output audio data (e.g., mel-spectrogram) <NUM> corresponding to a translated synthesized speech representation of a translated utterance <NUM> spoken in a different second language (e.g., English) by the source speaker <NUM>. The direct S2ST model <NUM> may convert an input spectrogram corresponding to the input audio data <NUM> directly into an output spectrogram corresponding to the output audio data <NUM> without performing speech recognition and text-to-text machine translation, or otherwise without requiring the generation of any intermediate discrete representations (e.g., text or phonemes) from the input data <NUM>. While described in greater detail with reference to <FIG> and <FIG>, the direct S2ST model <NUM> includes a spectrogram encoder <NUM>, an attention module <NUM>, a decoder <NUM>, and a synthesizer (e.g., a spectrogram decoder) <NUM>.

A vocoder <NUM> may synthesize the output audio data <NUM> output from the direct S2ST model <NUM> into a time-domain waveform for audible output as the translated utterance <NUM> spoken in the second language and in the voice of the source speaker. A time-domain audio waveform includes an audio waveform that defines an amplitude of an audio signal over time. The lieu of a vocoder <NUM>, a unit selection module or a WaveNet module may instead synthesize the output audio data <NUM> into time-domain waveforms of synthesized speech in the translated second language and in the voice of the source speaker <NUM>. In some implementations, the vocoder <NUM> includes a vocoder network, i.e., neural vocoder, which is separately trained and conditioned on mel-frequency spectrograms for conversion into time-domain audio waveforms.

In the example shown, the source speaker <NUM> is a native speaker of the first/source language of Spanish. The direct S2ST <NUM> is accordingly trained to covert the input audio data <NUM> corresponding to utterances <NUM> spoken by the source speaker <NUM> in Spanish directly into the output audio data <NUM> corresponding to the translated synthesized speech representation corresponding to a translated utterance <NUM> in English (e.g., second/target language). That is, the translated utterance <NUM> in English (e.g., "Hi, what are your plans this afternoon?) includes synthesized audio for a translated version of the input utterance <NUM> that was spoken by the source speaker <NUM> in Spanish (e.g., "Hola, cuales son tus planes esta tarde?"). Thus, the translated synthesized representation provided by the output audio data <NUM> in English allows the native Spanish speaker to convey the utterance <NUM> spoken in Spanish to a recipient user <NUM> that natively speaks English. In some examples, the source speaker <NUM> does not speak English and the recipient speaker <NUM> does not speak/understand Spanish. In some implementations, the direct S2ST model <NUM> is a multilingual and trained to also convert input utterances spoken in English by speaker <NUM> into translated utterances in Spanish. In these implementations, the direct S2ST model <NUM> may be configured to convert speech between one or more other pairs of languages in addition to, or in lieu of, Spanish and English.

Notably, the direct S2ST model <NUM> is trained to retain voice characteristics of the source speaker such that the output audio data <NUM> corresponding to the synthesized speech representation and resulting translated utterance <NUM> conveys the voice of the source speaker, but in the different second language. Put another way, the translated utterance <NUM> conveys the voice characteristics of the source speaker <NUM> (e.g., speaking style/prosody) as the source speaker <NUM> would actually speak the different second language. In some examples, and described in greater detail below, the direct S2ST model <NUM> is trained to not only retain the voice characteristics of the source speaker in output audio data <NUM>, but also prevent the ability to generate speech in a voice different from the source speaker to mitigate misuse of the model <NUM> for creating spoofing audio artifacts.

A computing device associated with the source speaker <NUM> may capture the utterance <NUM> spoken by the source speaker <NUM> in the source/first language (e.g., Spanish) and transmit the corresponding input audio data <NUM> to the direct S2ST model <NUM> for conversion into the output audio data <NUM>. Thereafter, the direct S2ST model <NUM> may transmit the output audio data <NUM> corresponding to the translated synthesized speech representation of the translated utterance <NUM> to another computing device <NUM> associated with recipient user <NUM>, whereby the other computing device <NUM> audibly outputs the translated synthesized speech representation as the translated utterance <NUM> in the different second language (e.g., English). In this example, the source speaker <NUM> and the user <NUM> are speaking with each other through their respective computing devices <NUM>, <NUM>, such as over an audio/video call (e.g., video meeting/chat) telephone call or other type of voice communication protocol, for example, voice over internet protocol.

Notably, the direct S2ST model <NUM> may be trained to retain the same speaking style/prosody in the output audio data <NUM> corresponding to the translated synthesized speech representation that was used in the input audio data <NUM> corresponding to the utterance <NUM> spoken by the source speaker <NUM>. For instance, in the example shown, since the input audio data <NUM> for the Spanish utterance <NUM> conveys a style/prosody associated with the speaking of a question, the S2ST model <NUM> generates the output audio data <NUM> corresponding to the translated synthesized speech representation having the style/prosody associated with the speaking of the question.

In some other examples, the S2ST conversion model <NUM> instead sends the output audio data <NUM> corresponding to the translated synthesized speech representation of the utterance spoken by the source speaker <NUM> to an output audio device for audibly outputting the translated synthesized speech representation in the voice of the source speaker <NUM> to an audience. For instance, the source speaker <NUM> who natively speaks Spanish may be a lecturer providing a lecture to an English speaking audience, in which utterances spoken by the source speaker <NUM> in Spanish are converted into translated synthesized speech representations audibly output from the audio device to the English speaking audience as translated utterances in English.

Alternatively, the other computing device <NUM> may be associated with a downstream automated speech recognition (ASR) system in which the S2ST model <NUM> functions as a front-end to provide the output audio data <NUM> corresponding to the synthesized speech representation as an input to the ASR system for conversion into recognized text. The recognized text could be presented to the other user <NUM> and/or could be provided to a natural language understanding (NLU) system for further processing.

The functionality of the S2ST model <NUM> can reside on a remote server <NUM>, on either or both of the computing devices <NUM>, <NUM>, or any combination of the remote server and computing devices <NUM>, <NUM>. In particular, data processing hardware of the computing devices <NUM>, <NUM> may execute the S2ST model <NUM>. In some implementations, the S2ST model <NUM> continuously generates output audio data <NUM> corresponding to synthesized speech representations of an utterance as the source speaker <NUM> speaks corresponding portions of the utterance in a first/source language. By continuously generating output audio data <NUM> corresponding to synthesized speech representations of portions of the utterance <NUM> spoken by the source speaker <NUM>, the conversation between the source speaker <NUM> and the user <NUM> (or audience) may be more naturally paced. In some additional implementations, the S2ST model <NUM> waits to determine/detect when the source speaker <NUM> stops speaking, using techniques such as voice activity detection, end pointing, end of query detection, etc., before converting the corresponding input audio data <NUM> of the utterance <NUM> in the first language into the corresponding output audio data <NUM> corresponding to the translated synthesized speech representation of the same utterance <NUM>, but in the different second language.

<FIG> shows the direct S2ST model <NUM> of <FIG> including the encoder <NUM>, attention module <NUM>, decoder <NUM>, and synthesizer <NUM>. The encoder <NUM> is configured to encode the input audio data <NUM> into a hidden feature representation (e.g., a series of vectors) <NUM>. Here, the input audio data <NUM> includes a sequence of input spectrograms that correspond to the utterance <NUM> spoken by the source speaker <NUM> in the source/first language (e.g., Spanish). The sequence of input phonemes may include an <NUM>-channel mel-spectrogram sequence. In some implementations, the encoder <NUM> includes a stack of Conformer layers. In these implementations, the encoder subsamples the input audio data <NUM> including the input mel-spectrogram sequence using a convolutional layer, and then processes the input mel-spectrogram sequence with the stack of Conformer blocks. Each Conformer block may include a feed-forward layer, a self-attention layer, a convolution layer, and a second feed-forward layer. In some examples, the stack of Conformer blocks includes <NUM> layers of Conformer blocks with a dimension of <NUM>, and a subsampling factor of four (<NUM>). <FIG> provides a schematic view of an example Conformer block. The encoder <NUM> may instead use a stack of transformer blocks or lightweight convolutional blocks in lieu of Conformer blocks.

The attention module <NUM> is configured to generate a context vector <NUM> that attends to the hidden feature representation <NUM> encoded by the encoder <NUM>. The attention module <NUM> may include a multi-headed attention mechanism. The decoder <NUM> is configured to receive, as input, the context vector <NUM> indicating the hidden feature representation <NUM> as source values for attention, and predict, as output, a phoneme representation <NUM> representing a probability distribution of possible phonemes in a phoneme sequence <NUM> corresponding to the audio data (e.g., target translated synthesized speech representation) <NUM>. That is, the phoneme representation <NUM> corresponds to a translation of the utterance <NUM> in a second different utterance (e.g., in the second language). A fully-connected network plus softmax <NUM> layer may select, at each of a plurality of output steps, a phoneme in the sequence of phonemes <NUM> (e.g., English phonemes) based on using the phoneme with the highest probability in the probability distribution of possible phonemes represented by the phoneme representation <NUM>. In the example shown, the decoder <NUM> is autoregressive and generates, at each output step, the probability distribution of possible phonemes for the given output step based on each previous phoneme in the phoneme sequence <NUM> selected by the Softmax <NUM> during each of the previous output steps. In some implementations, the decoder <NUM> includes a stack of long short-term memory (LSTM) cells assisted by the attention module <NUM>. Notably, the combination of the encoder <NUM>, attention module <NUM>, and decoder <NUM> is similar to a direct speech-to-text translation (ST) component typically found in a cascaded S2ST system.

The synthesizer <NUM> receives, as input during each of a plurality of output steps, a concatenation of the phoneme representation <NUM> (or phoneme sequence <NUM>) and the context vector <NUM> at the corresponding output step and generates, as output at each of the plurality of output steps, the output audio data <NUM> corresponding to the translated synthesized speech representation in the target/second language and in the voice of the source speaker <NUM>. Alternatively, the synthesizer <NUM> may receive the phoneme representation <NUM> and the context vector <NUM> (e.g., without any concatenation). The synthesizer <NUM> may also be referred to as a spectrogram decoder. In some examples, the synthesizer is autoregressive where each output spectrogram predicted is based on the sequence of previously predicted spectrograms. In other examples, the synthesizer <NUM> is parallel and non-autoregressive.

<FIG> provides an example of the synthesizer <NUM> of <FIG>. Here, the synthesizer <NUM> may include a phoneme duration modeling network (i.e., duration predictor) <NUM>, and upsampler module <NUM>, a recurrent neural network (RNN) <NUM>, and a convolutional layer <NUM>. The duration modeling network receives the phoneme representation <NUM> from the decoder <NUM> and the context vector <NUM> from the attention module <NUM> as input. Moreover, the duration modeling network <NUM> is tasked with predicting a duration <NUM> for each phoneme in the phoneme representation <NUM> corresponding to the output audio data <NUM> that represents the translated synthesized speech representation in the target/second language. During training, an individual target duration <NUM> for each phoneme is unknown, thus, the duration model network <NUM> determines a target average duration based on a proportion of the T total frame duration of an entire reference mel-frequency spectrogram sequence and K total number of phonemes (e.g., tokens) in a reference phoneme sequence corresponding to the reference mel-frequency spectrogram sequence. That is, the target average duration is the average duration for all phonemes using the reference mel-frequency spectrogram sequence and the reference phoneme sequence used during training. During training, a loss term (e.g., L2 loss term) is then determined between the predicted phoneme durations and the target average duration. As such, the duration model network <NUM> learns to predict phoneme durations in an unsupervised manner without the use of supervised phoneme duration labels provided from an external aligner. While external aligners are capable of providing reasonable alignments between phonemes and mel-spectral frames, phoneme duration rounding is required by a length regulator to upsample phonemes in the reference phoneme sequence according to their duration which leads to rounding errors that may persist. In some instances, using supervised duration labels from the external aligner during training and using predicted durations during inference creates phoneme duration discrepancies between training the S2ST model <NUM> and inference of the S2ST model <NUM>. Moreover, such rounding operations are not differentiable, and thus, an error gradient is unable to propagate through the duration model network.

The upsampler <NUM> receives the predicted durations <NUM>, the context vector <NUM>, and the phoneme representations as input and generates an output <NUM>. In particular, the upsampler <NUM> is configured to upsample the input sequence (e.g., the phoneme representation <NUM> or the phoneme sequence <NUM>) based on the predicted durations <NUM> from the duration model network <NUM>. The RNN <NUM> receives the output <NUM> and is configured to predict the target mel-spectrogram <NUM> autoregressively, which corresponds to the audio data <NUM> (e.g., the target translated synthesized speech representation in the target/second language). The RNN <NUM> provides the target mel-spectrograph <NUM> to the convolutional layer <NUM> and a concatenator <NUM>. The convolutional layer <NUM> provides a residual convolutional post-net configured to further refine the target mel-spectrogram <NUM> and generate an output <NUM>. That is, the convolutional layer <NUM> further refines the predicted translated synthesized speech representation in the second language. The concatenator <NUM> concatenates the output <NUM> and the target mel-spectrogram <NUM> to generate a translated synthesized speech representation <NUM> that corresponds to a translation of the utterance <NUM> spoken in the different second language. As such, the translated synthesized speech representation <NUM> may correspond to the audio data <NUM> (<FIG>). Notably, the translated synthesized speech representation <NUM> retains the speaking style/prosody of the source speaker <NUM>.

Implementations herein are further directed toward voice retaining techniques that restrict the trained S2ST model <NUM> to retain only the source speaker's voice, without the ability to generate synthesized speech in a different speaker's voice. This technique includes training on parallel utterances with the same speaker's voice on both the input utterance in a first language and the output utterance in a second language. Since fluent bilingual speakers are not prevalent, a cross-lingual TTS model may be employed to synthesize training utterances in the target second language that include the voice of the source speaker. Thus, the S2ST model <NUM> may train using utterances from the source speaker <NUM> in the first language and the synthesized training utterances of the source speaker <NUM> in the target second language. The S2ST model <NUM> can be further be trained to retain the source speakers voice in translated synthesized speech for each source speaker during speaker turns.

<FIG> provides an example of a Conformer block <NUM> from the stack of Conformer layers of the encoder <NUM>. The Conformer block <NUM> includes a first half feed-forward layer <NUM>, a second half feed-forward layer <NUM>, with a multi-head self attention block <NUM> and a convolution layer <NUM> disposed between the first and second half feed-forward layers <NUM>, <NUM>, and concatenation operators <NUM>. The first half feed-forward layer <NUM> processes the input audio data <NUM> including the input mel-spectrogram sequence. Subsequently, the multi-head self attention block <NUM> receives the input audio data <NUM> concatenated with the output of the first half-feed forward layer <NUM>. Intuitively, the role of the multi-head self attention block <NUM> is to summarize noise context separately for each input frame that is to be enhanced. A convolution layer <NUM> subsamples the output of the multi-head self attention block <NUM> concatenated with the output of the first half feed forward layer <NUM>. Thereafter, a second half-feed forward layer <NUM> receives a concatenation of the convolution layer <NUM> output and the multi-head self attention block <NUM>. A layernorm module <NUM> processes the output from the second half feed-forward layer <NUM>. Mathematically, the conformer block <NUM> transforms input features x, using modulation features m, to produce output features y, as follows: <MAT>.

<FIG> is a flow chart of an exemplary arrangement of operations for a computer-implemented method <NUM> for performing direct speech-to-speech translation. At operation <NUM>, the method <NUM> includes receiving an input speech representation <NUM> that corresponds to an utterance <NUM> spoken by a source speaker <NUM> in a first language. At operation <NUM>, the method <NUM> includes an encoder <NUM> of the S2ST model <NUM> encoding the input speech representation <NUM> into a hidden feature representation <NUM>. At operation <NUM>, the method <NUM> includes an attention module <NUM> of the S2ST model <NUM> generating a context vector <NUM> that attends to the hidden feature representation <NUM> encoded by the encoder <NUM>. At operation <NUM>, the method <NUM> includes receiving the context vector <NUM> at a decoder <NUM> of the S2ST model <NUM>. At operation <NUM>, the method <NUM> includes the decoder <NUM> predicting a phoneme representation <NUM> corresponding to a translation of the utterance <NUM> in a second different language. At operation <NUM>, the method <NUM> includes receiving the context vector <NUM> and the phoneme representation <NUM> at a synthesizer <NUM> of the S2ST model <NUM>. At operation <NUM>, the method <NUM> includes generating, by the synthesizer <NUM>, a translated speech representation <NUM> corresponding to the translation of the utterance <NUM> spoken in the different second language.

Claim 1:
A direct speech-to-speech translation (S2ST) model (<NUM>) comprising:
an encoder (<NUM>) configured to:
receive an input speech representation (<NUM>) corresponding to an utterance (<NUM>) spoken by a source speaker (<NUM>) in a first language; and
encode the input speech representation (<NUM>) into a hidden feature representation (<NUM>);
an attention module (<NUM>) configured to generate a context vector (<NUM>) that attends to the hidden feature representation (<NUM>) encoded by the encoder (<NUM>);
a decoder (<NUM>) configured to:
receive the context vector (<NUM>) generated by the attention module (<NUM>); and
predict a phoneme representation (<NUM>) corresponding to a translation of the utterance (<NUM>) in a second different language; and
a synthesizer (<NUM>) configured to:
receive the context vector (<NUM>) and the phoneme representation (<NUM>); and
generate a translated synthesized speech representation (<NUM>) corresponding to the translation of the utterance (<NUM>) spoken in the different second language.