Patent ID: 11960852
Assignee: GOOGLE LLC
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that, when executed on the data processing hardware, cause the data processing hardware to execute a direct speech-to-speech translation (S2ST) model, the S2ST model comprising:
an encoder configured to:
receive an input speech representation corresponding to an utterance spoken by a source speaker in a first language and in a first voice; and
encode the input speech representation into a hidden feature representation;

an attention module configured to generate a context vector that attends to the hidden feature representation encoded by the encoder;
a decoder configured to:
receive the context vector generated by the attention module; and
predict a phoneme representation corresponding to a translation of the utterance in a second different language, the phoneme representation comprising a sequence of phonemes; and

a synthesizer configured to:
receive the context vector and the phoneme representation; and
generate a translated synthesized speech representation corresponding to the translation of the utterance spoken in the different second language and in the first voice,

wherein the S2ST model is trained end-to-end on pairs of parallel source language and target language utterances each in a same voice, the S2ST model trained to not translate input speech representations corresponding to the source language utterances into corresponding translated synthesized speech representations that are in a different voice than the corresponding source language utterances.