Translatotron 2: Robust direct speech-to-speech translation

1 INTRODUCTION . Speech-to-speech translation ( S2ST ) is highly beneficial for breaking down communication barriers between people not sharing a common language . Conventional S2ST systems are often composed of a cascade of three components : automatic speech recognition ( ASR ) , text-to-text machine translation ( MT ) , and text-to-speech ( TTS ) synthesis ( Lavie et al. , 1997 ; Wahlster , 2000 ; Nakamura et al. , 2006 ) . Very recently , direct speech-to-text translation ( ST ) is rapidly emerging , and has outperformed the cascade of ASR and MT ( Weiss et al. , 2017 ; Jia et al. , 2019a ; Di Gangi et al. , 2019 ; McCarthy et al. , 2020 ; Wang et al. , 2021b ; Ansari et al. , 2020 ; Anastasopoulos et al. , 2021 ) , which makes the cascade of ST and TTS as S2ST feasible ( Jia et al. , 2019b ) . However , works on direct S2ST is very limited . Compared to cascaded systems , direct S2ST has the potential benefits of : 1 ) retaining paralinguistic and non-linguistic information during the translation , such as speaker ’ s voice ( Jia et al. , 2019b ) , emotion and prosody ; 2 ) working on languages without written form ( Tjandra et al. , 2019 ; Zhang et al. , 2021 ; Lee et al. , 2021a ) ; 3 ) reduced computational requirements and lower inference latency ; 4 ) avoiding error compounding across sub-systems ; 5 ) easier on handling contents that do not need to be translated , such as names and proper nouns ( Jia et al. , 2019b ) . Translatotron ( Jia et al. , 2019b ) is the first model that is able to directly translate speech in one language to speech in another language . It is also able to retain the source speaker ’ s voice in the translated speech . However , its translation quality underperforms a cascade baseline by a large margin , and its synthesized translated speech suffers from robustness issues , such as babbling and long pause , which is an issue shared with the Tacotron 2 TTS model ( Shen et al. , 2018 ; He et al. , 2019 ; Zheng et al. , 2019 ; Battenberg et al. , 2020 ; Shen et al. , 2020 ) , since they share the same attention-based approach for synthesizing speech . In this work we present Translatotron 2 . The main contributions include : 1 . We propose Translatotron 2 , a novel direct S2ST model that is able to be trained end-to-end ; 2 . We conduct experiments suggesting that Translatotron 2 significantly outperforms Trans- latotron , and is comparable to a cascade system , in terms of translation quality , speech naturalness , and speech robustness ; 3 . We propose a new method for voice retention in S2ST without relying on any explicit speaker embedding or ID . The trained model is only able to retain the source speaker ’ s voice but not able to generate speech in a different speaker ’ s voice , making it more robust for production deployment by mitigating potential misuse for creating spoofing audio artifacts ; 4 . We propose a simple concatenation data augmentation , ConcatAug , to enable Translatotron 2 to retain each speaker ’ s voice when the input includes speaker turns ; 5 . We conduct experiment suggesting that Translatotron 2 is efficient on multilingual direct S2ST , in which it obtaines translation quality very close to an ST baseline . Audio samples from Translatotron 2 are available in the supplementary material . 2 RELATED WORKS . S2ST Until very recently , S2ST systems are typically composed of a cascade of ASR , MT , and TTS components ( Lavie et al. , 1997 ; Wahlster , 2000 ; Nakamura et al. , 2006 ; ITU , 2016 ) . Translatotron ( Jia et al. , 2019b ) is the first direct S2ST model , which is a sequence-to-sequence model trained endto-end in a multi-objective task . It has shown reasonable translation quality and speech naturalness , but still underperformed a baseline of ST + TTS cascade by a large margin . It also demonstrated the capacity of retaining speaker ’ s voice during the translation , by leveraging a speaker encoder separately trained in a speaker verification task ( Wan et al. , 2018 ; Jia et al. , 2018 ) . A few recent works proposed cascade S2ST systems using learned discrete representation as the intermediate representation instead of text or phoneme . Tjandra et al . ( 2019 ) introduced such an S2ST system that first translated the source speech into a discrete representation of the target speech which was predicted from a separately trained VQ-VAE ( Oord et al. , 2017 ) , and then used the VQ-VAE decoder to predict the target speech spectrogram from the discrete representation . Zhang et al . ( 2021 ) additionally trained the VQ-VAE jointly with a supervised phoneme recognition objective in different languages . Lee et al . ( 2021a ) used a separately trained vocoder to directly predict waveform from the discrete representation without relying on spectrogram ; for the best performance , this vocoder included a duration predictor and an upsampler , akin to a generative TTS model . All these works require multiple components being trained in multiple steps , but are not able to be trained end-to-end . Kano et al . ( 2021 ) introduced an end-to-end S2ST model with a cascade of three auto-regressive decoders , and used pre-trained MT and TTS models as teacher models to facilitate the training of the end-to-end model . It requires pre-trained ASR , MT , and TTS models , and the end-to-end model itself has to be trained in multiple steps . Unfortunately , despite that these recent works generated translated speech in novel ways without relying on TTS subsystems , most of these works ( except for Jia et al . ( 2019b ) ) focused only on the translation quality , but did not assess the perceptual quality ( Wagner et al. , 2019 ) of the translated speech ( e.g . naturalness ) , which is critical to S2ST . TTS Translatotron uses a decoder similar to Tacotron 2 ( Shen et al. , 2018 ; Jia et al. , 2018 ) , which is an attention-based auto-regressive decoder . Due to the flexibility of the attention mechanism , they both suffer from robustness issues such as over-generation . Recent TTS models such as FastSpeech ( Ren et al. , 2019 ; 2021 ) , Non-Attentive Tacotron ( NAT ) ( Shen et al. , 2020 ; Jia et al. , 2021 ) and Parallel Tacotron ( Elias et al. , 2021b ; a ) , demonstrate that replacing the attention module with a duration-based upsampler yields more robust synthesized speech , as quantitatively evaluated at a large scale in Shen et al . ( 2020 ) . The synthesizer component in this work resembles these works . Voice conversion and anti-spoofing The performance of voice conversion ( VC ) has progressed rapidly in the recent years , and is reaching a quality that is hard for automatic speaker verification ( ASV ) systems to detect ( Yi et al. , 2020 ) . ASVspoof 2019 ( Todisco et al. , 2019 ; Wang et al. , 2020 ) found that it was challenging to detect spoof audios generated from Jia et al . ( 2018 ) , which uses the same speaker encoder-based approach as in the original Translatotron . Such progress poses concerns on related techniques being misused for creating spoofing artifacts . We designed Translatotron 2 with the motivation of avoiding such potential misuse . 3 TRANSLATOTRON 2 . The motivation of the architectural design of Translatotron 2 is to improve on three weaknesses existing in the original Translatotron : 1 ) The valuable auxiliary supervision on textual features is not fully utilized during training ; 2 ) Long-sequence-to-long-sequence modelling with attention is difficult because of the flexibility of the attention mechanism ; 3 ) Attention-based speech generation is known to suffer from robustness issues such as over-generation and under-generation . The proposed Translatotron 2 model solves these problems by using an architecture composed of a source speech encoder , a target phoneme decoder , and a target mel-spectrogram synthesizer . These three components are connected together by a single attention module . See Figure 1 for an illustration . The model is jointly trained with a speech-to-speech translation objective and a speech-to-phoneme translation objective . As the result , the auxiliary supervision is fully utilized , and the attention is used only for modeling shorter target phoneme sequence but is not directly involved in speech generation . The following subsections describe the architecture of each components as used in our main experiments . As shown in Appendix D , using alternative architectures for these components do not significantly impact the performance of Translatotron 2 . 3.1 ENCODER . The encoder of Translatotron 2 takes mel-spectrogram of the source speech as the input , and produces a hidden representation . We use Conformer ( Gulati et al. , 2020 ) as the architecture for the encoder . It first subsamples the input mel-spectrogram with a convolutional layer , and then processes it with a stack of Conformer blocks . Each Conformer block is composed of a feed-forward layer , a selfattention layer , a convolution layer , and a second feed-forward layer . SpecAugment ( Park et al. , 2019 ) is applied at the training time . 3.2 DECODER . The autoregressive decoder is assisted with an attention module . It takes the encoder output as the source values for the attention , and predicts phoneme sequences corresponding to the target translated speech . We use a stack of LSTM cells as the decoder , along with a multi-head attention ( Vaswani et al. , 2017 ) . The combination of the encoder , the decoder , and the attention module is similar to a typical ST model , except that it predicts phonemes instead of subword tokens . 3.3 SYNTHESIZER . The synthesizer takes the hidden output from the decoder , as well as the context output from the attention module as its input , and synthesizes the target mel-spectrogram . It is similar to the decoders in typical neural TTS models , such as in Shen et al . ( 2018 ; 2020 ) ; Ren et al . ( 2021 ) . The predicted mel-spectrogram can be converted into waveform using either an estimation algorithm such as Griffin & Lim ( 1984 ) or a neural vocoder . We experimented with a number of different non-attentive architectures for the synthesizer , including autoregressive ones and parallel ones , and found that autoregressive synthesizers worked best . We followed the architecture and the hyperparameters of the mel-spectrogram decoder from NAT ( Shen et al. , 2020 ) . It first predicts a duration for each of its input elements using a bidirectional LSTM stack , then upsamples the input sequence with Gaussian weights based on the predicted duration , and finally uses an auto-regressive LSTM stack to predict the target mel-spectrogram . The predicted melspectrogram is further refined with a residual convolutional post-net . This synthesizer is trained jointly with a mel-spectrogram reconstruction loss and a duration prediction loss . Figure 1 ( b ) illustrates the architecture of this synthesizer . Unlike in Shen et al . ( 2020 ) , we do not use per-phoneme duration labels for training the duration predictor , nor do we apply the FVAE-based alignment . Instead , only an L2 loss on the total predicted duration of the entire sequence is used ( i.e . the “ naı̈ve approach ” of unsupervised duration modelling in Shen et al . ( 2020 ) ) . We anticipate that adopting these approaches could further improve the performance of Translatotron 2 , which we leave as future work .

Summary: Translatotron2, a speech-to-speech neural based translation system is proposed. The work is a modification of Translatotron (also a speech-to-speech translation system) and tries to address some of the  issues in Translatotron:  a) Translation quality fairly below a cascaded baseline  b) synthesized translated speech suffers from robustness issues, such as babbling and long pauses  c) Voice retention relies on explicit speaker embedding which can potentially be misused for generating spoofing audio with arbitrary content  Compared to Translatotron, following modifications are made in Translatotron2. The experiments suggest that Translatotron 2 significantly outperforms Translatotron, and is comparable to a cascaded system, in terms of translation quality, speech naturalness and speech robustness.  a)  Output from the auxiliary target phoneme decoder is used as an input to the spectrogram synthesizer   b) Conformer Encoder with SpecAugment is used, which is known to improve the speech to text performance  c) Duration-based  Spectrogram synthesizer is used. Specifically, architecture and hyperparameters similar to Non-Attentive Tacotron spectrogram synthesizer is used, which is known to improve the robustness issue in Tacotron2   d) To retain speakers’ voices across translation, Translatotron2 is trained on parallel utterances with the same speaker’s voice on both sides. This enables the trained model to restrict generation to only the source speaker’s voice, mitigating the risk of potential misuse for creating spoofing audio artifacts. Parallel utterances are synthesized using a TTS model with cross lingual voice transfer capacity.  In addition, authors also trained Translatotron 2 with examples that contain two speakers’ voices in both the source and the target. The experiments suggest that Translatotron 2 has capability to retain voice identity when the input contains speaker turns.  For multilingual experiments, Translatotron 2 was trained with 4 high resource languages  and BLEU scores suggest Translatotron2 outperforms Translatotron in multilingual settings as well.