Translatotron 2: Robust direct speech-to-speech translation

1 INTRODUCTION . Speech-to-speech translation ( S2ST ) is highly beneficial for breaking down communication barriers between people not sharing a common language . Conventional S2ST systems are often composed of a cascade of three components : automatic speech recognition ( ASR ) , text-to-text machine translation ( MT ) , and text-to-speech ( TTS ) synthesis ( Lavie et al. , 1997 ; Wahlster , 2000 ; Nakamura et al. , 2006 ) . Very recently , direct speech-to-text translation ( ST ) is rapidly emerging , and has outperformed the cascade of ASR and MT ( Weiss et al. , 2017 ; Jia et al. , 2019a ; Di Gangi et al. , 2019 ; McCarthy et al. , 2020 ; Wang et al. , 2021b ; Ansari et al. , 2020 ; Anastasopoulos et al. , 2021 ) , which makes the cascade of ST and TTS as S2ST feasible ( Jia et al. , 2019b ) . However , works on direct S2ST is very limited . Compared to cascaded systems , direct S2ST has the potential benefits of : 1 ) retaining paralinguistic and non-linguistic information during the translation , such as speaker ’ s voice ( Jia et al. , 2019b ) , emotion and prosody ; 2 ) working on languages without written form ( Tjandra et al. , 2019 ; Zhang et al. , 2021 ; Lee et al. , 2021a ) ; 3 ) reduced computational requirements and lower inference latency ; 4 ) avoiding error compounding across sub-systems ; 5 ) easier on handling contents that do not need to be translated , such as names and proper nouns ( Jia et al. , 2019b ) . Translatotron ( Jia et al. , 2019b ) is the first model that is able to directly translate speech in one language to speech in another language . It is also able to retain the source speaker ’ s voice in the translated speech . However , its translation quality underperforms a cascade baseline by a large margin , and its synthesized translated speech suffers from robustness issues , such as babbling and long pause , which is an issue shared with the Tacotron 2 TTS model ( Shen et al. , 2018 ; He et al. , 2019 ; Zheng et al. , 2019 ; Battenberg et al. , 2020 ; Shen et al. , 2020 ) , since they share the same attention-based approach for synthesizing speech . In this work we present Translatotron 2 . The main contributions include : 1 . We propose Translatotron 2 , a novel direct S2ST model that is able to be trained end-to-end ; 2 . We conduct experiments suggesting that Translatotron 2 significantly outperforms Trans- latotron , and is comparable to a cascade system , in terms of translation quality , speech naturalness , and speech robustness ; 3 . We propose a new method for voice retention in S2ST without relying on any explicit speaker embedding or ID . The trained model is only able to retain the source speaker ’ s voice but not able to generate speech in a different speaker ’ s voice , making it more robust for production deployment by mitigating potential misuse for creating spoofing audio artifacts ; 4 . We propose a simple concatenation data augmentation , ConcatAug , to enable Translatotron 2 to retain each speaker ’ s voice when the input includes speaker turns ; 5 . We conduct experiment suggesting that Translatotron 2 is efficient on multilingual direct S2ST , in which it obtaines translation quality very close to an ST baseline . Audio samples from Translatotron 2 are available in the supplementary material . 2 RELATED WORKS . S2ST Until very recently , S2ST systems are typically composed of a cascade of ASR , MT , and TTS components ( Lavie et al. , 1997 ; Wahlster , 2000 ; Nakamura et al. , 2006 ; ITU , 2016 ) . Translatotron ( Jia et al. , 2019b ) is the first direct S2ST model , which is a sequence-to-sequence model trained endto-end in a multi-objective task . It has shown reasonable translation quality and speech naturalness , but still underperformed a baseline of ST + TTS cascade by a large margin . It also demonstrated the capacity of retaining speaker ’ s voice during the translation , by leveraging a speaker encoder separately trained in a speaker verification task ( Wan et al. , 2018 ; Jia et al. , 2018 ) . A few recent works proposed cascade S2ST systems using learned discrete representation as the intermediate representation instead of text or phoneme . Tjandra et al . ( 2019 ) introduced such an S2ST system that first translated the source speech into a discrete representation of the target speech which was predicted from a separately trained VQ-VAE ( Oord et al. , 2017 ) , and then used the VQ-VAE decoder to predict the target speech spectrogram from the discrete representation . Zhang et al . ( 2021 ) additionally trained the VQ-VAE jointly with a supervised phoneme recognition objective in different languages . Lee et al . ( 2021a ) used a separately trained vocoder to directly predict waveform from the discrete representation without relying on spectrogram ; for the best performance , this vocoder included a duration predictor and an upsampler , akin to a generative TTS model . All these works require multiple components being trained in multiple steps , but are not able to be trained end-to-end . Kano et al . ( 2021 ) introduced an end-to-end S2ST model with a cascade of three auto-regressive decoders , and used pre-trained MT and TTS models as teacher models to facilitate the training of the end-to-end model . It requires pre-trained ASR , MT , and TTS models , and the end-to-end model itself has to be trained in multiple steps . Unfortunately , despite that these recent works generated translated speech in novel ways without relying on TTS subsystems , most of these works ( except for Jia et al . ( 2019b ) ) focused only on the translation quality , but did not assess the perceptual quality ( Wagner et al. , 2019 ) of the translated speech ( e.g . naturalness ) , which is critical to S2ST . TTS Translatotron uses a decoder similar to Tacotron 2 ( Shen et al. , 2018 ; Jia et al. , 2018 ) , which is an attention-based auto-regressive decoder . Due to the flexibility of the attention mechanism , they both suffer from robustness issues such as over-generation . Recent TTS models such as FastSpeech ( Ren et al. , 2019 ; 2021 ) , Non-Attentive Tacotron ( NAT ) ( Shen et al. , 2020 ; Jia et al. , 2021 ) and Parallel Tacotron ( Elias et al. , 2021b ; a ) , demonstrate that replacing the attention module with a duration-based upsampler yields more robust synthesized speech , as quantitatively evaluated at a large scale in Shen et al . ( 2020 ) . The synthesizer component in this work resembles these works . Voice conversion and anti-spoofing The performance of voice conversion ( VC ) has progressed rapidly in the recent years , and is reaching a quality that is hard for automatic speaker verification ( ASV ) systems to detect ( Yi et al. , 2020 ) . ASVspoof 2019 ( Todisco et al. , 2019 ; Wang et al. , 2020 ) found that it was challenging to detect spoof audios generated from Jia et al . ( 2018 ) , which uses the same speaker encoder-based approach as in the original Translatotron . Such progress poses concerns on related techniques being misused for creating spoofing artifacts . We designed Translatotron 2 with the motivation of avoiding such potential misuse . 3 TRANSLATOTRON 2 . The motivation of the architectural design of Translatotron 2 is to improve on three weaknesses existing in the original Translatotron : 1 ) The valuable auxiliary supervision on textual features is not fully utilized during training ; 2 ) Long-sequence-to-long-sequence modelling with attention is difficult because of the flexibility of the attention mechanism ; 3 ) Attention-based speech generation is known to suffer from robustness issues such as over-generation and under-generation . The proposed Translatotron 2 model solves these problems by using an architecture composed of a source speech encoder , a target phoneme decoder , and a target mel-spectrogram synthesizer . These three components are connected together by a single attention module . See Figure 1 for an illustration . The model is jointly trained with a speech-to-speech translation objective and a speech-to-phoneme translation objective . As the result , the auxiliary supervision is fully utilized , and the attention is used only for modeling shorter target phoneme sequence but is not directly involved in speech generation . The following subsections describe the architecture of each components as used in our main experiments . As shown in Appendix D , using alternative architectures for these components do not significantly impact the performance of Translatotron 2 . 3.1 ENCODER . The encoder of Translatotron 2 takes mel-spectrogram of the source speech as the input , and produces a hidden representation . We use Conformer ( Gulati et al. , 2020 ) as the architecture for the encoder . It first subsamples the input mel-spectrogram with a convolutional layer , and then processes it with a stack of Conformer blocks . Each Conformer block is composed of a feed-forward layer , a selfattention layer , a convolution layer , and a second feed-forward layer . SpecAugment ( Park et al. , 2019 ) is applied at the training time . 3.2 DECODER . The autoregressive decoder is assisted with an attention module . It takes the encoder output as the source values for the attention , and predicts phoneme sequences corresponding to the target translated speech . We use a stack of LSTM cells as the decoder , along with a multi-head attention ( Vaswani et al. , 2017 ) . The combination of the encoder , the decoder , and the attention module is similar to a typical ST model , except that it predicts phonemes instead of subword tokens . 3.3 SYNTHESIZER . The synthesizer takes the hidden output from the decoder , as well as the context output from the attention module as its input , and synthesizes the target mel-spectrogram . It is similar to the decoders in typical neural TTS models , such as in Shen et al . ( 2018 ; 2020 ) ; Ren et al . ( 2021 ) . The predicted mel-spectrogram can be converted into waveform using either an estimation algorithm such as Griffin & Lim ( 1984 ) or a neural vocoder . We experimented with a number of different non-attentive architectures for the synthesizer , including autoregressive ones and parallel ones , and found that autoregressive synthesizers worked best . We followed the architecture and the hyperparameters of the mel-spectrogram decoder from NAT ( Shen et al. , 2020 ) . It first predicts a duration for each of its input elements using a bidirectional LSTM stack , then upsamples the input sequence with Gaussian weights based on the predicted duration , and finally uses an auto-regressive LSTM stack to predict the target mel-spectrogram . The predicted melspectrogram is further refined with a residual convolutional post-net . This synthesizer is trained jointly with a mel-spectrogram reconstruction loss and a duration prediction loss . Figure 1 ( b ) illustrates the architecture of this synthesizer . Unlike in Shen et al . ( 2020 ) , we do not use per-phoneme duration labels for training the duration predictor , nor do we apply the FVAE-based alignment . Instead , only an L2 loss on the total predicted duration of the entire sequence is used ( i.e . the “ naı̈ve approach ” of unsupervised duration modelling in Shen et al . ( 2020 ) ) . We anticipate that adopting these approaches could further improve the performance of Translatotron 2 , which we leave as future work .

Summary: This paper advances the previously proposed direct speech-to-speech translation (S2ST) model Translatotron.  Direct S2ST has many benefits but has quite limited research effort. The major contributions of this work can be summarized as follows. Translatotron 2 addressed the following major problems of Translatotron: 1)  translation quality, naturalness and robustness of predicted speech are significantly improved and now comparable to a cascaded system (i.e., speech-to-text translation followed by TTS).  2) Similar to Translatotron, the model retrains source speaker’s voice in predicted speech, but Translatotron 2 achieves this without relying on any explicit speaker embedding or speaker ID, hence Translatotron 2 will not be able to generate speech in a different speaker’s voice. This will address some ethical concerns for deployment. Translatotron 2 achieved these improvements through the following approaches. Compared to the previous Translatotron model,  1) Translatotron 2  uses  the output from the auxiliary target phoneme decoder as an input to the spectrogram synthesizer;  2) the spectrogram synthesizer is duration-based, while still keeping the benefits of the attention mechanism.