Abstract: We introduce a novel sequence-to-sequence (seq2seq) voice conversion (VC) model based on the Transformer architecture with text-to-speech (TTS) pretraining. Seq2seq VC models are attractive owing to their ability to convert prosody. While seq2seq models based on recurrent neural networks (RNNs) and convolutional neural networks (CNNs) have been successfully applied to VC, the use of the Transformer network, which has shown promising results in various speech processing tasks, has not yet been investigated. Nonetheless, their data-hungry property and the mispronunciation of converted speech make seq2seq models far from practical. To this end, we propose a simple yet effective pretraining technique to transfer knowledge from learned TTS models, which benefit from large-scale, easily accessible TTS corpora. VC models initialized with such pretrained model parameters are able to generate effective hidden representations for high-fidelity, highly intelligible converted speech. Experimental results show that such a pretraining scheme can facilitate data-efficient training and outperform an RNN-based seq2seq VC model in terms of intelligibility, naturalness, and similarity.

Introduction: Voice conversion (VC) aims to convert the speech from a source to that of a target without changing the linguistic content BIBREF0. Conventional VC systems follow an analysis—conversion —synthesis paradigm BIBREF1. First, a high quality vocoder such as WORLD BIBREF2 or STRAIGHT BIBREF3 is utilized to extract different acoustic features, such as spectral features and fundamental frequency (F0). These features are converted separately, and a waveform synthesizer finally generates the converted waveform using the converted features. Past VC studies have focused on the conversion of spectral features while only applying a simple linear transformation to F0. In addition, the conversion is usually performed frame-by-frame, i.e, the converted speech and the source speech are always of the same length. To summarize, the conversion of prosody, including F0 and duration, is overly simplified in the current VC literature. This is where sequence-to-sequence (seq2seq) models BIBREF4 can play a role. Modern seq2seq models, often equipped with an attention mechanism BIBREF5, BIBREF6 to implicitly learn the alignment between the source and output sequences, can generate outputs of various lengths. This ability makes the seq2seq model a natural choice to convert duration in VC. In addition, the F0 contour can also be converted by considering F0 explicitly (e.g, forming the input feature sequence by concatenating the spectral and F0 sequences) BIBREF7, BIBREF8, BIBREF9 or implicitly (e.g, using mel spectrograms as the input feature) BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15. Seq2seq VC can further be applied to accent conversion BIBREF13, where the conversion of prosody plays an important role. Existing seq2seq VC models are based on either recurrent neural networks (RNNs) BIBREF7, BIBREF8, BIBREF10, BIBREF11, BIBREF12, BIBREF13, BIBREF14, BIBREF15 or convolutional neural networks (CNNs) BIBREF9. In recent years, the Transformer architecture BIBREF16 has been shown to perform efficiently BIBREF17 in various speech processing tasks such as automatic speech recognition (ASR) BIBREF18, speech translation (ST) BIBREF19, BIBREF20, and text-to-speech (TTS) BIBREF21. On the basis of attention mechanism solely, the Transformer enables parallel training by avoiding the use of recurrent layers, and provides a receptive field that spans the entire input by using multi-head self-attention rather than convolutional layers. Nonetheless, the above-mentioned speech applications that have successfully utilized the Transformer architecture all attempted to find a mapping between text and acoustic feature sequences. VC, in contrast, attempts to map between acoustic frames, whose high time resolution introduces challenges regarding computational memory cost and accurate attention learning. Despite the promising results, seq2seq VC models suffer from two major problems. First, seq2seq models usually require a large amount of training data, although a large-scale parallel corpus, i.e, pairs of speech samples with identical linguistic contents uttered by both source and target speakers, is impractical to collect. Second, as pointed out in BIBREF11, the converted speech often suffers from mispronunciations and other instability problems such as phonemes and skipped phonemes. Several techniques have been proposed to address these issues. In BIBREF10 a pretrained ASR module was used to extract phonetic posteriorgrams (PPGs) as an extra clue, whereas PPGs were solely used as the input in BIBREF13. The use of context preservation loss and guided attention loss BIBREF22 to stabilize training has also been proposed BIBREF8, BIBREF9. Multitask learning and data augmentation were incorporated in BIBREF11 using additional text labels to improve data efficiency, and linguistic and speaker representations were disentangled in BIBREF12 to enable nonparallel training, thus removing the need for a parallel corpus. In BIBREF15 a large hand-transcribed corpus was used to generate artificial training data from a TTS model for a many-to-one (normalization) VC model, where multitask learning was also used. One popular means of dealing with the problem of limited training data is transfer leaning, where knowledge from massive, out-of-domain data is utilized to aid learning in the target domain. Recently, TTS systems, especially neural seq2seq models, have enjoyed great success owing to the vast large-scale corpus contributed by the community. We argue that lying at the core of these TTS models is the ability to generate effective intermediate representations, which facilitates correct attention learning that bridges the encoder and the decoder. Transfer learning from TTS has been successfully applied to tasks such as speaker adaptation BIBREF23, BIBREF24, BIBREF25, BIBREF26. In BIBREF27 the first attempt to apply this technique to VC was made by bootstrapping a nonparallel VC system from a pretrained speaker-adaptive TTS model. In this work, we propose a novel yet simple pretraining technique to transfer knowledge from learned TTS models. To transfer the core ability, i.e, the generation and utilization of fine representations, knowledge from both the encoder and the decoder is needed. Thus, we pretrain them in separate steps: first, the decoder is pretrained by using a large-scale TTS corpus to train a conventional TTS model. The TTS training ensures a well-trained decoder that can generate high-quality speech with the correct hidden representations. As the encoder must be pretrained to encode input speech into hidden representations that can be recognized by the decoder, we train the encoder in an autoencoder style with the pretrained decoder fixed. This is carried out using a simple reconstruction loss. We demonstrate that the VC model initialized with the above pretrained model parameters can generate high-quality, highly intelligible speech even with very limited training data. Our contributions in this work are as follows: We apply the Transformer network to VC. To our knowledge, this is the first work to investigate this combination. We propose a TTS pretraining technique for VC. The pretraining process provides a prior for fast, sample-efficient VC model learning, thus reducing the data size requirement and training time. In this work, we verify the effectiveness of this scheme by transferring knowledge from Transformer-based TTS models to a Transformer-based VC model.

Question: What is the baseline model?
Answer: a RNN-based seq2seq VC model called ATTS2S based on the Tacotron model