INTERSPEECH 2020
October 25–29, 2020, Shanghai, China

Voice Transformer Network: Sequence-to-Sequence Voice Conversion Using
Transformer with Text-to-Speech Pretraining
Wen-Chin Huang1 , Tomoki Hayashi1 , Yi-Chiao Wu1 , Hirokazu Kameoka2 , Tomoki Toda1
1

2

Nagoya University, JapanNTT Communication Science Laboratories, Japan
wen.chinhuang@g.sp.m.is.nagoya-u.ac.jp

Abstract

Transformer architecture [17] has been shown to perform efficiently [18] in various speech processing tasks such as automatic speech recognition (ASR) [19], speech translation (ST)[20, 21], and text-to-speech (TTS) [22]. On the basis of attention mechanism solely, the Transformer enables parallel training by avoiding the use of recurrent layers, and provides a receptive field that spans the entire input by using multi-head
self-attention rather than convolutional layers. Nonetheless, theabove-mentioned speech applications that have successfully utilized the Transformer architecture all attempted to find a mapping between text and acoustic feature sequences. VC, in contrast, attempts to map between acoustic frames, whose high time
resolution introduces challenges regarding computational memory cost and accurate attention learning.
Despite the promising results, seq2seq VC models sufferfrom two major problems. First, seq2seq models usually requirea large amount of training data, although a large-scale parallel corpus, i.e, pairs of speech samples with identical linguistic
contents uttered by both source and target speakers, is impractical to collect. Second, as pointed out in [11], the converted
speech often suffers from mispronunciations and other instability problems such as phonemes and skipped phonemes. Several
techniques have been proposed to address these issues. Some
used external resources such as text labels [11, 12], extra data
[15] or pretrained modules such as ASR [10, 13], while others
designed extra regularization objectives [9, 16].
In this work, we propose a novel yet simple pretraining
technique to transfer knowledge from learned TTS models to
seq2seq VC training. We pretrain the encoder and decoder inseparate steps to transfer the core abilities, which are to generate and utilize fine representations. First, the decoder is pretrained with a large-scale TTS corpus and a conventional TTS
objective. This ensures a well-trained decoder that can generate high-quality speech with the correct hidden representations.
Then, we train the encoder in an autoencoder style with the pretrained decoder fixed, so that the encoder is forced to encode
input speech into hidden representations that can be recognizedby the decoder. We demonstrate that the VC model initializedwith the pretrained model parameters can generate high-quality,
highly intelligible speech even with very limited training data.
Our contributions in this work are as follows:

We introduce a novel sequence-to-sequence (seq2seq) voice
conversion (VC) model based on the Transformer architecturewith text-to-speech (TTS) pretraining. Seq2seq VC models
are attractive owing to their ability to convert prosody. While
seq2seq models based on recurrent neural networks (RNNs)
and convolutional neural networks (CNNs) have been successfully applied to VC, the use of the Transformer network, which
has shown promising results in various speech processing tasks,
has not yet been investigated. Nonetheless, their data-hungry
property and the mispronunciation of converted speech makeseq2seq models far from practical. To this end, we propose
a simple yet effective pretraining technique to transfer knowledge from learned TTS models, which benefit from large-scale,easily accessible TTS corpora. VC models initialized withsuch pretrained model parameters are able to generate effective
hidden representations for high-fidelity, highly intelligible converted speech. Experimental results show that such a pretraining scheme can facilitate data-efficient training and outperform
an RNN-based seq2seq VC model in terms of intelligibility, naturalness, and similarity.
Index Terms: Voice Conversion, Sequence-to-Sequence
Learning, Transformer, pretraining

1. Introduction
Voice conversion (VC) aims to convert the speech from asource to that of a target without changing the linguistic content[1]. Conventional VC systems follow an analysis—conversion—synthesis paradigm [2]. First, a high quality vocoder suchas WORLD [3] or STRAIGHT [4] is utilized to extract different acoustic features, such as spectral features and fundamental
frequency (F0). These features are converted separately, and a
waveform synthesizer finally generates the converted waveformusing the converted features. Past VC studies have focused on
the conversion of spectral features while only applying a simple
linear transformation to F0. In addition, the conversion is usually performed frame-by-frame, i.e, the converted speech andthe source speech are always of the same length. To summarize, the conversion of prosody, including F0 and duration, isoverly simplified in the current VC literature.
This is where sequence-to-sequence (seq2seq) models [5]can play a role. Modern seq2seq models, often equipped withan attention mechanism [6, 7] to implicitly learn the alignment
between the source and output sequences, can generate outputsof various lengths. This ability makes the seq2seq model a natural choice to convert duration in VC. In addition, the F0 contour
can also be converted by considering F0 implicitly using mel
spectrograms as the feature representation.
Existing seq2seq VC models are based on either recurrent
neural networks (RNNs) [8, 9, 10, 11, 12, 13, 14, 15] or convolutional neural networks (CNNs) [16]. In recent years, theCopyright © 2020 ISCA

• We apply the Transformer network to VC. To our knowledge, this is the first work to investigate this combination.
• We propose a TTS pretraining technique for VC. The
pretraining process provides a prior for fast, sampleefficient VC model learning, thus reducing the data
size requirement and training time. In this work, we
verify the effectiveness of this scheme by transferringknowledge from Transformer-based TTS models to a
Transformer-based VC model.

4676

http://dx.doi.org/10.21437/Interspeech.2020-1066

2.2. Transformer-based text-to-speech synthesis
In this subsection we describe the Transformer-based TTS system proposed in [22], which we will refer to as TransformerTTS. Transformer-TTS is a combination of the Transformer[17] architecture and the Tacotron 2 [23] TTS system.
We first briefly introduce the Transformer model [17].
The Transformer relies solely on a so-called multi-head selfattention module that learns sequential dependences by jointly
attending to information from different representation subspaces. The main body of Transformer-TTS resembles theoriginal Transformer architecture, which consists of an encoder
stack and a decoder stack that are composed of L encoder layers
and L decoder layers, respectively. An encoder layer contains
a multi-head self-attention sublayer followed by a positionwise
fully connected feedforward network. A decoder layer, in addition to the two sub-layers in the encoder layer, contains a third
sub-layer, which performs multi-head attention over the outputof the encoder stack. Each layer is equipped with residual connections and layer normalization. Finally, since no recurrent relation is employed, sinusoidal positional encoding [24] is added
to the inputs of the encoder and decoder so that the model canbe aware of information about the relative or absolute positionof each element.
The model architecture of Transformer-TTS is depicted in
Figure 1. Since the Transformer architecture was originally designed for machine translation, several changes have been madeto the architecture in [22] to make it compatible in the TTS task.
First, as in Tacotron 2, prenets are added to the encoder and decoder sides. Since the text space and the acoustic feature space
are different, the positional embeddings are employed with corresponding trainable weights to adapt to the scale of each space.
In addition to the linear projection to predict the output acoustic
feature, an extra linear layer is added to predict the stop token[23]. A weighted binary cross-entropy loss is used so that themodel can learn when to stop decoding. As a common practice in recent TTS models, a five-layer CNN postnet predicts a
residual to refine the final prediction.
In this work, our implementation is based on the opensource ESPnet-TTS [25, 26], where the encoder prenet is discarded and the guided attention loss is applied [27] to partial
heads in partial decoder layers [18].

Figure 1: Model architecture of Transformer-TTS and VC.

2. Background

3. Voice Transformer Network

2.1. Sequence-to-sequence speech systhesis

In this section we describe the combination of Transformer andseq2seq VC. Our proposed model, called the Voice Transformer
Network (VTN), is largely based on Transformer-TTS introduced in Section 2.2. Our model consumes the source log-mel
spectrogram and outputs the converted log-mel spectrogram.
Since both TTS and VC respectively encode text and acoustic features to decode acoustic features, we make a very simple
modification to the TTS model, which is to replace the embedding lookup layer in the encoder with a linear projection layer,
as shown in Figure 1. Although more complicated networks
can be employed, we found that this simple design is sufficientto generate satisfying results. The rest of the model architecture as well as the training process remains the same as that forTransformer-TTS.
An important trick we found to be useful here is to use a reduction factor in both the encoder and the decoder for accurate
attention learning. In seq2seq TTS, since the time resolutionof acoustic features is usually much larger than that of the text
input, a reduction factor rd is commonly used on the decoder

Seq2seq models are used to find a mapping between a source
feature sequence x1:n = (x1 , · · · , xn ) and a target feature sequence y 1:m = (y 1 , · · · , y m ) which do not necessarily haveto be of the same length, i.e, n 6= m. Most seq2seq models have an encoder—decoder structure [5], where advanced
ones are equipped with an attention mechanism [6, 7]. First,
an encoder (Enc) maps x1:n into a sequence of hidden representations h1:n = (h1 , · · · , hn ) = Enc(x1:n ). The decoder (Dec) is autoregressive, which means that at time step t,
the previously generated features y 1:t−1 are considered when
decoding the current output y t . A context vector ct , which
is a weighted sum of h1:n calculated by an attention proba(1)(n)
bility vector at = (at , · · · , at ), is also used by the de(k)coder. Each attention probability at can be thought of as
the importance of the hidden representation hk at the tth time
step. The above-mentioned procedure can be formulated as:
y t = Dec(y 1:t−1 , ct ). During training, the model is optimized
via backpropagation using an L1 or L2 loss.

4677

EncSTTS can be obtained by minimizing the reconstruction lossof S TTS . As the decoder pretraining process described in Section 4.1 takes a hidden representation encoded from text as the
input, fixing it in the encoder pretraining process guarantees theencoder to behave similarly to EncTTTS , which is to extract finegrained, linguistic-information-rich representations.
4.3. VC model training
Finally, using D VC , we train the desired VC models, with theencoder and decoder initialized with EncSTTS and DecSTTS pretrained in Section 4.2 and Section 4.1, respectively. The pretrained model parameters serve as a very good prior to adapt
to the relatively scarce VC data, as we will show later. Also,
compared with training from scratch, the model takes less thanhalf the training time to converge with the pretraining scheme,
enabling extremely efficient training.

Figure 2: Illustration of proposed TTS pretraining techniquefor VC.
side [28], where multiple stacked frames are decoded at eachtime step. On the other hand, although the input and output of
VC are both acoustic features, the high time resolution (about
100 frames per second) not only makes attention learning difficult but also increases the training memory footprint. Whilepyramid RNNs were used to reduce the time resolution in [10],here we simply introduce an encoder reduction factor re , where
adjacent frames are stacked to reduce the time axis. We foundthat this not only leads to better attention alignment but also reduces the training memory footprint by half and subsequently
the number of required gradient accumulation steps [26].

5. Experimental evaluation
5.1. Experimental settings
We conducted our experiments on the CMU ARCTIC database[29], which contains parallel recordings of professional US English speakers sampled at 16 kHz. One female (slt) was chosen
as the target speaker and one male (bdl) and one female (clb)were chosen as sources. The number of train/valid/test utterances were 932/100/100. For the TTS corpus, we chose a US female English speaker (judy bieber) from the M-AILABS speechdataset [30], which contained 15,200 utterances (32 hours).
The entire implementation was carried out on the opensource ESPnet toolkit [26, 25], including feature extraction,
training and benchmarking. We extracted 80-dimensional mel
spectrograms with 1024 FFT points and a 256 point frame shift.
The base settings for the TTS model and training follow theTransformer.v1 configuration in [26], and we made minimal
modifications to it for VC. The reduction factors re , rd areboth 2 in all VC models. For the waveform synthesis module, we used Parallel WaveGAN (PWG) [31], which is a nonautoregressive variant of the WaveNet vocoder [32, 33]1 . We
trained a speaker-dependent PWG by conditioning on natural
mel spectrograms using the full training data of slt.
We carried out two types of objective evaluations between
the converted speech and the ground truth: the mel cepstrum
distortion (MCD), a commonly used measure of spectral distortion in VC, and the character error rate (CER) as well as theword error rate (WER), which estimate the intelligibility. The
ASR engine is based on the Transformer [19] and is trained using the LibriSpeech dataset [34]. The CER and WER for the
ground-truth test set of slt were 0.9% and 3.8%, respectively.

4. Proposed training strategy with
text-to-speech pretraining
We present a text-to-speech pretraining technique that enables
fast, sample-efficient training without introducing additional
modification or loss to the original model structure or training
loss. Assume that, in addition to a small, parallel VC dataset
D VC = {S src , S trg }, access to a large single-speaker TTS corpus D TTS = {T TTS , S TTS } is also available. S src , S trg denotethe source, target speech respectively, and T TTS , S TTS denotethe text and speech of the TTS speaker respectively. Our setup
is highly flexible in that we do not require any of the speakers
to be the same, nor any of the sentences between the VC andTTS corpus to be parallel. We employ a two-stage training procedure, where in the first stage we use D TTS to learn the initial
parameters as a prior, and then use D VC to adapt to the VC
model in the second stage. As argued in Section 1, the ability
to generate fine-grained hidden representations H is the key toa good VC model, so our goal is to find a set of prior model
parameters to train the final encoder EncSVC and decoder DecSVC .
The overall procedure is depicted in Figure 2.
4.1. Decoder pretraining
The decoder pretraining is as simple as training a conventional
TTS model using D TTS . Since text itself contains pure linguistic
information, the text encoder EncTTTS here is ensured to learn toencode an effective hidden representation that can be consumed
by the decoder DecSTTS . Furthermore, by leveraging large-scale
corpora, the decoder is expected to be more robust by capturing
various speech features, such as articulation and prosody.

5.2. Effectiveness of TTS pretraining
To evaluate the importance and the effectiveness of each pretraining scheme we proposed, we conducted a systematic comparison between different training processes and different sizesof training data. The objective results are in Table 1. First, when
the network was trained from scratch without any pretraining,
the performance was not satisfactory even with the full training set. With decoder pretraining, a performance boost in MCD
was obtained, whereas the ASR results were similar. Nonetheless, as we reduced the training size, the performance dropped
dramatically. Finally, by incorporating encoder pretraining, the4.2. Encoder pretraining
A well pretrained encoder should be capable of encoding acoustic features into hidden representations that are recognizable by
the pretrained decoder. With this goal in mind, we train an autoencoder whose decoder is the one pretrained in Section 4.1
and kept fixed during training. The desired pretrained encoder

1 https://github.com/kan-bayashi/ParallelWaveGAN

4678

Table 1: Validation-set objective results of adapted TTS, baseline (ATTS2S), and variants of the proposed VTN.
pretraining
Speaker

250 training utterances

80 training utterances

Description

Encoder

Decoder

MCD

CER

WER

MCD

CER

WER

MCD

CER

WER

VTN

X

X
X

7.16
6.84
6.44

15.1
9.9
2.4

23.2
18.1
6.3

8.20
6.82

52.3
2.9

71.7
7.0

7.15

10.6

16.1

ATTS2S

-

-

7.18

7.5

14.1

7.83

19.9

30.4

8.46

40.4

58.5

VTN

X

X
X

7.32
7.07
6.56

20.9
20.3
6.1

32.6
31.7
12.0

8.52
7.14

58.9
9.6

80.7
16.2

7.34

15.0

23.7

ATTS2S

-

-

7.36

13.0

22.7

8.01

28.4

43.9

8.76

51.2

72.7

TTS adaptation

-

-

-

3.3

6.2

-

4.5

7.2

-

4.8

7.8

clb-slt

bdl-slt

slt

932 training utterances

Table 2: Test set subjective evaluation results with 95%
confidence intervals of the ground truth, the TTS model, thebaseline ATTS2S and the proposed model, VTN. The numbersin the parenthesis indicate the number of training utterances.
Description

Naturalness

Similarity

Ground truth
TTS adaptation (932)

4.65 ± 0.19
3.80 ± 0.17-

ATTS2S (932)VTN (932)
VTN (80)

2.67 ± 0.23
4.02 ± 0.17
3.75 ± 0.17

50% ± 16%83% ± 9%72% ± 12%

i.e., sure or not sure. Ten listeners were recruited.2
Table 2 shows the subjective results on the test set. First,
with the full training set, our proposed VTN model significantly
outperformed the baseline ATTS2S by over one point for naturalness and 30% for similarity. Moreover, when trained with
80 utterances, our proposed method showed only a slight dropin performance, and was still superior to the baseline method.
This result justifies the effectiveness of our method and also
showed that the pretraining technique can greatly increase data
efficiency without severe performance degradation.

6. Conclusion

model exhibited a significant improvement in all objective measures, where the effectiveness was robust against the reductionin the size of training data.

In this work, we successfully applied the Transformer structure
to seq2seq VC. Also, to address the problems of data efficiency
and mispronunciation in seq2seq VC, we proposed the transfer of knowledge from easily accessible, large-scale TTS corpora by initializing the VC models with pretrained TTS models. A two-stage training strategy that pretrains the decoder andthe encoder subsequently ensures that fine-grained intermediate
representations are generated and fully utilized. Objective andsubjective evaluations showed that our pretraining scheme cangreatly improve speech intelligibility, and it significantly outperformed an RNN-based seq2seq VC baseline. Even with limited training data, our system can be successfully trained without significant performance degradation. In the future, we plan
to more systematically examine the effectiveness of the Transformer architecture compared with RNN-based models. Extension of our pretraining methods to more flexible training conditions, such as nonparallel training [12], is also an important
future task.

5.3. Comparison with baseline method
Next, we compared our VTN model with an RNN-based
seq2seq VC model called ATTS2S [9]. This model is based on
the Tacotron model [28] with the help of context preservation
loss and guided attention loss to stabilize training and maintainlinguistic consistency after conversion. We followed the configurations in [9] but used mel spectrograms instead of WORLDfeatures.
The objective evaluation results of the baseline are reported
in Table 1. For the different sizes of training data, our system not only consistently outperformed the baseline method but
also remained robust, whereas the performance of the baseline
method dropped dramatically as the size of training data was
reduced. This proves that our proposed method can improvedata efficiency as well as pronunciation. We also observed thatwhen trained from scratch, our VTN model had a similar MCD
and inferior ASR performance compared with the baseline. As
the ATTS2S employed an extra mechanism to stabilize training,
this result may indicate the superiority of using the Transformer
architecture over RNNs. We will investigate this in the future.
Systemwise subjective tests on naturalness and conversion
similarity were also conducted to evaluate the perceptual performance. For naturalness, participants were asked to evaluate
the naturalness of the speech by the mean opinion score (MOS)
test on a five-point scale. For conversion similarity, each listener was presented a natural speech of the target speaker anda converted speech, and asked to judge whether they were produced by the same speaker with the confidence of the decision,

7. Acknowledgements
This work was supported in part by JST PRESTO Grant
Number JPMJPR1657 and JST CREST Grant Number JPMJCR19A3, Japan.

8. References[1] Y. Stylianou, O. Cappe, and E. Moulines, “Continuous probabilistic transform for voice conversion,” IEEE Transactions on Speech
and Audio Processing, vol. 6, no. 2, pp. 131–142, 1998.
2 Audiosamples:
https://unilight.github.io/
Publication-Demos/publications/transformer-vc/

4679

[2] T. Toda, A. W. Black, and K. Tokuda, “Voice Conversion Based on
Maximum-Likelihood Estimation of Spectral Parameter Trajectory,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 15, no. 8, pp. 2222–2235, 2007.

[19] L. Dong, S. Xu, and B. Xu, “Speech-Transformer: A NoRecurrence Sequence-to-Sequence Model for Speech Recognition,” in IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP), 2018, pp. 5884–5888.

[3] M. Morise, F. Yokomori, and K. Ozawa, “WORLD: A VocoderBased High-Quality Speech Synthesis System for Real-Time Applications,” IEICE Transactions on Information and Systems,
vol. 99, pp. 1877–1884, 2016.

[20] L. Cross Vila, C. Escolano, J. A. R. Fonollosa, and M. R. CostaJuss, “End-to-End Speech Translation with the Transformer,” inProc. IberSPEECH, 2018, pp. 60–63.
[21] M. A. D. Gangi, M. Negri, R. Cattoni, R. Dessi, and M. Turchi,“Enhancing Transformer for End-to-end Speech-to-Text Translation,” in Proceedings of Machine Translation Summit XVII Volume
1: Research Track, 2019, pp. 21–31.

[4] H. Kawahara, I. Masuda-Katsuse, and A. de Cheveign, “Restructuring speech representations using a pitch-adaptive timefrequency smoothing and an instantaneous-frequency-based F0
extraction: Possible role of a repetitive structure in sounds,”
Speech Communication, vol. 27, no. 3, pp. 187–207, 1999.

[22] N. Li, S. Liu, Y. Liu, S. Zhao, and M. Liu, “Neural Speech Synthesis with Transformer Network,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 6706–6713.

[5] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to Sequence
Learning with Neural Networks,” in Advances in Neural Information Processing Systems, 2014, pp. 3104–3112.

[23] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang,
Z. Chen, Y. Zhang, Y. Wang, R. Skerry-Ryan, R. A. Saurous,
Y. Agiomyrgiannakis, and Y. Wu, “Natural TTS Synthesis by
Conditioning WaveNet on MEL Spectrogram Predictions,” inIEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2018, pp. 4779–4783.

[6] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint
arXiv:1409.0473, 2014.
[7] T. Luong, H. Pham, and C. D. Manning, “Effective approaches toattention-based neural machine translation,” in Proceedings of theConference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, Sep. 2015, pp. 1412–1421.

[24] J. Gehring, M. Auli, D. Grangier, D. Yarats, and Y. N. Dauphin,“Convolutional Sequence to Sequence Learning,” in Proceedings of the 34th International Conference on Machine Learning,
vol. 70, 2017, pp. 1243–1252.

[8] H. Miyoshi, Y. Saito, S. Takamichi, and H. Saruwatari, “Voice
Conversion Using Sequence-to-Sequence Learning of Context
Posterior Probabilities,” in Proc. Interspeech, 2017, pp. 1268–
1272.

[25] S. Watanabe, T. Hori, S. Karita, T. Hayashi, J. Nishitoba, Y. Unno,
N. E. Y. Soplin, J. Heymann, M. Wiesner, N. Chen, A. Renduchintala, and T. Ochiai, “ESPnet: End-to-End Speech Processing
Toolkit,” in Proc. Interspeech, 2018, pp. 2207–2211.

[9] K. Tanaka, H. Kameoka, T. Kaneko, and N. Hojo, “ATTS2S-VC:
Sequence-to-sequence Voice Conversion with Attention and Context Preservation Mechanisms,” in IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP), May 2019,
pp. 6805–6809.

[26] T. Hayashi, R. Yamamoto, K. Inoue, T. Yoshimura, S. Watanabe,
T. Toda, K. Takeda, Y. Zhang, and X. Tan, “ESPnet-TTS: Unified,
Reproducible, and Integratable Open Source End-to-End Text-toSpeech Toolkit,” arXiv preprint arXiv:1910.10909, 2019.

[10] J. Zhang, Z. Ling, L. Liu, Y. Jiang, and L. Dai, “Sequence-toSequence Acoustic Modeling for Voice Conversion,” IEEE/ACM
Transactions on Audio, Speech, and Language Processing,vol. 27, no. 3, pp. 631–644, 2019.

[27] H. Tachibana, K. Uenoyama, and S. Aihara, “Efficiently Trainable Text-to-Speech System Based on Deep Convolutional Networks with Guided Attention,” in IEEE International Conferenceon Acoustics, Speech and Signal Processing (ICASSP), 2018, pp.
4784–4788.

[11] J.-X. Zhang, Z.-H. Ling, Y. Jiang, L.-J. Liu, C. Liang, and
L.-R. Dai, “Improving Sequence-to-sequence Voice Conversion
by Adding Text-supervision,” IEEE International Conference on
Acoustics, Speech and Signal Processing (ICASSP), pp. 6785–
6789, 2019.

[28] Y. Wang, R. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss,
N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le,
Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous, “Tacotron:
Towards end-to-end speech synthesis,” in Proc. Interspeech, 2017,
pp. 4006–4010.

[12] J.-X. Zhang, Z.-H. Ling, and L.-R. Dai, “Non-Parallel Sequenceto-Sequence Voice Conversion with Disentangled Linguistic
and Speaker Representations,” arXiv preprint arXiv:1906.10508,
2019.

[29] J. Kominek and A. W. Black, “The CMU ARCTIC speech
databases,” in Fifth ISCA Workshop on Speech Synthesis, 2004.
[30] Munich Artificial Intelligence Laboratories GmbH, “TheM-AILABS speech dataset,” 2019, accessed 30 November 2019. [Online]. Available: https://www.caito.de/2019/01/
the-m-ailabs-speech-dataset/[13] G. Zhao, S. Ding, and R. Gutierrez-Osuna, “Foreign Accent Conversion by Synthesizing Speech from Phonetic Posteriorgrams,”
in Proc. Interspeech, 2019, pp. 2843–2847.
[14] P. Narayanan, P. Chakravarty, F. Charette, and G. Puskorius, “Hierarchical sequence to sequence voice conversion with limited
data,” arXiv preprint arXiv:1907.07769, 2019.

[31] R. Yamamoto, E. Song, and J.-M. Kim, “Parallel WaveGAN: A
fast waveform generation model based on generative adversarial networks with multi-resolution spectrogram,” arXiv preprint
arXiv:1910.11480, 2019.

[15] F. Biadsy, R. J. Weiss, P. J. Moreno, D. Kanvesky, and Y. Jia, “Parrotron: An End-to-End Speech-to-Speech Conversion Model andits Applications to Hearing-Impaired Speech and Speech Separation,” in Proc. Interspeech, 2019, pp. 4115–4119.

[32] A. v. d. Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals,
A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu,“Wavenet: A generative model for raw audio,” arXiv preprint
arXiv:1609.03499, 2016.

[16] H. Kameoka, K. Tanaka, T. Kaneko, and N. Hojo, “ConvS2SVC: Fully convolutional sequence-to-sequence voice conversion,”
CoRR, vol. abs/1811.01609, 2018.

[33] A. Tamamori, T. Hayashi, K. Kobayashi, K. Takeda, and T. Toda,“Speaker-dependent wavenet vocoder,” in Proc. Interspeech,
2017, pp. 1118–1122.

[17] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N.
Gomez, L. u. Kaiser, and I. Polosukhin, “Attention is All you
Need,” in Advances in Neural Information Processing Systems,
2017, pp. 5998–6008.

[34] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: An ASR corpus based on public domain audio books,”in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2015, pp. 5206–5210.

[18] S. Karita, N. Chen, T. Hayashi, T. Hori, H. Inaguma, Z. Jiang,
M. Someki, N. E. Y. Soplin, R. Yamamoto, X. Wang et al., “A
comparative study on transformer vs RNN in speech applications,” arXiv preprint arXiv:1909.06317, 2019.

4680