ATTS2S-VC: SEQUENCE-TO-SEQUENCE VOICE CONVERSIONWITH ATTENTION AND CONTEXT PRESERVATION MECHANISMS
Kou Tanaka, Hirokazu Kameoka, Takuhiro Kaneko, Nobukatsu Hojo
NTT Communication Science Laboratories, NTT Corporation, Japan{tanaka.ko, kameoka.hirokazu, kaneko.takuhiro, hojo.nobukatsu}@lab.ntt.co.jp

ABSTRACTThis paper describes a method based on a sequenceto-sequence learning (Seq2Seq) with attention and context
preservation mechanism for voice conversion (VC) tasks.
Seq2Seq has been outstanding at numerous tasks involving
sequence modeling such as speech synthesis and recognition,
machine translation, and image captioning. In contrast to
current VC techniques, our method 1) stabilizes and accelerates the training procedure by considering guided attention
and proposed context preservation losses, 2) allows not only
spectral envelopes but also fundamental frequency contours
and durations of speech to be converted, 3) requires no context information such as phoneme labels, and 4) requires notime-aligned source and target speech data in advance. In our
experiment, the proposed VC framework can be trained inonly one day, using only one GPU of an NVIDIA Tesla K80,
while the quality of the synthesized speech is higher than thatof speech converted by Gaussian mixture model-based VC
and is comparable to that of speech generated by recurrent
neural network-based text-to-speech synthesis, which can be
regarded as an upper limit on VC performance.
Index Terms— Voice conversion, deep learning, sequenceto-sequence, attention mechanism, context preservation mechanism
1. INTRODUCTION
Voice conversion (VC) systems aim to convert para/nonlinguistic information included in a given speech waveform
while preserving its linguistic information. VC has been
applied to various tasks, such as speaker conversion [1–3]
for impersonating or hiding a speaker’s identity, as a speaking aid [4, 5] for overcoming speech impairments, as a style
conversion [6, 7] for controlling speaking styles including
emotion, and for pronunciation/accent conversion [8, 9] inlanguage learning.
A popular form of VC is a statistical one based on a Gaussian mixture model (GMM) [10]; it requires time-aligned
parallel data of the source and target speech for training theconversion models. For frameworks requiring time-aligned
parallel data, other researchers have proposed exemplar-based
VCs using non-negative matrix factorization (NMF) [11, 12]
and neural network (NN)-based VCs using restricted Boltzmann machines [13,14], feed-forward NNs [15,16], recurrent
NNs [17, 18], variational autoencoders [19, 20], and generative adversarial nets [9]. On the other hand, frameworks

978-1-5386-4658-8/18/$31.00 ©2019 IEEE

requiring no parallel data, called parallel-data-free VCs, havebeen proposed [1, 3] to avoid the time-consuming job ofrecording speech for parallel data collection. Notably, thedrawbacks of these VCs are the prerequisite of a large number of transcripts and/or difficulty converting the durations of
the source speech.
Recently, sequence-to-sequence (Seq2Seq) learning [21,22] has proved to be outstanding at various research tasks
such as text-to-speech synthesis (TTS) [23–25] and automatic speech recognition (ASR) [26, 27]. The early Seq2Seq
model [21] has encoder and decoder architectures for mapping an input sequence to an encoded representation used by
the decoder network to generate an output sequence. To selectcritical information from the encoded representation in accordance with the output sequence representation, later Seq2Seq
models [22, 28] introduce an attention mechanism. The key
advantages of the Seq2Seq learning approach are the ability to
train a single end-to-end model directly on the source and target sequences and the capacity to handle input and output sequences of different lengths. In particular, we expect that the
Seq2Seq model makes it possible to convert not only acoustic
features but also the durations of the source speech to those of
the target speech. Moreover, Seq2Seq learning is extensible
to semi-supervised learning [29], where it can avoid the timeconsuming task of collecting parallel data. In a supervised
learning task, Seq2Seq learning requires parallel data of thesource and target sequences rather than time-aligned parallel data. Considering dual learning [30,31], Seq2Seq learning
can be trained with a small amount of parallel data and a large
amount of non-parallel data.
In this paper, we propose a Seq2Seq-based VC with attention and context preservation mechanisms1 . Our contributions are as follows:• Our VC method makes it possible to stabilize and accelerate the training procedure by considering guided
attention and context preservation losses.
• It makes it possible to convert not only spectral envelopes but also fundamental frequency contours anddurations of the speech.
• It requires no context information such as phoneme
labels, unlike [32, 33] which introduced the Seq2Seq
model and used context information.
1 Audio samples can be accessed on our web page: http://www.
kecl.ntt.co.jp/people/tanaka.ko/projects/atts2svc/
attention_based_seq2seq_voice_conversion.html

6805

ICASSP 2019• It requires no time-aligned source or target speech datain advance.

a)TargetWe conducted an our experiment demonstrating that the quality of the synthesized speech generated by our VC framework
is higher than that of speech generated by the conventional
GMM-based VC, and it is comparable to that of speech generated by recurrent-NN based TTS in terms of both naturalness and speaker similarity. Note that the proposed model wastrained in only one day, using only one GPU of an NVIDIA
Tesla K80.

Source speech

Speech analysis
Time alignment
Model training

Conversion
Speechsynthesis

Conversion
model

Converted speechb)

2. CONVENTIONAL VC

Source speech

LSTM (decoder)

Speech analysis

BiLSTM3 (TTS)BiLSTM1 (ASR)

Converted feature

CPPs

Speech synthesis

BiLSTM2 (encoder)

Converted speech

2.2. Seq2Seq-based VC
In contrast to the frame/sequence- based VC frameworks,
Seq2Seq-based VC frameworks make it possible to convert
not only the acoustic features but also the durations of thesource speech. Most Seq2Seq-based VCs consist of ASR andTTS modules which are trainable with pairs of speech andits transcript rather than the source and target speech. The
ASR module converts the acoustic feature sequence of thesource speech into a sequence of context information such as
phoneme labels and context posterior probabilities [32, 37],
and the TTS module generates the acoustic feature sequenceof the desired speech from the sequence of context information. One approach to changing the duration of the source
speech uses a re-generation method that generates the duration information from the text symbols after converting
the acoustic features of the source speech into text symbols
once. Namely, the duration information is erased once andre-generated. Another approach [32, 33] involves Seq2Seq
learning, as shown in Fig. 1b. In this approach, the context
posterior probability sequence of the source speech including
the duration information is directly converted into a context
posterior probability sequence of the desired speech including
the duration information. Both approaches work well if ASR

Target speech

Speech analysis

Target feature

There are two types of frame/sequence- based VC: VCs requiring parallel data [10, 34, 35] and parallel-data-free VC [1,3]. The first framework has different procedures of training
and conversion, as shown in Fig. 1a. The conversion procedure does not have a time warping function, despite that thetraining procedure includes a time-alignment step to handle
source and target sequences having different lengths. The second framework is a parallel-data-free VC that does not requireparallel source and target speech data. To realize paralleldata-free VC, the second framework uses context information [36,37], adaptation techniques [38,39], a pre-constructedspeaker space [40, 41], and cycle consistency [1, 3]. Althoughthese VC techniques have various training procedures, theconversion procedure does not involve the time warping function. Consequently, the frame/sequence- based VC frameworks do not allow us to convert the durations and the acoustic features of the source speech at the same time. In contrast,
our model allows both the acoustic features and the durations
to be converted at the same time.

Calculate loss

CPPs

2.1. Frame/Sequence- based VC

Fig. 1.
System overviews of conventional VC, a)frame/sequence- based VC using parallel data (see Sec. 2.1)
and b) Seq2Seq-based VC (see Sec. 2.2). “CPPs” and “BiLSTM” denote context posterior probabilities and bidirectional
LSTM, respectively.
performs robustly and accurately enough, but they require alarge number of transcripts to train each module. In contrast,
our model does not use any transcript.
3. ATTS2S-VC
Our method consists of 1) four basic components of the
Seq2Seq model and 2) two additional components as a context preservation mechanism. The four basic components area source encoder, target encoder, target autoregressive (AR)decoder, and attention mechanism. The two additional components are a source decoder and another target decoder tokeep linguistic information of the source speech. Figure 2 isan overview of the system.
3.1. Seq2Seq Model with Attention Mechanism
Let us use X = [x1 , · · · , xI ] and Y = [y 1 , · · · , y J ] to denote sequences of acoustic features of the source and target
speech, respectively. The source encoder network fSrcEnc andtarget encoder network fTarEnc encode the input sequences
X and Y to the embeddings K = [k1 , · · · , kI ] and Q =[q 1 , · · · , q J ], as follows:
K = fSrcEnc (X),Q = fTarEnc (Y ).

(1)
(2)

In order to accurately predict the output sequence, [22,28]
introduced an attention mechanism. At each time frame of theembeddings Q, the attention mechanism gives a probability
distribution that describes the relationship between the given

6806

3.2. Stabilizing and Accelerating Training Procedure

Context preservation loss
L1 loss

Speech analysis

Target speech

Converted speechY

To accelerate the training of an attention module, [25] introduced a guided attention loss. Generally speaking, most
speech signal processing applications, such as ASR, TTS, andVC, are time incremental algorithms. It is natural to assume
that the time frame i of the source speech waveform progresses nearly linearly with respect to the time frame j of thetarget speech waveform, i.e., i ∼ αj, where α ∼ JI . Therefore, the attention matrix A should be a nearly diagonal. A
penalty matrix G is designed, as follows:{}−( Ii − Jj )2(8)
gi,j = 1 − exp,
2σg2

Conversion
step

Training step

<latexit sha1_base64="OTVT4pmtA7JNuO5s4z1w/wSt65o=">AAAGKHicnVQ9bxNBEJ0YDCF8JIEGicbCCUoorHUUCUQViSAh8ZUP7ATFluU7r83F57vT3dnYPpySgooOARVIFMDPoKGkQcgtHaIMEg0Fb8dnwLbiQ9zK3pk3897tzuyt5piG5wvRnYgdOhw/cnTy2NTxEydPTc/Mns56dt3VZUa3Tdvd1oqeNA1LZnzDN+W248piTTPllla9quJbDel6hm3d9VuOzNeKFcsoG3rRB5TNNWrBvU5hJilSgp/EqJEOjSSFz5o9G3tEOSqRTTrVqUaSLPJhm1QkD2OH0iTIAZanAJgLy+C4pA5NgVtHlkRGEWgV/xV4OyFqwVeaHrN1vMXEzwUzQfPik3gj9sUH8U58FT8P1ApYQ62lhVnrcaVTmH58dvNHJKuG2af7f1iRjDJWqGrgjdlfP9cGbkXUwYfiZd6/gVyHEVUZvbemRvvp/uaVjfnggnglvqEmL0VXvEdVrMZ3/fW63HgBdaVvgfWAa1jjXVnoWgC8Aa/Dfcqzr9Sb3CWffY2rXvpdiQTNURLZc2B1xipXmae0l1h7lQpA+zourIBu0E3eR45PRBmRBXhK/yFGEkwVc7GaCjg+LcLvhDtaBaN30lxYt0LdO6iRBKI8dVYu8lpUToXf3QQ/GPL/V83g7gVD/r+rKS0PHdVol/N9xpXiHsZgZI9jCu9EdFRVtsQqw9UuRfasGfIanN+MzG8N5Lci89sD+e3I/GtD+9AwAkbH8wz+/lQdEyNV6MUSYSUP1qjArvPN8ff30dfSMZtAbiO6wCd2kdT9mR6+LUeN7FIqLVLp9eXkynJ4k07SOToPnTRdohW6TmuUwRt26Qk9o+fxt/GP8c/xbi81NhFyztDAE//yCxGyS08=</latexit><latexit

3.2.1. Guided Attention Loss

Speechsynthesis

Start
token

Ŷ
<latexit sha1_base64="7bRnO8wUY4qe9cIqOxJ6lURFTLY=">AAAGLnicnVQ7b9RAEJ4EDkJ4JIEGiebEJSihiNZRJBBVJIKExCsP8kC5KLJ9m4uJX7J95i7GKSn4AVBQ8JAoEA3/gYY/AFIKRAuiDBINBd/OOYS7U84Ir+525pv5Pu/OrNfwbSuMhNjp6T10uHDkaN+x/uMnTp4aGBw6vRh6tcCUC6Zne8GyoYfStly5EFmRLZf9QOqOYcslY/Oqii/FMggtz70bNXy56uhV11q3TD0CVC5v6FFSjp3kXpquDZbEuOCn2GlomVGi7JnxhnofUZkq5JFJNXJIkksRbJt0CjFWSCNBPrBVSoAFsCyOS0qpH9wasiQydKCb+K/CW8lQF77SDJlt4i02fgGYRRoRH8UbsSs+iLfim/h1oFbCGmotDcxGkyv9tYHHZ+d/5rIczBFt7LNyGetYoapB2GV/e7kecDenDhEUL/P+LeT6jKjKmM01xVtPd+evzI0kF8Qr8R01eSl2xHtUxY1/mK9n5dwzqCt9F6wHXEOHd+WiawnwGF7KfVplX6nXuUsR+wZXvfKnEkUaphKyh8FKuypvMk9pT7D2NK0B3dMJYCV0g27yPsp8ItYRGYWn9B9ilMBUsQCrqYIT0Rj8NNvRNBjNkxbAupXp3kGNJBDlqbNykdeicqr87jr4SZv/v2oWdy9p8/9dTWmF6KhB9zk/YlwpbmO0RrY5pvA0p6OqshVWaa92Jbdn9YwXc349N7/Rkt/Izd9qyd/Kzb/Wtg8DI2G0O8/i70/VsdhRhWasmFXyYI0q7BrfHH9/H3taJmYbyG1ER/nEjpG6P7X227LTWJwY18S4NjtZmprMbtI+OkfnoaPRJZqi6zRDC3iDT0/oOb0ovCt8KnwufGmm9vZknDPU8hS+/gbd0E4c</latexit>

Target encoder

Target AR decoder

Q
Source encoder

Speech analysis

Source speech<latexit sha1_base64="Kl460FRpZTxx0rrk3R9DKalgZtA=">AAAGKHicnVS/b9NAFH4NGEr50RYWJJaItKhliC5VJRBTJYqExK+mJWmlJopi5xLcOHZkOyGJSUcGJjYETCAxAH8GCyMLQlnZEGORWBj47sUFkqhxhU/Jvfe9932+e+98et0yPV+I3kTsyFHt2PHJE1MnT50+Mz0zezbrOQ3XkBnDsRx3Sy960jJtmfFN35JbdVcWa7olN/XqdRXfbErXMx37vt+uy3ytWLHNsmkUfUDZXLMWpLuFmYRICn7io0YqNBIUPmvObOwx5ahEDhnUoBpJssmHbVGRPIxtSpGgOrA8BcBcWCbHJXVpCtwGsiQyikCr+K/A2w5RG77S9Jht4C0Wfi6YcZoXn8VbsSc+ivfim/h1oFbAGmotbcx6nyvrhekn5zd+RrJqmH168JcVyShjhaoG3pj97ec6wO2IOvhQvMr7N5FbZ0RVxuivqdl5trdxbX0+uCRei++oySvREx9QFbv5w3iTlusvoa70bbAecg1rvCsbXQuAN+F1uU959pV6i7vks69z1Ut/KhGnOUogew6s7ljlKvOU9hJrr1IB6L6OCyugW3Sb95HjE1FGZAGe0n+EkQBTxVyspgKOT4vwu+GOVsHonzQX1p1Q9x5qJIEoT52Vy7wWlVPhd7fAD4b8/1UzuXvBkH94NaXloaM67XC+z7hS3MUYjOxyTOHdiI6qypZYZbjapcietUJek/Nbkfntgfx2ZH5nIL8TmX9jaB86RsDoeJ7J35+qY3ykCv1YPKzkwRoV2A2+Of79Pva1DMwWkLuILvCJXSR1f6aGb8tRI7uUTIlkKr2cWFkOb9JJukAXoZOiK7RCN2mNMnjDDj2l5/RCe6d90r5ovX5qbCLknKOBR/v6G+OjS0c=</latexit>

X

K

G

<latexit sha1_base64="oclLUNSua7PxrAXh7MBTYiebDxE=">AAAGKHicnVQ9bxNBEJ0YDCF8JIEGicbCCUoorL0oEogqEkFCCh/5wE6k2LJ857W5+Hxn3Z2N7cMpKajoEFCBRAH8DBpKGoTc0iHKINFQ8HZ8BmwrPsSt7J15M+/d7sze6jXL9HwhuhOxI0fjx45Pnpg6eer0memZ2bMZz6m7hkwbjuW4O3rBk5Zpy7Rv+pbcqbmyUNUtua1Xrqv4dkO6nunY9/xWTeaqhbJtlkyj4APKZBvVYK2Tn0mKlOAnMWpooZGk8Fl3ZmOPKEtFcsigOlVJkk0+bIsK5GHskkaCasByFABzYZkcl9ShKXDryJLIKACt4L8MbzdEbfhK02O2gbdY+LlgJmhefBJvxIH4IN6Jr+LnoVoBa6i1tDDrPa6s5acfn9/6EcmqYvbp/h9WJKOEFaoaeGP21891gNsRdfCheJX3byK3xoiqjNFbU6P99GDr2uZ8cEm8Et9Qk5eiK96jKnbju/F6Q26+gLrSt8F6wDWs8q5sdC0A3oDX4T7l2FfqTe6Sz77OVS/+rkSC5iiJ7DmwOmOVK8xT2kusvUp5oH0dF1ZAa3SL95HlE1FCZAGe0n+IkQRTxVyspgyOT4vwO+GOVsHonTQX1u1Q9y5qJIEoT52Vy7wWlVPmdzfBD4b8/1UzuXvBkP/vakrLQ0d12uN8n3GluI8xGNnnmMI7ER1VlS2yynC1i5E9a4a8Buc3I/NbA/mtyPz2QH47Mv/G0D50jIDR8TyTvz9Vx8RIFXqxRFjJwzXKsOt8c/z9ffS1DMwWkDuILvCJXSR1f2rDt+WokVlKaSKlbSwnV5bDm3SSLtBF6Gh0hVboJq1TGm/Yoyf0jJ7H38Y/xj/Hu73U2ETIOUcDT/zLL8EjS0E=</latexit><latexit<latexit sha1_base64="rjFJnS4m8oD7auPGJxparcwGZiE=">AAAGKHicnVQ9bxNBEJ0YDCF8JIEGicbCCUoorL0oEogqEkEg8ZUP7ESKLct3XpuLz3fW3dnYPpySgooOARVIFMDPoKGkQcgtHaIMEg0Fb8dnwLbiQ9zK3pk3897tzuytXrNMzxeiOxE7dDh+5OjksanjJ06emp6ZPZ3xnLpryLThWI67rRc8aZm2TPumb8ntmisLVd2SW3rlqopvNaTrmY59z2/VZK5aKNtmyTQKPqBMtlENrnfyM0mREvwkRg0tNJIUPmvObOwRZalIDhlUpypJssmHbVGBPIwd0khQDViOAmAuLJPjkjo0BW4dWRIZBaAV/Jfh7YSoDV9pesw28BYLPxfMBM2LT+KN2BcfxDvxVfw8UCtgDbWWFma9x5W1/PTjs5s/IllVzD7d/8OKZJSwQlUDb8z++rkOcDuiDj4UL/P+TeTWGFGVMXprarSf7m9e2ZgPLohX4htq8lJ0xXtUxW58N16vy40XUFf6NlgPuIZV3pWNrgXAG/A63Kcc+0q9yV3y2de56sXflUjQHCWRPQdWZ6xyhXlKe4m1VykPtK/jwgroJt3ifWT5RJQQWYCn9B9iJMFUMRerKYPj0yL8TrijVTB6J82FdTvUvYsaSSDKU2flIq9F5ZT53U3wgyH/f9VM7l4w5P+7mtLy0FGddjnfZ1wp7mEMRvY4pvBOREdVZYusMlztYmTPmiGvwfnNyPzWQH4rMr89kN+OzL82tA8dI2B0PM/k70/VMTFShV4sEVbyYI0y7DrfHH9/H30tA7MF5A6iC3xiF0ndn9rwbTlqZJZSmkhp68vJleXwJp2kc3QeOhpdohW6QWuUxht26Qk9o+fxt/GP8c/xbi81NhFyztDAE//yC6ojSz0=</latexit><latexit

A<latexit sha1_base64="SEctMWwADywCOP/MdanJ0G2Mmhk=">AAAGKHicnVQ9bxNBEJ0YDkL4SAINEo2FE5RQWOsoEogqEkFC4isf2LEUW5bvvDYXn++su7OxfTglBRUdAiqQKICfQUNJg5BbOkQZJBoK3o4vgG3FF3Ere2fezHu3O7O3et0yPV+I3kTsyFHt2PHJE1MnT50+Mz0zezbjOQ3XkGnDsRw3qxc9aZm2TPumb8ls3ZXFmm7JLb16XcW3mtL1TMe+77frMl8rVmyzbBpFH1Am16wF2W5hJiGSgp/4qJEKjQSFz5ozG3tMOSqRQwY1qEaSbPJhW1QkD2ObUiSoDixPATAXlslxSV2aAreBLImMItAq/ivwtkPUhq80PWYbeIuFnwtmnObFZ/FW7ImP4r34Jn4dqBWwhlpLG7Pe58p6YfrJ+c2fkawaZp8e/GVFMspYoaqBN2Z/+7kOcDuiDj4Ur/L+TeTWGVGVMfpranae7W1e25gPLonX4jtq8kr0xAdUxW7+MN6sy42XUFf6NlgPuYY13pWNrgXAm/C63Kc8+0q9xV3y2de56qU/lYjTHCWQPQdWd6xylXlKe4m1V6kAdF/HhRXQLbrN+8jxiSgjsgBP6T/CSICpYi5WUwHHp0X43XBHq2D0T5oL606oew81kkCUp87KZV6Lyqnwu1vgB0P+/6qZ3L1gyD+8mtLy0FGddjjfZ1wp7mIMRnY5pvBuREdVZUusMlztUmTPWiGvyfmtyPz2QH47Mr8zkN+JzL8xtA8dI2B0PM/k70/VMT5ShX4sHlbyYI0K7AbfHP9+H/taBmYLyF1EF/jELpK6P1PDt+WokVlKpkQytb6cWFkOb9JJukAXoZOiK7RCN2mN0njDDj2l5/RCe6d90r5ovX5qbCLknKOBR/v6GwvyS04=</latexit><latexit<latexit sha1_base64="Jklse+oe9wj2twEoxlKZM3nGVOY=">AAAGKHicnVQ9bxNBEJ0YDCF8JIEGicbCCUoorL0oEogqiCAh8ZUP7ESKLct3XpuLz3fW3dnYPpySgooOARVIFMDPoKGkQcgtHaIMEg0Fb8dnwLbiQ9zK3pk3897tzuytXrNMzxeiOxE7dDh+5OjksanjJ06emp6ZPZ3xnLpryLThWI67rRc8aZm2TPumb8ntmisLVd2SW3rlmopvNaTrmY59z2/VZK5aKNtmyTQKPqBMtlENrnbyM0mREvwkRg0tNJIUPmvObOwRZalIDhlUpypJssmHbVGBPIwd0khQDViOAmAuLJPjkjo0BW4dWRIZBaAV/Jfh7YSoDV9pesw28BYLPxfMBM2LT+KN2BcfxDvxVfw8UCtgDbWWFma9x5W1/PTjs5s/IllVzD7d/8OKZJSwQlUDb8z++rkOcDuiDj4UL/P+TeTWGFGVMXprarSf7m9e2ZgPLohX4htq8lJ0xXtUxW58N16vy40XUFf6NlgPuIZV3pWNrgXAG/A63Kcc+0q9yV3y2de56sXflUjQHCWRPQdWZ6xyhXlKe4m1VykPtK/jwgroJt3ifWT5RJQQWYCn9B9iJMFUMRerKYPj0yL8TrijVTB6J82FdTvUvYsaSSDKU2flIq9F5ZT53U3wgyH/f9VM7l4w5P+7mtLy0FGddjnfZ1wp7mEMRvY4pvBOREdVZYusMlztYmTPmiGvwfnNyPzWQH4rMr89kN+OzL8+tA8dI2B0PM/k70/VMTFShV4sEVbyYI0y7DrfHH9/H30tA7MF5A6iC3xiF0ndn9rwbTlqZJZSmkhp68vJleXwJp2kc3QeOhpdohW6QWuUxht26Qk9o+fxt/GP8c/xbi81NhFyztDAE//yC4ejSzc=</latexit><latexit

Context
preservation
loss

R

Guidedattention
loss

where σg controls how close A is to a diagonal matrix. The
guided attention loss Lga is defined as<latexit sha1_base64="JLlsjlIkXzt3LS8rs2Kdi5hUpW0=">AAAGKHicnVS/b9NAFH4NGEr50RYWJJaItKhliC5VJRBTJYqExK82JWmlJopi5xLcOHZkOyGJSUcGJjYETCAxAH8GCyMLQlnZEGORWBj47sUFkqhxhU/Jvfe9932+e+98et0yPV+I3kTsyFHt2PHJE1MnT50+Mz0zezbrOQ3XkBnDsRx3Sy960jJtmfFN35JbdVcWa7olN/XqdRXfbErXMx37vt+uy3ytWLHNsmkUfUDZXLMWpLuFmYRICn7io0YqNBIUPmvObOwx5ahEDhnUoBpJssmHbVGRPIxtSpGgOrA8BcBcWCbHJXVpCtwGsiQyikCr+K/A2w5RG77S9Jht4C0Wfi6YcZoXn8VbsSc+ivfim/h1oFbAGmotbcx6nyvrhekn5zd+RrJqmH168JcVyShjhaoG3pj97ec6wO2IOvhQvMr7N5FbZ0RVxuivqdl5trdxLT0fXBKvxXfU5JXoiQ+oit38YbxZl+mXUFf6NlgPuYY13pWNrgXAm/C63Kc8+0q9xV3y2de56qU/lYjTHCWQPQdWd6xylXlKe4m1V6kAdF/HhRXQLbrN+8jxiSgjsgBP6T/CSICpYi5WUwHHp0X43XBHq2D0T5oL606oew81kkCUp87KZV6Lyqnwu1vgB0P+/6qZ3L1gyD+8mtLy0FGddjjfZ1wp7mIMRnY5pvBuREdVZUusMlztUmTPWiGvyfmtyPz2QH47Mr8zkN+JzL8xtA8dI2B0PM/k70/VMT5ShX4sHlbyYI0K7AbfHP9+H/taBmYLyF1EF/jELpK6P1PDt+WokV1KpkQytb6cWFkOb9JJukAXoZOiK7RCN2mNMnjDDj2l5/RCe6d90r5ovX5qbCLknKOBR/v6G+ljS0g=</latexit><latexit

Source decoder

Target decoderX̃

Lga = ||G ⊙ A||1 ,where ⊙ indicates an element-wise product.

Ỹ<latexit sha1_base64="n2PYCwrCvS5vrK5R1T1eNvzpqCM=">AAAGMHicnVRNb9NAEJ0GAqV8tIULEpeItKjlEG2qSiBOlSgSEl/9IG2lJkSxswkmjm3ZTkhi3CMH/gBInFqJA+LGX+DCH0CiJ8QROBaJCwfeThwgiRojvEp25s28592Z9WqOaXi+EPtjiSNHk8eOj5+YOHnq9JnJqemzG57dcHWZ023Tdre0kidNw5I53/BNueW4slTXTLmp1a6r+GZTup5hW/f9tiML9VLVMiqGXvIBPcj7hlmWQb5ZD7bCsDiVFhnBT2rYyEZGmqJnxZ5OPKU8lckmnRpUJ0kW+bBNKpGHsU1ZEuQAK1AAzIVlcFxSSBPgNpAlkVECWsN/Fd52hFrwlabHbB1vMfFzwUzRrPggXosD8V68EV/Ez0O1AtZQa2lj1rpc6RQnn51f/xHLqmP26eEfViyjghWqGngj9tfLtYFbMXXwoXiV928g12FEVUbvrqnZeX6wfm1tNrgk9sQ31GRX7It3qIrV/K6/WpVrL6Gu9C2wHnMN67wrC10LgDfhhdynAvtKvcVd8tnXuOrl35VI0QylkT0DVjhSucY8pb3A2stUBNrTcWEFdItu8z7yfCIqiMzBU/pPMNJgqpiL1VTB8WkefhjtaBmM7klzYd2JdO+hRhKI8tRZucxrUTlVfncL/GDA/181g7sXDPj/rqa0PHRUo0ec7zOuFHcw+iM7HFN4GNNRVdkyqwxWuxzbs1bEa3J+Kza/3Zffjs3v9OV3YvNvDOxDwwgYHc0z+PtTdUwNVaEbS0WVPFyjCrvBN8ff30dPS8dsArmL6Byf2HlS92d28LYcNjYWMlmRya4uppcWo5t0nC7QRehk6Qot0U1aoRze4NIL2qW95Nvkx+Sn5OduamIs4pyjvif59Rf9J08E</latexit><latexit sha1_base64="J3GRwYGWTu+Az/jzEo0bzUg7Mik=">AAAGMHicnVRNb9NAEJ0WDKV8tIULEpeItKjlEG2qSiBOlSgSEl/9IG1RE6LY2QQTx45sJyQx7pEDfwAkTq3EAXHjL3DhDyDRE+IIHIvEhQNvJw40iRojvEp25s28592Z9eo1y/R8IfZGRo8c1Y4dHzsxfvLU6TMTk1Nn1z2n7hoyYziW427qBU9api0zvulbcrPmykJVt+SGXrmu4hsN6XqmY9/3WzWZqxbKtlkyjYIP6GHWN62iDLKNavAgDPOTSZES/CQGjXRkJCl6lp2p0WeUpSI5ZFCdqiTJJh+2RQXyMLYoTYJqwHIUAHNhmRyXFNI4uHVkSWQUgFbwX4a3FaE2fKXpMdvAWyz8XDATNCM+ijdiX3wQb8VX8etQrYA11FpamPUOV9byE8/Pr/2MZVUx+/ToLyuWUcIKVQ28Ifvr5jrA7Zg6+FC8yvs3kVtjRFXG6Kyp0X6xv3ZtdSa4JHbFd9RkR+yJ96iK3fhhvF6Rq6+grvRtsJ5wDau8KxtdC4A34IXcpxz7Sr3JXfLZ17nqxT+VSNA0JZE9DVY4VLnCPKU9z9pLlAfa1XFhBXSLbvM+snwiSojMwlP6TzGSYKqYi9WUwfFpDn4Y7WgJjM5Jc2HdiXTvoUYSiPLUWbnMa1E5ZX53E/ygz/9fNZO7F/T5/66mtDx0VKfHnO8zrhS3MXoj2xxTeBjTUVXZIqv0V7sY27NmxGtwfjM2v9WT34rNb/fkt2Pzb/TtQ8cIGB3OM/n7U3VMDFShE0tElTxcowy7zjfHwe+jq2VgtoDcRXSWT+wcqfsz3X9bDhrr86m0SKVXFpKLC9FNOkYX6CJ00nSFFukmLVMGb3DpJe3QrvZO+6R91r50UkdHIs456nm0b78BAvdPBQ==</latexit>

Fig. 2. System overviews of proposed VC. The solid lines
indicate the training and conversion procedures. The dashedlines indicate calculations of the differences during training.
Green boxes and red boxes respectively denote the originalcomponents and the proposed additional components.

time frame feature q j and the embeddings K. Consequently,
the attention matrix A can be written as
ei,j = fFFNN (ki , q j ),(3)

exp(ei,j )
ai,j = ∑.
i exp(ei,j )(4)where fFFNN indicates a function described by feed-forward
NNs and ai,j is an element (i, j) of the attention matrix A.
A seed R = [r 1 , · · · , r J ] of the target AR decoder is obtained by considering the long-range temporal dependencies
between the source and target sequences as follows:r j = Kaj .

(6)

Finally, we minimize the objective function LSeq2Seq of
Seq2Seq learning:
LSeq2Seq = ||Ŷ − Y ||1 .

3.2.2. Context Preservation Loss
To stabilize the training procedure, we propose a context
preservation loss. In preliminary experiments, we found thatthe training procedure sometimes failed even if it took into
account the guided attention loss (see speech samples on our
web page 1). In particular, the converted speech seemed
like randomly generated speech or speech repeating several
phonemes. One possible reason is that minimizing the objective function LSeq2Seq sometimes makes the target AR
decoder a network just reconstructing the input of the target
encoder. It is because we use Y rather than Ŷ as the inputof the target encoder in the training. As a result, the source
encoder is not required to control the output of the target AR
decoder and preserve the context information of the source
speech.
To make the source encoder meaningful, we introduce two
additional networks to the original Seq2Seq model as a context preservation mechanism. One is a source decoder fSrcDec
for reconstructing the source speech X̃ from the embeddings
K. The other is a target decoder fTarDec for predicting thetarget speech Ỹ from the seed R.

(5)

As the name implies, the target AR decoder involves all previous outputs of itself. Hence, the input of the target AR decoder is R′ combined with the seed R and the embeddings
Q. The output Ŷ of the Seq2Seq model is obtained throughthe target AR decoder fTarDecAR ,Ŷ = fTarDecAR (R′ ).

(9)X̃ = fSrcDec (K),(10)

Ỹ = fTarDec (R).

(11)

From another point of view, the source decoder fSrcDec helps
the source encoder to preserve the linguistic information of
the source speech X, while the target decoder fTarDec helps
the source encoder to encode the source speech X to theshared space of the source and target speech. Note that in
the preliminary experiments, the target decoder was more important than the source decoder. The full objective functionof our model is formulated as

(7)

6807

Lproposed = LSeq2Seq + λga Lga+ λcp (||X̃ − X||1 + ||Ỹ − Y ||1 ),

(12)

100

4.2. Experimental Results
As shown in Fig. 3, we conducted two subjective evaluations,preference tests on naturalness and speaker similarity. The
number of listeners was 15, and each listener evaluated 80
shots consisting of randomly selected 10 speech samples × 4

5075

Proposed

a
rm ry
s-b
d
rm l
s-s
lt
clb
-bd
l
clb
-sl
t

mm
su

su

mm

0
100

a
rm ry
s-b
d
rm l
s-s
lt
clb
-bd
l
clb
-sl
t

25

Fair

GMM-VC-wGV

LSTM-TTS

50
25

Fair

d
rm l
s-s
lt
clb
-bd
l
clb
-sl
t

rm

s-b

ary
su

rm

Proposed

mm

ary

s-b
d
rm l
s-s
lt
clb
-bd
l
clb
-sl
t

0

mm

Datasets: We used the CMU Arctic database [42] consisting of utterances by two male speakers (rms and bdl) andtwo female speakers (clb and slt). To train the models, weused about 1,000 sentences (speech section of 50 min) of
each speaker. To evaluate the performance, we used 132 sentences of each speaker. The sampling rate of the speech signals was 16 kHz. We treated rms and clb as source speakers
and bdl and slt as target speakers. For the evaluations, we
conducted intra-gender pairs, rms-bdl and clb-slt, and crossgender pairs, rms-slt and clb-bdl. Note that we trained theconversion models for every speaker pair, independently.
Baseline system 1 (GMM-VC-wGV): We used a GMMbased VC method [10] as the baseline for frame/sequencebased VC described in Sec. 2.1. To train the conversion models, we used an open source VC toolkit sprocket [43] and its
default settings, except for F0 ranges and power thresholds.
Note that a global variance (GV) [10] was also considered.
Baseline system 2 (LSTM-TTS): By assuming the ASR
module and the encoder part of the encoder-decoder module in [32] work perfectly, we can focus on the TTS module. Therefore, we used an LSTM-based TTS method as thebaseline of Seq2Seq-based VC described in Sec. 2.2. The
contextual features used as input were 416-dimensional linguistic features obtained using the default question set of theopen source TTS toolkit Merlin [44]. From the speech data,
60 Mel-cepstral coefficients, logarithmic F0 , and coded aperiodicities were extracted every 5 ms with the WORLD analysis
system [45]. As the duration model, we stacked three LSTMs
with 256 cells followed by a linear projection. As the acoustic
model, we stacked three bidirectional LSTMs with 256 cells
followed by a linear projection.
Proposed system (Proposed): Inspired by Tacotron [23],we used the architecture described in open Tacotron [46].
Note that we replaced all ReLU activations [47] with a gating
mechanism of gated linear units [48]. Although the proposed
method worked well for not only acoustic features of theWORLD vocoder but also raw spectral features, we chose to
use the acoustic features of WORLD vocoder to balance theexperimental conditions of LSTM-TTS. Note that the target
AR decoder also generated the stop tokens. As the additionalsource decoder and target decoder networks, we used thesame architectures as in the source encoder. The hyperparameters σg , λga , λcp were 0.4, 10,000, and 10, respectively.
The batch size, number of epochs, and reduction factor [49]were 32, 1,000 and 5. We used the Adam optimizer [50] andvaried the learning rate over the course of training [51].

75su4.1. Experimental Conditions

Preference score [%]4. EXPERIMENTS

Preference score [%]where λcp controls the context preservation loss.

GMM-VC-wGV

LSTM-TTS

Fig. 3. Results of preference tests of naturalness (upper) andspeaker similarity (lower).
pairs of intra/cross- gender × 2 comparisons, v.s. GMM-VCwGV and v.s. LSTM-TTS.
The evaluations indicated that Proposed outperformed
GMM-VC-wGV in terms of both naturalness and speakersimilarity. This is because our method makes it possible toconvert not only the acoustic features but also the durations ofspeech. In contrast, baseline system 1 forces the conversion
while preserving the durations of the source speech. Consequently, durations not used in the target speech make theconversion errors larger.
Moreover, Proposed was comparable to LSTM-TTS.
This result demonstrates that our method makes it possible
to learn the key components for changing the individualityof the speaker while preserving the linguistic information.
Notably, our model was trained without any transcript while[32, 33] used a large number of transcripts.
5. CONCLUSIONSWe proposed a method based on Seq2Seq learning with attention and context preservation mechanisms for VC tasks.
Experimental results demonstrated that the proposed method
outperformed the conventional GMM-based VC and wascomparable to LSTM-based TTS. Extending the proposed
method so that it can be used in semi-supervised learning
tasks is ongoing work. Note that since we also progressed ina convolutional version of the proposed method [52] simultaneously, we will conduct further evaluations and report theresults.
Acknowledgements: This work was supported by a grantfrom the Japan Society for the Promotion of Science (JSPS
KAKENHI 17H01763).

6808

6. REFERENCES[1] Takuhiro Kaneko and Hirokazu Kameoka, “Parallel-data-free voice conversion
using cycle-consistent adversarial networks,” arXiv preprint arXiv:1711.11293,
2017. 1, 2
[2] Yang Gao, Rita Singh, and Bhiksha Raj, “Voice impersonation using generative
adversarial networks,” arXiv preprint arXiv:1802.06840, 2018. 1[3] Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo,“StarGAN-VC: Non-parallel many-to-many voice conversion with star generative
adversarial networks,” arXiv preprint arXiv:1806.02169, 2018. 1, 2
[4] Alexander B Kain, John-Paul Hosom, Xiaochuan Niu, Jan PH van Santen,
Melanie Fried-Oken, and Janice Staehely, “Improving the intelligibility ofdysarthric speech,” Speech communication, vol. 49, no. 9, pp. 743–759, 2007.
1[5] Keigo Nakamura, Tomoki Toda, Hiroshi Saruwatari, and Kiyohiro Shikano,“Speaking-aid systems using GMM-based voice conversion for electrolaryngeal
speech,” Speech Communication, vol. 54, no. 1, pp. 134–146, 2012. 1[6] Zeynep Inanoglu and Steve Young, “Data-driven emotion conversion in spoken
english,” Speech Communication, vol. 51, no. 3, pp. 268–283, 2009. 1[7] Tomoki Toda, Mikihiro Nakagiri, and Kiyohiro Shikano, “Statistical voice conversion techniques for body-conducted unvoiced speech enhancement,” IEEE Transactions on Audio, Speech and Language Processing, vol. 20, no. 9, pp. 2505–2517,
2012. 1[8] Daniel Felps, Heather Bortfeld, and Ricardo Gutierrez-Osuna, “Foreign accent
conversion in computer assisted pronunciation training,” Speech communication,
vol. 51, no. 10, pp. 920–932, 2009. 1[9] Takuhiro Kaneko, Hirokazu Kameoka, Kaoru Hiramatsu, and Kunio Kashino,“Sequence-to-sequence voice conversion with similarity metric learned using generative adversarial networks,” Proc. Interspeech 2017, pp. 1283–1287, 2017. 1[10] Tomoki Toda, Alan W Black, and Keiichi Tokuda, “Voice conversion based on
maximum-likelihood estimation of spectral parameter trajectory,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 15, no. 8, pp. 2222–2235,
2007. 1, 2, 4
[11] Ryoichi Takashima, Tetsuya Takiguchi, and Yasuo Ariki, “Exemplar-based voice
conversion using sparse representation in noisy environments,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences,vol. 96, no. 10, pp. 1946–1953, 2013. 1[12] Zhizheng Wu, Tuomas Virtanen, Eng Siong Chng, and Haizhou Li, “Exemplarbased sparse representation with residual compensation for voice conversion,”
IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 22,
no. 10, pp. 1506–1521, 2014. 1[13] Ling-Hui Chen, Zhen-Hua Ling, Li-Juan Liu, and Li-Rong Dai, “Voice conversion using deep neural networks with layer-wise generative training,” IEEE/ACM
Transactions on Audio, Speech and Language Processing (TASLP), vol. 22, no.
12, pp. 1859–1872, 2014. 1[14] Toru Nakashika, Tetsuya Takiguchi, and Yasuo Ariki, “Voice conversion based on
speaker-dependent restricted boltzmann machines,” IEICE TRANSACTIONS on
Information and Systems, vol. 97, no. 6, pp. 1403–1410, 2014. 1[15] Srinivas Desai, Alan W Black, B Yegnanarayana, and Kishore Prahallad, “Spectral mapping using artificial neural networks for voice conversion,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 5, pp. 954–964,
2010. 1[16] Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari, “Voice conversion
using input-to-output highway networks,” IEICE Transactions on Informationand Systems, vol. 100, no. 8, pp. 1925–1928, 2017. 1[17] Lifa Sun, Shiyin Kang, Kun Li, and Helen Meng, “Voice conversion using deep
bidirectional long short-term memory based recurrent neural networks,” in Acoustics, Speech and Signal Processing (ICASSP), 2015 IEEE International Conference on. IEEE, 2015, pp. 4869–4873. 1[18] Toru Nakashika, Tetsuya Takiguchi, and Yasuo Ariki, “High-order sequence modeling using speaker-dependent recurrent temporal restricted boltzmann machinesfor voice conversion,” in Fifteenth Annual Conference of the International SpeechCommunication Association, 2014. 1[19] Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, and Hsin-Min Wang,“Voice conversion from non-parallel corpora using variational auto-encoder,” in
Signal and Information Processing Association Annual Summit and Conference(APSIPA), 2016 Asia-Pacific. IEEE, 2016, pp. 1–6. 1[20] Yuki Saito, Yusuke Ijima, Kyosuke Nishida, and Shinnosuke Takamichi, “Nonparallel voice conversion using variational autoencoders conditioned by phonetic
posteriorgrams and d-vectors,” in Proc. ICASSP, 2018, pp. 5274–5278. 1[21] Ilya Sutskever, Oriol Vinyals, and Quoc V Le, “Sequence to sequence learningwith neural networks,” in Advances in neural information processing systems,
2014, pp. 3104–3112. 1[22] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473,
2014. 1, 2
[23] Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss,
Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio,
et al., “Tacotron: Towards end-to-end speech synthesis,” arXiv preprint
arXiv:1703.10135, 2017. 1, 4
[24] Jonathan Shen, Ruoming Pang, Ron J Weiss, Mike Schuster, Navdeep Jaitly,
Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, Rj Skerrv-Ryan, et al.,“Natural TTS synthesis by conditioning wavenet on mel spectrogram predictions,”in ICASSP. IEEE, 2018, pp. 4779–4783. 1[25] Hideyuki Tachibana, Katsuya Uenoyama, and Shunsuke Aihara, “Efficiently
trainable text-to-speech system based on deep convolutional networks with guided
attention,” in 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2018, pp. 4784–4788. 1, 3
[26] William Chan, Navdeep Jaitly, Quoc Le, and Oriol Vinyals, “Listen, attend andspell: A neural network for large vocabulary conversational speech recognition,”in ICASSP. IEEE, 2016, pp. 4960–4964. 1[27] Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick
Nguyen, Zhifeng Chen, Anjuli Kannan, Ron J Weiss, Kanishka Rao, Ekaterina
Gonina, et al., “State-of-the-art speech recognition with sequence-to-sequencemodels,” in ICASSP. IEEE, 2018, pp. 4774–4778. 1[28] Colin Raffel, Minh-Thang Luong, Peter J Liu, Ron J Weiss, and Douglas Eck,
“Online and linear-time attention by enforcing monotonic alignments,” arXiv
preprint arXiv:1704.00784, 2017. 1, 2
[29] Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura, “Listening while speaking: Speech chain by deep learning,” CoRR, vol. abs/1707.04879, 2017. 1[30] Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tieyan Liu, and Wei-Ying
Ma, “Dual learning for machine translation,” in Advances in Neural Information
Processing Systems, 2016, pp. 820–828. 1[31] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros, “Unpaired imageto-image translation using cycle-consistent adversarial networks,” arXiv preprint
arXiv:1703.10593, 2017. 1[32] Hiroyuki Miyoshi, Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari,“Voice conversion using sequence-to-sequence learning of context posterior probabilities,” arXiv preprint arXiv:1704.02360, 2017. 1, 2, 4[33] Jing-Xuan Zhang, Zhen-Hua Ling, Li-Rong Dai, Li-Juan Liu, and Yuan Jiang,“Sequence-to-sequence acoustic modeling for voice conversion,” arXiv preprint
arXiv:1810.06865, 2018. 1, 2, 4[34] Yannis Stylianou, Olivier Cappé, and Eric Moulines, “Continuous probabilistictransform for voice conversion,” IEEE Transactions on speech and audio processing, vol. 6, no. 2, pp. 131–142, 1998. 2[35] Elina Helander, Tuomas Virtanen, Jani Nurminen, and Moncef Gabbouj, “Voice
conversion using partial least squares regression,” IEEE Transactions on Audio,
Speech, and Language Processing, vol. 18, no. 5, pp. 912–921, 2010. 2[36] Sun Lifa, Li Kun, Wang Hao, Kang Shiyin, and Meng Helen, “Phonetic posteriorgrams for many-to-one voice conversion without parallel data training,” in ICME,
2016. 2[37] Li-Juan Liu, Zhen-Hua Ling, Yuan Jiang, Ming Zhou, and Li-Rong Dai, “Wavenet
vocoder with limited training data for voice conversion,” Proc. Interspeech 2018,
pp. 1983–1987, 2018. 2[38] Athanasios Mouchtaris, Jan Van der Spiegel, and Paul Mueller, “Nonparallel training for voice conversion based on a parameter adaptation approach,” 2006. 2[39] Chung-Han Lee and Chung-Hsien Wu, “Map-based adaptation for speech conversion using adaptation data selection and non-parallel training,” in Ninth International Conference on Spoken Language Processing, 2006. 2[40] Toda Tomoki, Ohtani Yamato, and Shikano Kiyohiro, “Eigenvoice conversion
based on gaussian mixture model,” in INTERSPEECH, 2006, pp. 2446–2449. 2[41] Daisuke Saito, Keisuke Yamamoto, Nobuaki Minematsu, and Keikichi Hirose,“One-to-many voice conversion based on tensor representation of speaker space,”in Twelfth Annual Conference of the International Speech Communication Association, 2011. 2[42] John Kominek and Alan W Black, “The CMU Arctic speech databases,” in Fifth
ISCA workshop on speech synthesis, 2004. 4[43] Kazuhiro Kobayashi and Tomoki Toda, “sprocket: Open-source voice conversion software,” in Proc. Odyssey 2018 The Speaker and Language Recognition
Workshop, 2018, pp. 203–210. 4[44] Zhizheng Wu, Oliver Watts, and Simon King, “Merlin: An open source neuralnetwork speech synthesis system,” Proc. SSW, Sunnyvale, USA, 2016. 4[45] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa, “WORLD: a vocoderbased high-quality speech synthesis system for real-time applications,” IEICE
TRANSACTIONS on Information and Systems, vol. 99, no. 7, pp. 1877–1884,
2016. 4[46] Ito Keith,“Tacotron speech synthesis implemented in tensorflow,”inhttps://github.com/keithito/tacotron. 4[47] Vinod Nair and Geoffrey E Hinton, “Rectified linear units improve restrictedboltzmann machines,” in Proceedings of the 27th international conference on
machine learning (ICML-10), 2010, pp. 807–814. 4[48] Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier, “Language
modeling with gated convolutional networks,” arXiv preprint arXiv:1612.08083,
2016. 4[49] Heiga Zen, Yannis Agiomyrgiannakis, Niels Egberts, Fergus Henderson, andPrzemysław Szczepaniak, “Fast, compact, and high quality LSTM-RNN based
statistical parametric speech synthesizers for mobile devices,” arXiv preprint
arXiv:1606.06061, 2016. 4[50] Diederik Kingma and Jimmy Ba, “Adam: A method for stochastic optimization,”
arXiv preprint arXiv:1412.6980, 2014. 4[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones,
Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin, “Attention is all you need,”
CoRR, vol. abs/1706.03762, 2017. 4[52] Hirokazu Kameoka, Kou Tanaka, Takuhiro Kaneko, and Nobukatsu Hojo,“Convs2s-vc: Fully convolutional sequence-to-sequence voice conversion,” arXiv
preprint arXiv:1811.01609, 2018. 4

6809