INTERSPEECH 2017
August 20–24, 2017, Stockholm, Sweden

Sequence-to-Sequence Voice Conversion with Similarity Metric
Learned Using Generative Adversarial Networks
Takuhiro Kaneko, Hirokazu Kameoka, Kaoru Hiramatsu, Kunio Kashino
NTT Communication Science Laboratories, NTT Corporation, Japan{kaneko.takuhiro,kameoka.hirokazu,hiramatsu.kaoru,kashino.kunio}@lab.ntt.co.jp

Abstract

exemplar-based framework based on non-negative matrix factorization (NMF) [7, 8] have also been proposed with notablesuccess.
A well-known problem regarding the conventional VC
frameworks is that acoustic-feature sequences generated froma converter tend to be over-smoothed, resulting in buzzysounding speech. This is caused by a side effect of assuming
a particular form of similarity metric (e.g., mean squared error(MSE)) or distribution (e.g., Gaussian) for parameter training of
the acoustic model so that the generated feature sequence that
averagely fits the training target example is considered optimal.
This over-smoothing occurs as long as a manually constructedsimilarity metric is used. Postfiltering methods based on the
global variance [9, 10] or modulation spectrum [11] have been
proposed with the aim of reconstructing the original spectrotemporal fine texture. However, these methods rely on heuristics since it is usually difficult to model the exact probability density of the features of real speech. Recently, a direct
waveform-modification method using the spectrum differential
feature has been proposed [12]. This method avoids generating
over-smoothed spectra by transplanting the spectro-temporalfine texture of the source speech into the generated speech.
Most conventional VC frameworks are designed to convert acoustic features frame-by-frame. When it comes to apronunciation-conversion task, we must consider the contextual
or temporal dependencies of the conversion rules since the way
speakers pronounce each phoneme may differ depending on thepreceding, current, and succeeding contexts. To model longterm dependencies, some attempts have been made to model the
mapping function using recurrent NNs (RNNs) [13] including
the long short-term memory networks [14]. However, RNNs are
not well suited to parallel implementations; thus, the conversion
process becomes computationally demanding.
Except for some parallel-data-free frameworks [15, 16, 17],
many VC frameworks require accurately aligned parallel data ofsource and target speech. When there is a large acoustic gap between source and target speech, collecting parallel speech data
can be a painstaking process since automatic time alignment
between the pair of acoustic-feature sequences becomes relatively difficult. Since many frameworks are weak against misalignment involved in parallel data, careful pre-screening andmanual correction may be required to make these frameworks
reliable. It is difficult to completely avoid misalignment errors
particularly in a pronunciation-conversion task where the acoustic characteristics of each phoneme pronounced by source andtarget speakers can be very dissimilar.
In this paper, we propose a VC framework that (1) avoids
over-smoothing of converted feature sequences, (2) provides areasonable way of modeling time-dependent conversion rules,and (3) is robust against misalignment errors involved in the parallel data. We use a similarity metric implicitly obtained witha generative adversarial network (GAN) [18] that has been suc-

We propose a training framework for sequence-to-sequence
voice conversion (SVC). A well-known problem regarding a
conventional VC framework is that acoustic-feature sequences
generated from a converter tend to be over-smoothed, resulting in buzzy-sounding speech. This is because a particularform of similarity metric or distribution for parameter training of the acoustic model is assumed so that the generated feature sequence that averagely fits the training target example isconsidered optimal. This over-smoothing occurs as long as amanually constructed similarity metric is used. To overcome
this limitation, our proposed SVC framework uses a similarity
metric implicitly derived from a generative adversarial network,
enabling the measurement of the distance in the high-level abstract space. This would enable the model to mitigate the oversmoothing problem caused in the low-level data space. Furthermore, we use convolutional neural networks to model the
long-range context-dependencies. This also enables the similarity metric to have a shift-invariant property; thus, making themodel robust against misalignment errors involved in the parallel data. We tested our framework on a non-native-to-native VC
task. The experimental results revealed that the use of the proposed framework had a certain effect in improving naturalness,clarity, and speaker individuality.
Index Terms: voice conversion, deep neural network, generative adversarial network, similarity metric learning

1. Introduction
In this paper, we address the problem of converting the voice
characteristics of speech including speaker identity and pronunciation quality. We are particularly interested in developing a real-time (or low-latency) pronunciation-conversion system, which converts non-native speech into native-like intelligible speech in an online manner. The problem of converting
non/para-linguistic information of speech while preserving linguistic information is called voice conversion (VC). Since pronunciation quality is part of a speaker’s identity in a wide sense,
the pronunciation-conversion problem can be seen as a specialclass of the VC problem.
One successful VC framework involves statistical methods
based on Gaussian mixture models (GMMs) [1, 2, 3]. This
framework typically consists of collecting parallel data, namely
a pair of time-aligned feature sequences of source and target
speech and training a mapping function between the acoustic
features of the source and target speech using the parallel data.
At test time, the feature sequence of input speech is converted
using the trained mapping function. This GMM-based framework uses a GMM to represent the conditional distribution of
the target feature given the source feature. Recently, a neural network (NN)-based framework based on restricted Boltzmann machines [4] and feed-forward deep NNs [5, 6] and an

Copyright © 2017 ISCA

1283

http://dx.doi.org/10.21437/Interspeech.2017-970

cessful in image generation [19, 20] and has begun to be appliedin speech synthesis [21, 22, 23]. A GAN offers a framework fortraining a generator in such a way that it can deceive a discriminator that attempts to distinguish real data from the samples
drawn from the generator. Thus, we expect that measuring thedifference in the units in a particular layer in the GAN discriminator between the generated and target speech will enable themeasurement of the similarity in high-level abstract space andmitigate the over-smoothing problem caused in the low-level
data space. To model time-dependent conversion rules, we use
a segmental feature (a sequence of acoustic features with a duration of hundreds of frames) as the input/output. For the generator, converter, and discriminator networks, we use a gatedconvolutional NN (CNN) [24], so that this network structure
not only allows for modeling long-term dependencies by deepening the layers but also offers robustness against misalignment
errors involved in the parallel data due to the nature of the CNN,
known as the shift invariance property.
We previously proposed a learning-based postfilter using a
GAN that adds a high-fidelty spectro-temporal texture to thefeature sequence obtained with parametric speech synthesis sothat the synthesized speech becomes as indistinguishable aspossible from real speech [21]. The proposed framework combines the goals of VC and postfiltering, namely converting input
speech as close as possible to target speech and making the output speech as indistinguishable as possible from real speech.
We tested our framework on a non-native-to-native VC task.
The experimental results showed that our proposed framework
can be used to improve naturalness, clarity, and speaker individuality.
This paper is organized as follows. In Section 2, we explain
the proposed sequence-to-sequence VC (SVC) framework witha GAN and gated CNN. We present the experimental results in
Section 3 and summarize our findings in Section 4.

Source Speech

FeatureExtraction

Acoustic
FeatureFrame(a) Conventional
Frame-by-Frame
Voice Conversion

Converter C

C

C

Frame

(b) Proposed
Sequence-to-Sequence
Voice Conversion

Target Speech

Synthesis

Acoustic
FeatureSource Speech

FeatureExtraction

Acoustic
FeatureSequence
C
Sequence
Target Speech

Synthesis

Acoustic
FeatureFigure 1: Comparison of voice-conversion frameworks
the following objective function:
LSVC = S(y, C(x)),(1)

where S(x1 , x2 ) is a similarity metric.
In conventional VC frameworks, the similarity between thesource and target acoustic features is measured in the data space
using a particular form of similarity metric, e.g., MSE:
LData
SVC =

1
||y − C(x)||2 ,M0(2)

where M0 is the dimensions of y. The weak point of theseframeworks is that they are sensitive to the slight translation in
the data space. For example, this metric imposes a large error
when a sharp peak is shifted by only one frame. This causes
over-smoothing, resulting in buzzy-sounding speech.
To solve this problem, we use a similarity metric learned
using a GAN [18]. In the following sub-sections, we first give a
brief overview of a GAN then explain our proposed frameworkwith the similarity metric learned using a GAN.

2. Sequence-to-sequence voice conversion

2.2.1. Background: GAN

2.1. Framework overview

A GAN [18] is a framework for training a deep generativemodel using a minimax game. The goal is to learn a generator distribution PG (y) that matches the true data distribution
PData (y). It consists of two networks: a generator G that transforms noise variables z ∼ PNoise (z) to data space y = G(z)and a discriminator D that assigns probability p = D(y) when
y is a sample from the PData (y) and assigns probability 1 − pwhen y is a sample from the PG (y). In a GAN, the following
binary cross entropy is maximized and minimized with respect
to the D and G, respectively.

In a conventional VC framework, the goal is to convert voice
characteristics, and a frame-by-frame approach, as shown in
Figure 1(a), has been widely used for this conversion. In contrast, our goal is not only to convert voice characteristics but
also convert pronunciation, e.g., to convert non-native speech tonative-like speech. To achieve this goal, we need to model the
temporal dependencies of a speech sequence. To achieve this,the conventional frame-by-frame framework is difficult to apply
since it is limited to modeling the mapping function (namely,
the converter (C)) in a short time (e.g., 5 ms) and is not able tomodel the temporal dependencies. Our proposed SVC framework was developed to overcome this limitation, as shown in
Figure 1(b), which converts speech on a sequence-to-sequencebasis using a C that models long-range context dependencies.
To obtain a suitable C, we need to find a suitable answers
to the following two questions: “How is the C optimized?” and“How is the sequence structure modeled?”. We answer the firstquestion in Section 2.2 and the second question in Section 2.3.

LGAN = log D(y) + log(1 − D(G(z))).

(3)This enables the D to find the binary classifier that provides thebest possible discrimination between true and generated dataand simultaneously enables the G to fit PData (y). Both G and
D can be trained using back-propagation.
2.2.2. Similarity metric learning with GAN
In a GAN, its D’s hidden feature space is optimized to determine the key difference between true and generated data to discriminate them. This means that a more abstract structure isdetermined in the D’s hidden feature space than that in the data
space. We use this metric to measure similarity. This technique
is also used in computer vision [25], in which this similarity
metric enables a high-fidelity image to be generated. In the term

2.2. Optimization with learned similarity metric and GAN
Sequence-to-sequence VC is considered a matching problem.
Given a source acoustic-feature sequence x = (x1 , ..., xT ) andtarget acoustic-feature sequence y = (y1 , ..., yT ) as training
data, the C that maps x to y can be optimized by minimizing

1284

Random
NoiseK

Real Data

ReshapeUp-sample
Fully
Connected

2D CNN

G

Transpose

Acoustic
FeatureT
TransposeF

Down-sample
F

Figure 2: Flow for training C with similarity metric learned
with GAN. Red lines indicate terms in objective.

D

1||Dl (y) − Dl (C(x))||2 ,
Ml

D

F


T

T

(l-1)-th layer

Real Data F

Acoustic
FeatureT

2D CNNReal
or notp

D

Acoustic
FeatureTσl-th layer

Figure 4: Architectures of gated CNN(5)where σ is the sigmoid function and ⊗ is the element-wise product between matrices. We show the gated CNN architecture in
Figure 4. This gated mechanism enables the information to be
propagated depending on the previous layer states and is well
suited to represent region dependency.

3. Experiments
3.1. Experimental conditions(6)

In our VC experiments, we used an Indian male speaker as thesource (hereafter, the SRC) and an American male speaker as
the target (the TGT). The data consisted of 842 pair utterances(roughly 1 hour). We used 83 utterances for evaluation andthe others for training. The data were sampled at 16 kHz, then
25 mel-cepstral coefficients (MCEPs), logarithmic fundamental
frequency (log F0 ), and 5-band aperiodicities were extracted every 5 ms by using the STRAIGHT analysis system [29]. To obtain parallel utterances, we used dynamic time warping (DTW)
to align MCEP sequences of the source and target speakers [1].
The MCEPs were converted using the C described in Section 2.
Log F0 was converted using a logarithm Gaussian-normalized
transformation [30], i.e., linear conversion based on the meanand standard deviation of the source and target speech, which
is widely used in VC. Aperiodicities were directly used without modification, since a previous study showed that converting
aperiodicites does not provide a statistically significant difference on synthesized speech [31].
Table 1 details the network architectures of the G, C, and
D. The symbols ↓ and ↑ indicate down-sampling and upsampling, respectively. To upscale and downscale, we respectively used convolutions and backward convolutions with stride
2. The terms LReLU and BNorm denote leaky ReLU [32, 33]
and batch normalization [34], respectively. We set the dimensions of input and output as K = 100, F = 25, and T = 200(this means that a temporal dependency in 200 frames × 5 ms
= 1 sec. was captured). In training, we fixed the frame lengthof the converter input, but we designed it as a fully convolutional network (FCN) [35], so we could take inputs of arbitrary
length in synthesizing. During preprocessing, we normalized
the source and target MCEPs to zero-mean and unit-variancefor each dimension using their training sets, respectively. We
trained the G, C, and D using the Adam optimizer [36] witha minibatch of size 16. The learning rate was set to 0.0002 for
the C, 0.0001 for the G, and 0.0002 for the D, respectively; themomentum term β1 was set to 0.5.
To clarify the characteristics of our framework with alearned similarity metric (the LSM), we implemented two

We optimize the G by minimizing LGAN , optimize the C byDlminimizing γLSVC+LGAN , and optimize the D by minimizing
Dl−LGAN . Here, γ is the trade-off parameter between LSVC
andLGAN , which we set to 1 in the experiments.
2.3. Sequence modeling with gated CNN
To achieve SVC, we need to model the long-range contextdependencies. To do so with a reasonably small number ofparameters, we use a CNN. Figure 3 shows the network architectures of the G, C, and D. We consider converting the F × T
time-sequential acoustic features (e.g., mel-cepstrum), where F
is the feature dimension and T is the frame length. We carefully design the network architectures considering the objectiveof each network. To convert the source acoustic features to thetarget ones considering the temporal dependency in the overall
features, we design the C using a one-dimensional (1D) CNN[26, 27]. In this model, we regard the feature dimension as achannel (i.e., RGB in the case of an image) and convolve thefeature using a (1, kT ) kernel, where kT is the kernel width in
the time direction. In the G and D, we focus on the 2D texture(i.e., spectro-temporal structure) of acoustic features. Based on
this intuition, we treat them as image-like data [19, 20]. We
design the G to generate the 2D texture from K-dimensionalrandom noise using an up-sampling 2D CNN and design the D
to discriminate the 2D texture using a down-sampling 2D CNN.
Based on the observation that acoustic features have region
dependency (e.g., they have different structures in voiced segments and unvoiced segments), we design all the CNNs using agated CNN [24]. In a gated CNN, gated linear units (GLUs) areused as an activation function instead of regular rectified linear
units (ReLUs) [28] or Tanh activations. A GLU is a data-driven
activation function, and the l-th-layer output Hl is calculatedusing the (l − 1)-th-layer output Hl−1 and model parameters
W , V , b, and c.
Hl = (Hl−1 ∗ W + b) ⊗ σ(Hl−1 ∗ V + c),(4)

Figure 2 gives an overview of the training procedure. In practice, we obtain better results when using not only a sample fromthe G but also a sample from the C as the discriminator input.
This is because similar positive and negative samples are more
useful for the D to determine the decision boundary. In thissetting, the GAN objective function is rewritten as
LGAN = log D(y) + log(1 − D(G(z)))+ log(1 − D(C(x))).

F

1D CNN

Figure 3: Network architectures of G, C, and D

where Ml is the dimension of Dl (y).
Following a previous study [25], we train our model withthe joint objectives of the GAN objective with Equation 3 andSVC objective with Equation 4:l
L = LSVC+ LGAN .

FC

Dl (y), which denotes the l-th layer hidden feature in the D, a
new similarity metric is written as
l
LSVC
=

Acoustic
Feature
T

(7)

1285

(a) SRC

25(b) TGT (Aligned by DTW)

25(c) FVC

25(d) MSE (Proposed)

25

20

20

20

20

20

15

15

15

15

15

10

10

10

10

10

5

5

5

5

1

2

Time (s)

3

6

4

1

2

Time (s)

3

0

2

3

Time (s)12Time (s)

3

0

1

8

2

3

Time (s)203Time (s)

2

3

2

3

Time (s)

4

2

1

1

6

4

0

0
8

6

2

0

5
0

4

2

0

1

6

4

2

0
8

Frequency (kHz)

8

6

0

0

Frequency (kHz)Time (s)3Frequency (kHz)82Frequency (kHz)1

Frequency (kHz)0(e) LSM (Proposed)

25

2

0

1

2

3

Time (s)001Time (s)Figure 5: Comparison of MCEPs (upper row) and STFT spectrograms (lower row) of SRC, TGT, FVC, MSE, and LSM
Table 1: Details of network architectures of G, C, and D

Tables 2 and 3 list the average preference scores for naturalness and clarity, respectively. These results indicate that
the LSM outperformed the other two. In particular, the difference was marked regarding clarity, indicating that the LSM iseffective against the over-smoothing problem. Figures 6 shows
the similarity to the TGT and SRC with the three frameworks.
All frameworks were competitive regarding dissimilarity to the
SRC, but the LSM outperformed the MSE and FVC and exceeded the neutral rate regarding similarity to the TGT. Hence,
we can conclude that the proposed framework can improve thequality of VC.

G (Input: K Random noise, Output: F × T MCEP)
1024 fully connected, BNorm, ReLU
[F/4] · [T /4] · 128 fully connected, BNorm, ReLU
4 × 4 128 conv. ↑, BNorm, GLU
4 × 4 2 conv. ↑, GLU
C (Input: F × T MCEP, Output: F × T MCEP)(1 × 11 1024 conv., GLU ) × 5 stacks
1 × 11 2 · F conv., GLU
D (Input: F × T MCEP, Output: 1 Probability)
4 × 4 128 conv. ↓, GLU
4 × 4 256 conv. ↓, BNorm, GLU
1024 fully connected, BNorm, LReLU
1 fully connected, sigmoid

Table 2: Average preference scores (%) for naturalness with
95% confidence intervals. Bold font indicates top scores.

frameworks for comparison. One is a state-of-the art frameby-frame VC framework (the FVC) [6] that uses a deep autoencoder to obtain a compact representation of MCEPs. It has 7
hidden layers, and the numbers of units in input, hidden, andoutput layers are [25, 100, 40, 15, 150, 15, 40, 100, 25]. The
other framework has the same converter as the LSM but it is
optimized using MSE (the MSE).

FVC vs. LSM
MSE vs. LSM

FormerLatterNeutral32.9 ± 7.9
27.1 ± 7.5

54.3 ± 8.4
65.0 ± 8.0

12.9 ± 5.6
7.9 ± 4.5

Table 3: Average preference scores (%) for clarity with 95%
confidence intervals. Bold font indicates top scores.
FVC vs. LSM
MSE vs. LSM

Neutral10.0 ± 5.0
5.7 ± 3.9

100

100

80

80

Percentage

Figure 5 shows the comparison of the MCEPs and STFT spectrograms of the SRC, TGT, FVC, MSE, and LSM. In the FVC
and MSE, the spectral textures were over-smoothed both in themel-cepstral and spectral domains. In the LSM, the spectral textures similar to the target ones were reproduced not only in themel-cepstral domain but also in the spectral domain, although
the spectral conversion is performed in the mel-cepstral domain.

Latter
72.9 ± 7.5
84.3 ± 6.1

Percentage

3.2. Objective evaluation

Former
17.1 ± 6.3
10.0 ± 5.0

60
40

60
40
20

20
0

NeutralSame: absolutely sureSame: not sure
Different: not sure
Different: absolutely sure

TGT LSM MSE FVC SRC(a) Similarity to TGT

0

SRCTGT LSM FVC MSE(b) Similarity to SRC

Figure 6: Similarity to TGT and SRC with VC frameworks

4. Conclusions
3.3. Subjective evaluation

We explored sequence-to-sequence VC. To measure the similarity for complex sequential data, we proposed a VC frameworkthat uses a similarity metric learned using a GAN. To model thesequence structure, we introduced a gated CNN into our model.
In the experiments, we applied our framework to non-native-tonative VC tasks. The results showed that our proposed framework outperformed the state-of-the-art framework, as well asthat optimized using a conventional similarity metric, in termsof naturalness, clarity, and speaker individuality. Our proposed
framework using a learned similarity metric is not limited to aparticular task, so we plan to apply it to other tasks such as TTS.
Acknowledgements: We thank Mr. Keisuke Oyamada (Univ.
of Tsukuba), who kindly provided us with the source code of thestate-of-the-art VC framework [6]. This work was supported by
JSPS KAKENHI Grant Numbers JP26730100 and JP26280060.

We conducted listening tests by referring to the Voice Conversion Challenge 2016 (VCC 2016) [37] at which naturalness andsimilarity were used as evaluation metrics. We also evaluatedfor clarity to assess the effectiveness in pronunciation conversion. We conducted an AB test on naturalness and clarity: participants were asked to select the preferred one or to opt forneutral if they did not have any preference. To measure thesimilarity of the VC samples, the same/different paradigm wasused: participants were given two samples and asked whether
the samples were produced by the same speaker. For evaluation, ten sample pairs were selected from the test data. In the
AB test, all pairs were presented in both orders (AB and BA) toeliminate bias in the order of stimuli. There were seven participants who were well-educated English speakers.

1286

5. References[20] A. Radford, L. Metz, and S. Chintala, “Unsupervised representation learning with deep convolutional generative adversarial networks,” in Proc. ICLR, 2016.

[1] Y. Stylianou, O. Cappe, and E. Moulines, “Statistical methods
for voice quality transformation,” in Proc. Eurospeech, 1995, pp.
447–450.

[21] T. Kaneko, H. Kameoka, N. Hojo, Y. Ijima, K. Hiramatsu, and
K. Kashino, “Generative adversarial network-based postfilter for
statistical parametric speech synthesis,” in Proc. ICASSP, 2017,
pp. 4910–4914.

[2] Y. Stylianou and O. Cappe, “A system for voice conversion basedon probabilistic classification and a harmonic plus noise model,”in Proc. ICASSP, 1998, pp. 281–284.

[22] Y. Saito, S. Takamichi, and H. Saruwatari, “Training algorithm todeceive anti-spoofing verification for DNN-based speech synthesis,” in Proc. ICASSP, 2017, pp. 4900–4904.

[3] T. Toda, A. W. Black, and K. Tokuda, “Voice conversion based on
maximum-likelihood estimation of spectral parameter trajectory,”
IEEE/ACM Trans. Audio Speech Lang. Process., vol. 15, no. 8,
pp. 2222–2235, 2007.

[23] T. Kaneko, S. Takaki, H. Kameoka, and J. Yamagishi, “Generative
adversarial network-based postfilter for STFT spectrograms,” inProc. Interspeech, 2017.

[4] T. Nakashika, T. Takiguchi, and Y. Ariki, “Voice conversion
based on speaker-dependent restricted Boltzmann machines,” IEICE Trans. Inf. Syst., vol. 97, no. 6, pp. 1403–1410, 2014.

[24] Y. N. Dauphin, A. Fan, M. Auli, and D. Grangier, “Language
modeling with gated convolutional networks,” in arXiv preprint
arXiv:1612.08083, 2016.

[5] L.-H. Chen, Z.-H. Ling, L.-J. Liu, and L.-R. Dai, “Voice conversion using deep neural networks with layer-wise generative training,” IEEE/ACM Trans. Audio Speech Lang. Process., vol. 22,
no. 12, pp. 1859–1872, 2014.

[25] A. B. L. Larsen, S. K. Sønderby, H. Larochelle, and O. Winther,“Autoencoding beyond pixels using a learned similarity metric,”in Proc. ICML, 2016, pp. 1558–1566.

[6] S. H. Mohammadi and A. Kain, “Voice conversion using deepneural networks with speaker-independent pre-training,” in Proc.
SLT, 2014, pp. 19–23.

[26] Y. Kim, “Convolutional neural networks for sentence classification,” in arXiv preprint arXiv:1408.5882, 2014.

[7] R. Takashima, T. Takiguchi, and Y. Ariki, “Exampler-based voice
conversion using sparse representation in noisy environments,”
IEICE Trans. Inf. Syst., vol. E96-A, no. 10, pp. 1946–1953, 2013.

[27] X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional
networks for text classification,” in Proc. NIPS, 2015, pp. 649–
657.

[8] Z. Wu, T. Virtanen, E. S. Chng, and H. Li, “Exemplar-based
sparse representation with residual compensation for voice conversion,” IEEE/ACM Trans. Audio Speech Lang. Process., vol. 22,
no. 10, pp. 1506–1521, 2014.

[28] V. Nair and G. E. Hinton, “Rectified linear units improve restricted
Boltzmann machines,” in Proc. ICML, 2010, pp. 807–814.
[29] H. Kawahara, J. Estill, and O. Fujimura, “Aperiodicity extraction
and control using mixed mode excitation and group delay manipulation for a high quality speech analysis, modification and synthesis system STRAIGHT,” in Proc. MAVEBA, 2001, pp. 59–64.

[9] T. Toda and K. Tokuda, “A speech parameter generation algorithm
considering global variance for HMM-based speech synthesis,”
IEICE Trans. Inf. Syst., vol. E90-D, no. 5, pp. 816–824, 2007.

[30] K. Liu, J. Zhang, and Y. Yan, “High quality voice conversion
through phoneme-based linear mapping functions with straight for
mandarin,” in Proc. FSKD, 2007, pp. 410–414.

[10] H. Silén, E. Helander, J. Nurminen, and M. Gabbouj, “Waysto implement global variance in statistical speech synthesis,” inProc. Interspeech, 2012, pp. 1436–1439.

[31] Y. Ohtani, T. Toda, H. Saruwatari, and K. Shikano, “Maximum
likelihood voice conversion based on GMM with STRAIGHT
mixed excitation,” in Proc. Interspeech, 2006, pp. 2266–2269.

[11] S. Takamichi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura,“A postfilter to modify the modulation spectrum in HMM-based
speech synthesis,” in Proc. ICASSP, 2014, pp. 290–294.

[32] A. Maas, A. Y. Hannun, and A. Y. Ng, “Rectifier nonlinearities
improve neural network acoustic models,” in Proc. ICML Workshop, 2013.

[12] K. Kobayashi, T. Toda, G. Neubig, S. Sakti, and S. Nakamura,“Statistical singing voice conversion with direct waveform modification based on the spectrum differential.” in Proc. Interspeech,
2014, pp. 2514–2518.

[33] B. Xu, N. Wang, T. Chen, and M. Li, “Empirical evaluation of rectified activations in convolutional network,” in Proc. ICML Workshop, 2015.

[13] T. Nakashika, T. Takiguchi, and Y. Ariki, “High-order sequencemodeling using speaker-dependent recurrent temporal restricted
Boltzmann machines for voice conversion.” in Proc. Interspeech,
2014, pp. 2278–2282.

[34] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep
network training by reducing internal covariate shift,” in Proc.
ICML, 2015, pp. 448–456.

[14] L. Sun, S. Kang, K. Li, and H. Meng, “Voice conversion using
deep bidirectional long short-term memory based recurrent neuralnetworks,” in Proc. ICASSP, 2015, pp. 4869–4873.

[35] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in Proc. CVPR, 2015, pp.
3431–3440.

[15] D. Erro, A. Moreno, and A. Bonafonte, “INCA algorithm fortraining voice conversion systems from nonparallel corpora,”
IEEE/ACM Trans. Audio Speech Lang. Process., vol. 18, no. 5,
pp. 944–953, 2010.

[36] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in Proc. ICLR, 2015.
[37] T. Toda, L.-H. Chen, D. Saito, F. Villavicencio, M. Wester, Z. Wu,
and J. Yamagishi, “The voice conversion challenge 2016,” in Proc.
Interspeech, 2016, pp. 1632–1636.

[16] T. Nakashika, T. Takiguchi, and Y. Ariki, “Parallel-data-freemany-to-many voice conversion using an adaptive restricted
Boltzmann machine,” in Proc. MLSP, 2016.
[17] T. Kinnunen, L. Juvela, P. Alku, and J. Yamagishi, “Non-parallelvoice conversion using i-vector PLDA: Towards unifying speakerverification and transformation,” in Proc. ICASSP, 2017, pp.
5535–5539.
[18] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. WardeFarley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. NIPS, 2014, pp. 2672–2680.
[19] E. Denton, S. Chintala, S. A., and R. Fergus, “Deep generative image models using a Laplacian pyramid of adversarial networks,”
in Proc. NIPS, 2015, pp. 1486–1494.

1287