PRVAE-VC: Non-Parallel Many-to-Many Voice Conversion
with Perturbation-Resistant Variational Autoencoder
Kou Tanaka, Hirokazu Kameoka, Takuhiro Kaneko
NTT Communication Science Laboratories, NTT Corporation, Japan{kou.tanaka.ef, hirokazu.kameoka.uh, takuhiro.kaneko.tb}@hco.ntt.co.jp

Abstract
This paper describes a novel approach to non-parallel many-tomany voice conversion (VC) that utilizes a variant of the conditional variational autoencoder (VAE) called a perturbationresistant VAE (PRVAE). In VAE-based VC, it is commonly assumed that the encoder extracts content from the input speech
while removing source speaker information. Following this extraction, the decoder generates output from the extracted content and target speaker information. However, in practice,
the encoded features may still retain source speaker information, which can lead to a degradation of speech quality during
speaker conversion tasks. To address this issue, we propose
a perturbation-resistant encoder trained to match the encodedfeatures of the input speech with those of a pseudo-speech generated through a content-preserving transformation of the input
speech’s fundamental frequency and spectral envelope using a
combination of pure signal processing techniques. Our experimental results demonstrate that this straightforward constraint
signiﬁcantly enhances the performance in non-parallel many-tomany speaker conversion tasks. Audio samples can be accessedat our webpage 1 .
Index Terms: Voice conversion, variational autoencoder, perturbation resistance, representation learning, non-parallel

1. Introduction
Voice conversion (VC) is a technique that transforms the speechof one speaker to sound like that of another while preserving
linguistic content. This technique ﬁnds applications in various domains, including speaker conversion [1, 2], assistive systems [3, 4] aimed at overcoming speech and hearing impairments, and pronunciation and accent conversions [5] for language learning.
There are two frameworks for learning conversion models:parallel VC and non-parallel VC. Parallel VC [2, 6] requiresa parallel speech corpus consisting of recordings of the sametext spoken by both the source and target speakers. While collecting such a corpus can be time-consuming and expensive, it
has the potential to produce high-quality results since it allows
for direct optimization based on the target speech. In contrast,
non-parallel VC involves converting the source speech to thetarget speech without explicitly aligning the source and target
utterances. This makes the task more challenging, as the modelhas to learn the correspondence between the source and target
speech without any guidance from parallel data. However, nonparallel VC has become an active research area in recent years
due to the availability of a large amount of non-parallel speechdata.
1 http://www.kecl.ntt.co.jp/people/tanaka.ko/
projects/prvaevc/

There are two primary methodologies for developing nonparallel VC: one involving text supervision and the other being
unsupervised. Non-parallel VC using text supervision [7, 8] isalso known as an approach cascading automatic speech recognition (ASR) and text-to-speech synthesis (TTS). It utilizes aphoneme recognizer to extract phonetic information from the
input speech, which is then fed to TTS to generate the output
speech. While this approach can produce high-quality conversion results if the ASR works well, it requires paired data oftext and speech for training ASR and TTS, which can be a limiting factor. In contrast, non-parallel VC without text supervision [9–11] typically employs techniques such as autoencoders(AE) [12], variational autoencoders (VAE) [13], and generative
adversarial networks (GAN) [14]. This work focuses on nonparallel VC based on a VAE-based system without text supervision, as it has the potential to utilize latent space to represent common hidden features of speech signals among different
speakers.
The VAE-based VC [15] employs a latent space typicallyassumed to follow a Gaussian distribution to encode a set of
input acoustic features such as Mel-spectrogram. Then, thespeaker information is added to the encoded latent features in
the generation phase to obtain the output acoustic features. In
the decoder, the source speaker information is used to estimate
the reconstruction of the input acoustic features, while that of
the target speaker is used to estimate the converted acoustic features. Although speaker conversion can be achieved by setting
the appropriate hyperparameters, such as the number of dimensions of the model, various improvements have been proposedto achieve better conversion. An example of such an approach is
cycle consistency [16], which ensures that the converted speech
can be converted back to its original form with the output being
as close to the original speech as possible. Another variation involves incorporating an auxiliary classiﬁer [10], which prevents
the decoder from disregarding the speaker information. Unfortunately, according to our initial experiments, these variants
still suffer from hyperparameter tuning of the model. There is alarge difference in conversion performance between small andlarge model sizes. One possible reason is that the latent space
is not uniform across all speakers, and as the model size expands, it forms different distributions for each speaker to matchthe training data better.
To address this issue, we propose a variant of the conditional VAE called a perturbation-resistant VAE (PRVAE). In our
approach, a perturbation-resistant encoder is trained to matchthe encoded features of the input speech with those of a pseudospeech. The pseudo-speech is generated by applying contentpreserving transformations to the input speech using pure signal
processing techniques. This work deﬁnes content-preserving
transformations as linear transformations of fundamental fre-

Inputa: CVAE

Encoder

Decoder

Lrec

Decoder

Lrec

Decoder

Lrec<latexit sha1_base64="Ss5LFVEjJnTdDkoDhTijaxtH4HA=">AAAC4nichVG7TtxAFD2YZ3gu0ERK47CAqFaziJeoUNKkSMFrAYldrezZASz80nh2eViuI9FFKVIkTSKlQFFa+IA0/AAFn4AoiZQmRa5nLRAgYCx77j1zz/GZe+3QdSLF2EWL0drW3tHZ9aK7p7evfyA3OLQWBXXJRYkHbiA3bCsSruOLknKUKzZCKSzPdsW6vfs2PV9vCBk5gb+qDkJR8axt39lyuKUIquZel7VGLEUtMcuepXa45cbvk2pclp4pBU+quTwrML3Mh0ExC/LI1mKQO0cZNQTgqMODgA9FsQsLET2bKIIhJKyCmDBJkaPPBRJ0E7dOVYIqLEJ36btN2WaG+pSnmpFmc/qLS68kpokxds6O2TU7Yz/ZJfv3qFasNVIvB7TbTa4IqwNHL1f+PsvyaFfYuWU96VlhC3Paq0PeQ42kt+BNfuPw8/XK/PJYPM6+syvy/41dsN90A7/xh/9YEstfSD3V94m1p+/raQc+dTgmvEFZonta0Xmqvq87qnRu6w7VblybGEWeqkeJRd2m4Rbvj/JhsDZZKM4Uppem8gtvsjF34RVGMEFKs1jAOyyiRO4+4BdOcGrUjCPjo/GpWWq0ZJxh3FnG1/9Fqqo0</latexit>Lz<latexit sha1_base64="jVCugNxaq70v9IA2ZU3e3rr29gM=">AAAC4HichVE9T9tQFD0YSgMtEMpSqQNWA4gpekaUVkyILh06QCCARKLIdl6CFX/JfkkJlkekqgvqhAQTlTogVvgFLPwBBn4CYqRSFwauX6xWgIBn2e/e8+45Pu9ew7etUDB20aV097zofZnp63/1emBwKDv8ZiX0moHJi6Zne8GaoYfctlxeFJaw+ZofcN0xbL5qND4n56stHoSW5y6Lts/Ljl53rZpl6oKgSna0JDWigFdjteToYsPU7ehrXIlKgaNuxZVsjuWZXOrDQEuDHNK14GXPUUIVHkw04YDDhaDYho6QnnVoYPAJKyMiLKDIkuccMfqJ26QqThU6oQ361ilbT1GX8kQzlGyT/mLTGxBTxTg7Z4fsmp2xI3bJbh7ViqRG4qVNu9Hhcr8y9OPt0t9nWQ7tAhv/WU96Fqjhk/RqkXdfIsktzA6/tbV7vTRbGI8m2C92Rf4P2AU7pRu4rT/m70Ve2Cf1RN8l1jd5X0c6cKnDEeEtymLZ07LME/VN2VEhc0N2qPrPtYox5Kh6jFjUbRqudn+UD4OVqbw2k/+wOJ2bm0/HnME7vMckKX3EHL5gAUVyt40jHONEMZTvyo7ys1OqdKWcEdxZyt4tG1mpYA==</latexit>b: CVAE + Beta(+beta)Input

Encoder<latexit sha1_base64="Ss5LFVEjJnTdDkoDhTijaxtH4HA=">AAAC4nichVG7TtxAFD2YZ3gu0ERK47CAqFaziJeoUNKkSMFrAYldrezZASz80nh2eViuI9FFKVIkTSKlQFFa+IA0/AAFn4AoiZQmRa5nLRAgYCx77j1zz/GZe+3QdSLF2EWL0drW3tHZ9aK7p7evfyA3OLQWBXXJRYkHbiA3bCsSruOLknKUKzZCKSzPdsW6vfs2PV9vCBk5gb+qDkJR8axt39lyuKUIquZel7VGLEUtMcuepXa45cbvk2pclp4pBU+quTwrML3Mh0ExC/LI1mKQO0cZNQTgqMODgA9FsQsLET2bKIIhJKyCmDBJkaPPBRJ0E7dOVYIqLEJ36btN2WaG+pSnmpFmc/qLS68kpokxds6O2TU7Yz/ZJfv3qFasNVIvB7TbTa4IqwNHL1f+PsvyaFfYuWU96VlhC3Paq0PeQ42kt+BNfuPw8/XK/PJYPM6+syvy/41dsN90A7/xh/9YEstfSD3V94m1p+/raQc+dTgmvEFZonta0Xmqvq87qnRu6w7VblybGEWeqkeJRd2m4Rbvj/JhsDZZKM4Uppem8gtvsjF34RVGMEFKs1jAOyyiRO4+4BdOcGrUjCPjo/GpWWq0ZJxh3FnG1/9Fqqo0</latexit>Lz<latexit sha1_base64="e2Be/OVTA7jq+8Y9Bf6qKm2kyDA=">AAAC7nichVG7TtxAFD0YEggkYYEmUhorC1GarGaj8BAVShqKFLwWkPDKGnsHsBg/ZM8uD8s/kBaJFAghkCIF5QvShoYfoOATECVIaVLketYCJSjJWPbce+ae4zP3OpH0EsXYRYfR2fXgYXfPo96+x0+e9pcGBheTsBm7ouaGMoyXHZ4I6QWipjwlxXIUC+47Uiw5G+/z86WWiBMvDBbUdiTqPl8LvFXP5Yogu/Ta0hppLBqZaUkiNrhtOUJx0/K5Wne5TD9kdmrFvrmT2aUyqzC9zPtBtQjKKNZMWDqHhQZCuGjCh0AARbEER0LPCqpgiAirIyUspsjT5wIZeonbpCpBFZzQDfquUbZSoAHluWai2S79RdIbE9PECDtnJ+yanbGv7JL9/KtWqjVyL9u0O22uiOz+j8/mf/yX5dOusH7H+qdnhVVMaK8eeY80kt/CbfNbO5+u5yfnRtKX7Jhdkf8jdsFO6QZB68b9PCvm9kk91w+Itanv62sHAXU4JbxFWaZ7Wtd5rr6lO6p07ugONW5dmxhGmaqHiUXdpuFW/xzl/WDxTaU6VhmdfVueeleMuQfP8QKvSGkcU5jGDGrkbhff8B2nRmTsGQfGYbvU6Cg4Q/htGV9+AUJqryA=</latexit>Lzof the data x, given the latent space variable z and the attributecodes c. The log marginal distribution of the data x, given theattribute codes c, is given as:log pθ (x|c) =L(θ, ϕ) + KL[qϕ (z|x, c)|p(z)],(1)<latexit sha1_base64="jVCugNxaq70v9IA2ZU3e3rr29gM=">AAAC4HichVE9T9tQFD0YSgMtEMpSqQNWA4gpekaUVkyILh06QCCARKLIdl6CFX/JfkkJlkekqgvqhAQTlTogVvgFLPwBBn4CYqRSFwauX6xWgIBn2e/e8+45Pu9ew7etUDB20aV097zofZnp63/1emBwKDv8ZiX0moHJi6Zne8GaoYfctlxeFJaw+ZofcN0xbL5qND4n56stHoSW5y6Lts/Ljl53rZpl6oKgSna0JDWigFdjteToYsPU7ehrXIlKgaNuxZVsjuWZXOrDQEuDHNK14GXPUUIVHkw04YDDhaDYho6QnnVoYPAJKyMiLKDIkuccMfqJ26QqThU6oQ361ilbT1GX8kQzlGyT/mLTGxBTxTg7Z4fsmp2xI3bJbh7ViqRG4qVNu9Hhcr8y9OPt0t9nWQ7tAhv/WU96Fqjhk/RqkXdfIsktzA6/tbV7vTRbGI8m2C92Rf4P2AU7pRu4rT/m70Ve2Cf1RN8l1jd5X0c6cKnDEeEtymLZ07LME/VN2VEhc0N2qPrPtYox5Kh6jFjUbRqudn+UD4OVqbw2k/+wOJ2bm0/HnME7vMckKX3EHL5gAUVyt40jHONEMZTvyo7ys1OqdKWcEdxZyt4tG1mpYA==</latexit>c: CVAE +Cycle consistency
(+cc)

Input

Encoder’
Lrec

Decoder<latexit sha1_base64="Ss5LFVEjJnTdDkoDhTijaxtH4HA=">AAAC4nichVG7TtxAFD2YZ3gu0ERK47CAqFaziJeoUNKkSMFrAYldrezZASz80nh2eViuI9FFKVIkTSKlQFFa+IA0/AAFn4AoiZQmRa5nLRAgYCx77j1zz/GZe+3QdSLF2EWL0drW3tHZ9aK7p7evfyA3OLQWBXXJRYkHbiA3bCsSruOLknKUKzZCKSzPdsW6vfs2PV9vCBk5gb+qDkJR8axt39lyuKUIquZel7VGLEUtMcuepXa45cbvk2pclp4pBU+quTwrML3Mh0ExC/LI1mKQO0cZNQTgqMODgA9FsQsLET2bKIIhJKyCmDBJkaPPBRJ0E7dOVYIqLEJ36btN2WaG+pSnmpFmc/qLS68kpokxds6O2TU7Yz/ZJfv3qFasNVIvB7TbTa4IqwNHL1f+PsvyaFfYuWU96VlhC3Paq0PeQ42kt+BNfuPw8/XK/PJYPM6+syvy/41dsN90A7/xh/9YEstfSD3V94m1p+/raQc+dTgmvEFZonta0Xmqvq87qnRu6w7VblybGEWeqkeJRd2m4Rbvj/JhsDZZKM4Uppem8gtvsjF34RVGMEFKs1jAOyyiRO4+4BdOcGrUjCPjo/GpWWq0ZJxh3FnG1/9Fqqo0</latexit>

EncoderLz’<latexit sha1_base64="jVCugNxaq70v9IA2ZU3e3rr29gM=">AAAC4HichVE9T9tQFD0YSgMtEMpSqQNWA4gpekaUVkyILh06QCCARKLIdl6CFX/JfkkJlkekqgvqhAQTlTogVvgFLPwBBn4CYqRSFwauX6xWgIBn2e/e8+45Pu9ew7etUDB20aV097zofZnp63/1emBwKDv8ZiX0moHJi6Zne8GaoYfctlxeFJaw+ZofcN0xbL5qND4n56stHoSW5y6Lts/Ljl53rZpl6oKgSna0JDWigFdjteToYsPU7ehrXIlKgaNuxZVsjuWZXOrDQEuDHNK14GXPUUIVHkw04YDDhaDYho6QnnVoYPAJKyMiLKDIkuccMfqJ26QqThU6oQ361ilbT1GX8kQzlGyT/mLTGxBTxTg7Z4fsmp2xI3bJbh7ViqRG4qVNu9Hhcr8y9OPt0t9nWQ7tAhv/WU96Fqjhk/RqkXdfIsktzA6/tbV7vTRbGI8m2C92Rf4P2AU7pRu4rT/m70Ve2Cf1RN8l1jd5X0c6cKnDEeEtymLZ07LME/VN2VEhc0N2qPrPtYox5Kh6jFjUbRqudn+UD4OVqbw2k/+wOJ2bm0/HnME7vMckKX3EHL5gAUVyt40jHONEMZTvyo7ys1OqdKWcEdxZyt4tG1mpYA==</latexit>Lz<latexit sha1_base64="jVCugNxaq70v9IA2ZU3e3rr29gM=">AAAC4HichVE9T9tQFD0YSgMtEMpSqQNWA4gpekaUVkyILh06QCCARKLIdl6CFX/JfkkJlkekqgvqhAQTlTogVvgFLPwBBn4CYqRSFwauX6xWgIBn2e/e8+45Pu9ew7etUDB20aV097zofZnp63/1emBwKDv8ZiX0moHJi6Zne8GaoYfctlxeFJaw+ZofcN0xbL5qND4n56stHoSW5y6Lts/Ljl53rZpl6oKgSna0JDWigFdjteToYsPU7ehrXIlKgaNuxZVsjuWZXOrDQEuDHNK14GXPUUIVHkw04YDDhaDYho6QnnVoYPAJKyMiLKDIkuccMfqJ26QqThU6oQ361ilbT1GX8kQzlGyT/mLTGxBTxTg7Z4fsmp2xI3bJbh7ViqRG4qVNu9Hhcr8y9OPt0t9nWQ7tAhv/WU96Fqjhk/RqkXdfIsktzA6/tbV7vTRbGI8m2C92Rf4P2AU7pRu4rT/m70Ve2Cf1RN8l1jd5X0c6cKnDEeEtymLZ07LME/VN2VEhc0N2qPrPtYox5Kh6jFjUbRqudn+UD4OVqbw2k/+wOJ2bm0/HnME7vMckKX3EHL5gAUVyt40jHONEMZTvyo7ys1OqdKWcEdxZyt4tG1mpYA==</latexit>d: CVAE +Auxiliary classifier
(+ac)

Input

<latexit sha1_base64="Ss5LFVEjJnTdDkoDhTijaxtH4HA=">AAAC4nichVG7TtxAFD2YZ3gu0ERK47CAqFaziJeoUNKkSMFrAYldrezZASz80nh2eViuI9FFKVIkTSKlQFFa+IA0/AAFn4AoiZQmRa5nLRAgYCx77j1zz/GZe+3QdSLF2EWL0drW3tHZ9aK7p7evfyA3OLQWBXXJRYkHbiA3bCsSruOLknKUKzZCKSzPdsW6vfs2PV9vCBk5gb+qDkJR8axt39lyuKUIquZel7VGLEUtMcuepXa45cbvk2pclp4pBU+quTwrML3Mh0ExC/LI1mKQO0cZNQTgqMODgA9FsQsLET2bKIIhJKyCmDBJkaPPBRJ0E7dOVYIqLEJ36btN2WaG+pSnmpFmc/qLS68kpokxds6O2TU7Yz/ZJfv3qFasNVIvB7TbTa4IqwNHL1f+PsvyaFfYuWU96VlhC3Paq0PeQ42kt+BNfuPw8/XK/PJYPM6+syvy/41dsN90A7/xh/9YEstfSD3V94m1p+/raQc+dTgmvEFZonta0Xmqvq87qnRu6w7VblybGEWeqkeJRd2m4Rbvj/JhsDZZKM4Uppem8gtvsjF34RVGMEFKs1jAOyyiRO4+4BdOcGrUjCPjo/GpWWq0ZJxh3FnG1/9Fqqo0</latexit>

Encoder

Lrec

Decoder<latexit sha1_base64="Ss5LFVEjJnTdDkoDhTijaxtH4HA=">AAAC4nichVG7TtxAFD2YZ3gu0ERK47CAqFaziJeoUNKkSMFrAYldrezZASz80nh2eViuI9FFKVIkTSKlQFFa+IA0/AAFn4AoiZQmRa5nLRAgYCx77j1zz/GZe+3QdSLF2EWL0drW3tHZ9aK7p7evfyA3OLQWBXXJRYkHbiA3bCsSruOLknKUKzZCKSzPdsW6vfs2PV9vCBk5gb+qDkJR8axt39lyuKUIquZel7VGLEUtMcuepXa45cbvk2pclp4pBU+quTwrML3Mh0ExC/LI1mKQO0cZNQTgqMODgA9FsQsLET2bKIIhJKyCmDBJkaPPBRJ0E7dOVYIqLEJ36btN2WaG+pSnmpFmc/qLS68kpokxds6O2TU7Yz/ZJfv3qFasNVIvB7TbTa4IqwNHL1f+PsvyaFfYuWU96VlhC3Paq0PeQ42kt+BNfuPw8/XK/PJYPM6+syvy/41dsN90A7/xh/9YEstfSD3V94m1p+/raQc+dTgmvEFZonta0Xmqvq87qnRu6w7VblybGEWeqkeJRd2m4Rbvj/JhsDZZKM4Uppem8gtvsjF34RVGMEFKs1jAOyyiRO4+4BdOcGrUjCPjo/GpWWq0ZJxh3FnG1/9Fqqo0</latexit>

Lcls

ClassifierLz<latexit sha1_base64="Zr/HmYVljLIzkKLkKE8qnJUUTAM=">AAAC4nichVG7TtxAFD0YkvAIYSENUhqH3UVUq1kUHqJCSZOCgtfCSrurlT07gMX4IXt2gViuI9FFKVKEJkgUKEqbfEAafoCCT0CUREpDwfWslShBCWPZc++Ze47P3GsH0okUYxc9Rm/fg4eP+gcGhx4PPxnJjY5tRH475KLCfemHVduKhHQ8UVGOkqIahMJybSk27d1X6flmR4SR43vr6iAQDdfa9pwth1uKoGbueV1rxKFoJWbdtdQOt2S8lDTjeuiaXEZJM5dnJaaXeTcoZ0Ee2Vr2c+eoowUfHG24EPCgKJawENFTQxkMAWENxISFFDn6XCDBIHHbVCWowiJ0l77blNUy1KM81Yw0m9NfJL0hMU0U2Tk7ZdfsjH1ml+zmn1qx1ki9HNBud7kiaI4cjq/9vJfl0q6w85v1X88KW5jXXh3yHmgkvQXv8jtvPlyvLawW40l2zK7I/yd2wb7TDbzOD36yIlY/knqq7xFrT9/X1Q486nBMeIeyRPe0ofNUfV93VOnc1h1q/XJtooA8VReIRd2m4Zb/HuXdYGO6VJ4tzay8yC++zMbcj2eYwBQpzWERr7GMCrl7iy/4im9Gyzg03hnvu6VGT8Z5ij+WcXQLWHuqPA==</latexit><latexit sha1_base64="jVCugNxaq70v9IA2ZU3e3rr29gM=">AAAC4HichVE9T9tQFD0YSgMtEMpSqQNWA4gpekaUVkyILh06QCCARKLIdl6CFX/JfkkJlkekqgvqhAQTlTogVvgFLPwBBn4CYqRSFwauX6xWgIBn2e/e8+45Pu9ew7etUDB20aV097zofZnp63/1emBwKDv8ZiX0moHJi6Zne8GaoYfctlxeFJaw+ZofcN0xbL5qND4n56stHoSW5y6Lts/Ljl53rZpl6oKgSna0JDWigFdjteToYsPU7ehrXIlKgaNuxZVsjuWZXOrDQEuDHNK14GXPUUIVHkw04YDDhaDYho6QnnVoYPAJKyMiLKDIkuccMfqJ26QqThU6oQ361ilbT1GX8kQzlGyT/mLTGxBTxTg7Z4fsmp2xI3bJbh7ViqRG4qVNu9Hhcr8y9OPt0t9nWQ7tAhv/WU96Fqjhk/RqkXdfIsktzA6/tbV7vTRbGI8m2C92Rf4P2AU7pRu4rT/m70Ve2Cf1RN8l1jd5X0c6cKnDEeEtymLZ07LME/VN2VEhc0N2qPrPtYox5Kh6jFjUbRqudn+UD4OVqbw2k/+wOJ2bm0/HnME7vMckKX3EHL5gAUVyt40jHONEMZTvyo7ys1OqdKWcEdxZyt4tG1mpYA==</latexit>

e: CVAE +Perturbation resistance(+pr, ours)

Input

Encoder<latexit sha1_base64="h6mtHdH0KDcFY+Yu3EK5OQGSgUs=">AAAC8nichVHLThRBFD20DxAfjLoxcdNxwLiaVBNFw4roxoULHg6QMJNJdU8BHfqV6poR7MwP+AMkukLjwugvuDJRf4AFn0BYYuLGhadrOhglanW6695T95w+da+fRWFuhDgYcc6cPXd+dOzC+MVLl69M1K5eW87Tng5UM0ijVK/6MldRmKimCU2kVjOtZOxHasXfelSer/SVzsM0eWp2MtWO5UYSroeBNIQ6Na9lNQqtugO3FZHYlZ2ipWM30wRiaTYDGRVPBidgp1YXDWGXezrwqqCOas2ntX200EWKAD3EUEhgGEeQyPmswYNARqyNgphmFNpzhQHGye2xSrFCEt3id4PZWoUmzEvN3LID/iXiq8l0MSX2xTtxLL6K9+JQ/PirVmE1Si873P0hV2WdiRc3lr7/lxVzN9j8xfqnZ4N1PLBeQ3rPLFLeIhjy+893j5dmF6eK2+K1OKL/PXEgPvEGSf9b8HZBLb6ieqmfkPXM3je2DhJ2uCDeZzawPW3bvFTfth01Nvdth7onrl1Mos7qSbLYbQ7X+3OUp4Pl6YY307i3cLc+97Aa8xhu4hbuUOk+5vAY82jS3S4+4jO+OMZ56ew5b4alzkjFuY7flvPhJwdysO0=</latexit>

Decoderpr Lpr

Perturbation

Lrec<latexit sha1_base64="Ss5LFVEjJnTdDkoDhTijaxtH4HA=">AAAC4nichVG7TtxAFD2YZ3gu0ERK47CAqFaziJeoUNKkSMFrAYldrezZASz80nh2eViuI9FFKVIkTSKlQFFa+IA0/AAFn4AoiZQmRa5nLRAgYCx77j1zz/GZe+3QdSLF2EWL0drW3tHZ9aK7p7evfyA3OLQWBXXJRYkHbiA3bCsSruOLknKUKzZCKSzPdsW6vfs2PV9vCBk5gb+qDkJR8axt39lyuKUIquZel7VGLEUtMcuepXa45cbvk2pclp4pBU+quTwrML3Mh0ExC/LI1mKQO0cZNQTgqMODgA9FsQsLET2bKIIhJKyCmDBJkaPPBRJ0E7dOVYIqLEJ36btN2WaG+pSnmpFmc/qLS68kpokxds6O2TU7Yz/ZJfv3qFasNVIvB7TbTa4IqwNHL1f+PsvyaFfYuWU96VlhC3Paq0PeQ42kt+BNfuPw8/XK/PJYPM6+syvy/41dsN90A7/xh/9YEstfSD3V94m1p+/raQc+dTgmvEFZonta0Xmqvq87qnRu6w7VblybGEWeqkeJRd2m4Rbvj/JhsDZZKM4Uppem8gtvsjF34RVGMEFKs1jAOyyiRO4+4BdOcGrUjCPjo/GpWWq0ZJxh3FnG1/9Fqqo0</latexit>

Encoder

where KL[·|·] denotes the Kullback-Leibler (KL) divergence.
This implies we can minimize the KL divergence betweenqϕ (z|x, c) and p(z) by maximizing L(θ, ϕ) with respect toθ and ϕ. A typical way of modeling p(z), qϕ (z|x, c), andpθ (x|z, c), is to assume Gaussian distributions.
In the conditional VAE (CVAE) based VC [15], the encoderand decoder networks are designed to generate the sequences ofthe means and logarithmic variances of qϕ and pθ , given the
Mel-spectrogram xs and the speaker codes cs of the source
speaker:[]µzs ; log σ 2zs =Encoder(xs , cs ),(2)[]µxss ; log σ 2xss =Decoder(µzs + σ zs ⊙ ϵ, cs ),(3)

Encoder flow chart
Mel-spectrogram &speaker vector

Decoder flow chart
Latent space variable &speaker vector

Classifier flow chart
Mel-spectrogram

xss =µxss + σ xss ⊙ ϵ,

Encoder

DecoderClassifier

Mean & variance oflatent space variable

Mean & variance ofMel-spectrogram

Speaker posterior

where ϵ, [; ], and ⊙ denotes Gaussian noise, concatenation
along the channel dimension, and element-wise manipulation.
In the conversion process at the test time, given the speaker
codes ct of the target speaker, the converted Mel-spectrogram
xst is generated as follows:[]µxst ; log σ 2xst =Decoder(µzs + σ zs ⊙ ϵ, ct ), (5)Figure 1: System overview of VAE-based VCs. Black solid anddashed arrows in (a)-(e) indicate the reconstruction and conversion ﬂow. The solid red arrow indicates the loss calculation.

quency and spectral envelope without changing the linguistic
content. Our experimental results demonstrate that introducing
perturbation resistance successfully overcomes the unstable behavior caused by changes in model parameters. This ﬁnding
proves that increasing the model size can improve performance,
as shown in subjective and objective evaluations.

2. Conventional VAE-Based VC
The system overview is shown in Fig. 1. We only require thespeech waveform and the corresponding speaker ID as the training data.
As the speech parameters, we extract 80-dimensional Melspectrogram features over a range of 80-7600 Hz from the given
source speech signals sampled at 16 kHz. The requirements for
short-time Fourier transform are the same as reported in [17];a Hanning window, 64 ms frame length, eight ms frameshift,and 1024-point fast Fourier transform. Instead of using classical vocoders such as STRAIGHT [18] or WORLD [19],which were used in some conventional methods, we used HiFiGAN [20], a neural vocoder, to synthesize speech waveforms.
To ensure a fair comparison of all methods, in our experiment,
we 1) extracted the Mel-spectrogram from the speech waveform, 2) converted the Mel-spectrogram using each method, and3) ﬁnally generated the speech waveform using HiFiGAN.
2.1. Conditional VAE (CVAE)
A conditional variant [21] of VAE [13] is a neural networkmodel that includes an encoder network and a decoder network.
The encoder network produces parameters for the conditional
distribution qϕ (z|x, c) of a latent space variable z, given data
x and the attribute codes c. In contrast, the decoder network
generates parameters for the conditional distribution pθ (x|z, c)

xst =µxst + σ xst ⊙ ϵ.

(4)

(6)

Finally, the objective function Lcvae to be minimized isgiven as,Lcvae =Lz + Lrec ,)(Lz =FKLD N (µxss , σ 2xss )|N (0, I) , ,
)
(
2
) ,
Lrec =FGNLL xs , N (µxss , log σ x
ss

(7)
(8)
(9)

where N (·), FGNLL , and FKLD denote a Gaussian distribution,a Gaussian negative log-likelihood loss function, and a KL divergence loss function, respectively. As shown in Fig. 1(a),the well-known KL loss for the latent space variable and reconstruction loss for the data are Eqs. (8) and (9).
2.2. Beta variant
Beta-VAE [22] is a variant of the VAE model that emphasizes
the disentanglement of the latent space variables. In a typical
VAE, the latent space variables z follow a multidimensional
Gaussian distribution. However, the constraint may be weakened due to the balance between the KL divergence term Lz andthe reconstruction error Lrec in the objective function, Eq (7).
Beta-VAE strengthens the constraint by increasing the weightλbeta of the KL term to more than 1, promoting independence
and disentanglement of the latent space variables across dimensions (Fig. 1(b)). However, this weakens the importance ofthe reconstruction error, potentially resulting in blurred reconstructed data. The objective functions of Beta-VAE to be minimized is given as,
L+beta = λbeta Lz + Lrec .

(10)

Note that [23] uses Beta-VAE to model both speaker and content information in the encoder. In contrast, we explicitly incorporate speaker information using speaker IDs to compare withother methods under the same conditions.

CVAE-Small
CVAE-Large

2.3. Cycle-consistent variant

Since VAE is trained using unaligned speech data, the groundtruth Mel-spectrogram for xst does not exist in the training
data. However, since xsts is expected to be the input Melspectrogram xs , the losses L′z and L′rec can still be calculated,
as follows:()
L′z =FKLD N (µxst , σ 2xst )|N (0, I) ,
(13)
(
)
′
2
Lrec =FGNLL xs , N (µxsts , log σ xsts ) .
(14)This cyclic ﬂow can be continued by using the cyclic reconstructed features xsts as input xs for the next cycle. The objective functions L+cc of CycleVAE to be minimized is given
as,
1 ∑
′
(Lz + Lrec + L′z + Lrec).
(15)
L+cc =
Ncc N
cc

where Ncc indicates the total number of cycle.
2.4. Auxiliary classiﬁer variant
ACVAE [16] is a variant of the VAE model that considers both
the reconstruction ﬂow and the conversion ﬂow in the parameter optimization process (Fig. 1(d)). Unlike [16], ACVAE employs information-theoretic regularization during model training to ensure that the information contained in the attribute classlabel is preserved in the conversion process. In a standard CycleVAE, the encoder and decoder networks can still disregard
the attribute class labels. This results in limited control overthe speech characteristics during testing, potentially leading tosimple reconstruction without conversion.
To address this issue, ACVAE introduces an auxiliary classiﬁer that takes the Mel-spectrogram xs , xss , and xst as input
and estimates the logits y s , y ss , and y st of the speaker posteriors as output, as follows:y s =Classiﬁer(xs ),y ss , y st =Classiﬁer(xss ), Classiﬁer(xst ).

(16)
(17)

This enables us to optimize the conversion ﬂow directly bylearning the encoder, decoder, and classiﬁer. The objectivefunctions L+ac of ACVAE to be minimized is given as,
L+ac =Lcvae + Lclsreal + Lclsfake ,Lclsreal =FCE (y s , hs ),()
Lclsfake =0.5 ∗ FCE (y ss , hs ) + FCE (y st , ht ) .

(18)
(19)
(20)

where FCE , hs , and ht denote a cross-entropy loss function,
the index of the source speaker, and that of the target speaker,
respectively.

+ cc
+ cc

+ ac
+ ac

+ pr (ours)+ pr (ours)

105
104
103
PR loss

CycleVAE [16] is a variant of the VAE model that takes into
account not only the reconstruction ﬂow but also the conversion ﬂow in the parameter optimization (Fig. 1(c)). As shownin Eq. (7), the original CVAE objective function consisted of
ﬂows to reconstruct the input xs and did not consider the actual
conversion process. To address this problem, [16] indirectly
optimizes the conversion ﬂow by recycling the converted features xst back into the system to obtain corresponding cyclicreconstructed features xsts that can be directly optimized, as
follows:][(11)µzst ; log σ 2zst =Encoder(xst , ct ),[]2µxsts ; log σ xsts =Decoder(µzst + σ zst ⊙ ϵ, cs ). (12)+ beta+ beta

102
101
100
10 1
10 2

0

10000

20000
30000
Iterations

40000

50000

Figure 2: Comparison of perturbation resistance losses on conventional VAE-based voice conversions and the proposed. Solidand dashed lines indicate the results on the small and large
models, respectively.

3. Proposed Method
3.1. Concept
It is a common assumption that the encoder in voice conversion extracts the content from the input speech while eliminating any information about the source speaker. For example, incascading ASR and TTS approaches, ASR, which can be regarded as the encoder, extracts context information, a speakerindependent feature. The conventional VAE-based VCs introducing speaker codes are also assumed that the latent space variables, the output of the encoder, are the speaker-independentfeatures expected to represent phonetic information [16]. After
this extraction, the decoder uses the extracted content and target
speaker information to generate the output Mel-spectrogram.
As a preliminary experiment to conﬁrm the speaker independence of latent space variables, we calculated the KL divergence between the conditional distribution obtained when acertain Mel-spectrogram was given to the encoder and the conditional distribution obtained when a Mel-spectrogram of thepseudo-speech was given to the encoder, in which the mean offundamental frequency (F0 ) was randomly changed. Since therole of the encoder is to remove speaker bias, the KL divergencementioned above should be close to zero, as the difference inthe mean of F0 can be considered a form of speaker bias. However, as shown in Fig. 2, the results of the KL divergence arequite large, indicating that the encoded features may still retain
source speaker information. This could lead to a degradation of
speech quality during speaker conversion tasks. To address thisissue, we propose a training framework to learn less speakerdependent features as the latent space variables without text supervision.
3.2. Perturbation-resistant VAE
To learn a speech representation that is less speaker-dependent
in an unsupervised manner, pseudo-speech that manipulates
speaker biases such as the mean value of F0 and vocal tract
length is created and used for training. Inspired by [24], we use
WOLRD analyzer Fana and synthesizer Fsyn [19] to extract F0
f s , spectral envelopes es , and aperiodicities as from the original speech ws , and generate waveforms of pseudo-speech wm

from manipulated acoustic features, as follows:
f s , es , as =Fana (ws ),()
wm =Fsyn Ff0 (f s , αf ), Fenv (es , αe ), as ,

(21)
(22)

where Ff0 , αf , Fenv , and αe represent a function that randomizes the mean of F0 , a target mean value for F0 , a frequency
warping function [25], and a warping factor, respectively.
After generating the pseudo-speech, we extract the Melspectrogram xm from wm similarly to the extraction of xs
from ws . Unlike the conventional VAE-based VCs, a speaker
encoder is introduced to obtain the speaker codes cm from theMel-spectrogram of the pseudo-speech. Then, a set of the parameters, µzm and σ zm , for the conditional distribution of thelatent space variable z m is generated, as follow:
cm = SpeakerEncoder(xm ),[]µzm ; log σ 2zm = Encoder(xm , cm ).

(23)

Projection
Encoder

LSTM
Projection
Projection

Decoder

LSTM

Projection
Num. of Params

dim.
layer
dim.
residual
dim.
dim.
layer
dim.
residual
dim.

Small
128
4
128

Large
512
2
512
✓
16 × 2 32 × 2
128
512
4
2
128
512
✓
80 × 2
1.2M
8.8M

(24)

Our goal is to train the encoder to match the two distributionsof the latent space variables z s and z m . Hence, we deﬁne theperturbation resistance loss as follows:()
Lpr = FKLD N (µzs , σ 2xs )|N (µzm , σ 2xm ) .
(25)
The ﬁnal objective function L of PRVAE is given as,
L = Lcvae + λpr Lpr ,Table 1: Model architecture summary for Small and Large VAE
models.

(26)

where λpr is a regularization parameter, which weighs the importance of the perturbation-resistant regularization.

4. Experiments
4.1. Implementation details
As detailed in Table 1, CVAEs were designed using long shortterm memory (LSTM) [26]. To examine the conversion performance for different model sizes, we used two types of CVAEs
in the experiment: Small and Large. Therefore, we evaluated 10
VC systems: CVAE-Small, 4 variants of CVAE-Small (+beta:
beta, +cc: cycle-consistent, +ac: auxiliary classiﬁer, and +pr:
perturbation-resistant), CVAE-Large, and 4 variants of CVAELarge (+beta, +cc, +ac, and +pr). In the CVAE-Small, 4layer LSTMs with 128 hidden units were used for the encoderand decoder, respectively. In contrast, CVAE-Large uses a 2layer LSTM with 512 hidden units, and the second layer of thestacked LSTMs has a residual connection. Speaker vectors wereconcatenated in all layers.
The model was trained for 100k iterations using the Adam
optimizer [27] with a mini-batch size of 16. The learning rate,
the ﬁrst and second moments decay rates β1 , and β2 were set to0.001, 0.9, and 0.99, respectively. To train the small-sized models, we applied the KL term annealing technique [28], which
gradually increases the weight of the KL divergence term in
the objective function during training. This technique has beenshown to improve the quality of generated samples and preventthe model from ignoring the latent variables. After experimenting with different values (2, 3, 5, and 10), we set λβ to 3. We
also used Ncc of 3, similar to the setting in [16]. The settings
for the auxiliary classiﬁer are the same as those in [16]. After experimenting with different values (1, 10, and 100), we set
λpr to 10. As for the hyperparameters αf and αe , we randomly
sampled from uniform distributions of [90, 300] and [0.9, 1.1]
for each training iteration, respectively. As the speaker encoder,we used 2-layer LSTMs with 128 hidden units, followed by
a 32-dimensional linear projection to obtain a 32-dimensional
speaker vector.

4.2. Other experimental conditions
We conducted experimental evaluations using a phoneticallybalanced Japanese speech dataset [29] consisting of utterancesby six professional male speakers and four professional female
speakers. The speech was recorded in a quiet room with minimal reverberation, and the silent section was removed using
annotation labeled by experts. To train VC models, we used
450 sentences (speech section of around 0.5 hours) per speaker.
To evaluate the performance, we used 53 sentences per speaker.
All models were trained on many-to-many condition, which is
10-speaker input and 10-speaker output.
As the objective evaluation metrics, we used Mel-cepstral
distortion (MCD) [dB] [30], a correlation coefﬁcient of logarithmic F0 (F0 Corr), and character error rate (CER) [%]. We
used dynamic time warping [31] to get the alignment between
the converted sample and the reference sample. To calculate
the MCD and F0 Corr, we extracted 0-24 order Mel-cepstrum
and F0 from the raw speech and the converted speech synthesized by the neural vocoder. The CER was calculated by theTransformer-based ASR model trained on the corpus of spontaneous Japanese [32], provided by ESPnet [33]. Before calculating the CER, we converted kanji to hiragana to eliminate any
variation caused by kanji or hiragana.
As the subjective evaluation of sound quality, we conducted
a 5-scaled mean opinion score (MOS): 5 for excellent, 4 forgood, 3 for fair, 2 for poor, and 1 for bad. To conﬁrm speakersimilarity, we also conducted a 4-scaled preference test (PT): 4
for same (sure), 3 for same (not sure), 2 for different (not sure),and 1 for different (sure). Ten native Japanese speakers participated in each subjective evaluation. Each system was evaluatedover 270 times.
4.3. Results for generalization of latent space variables
To verify the degree of speaker independence of the latent spacevariables, we calculated the differences between the reconstruction error and the conversion error for each method, which are
shown in the second row of Table 2 (denoted by (·)). If the latent
space features are less speaker-dependent, the difference between the reconstruction and conversion errors should be small.
Conversely, if the difference is large, the latent space variables
contain speaker information, which may have caused the conversion error to be larger.
The objective evaluation results show that the proposed

+ cc
+ ac
+ pr
(ours)
CVAE-Large+ beta+ pr (ours)

+ ac

+ cc

+ beta

CVAE-Large

0

+ ac

20

Systems

Figure 4: Subjective evaluation results on speaker similarity.
The higher the rate of Same, the better the performance.

less speaker-dependent features as the latent space variables.
4.4. Results of subjective listening tests

4

Next, the results of the perceptual evaluation were shown in
Fig. 3 and 4. From these results, the proposed method (+pr)
with the large-sized model is the best system for sound quality
and speaker similarity. Similar to the objective experimental results, the difference between the methods is smaller when themodel size is small. However, as the model size increases, theconventional method deteriorates while the proposed method
improves. Moreover, while the beta variant (+beta) performs
less well than other methods when the model is compact, it outperforms other conventional methods when the model is large.
These results suggest that constraints on latent space variables
have a certain effect.

3

5. Conclusions

+ cc
+ ac
+ pr
(ours)

5
Re-synthsized+ pr (ours)

+ ac

+ cc

+ beta

CVAE-Large+ pr (ours)

+ ac

+ cc

1

+ beta2

CVAE-Small

Mean opinion scores

40

+ cc

CER ↓
7.5 ± 0.27
5.3 (2.2)
13.5 ± 0.35
10.4 (3.1)
10.6 ± 0.32
7.1 (3.5)
7.2 ± 0.26
4.9 (2.3)
9.2 ± 0.29
7.9 (1.3)
17.6 ± 0.52
3.5 (14.1)
12.6 ± 0.34
6.8 (5.8)
14.3 ± 0.42
4.5 (9.8)
13.5 ± 0.43
3.8 (9.7)
6.2 ± 0.23
4.8 (1.4)

60+ betaF0 Corr ↑0.73 ± 0.01
0.84 (0.10)
0.72 ± 0.01
0.82 (0.10)
0.72 ± 0.01
0.84 (0.12)
0.72 ± 0.01
0.85 (0.13)
0.73 ± 0.01
0.83 (0.10)
0.50 ± 0.01
0.86 (0.36)
0.62 ± 0.01
0.85 (0.23)
0.59 ± 0.01
0.85 (0.26)
0.60 ± 0.01
0.86 (0.26)
0.73 ± 0.01
0.84 (0.11)

80+ pr (ours)+ betaMCD ↓6.76 ± 0.04
5.20 (1.56)
6.62 ± 0.03
5.81 (0.81)
6.78 ± 0.04
5.40 (1.38)
6.75 ± 0.04
5.18 (1.57)
6.57 ± 0.03
5.79 (0.78)
7.45 ± 0.04
4.58 (2.87)
6.94 ± 0.04
5.50 (1.44)
7.24 ± 0.04
5.02 (2.22)
7.21 ± 0.04
4.79 (2.42)
6.52 ± 0.03
5.52 (1.00)

Different (not sure)
Different (sure)100

CVAE-Small

System
CVAE-Small

Same (sure)Same (not sure)

Percentage [%]Table 2: Objective evaluation results. The lower the MCD and
CER, the better the performance. The higher the F0 Corr, thebetter the performance. The values following ± indicate conﬁdence intervals. The ﬁrst terms of the second row for eachmethod represent the reconstruction errors on the evaluation
dataset. The second terms (represented by (·)) indicate differences between the reconstruction and conversion errors. The
conﬁdence intervals of the reconstruction error is omitted forbrevity.

Systems

Figure 3: Subjective evaluation results on sound quality. The
higher the value, the better the sound quality. The error bars
denote 95% conﬁdence intervals. Re-synthesized indicates the
speech synthesized from the ground-truth Mel-spectrogram.

method (+pr) has the smallest difference, indicating that thelatent space variables are more generalized. Moreover, whenthe model size is increased, the proposed method improves thereconstruction and conversion errors compared to the case witha smaller model size. In contrast, the conventional method has asmaller reconstruction error but a larger conversion error. Theseﬁndings support the claim that the proposed method can extract

This paper described a non-parallel many-to-many voice conversion method based on a perturbation-resistant variational
autoencoder. We introduced an encoder trained to match theencoded features of the input speech with those of a pseudospeech generated through a content-preserving transformationof the input speech’s fundamental frequency and spectral envelope. Experimental results showed that the proposed encoder
enabled us to extract less speaker-dependent features, leading
to the best performance in subjective evaluations. We plan toextend the proposed speech representation technique to otherdownstream tasks, such as automatic speech recognition andsource separation.

6. ACKNOWLEDGMENTS
Empty for double-blind review. This work was supported by
JST CREST Grant Number JP-MJCR19A3, Japan.

7. References[1] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu,“Neural discrete representation learning,” in NeurIPS, pp. 6309––6318, 2017.
[2] Jing-Xuan Zhang, Zhen-Hua Ling, Li-Juan Liu, Yuan Jiang, andLi-Rong Dai, “Sequence-to-sequence acoustic modeling for voice
conversion,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 27, no. 3, pp. 631–644, 2019.
[3] Kou Tanaka, Tomoki Toda, Graham Neubig, Sakriani Sakti, andSatoshi Nakamura, “A hybrid approach to electrolaryngeal speechenhancement based on noise reduction and statistical excitation
generation,” IEICE Transactions on Information and Systems, vol.
97, no. 6, pp. 1429–1437, 2014.
[4] Fadi Biadsy, Ron J Weiss, Pedro J Moreno, Dimitri Kanevsky, andYe Jia, “Parrotron: An end-to-end speech-to-speech conversion
model and its applications to hearing-impaired speech and speech
separation,” in INTERSPEECH, pp. 4115–4119, 2019.
[5] Daniel Felps, Heather Bortfeld, and Ricardo Gutierrez-Osuna,“Foreign accent conversion in computer assisted pronunciation
training,” Speech Communication, vol. 51, no. 10, pp. 920–932,
2009.
[6] Kou Tanaka, Hirokazu Kameoka, Takuhiro Kaneko, andNobukatsu Hojo, “AttS2S-VC: Sequence-to-sequence voice conversion with attention and context preservation mechanisms,” in
ICASSP, pp. 6805–6809, 2019.
[7] Li-Juan Liu, Yan-Nian Chen, Jing-Xuan Zhang, Yuan Jiang, YaJun Hu, Zhen-Hua Ling, and Li-Rong Dai, “Non-parallel voice
conversion with autoregressive conversion model and duration adjustment,” in Joint Workshop for the Blizzard Challenge and Voice
Conversion Challenge, pp. 126–130, 2020.
[8] Wen-Chin Huang, Shu-Wen Yang, Tomoki Hayashi, Hung-Yi
Lee, Shinji Watanabe, and Tomoki Toda, “S3PRL-VC: Opensource voice conversion framework with self-supervised speech
representations,” in ICASSP, pp. 6552–6556, 2022.
[9] Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, andMark Hasegawa-Johnson, “AutoVC: Zero-shot voice style transfer with only autoencoder loss,” in ICML, pp. 5210–5219, 2019.
[10] Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, andNobukatsu Hojo, “ACVAE-VC: Non-parallel voice conversion
with auxiliary classiﬁer variational autoencoder,” IEEE/ACM
Transactions on Audio, Speech, and Language Processing, vol.
27, no. 9, pp. 1432–1443, 2019.

[19] Masanori Morise, Fumiya Yokomori, and Kenji Ozawa,“WORLD: a vocoder-based high-quality speech synthesis system
for real-time applications,” IEICE Transactions on Informationand Systems, vol. 99, no. 7, pp. 1877–1884, 2016.
[20] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae, “HiFi-GAN:
Generative adversarial networks for efﬁcient and high ﬁdelity
speech synthesis,” in NeurIPS, pp. 17022–17033, 2020.
[21] Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende,and Max Welling, “Semi-supervised learning with deep generative models,” in NeurIPS, pp. 3581–3589, 2014.
[22] Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess,Xavier Glorot, Matthew Botvinick, Shakir Mohamed, andAlexander Lerchner, “beta-VAE: Learning basic visual concepts
with a constrained variational framework,” in ICLR, 2017.
[23] Manh Luong and Viet Anh Tran, “Many-to-Many Voice Conversion Based Feature Disentanglement Using Variational Autoencoder,” in INTERSPEECH, pp. 851–855, 2021.
[24] Chak Ho Chan, Kaizhi Qian, Yang Zhang, and Mark HasegawaJohnson, “SpeechSplit2.0: Unsupervised speech disentanglement
for voice conversion without tuning autoencoder bottlenecks,” in
ICASSP, pp. 6332–6336, 2022.
[25] Navdeep Jaitly and Geoffrey E Hinton, “Vocal tract length perturbation (VTLP) improves speech recognition,” in ICML, vol. 117,
p. 21, 2013.
[26] Sepp Hochreiter and Jürgen Schmidhuber, “Long short-term
memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780,
1997.
[27] Diederik Kingma and Jimmy Ba, “Adam: A method for stochastic
optimization,” in ICLR, 2015.
[28] Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai,
Rafal Jozefowicz, and Samy Bengio, “Generating sentences froma continuous space,” in SIGNLL, pp. 10–21, 2016.
[29] Akira Kurematsu, Kazuya Takeda, Yoshinori Sagisaka, Shigeru
Katagiri, Hisao Kuwabara, and Kiyohiro Shikano, “ATR Japanese
speech database as a tool of speech recognition and synthesis,”
Speech Communication, vol. 9, no. 4, pp. 357–363, 1990.

[11] Takuhiro Kaneko, Hirokazu Kameoka, Kou Tanaka, andNobukatsu Hojo, “MaskCycleGAN-VC: Learning non-parallelvoice conversion with ﬁlling in frames,” in ICASSP, pp. 5919–
5923, 2021.

[30] Tomoki Toda, Alan W Black, and Keiichi Tokuda, “Voice conversion based on maximum-likelihood estimation of spectral parameter trajectory,” IEEE Transactions on Audio, Speech and Language Processing, vol. 15, no. 8, pp. 2222–2235, 2007.

[12] Geoffrey E Hinton and Ruslan R Salakhutdinov, “Reducing thedimensionality of data with neural networks,” science, vol. 313,
no. 5786, pp. 504–507, 2006.

[31] Donald J Berndt and James Clifford, “Using dynamic time warping to ﬁnd patterns in time series.,” in KDD workshop, vol. 10,
pp. 359–370, 1994.

[13] Diederik P Kingma and Max Welling, “Auto-encoding variational
bayes,” arXiv preprint arXiv:1312.6114, 2013.

[32] Kikuo Maekawa, “Corpus of spontaneous Japanese: Its designand evaluation,” in Workshop on Spontaneous Speech Processing
and Recognition, pp. 7–12, 2003.

[14] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu,
David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua
Bengio, “Generative adversarial nets,” in NeurIPS, pp. 2672–
2680, 2014.
[15] Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, andHsin-Min Wang, “Voice conversion from non-parallel corpora
using variational auto-encoder,” in APSIPA, pp. 1–6, 2016.
[16] Patrick Lumban Tobing, Wen-Chin Huang, Tomoki Hayashi,
Kazuhiro Kobayashi, and Tomoki Toda, “Non-parallel voice conversion with cyclic variational autoencoder,” in INTERSPEECH,
pp. 674–678, 2019.
[17] Hirokazu Kameoka, Kou Tanaka, and Takuhiro Kaneko,“FastS2S-VC: Streaming non-autoregressive sequence-tosequence voice conversion,” arXiv preprint arXiv:2104.06900,
2021.
[18] Hideki Kawahara, Ikuyo Masuda-Katsuse, and Alain
De Cheveigne, “Restructuring speech representations using
a pitch-adaptive time-frequency smoothing and an instantaneousfrequency-based F0 extraction: Possible role of a repetitive
structure in sounds,” Speech communication, vol. 27, no. 3, pp.
187–207, 1999.

[33] Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi,
Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn
Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai, “ESPnet: End-to-end speech processing
toolkit,” in INTERSPEECH, pp. 2207–2211, 2018.