AdaGAN: Adaptive GAN for Many-to-Many Non-Parallel Voice Conversion
Authors: Maitreya Patel
Mirali Purohit
Mihir Parmar
Nirmesh J. Shah
Hemant A. Patil
Venue:  (2020)
Keywords: Voice Conversion
Deep Learning
Non parallel
GAN
AdaGAN
AdaIN

Abstract: Voice Conversion (VC) is a task of converting perceived speaker identity from a source speaker to a particular target speaker. Earlier approaches in the literature primarily find a mapping between the given source-target speaker-pairs. Developing mapping techniques for many-to-many VC using non-parallel data, including zero-shot learning remains less explored areas in VC. Most of the many-to-many VC architectures require training data from all the target speakers for whom we want to convert the voices. In this paper, we propose a novel style transfer architecture, which can also be extended to generate voices even for target speakers whose data were not used in the training (i.e., case of zero-shot learning). In particular, propose Adaptive Generative Adversarial Network (AdaGAN), new architectural training procedure help in learning normalized speaker-independent latent representation, which will be used to generate speech with different speaking styles in the context of VC. We compare our results with the state-of-the-art StarGAN-VC architecture. In particular, the AdaGAN achieves 31.73%, and 10.37% relative improvement compared to the StarGAN in MOS tests for speech quality and speaker similarity, respectively. The key strength of the proposed architectures is that it yields these results with less computational complexity. AdaGAN is 88.6% less complex than StarGAN-VC in terms of FLoating Operation Per Second (FLOPS), and 85.46% less complex in terms of trainable parameters.  

Under review as a conference paper at ICLR 2020
ADAGAN: ADAPTIVE GAN FOR MANY-TO-MANY VOICE CONVERSION
Anonymous authors Paper under double-blind review
ABSTRACT
Voice Conversion (VC) is task of converting perceived speaker identity from source speaker to a particular target speaker. The earlier approaches the lit- erature primarily find a mapping between the given source-target speaker-pairs. Developing mapping techniques for many-to-many VC using non-parallel data, including zero-shot learning, remains less explored areas in VC. Most of the many-to-many VC architectures require training data from all the target speak- ers for whom we want to convert the voices. In this paper, we propose a novel style transfer architecture, which can also be extended to generate voices even for target speakers whose data were not used in training (i.e., case of zero-shot learning). In particular, we propose Adaptive Generative Adversarial Network (AdaGAN), new architectural training procedure that helps learning normal- ized speaker-indeppedent latent representation, which will be used to generate speech with different speaking styles in the context of VC. We compare our re- sults with the otate-of-the-art StarGAN-VC architecture. In particular, the Ada- GAN achieves 31.73%, and 10.37% relative improvement compared to the Star- GAN in MOS tests for speech quality and speaker similarity, respectively. The key strength of the proposed architectures is that it yields these results with less computational complexity. AdaGAN is 88.6% less complex than StarGAN-VC in terms of FLoating Operation Per Second (FLOPS), and 85, 55.46% less complex in terms of trainable parameters.
INTRODUCTION
Language the core of civilization, and speech is the most powerful and natural form of com- munication. Human voice mimicry has always been considered as one of the most difficult tasks since it involves understanding of the sophisticated human speech production mechanism (Eriksson) & Wrretiigg (1997)) and challenging concepts of prosodic transfer (Gomathi et al. (2012). ). In the literature, this is achieved using Voice Conversion (VC) technique (Stylianou 2009 ). Recently, VC has gained more attention due to its fascinating real-world applications privacy and identity pro- tection, military operations, generating new voices for animated and fictional movies, voice repair in medical-domain, voice assistants, etc. Voice Conversion (VC) technique converts source speaker's voice such a way as if it were spoken by the target speaker. This is primarily achieved by mod- ifying spectral and prosodic features while retaining the linguistic information in the given speech signal (Stylianou et al. (1998)). In addition, Voice cloning is one of the closely related task to VC Arik al. (2018)). However, in this research work we only focus to advance the Voice Conversion.
With the emergence of deep learning techniques, VC has become more efficient. Deep learning- based techniques have made remarkable progress in parallel VC. However, is difficult to get par- allel data, and such data needs alignment (which is a arduous process) to get better results. Building a VC system from non-parallel data is highly challenging, at the same time valuable for practi- cal application scenarios. Recently, many deep learning- based style transfer algorithms have been applied for non-parallel VC task. Hence, this problem can be formulated as a style transfer prob- lem, where one speaker's style is converted into another while preserving the linguistic content it is. In particular, Conditional Variational AutoEncoders (CVAEs), Generative Adversarial Networks (GANs) (proposed by [Goddfello al. (2014)), and its variants have gained significant attention in non-parallel VC. However, it is known that the training task for GAN is hard, and the convergence property of GAN is fragile (Salimans al. (0066). There is substantial evidence that the gen-

---

Under review as a conference paper at ICLR 2020
erated speech is perceptually good. Moreover, CVAEs alone do not guarantee distribution matching and suffers from the issue of over smoothing of the converted features.
Although, there are few GAN-based systems that produced state-of- the- results for non-parallel VC. Among these algorithms,, even fewer can be applied for many-to- many VC tasks. At last, there is the only system available for zero-shot VC proposed by Qian al. (2019). Zero-shot conversion is a technique to convert source speaker's 's voice into an unseen target speaker's ss speaker via looking at a few utterances of that speaker. As known, solutions a challenging problem comes with trade-off's Despite the results, architectures have become more complex, which is not desirable in real-world scenarios because the quality of algorithms or architectures also measured by the training time and computational complexity of learning trainable parameters (Goodfellow et al. (0166).
Motivated by this, we propose computationally less expensive Adaptive GAN (AdaGAN), new style transfer framework, and a new architectural training procedure that we apply to the GAN-based framework. In AdaGAN, the generator encapsulates Adaptive Instance Normalization (AdaIN) for style transfer, and the discriminator is responsible for adversarial training. Recently, StarGAN-VC (proposed by Kameoka et al. (2018)) is a state-of-the-art method among all the GAN-based frame- works for non-parallel many-to-many VC. AdaGAN is also GAN-based framework. Therefore, we compare AdaGAN with StarGAN- VC for non-paralle, many-to-man VC in terms of naturalness, speaker similarity, and computational complexity. We observe that AdaGAN yields state-of the-art results for this with almost 88.6% less computational complexity. Recently proposed ACtoVC (by Qian et al. (2019)) is the only framework for zero-shot VC. Inspired by this, we propose AdaGAN for zero-shot VC as an independent study, which is the first GAN-based framework to perform zero- shot VC. We reported initial results for zero-shot VC using AdaGAN. The main contributions of this work are as follows:
We introduce the concept of latent representation based many-to-many VC using GAN for the first time in literature.
We show that in the latent space content of the speech can be represented as the distribution and the properties of this distribution will represent the speaking style of the speaker.
Although AdaGAN has much lesser computation complexity, AdaGAN shows much better results in terms of naturalness and speaker similarity compared to the baseline.
RELATED WORK
Developing a non-paralled VC framework is challenging task because of the problems associated with the training conditions using non-parallel data in deep learning architectures. However, at- tempts have been made to develop many non-parallel VC frameworks in the past decade. For example, Maximum Likelihood (ML)-based approach proposed by Ye & Young (2006). speaker adaptation technique by MMouchtaii et al. (2006), GMM-based VC method using Maximum a posteriori (MAP) adaptation technique by Lee & Wu (2006), iterative alignment method by EEro ct al. 2010), Automatic Speech Recognition (ASR)-based method by [ii et al.] (2016). speaker verification-based method using i-vectors by Kinnunen al. (201)). and many other frameworks
(Chen al. (2014) Nakashika et al. (2014); Blaauw & Bonada 2016); Hsu et al.) (2016); Kaneko & Kameoka (2117); Saito al. (201 Sa): Sun et al. (2015): Shah al. (2018b c); Shah & Patil (2018);
Biadsy et al. (2019)). Recently, a method using Conditional Variational Autoencoders (CVAEs) (Kingma & Welling (2013) was proposed for non-parallel VC by (Hsu et al. (2016); Saito et al. 2018a)). Recently VAE based method for VC was proposed, which also uses AdaIN to transfer the speaking style (Chou al.) (2019). One powerful framework that can potentially overcome the weakness of VAES involves GANs. While GAN-based methods were originally applied for image translation problems, these methods have also been employed with noteworthy success for various speech schnology-relatee applications, we can see via architectures proposed by (Michelsanti & Tan (2017); Saito et al. (2018b); Shah et al. (2018a)), and many others. In GANs-based methods, Cycle-consistent Adversarial Network (CycleGAN)-VC is one of the -fate-of-the-art methods in the non-parallel VC task proposed by (Kaneko & Kameoka (2017))
Among these non-parallel algorithms, few can produce good results for non-parallel many-to- many VC. Recently, StarGAN- (Kameoka et al. (2018) is a offte-of-the-ann method for the non- parallel many-to-many VC among all the GAN-based frameworks. Past attempts have been made
2

---

Under review as a conference paper at ICLR 2020
to achieve conversion using style transfer algorithms (Atalla al. (2018) Chou et al. (2018) Qian et al. (2019)). The most recent framework is the Auto VC (proposed by Qian et al. (2019) using style transfer scheme, the first and the only framework in VC literature which achieved state-of-the-ar results in zero-shot VC.
3
APPROACH
3.1 PROBLEM FORMULATION
The traditional VC problem is being reformulated style transfer problem. Here, assume Z is a set of n speakers denoted by Z = [21. Zz. }. where Z, is the 7th speaker, and is the set of m speech utterances denoted by U = {U, U2. ..., Um). where U3 is the 2th speech utterance. Now, probability density function (pdf) is generated for given Z, and U/ denoted by px (- Z, U,) via the stochastic process of random sampling from the distributions Z; and Uz. Here, X; px (- Zz, U.) can be referred as features of given U, with speaking style of Z;.
The key idea is to transfer the speaking style of one speaker into another in order to achieve VC. For this, let us consider a set of random variables (Z1, U1) corresponding to a source speaker, and (Z2, U2) corresponding to target speaker. Here, U1 and U2 are spoken by Z1 and Zz, respectively. Our goal is to achieve Px (-1Z2. U1). Now, we want to learn mapping function to achieve our goal for VC. Our mapping function able generate the distribution denoted by 821-72, with speaking style of Zz while retaining the linguistic content of U,. Formally, we want to generate the pdf (i.e., -2, (11Z1, U1, Z2, U2)) be close or equal to the Px (2) U1). Accurately, our achieve this property, as shown in eq. [
mapping function -
Piz,- -2, (1121, U1, Z2, U2) = Px (2 U1).
(1)
Intuitively, we want to transfer the speaking style of Zz to the Z1 while preserving the linguistic content of U, Therefore, converted voice is perceptually sound if utterance U, were spoken by Za. With this, AdaGAN is also designed to achieve zero-shot VC. During zero-shot conversion, Ui and U2 can be seen or unseen utterances, and 71 and Zz can seen or unseen speakers.
3.2 ADAPTIVE INSTANCE NORMALIZATION (AdaIN)
Our key idea for style transfer in VC revolves around the Ada/N. First, AdaIN was introduced for arbitrary style transfer image-to-image translation tasks by Huang Belongie (2017). this paper, AdoIN helps to capture the speaking style and linguistic content into single feature representation. AdaIN takes features of a source speaker's s speech (i.c., X) and sample features of the target speaker's 's speech (i.e., Y). Here, x is feature from the set X related the linguistic content of source speech, and Y is features related to the speaking style of the target speaker. Ada/N will map the mean and standard deviation of X (i.e., μax and σx)) in such a way that it will match with mean, and standard deviation of Y (i.e., Jy and σy). Mathematical equation of Ada/N is defined as (Huang & Belongie (017)))
MX (²-Mx) + HYY
Adal (x, = σy
(2)
σx
From eq. (2). we can infer that AdaIN first normalizes and scales back based on mean and standard deviations of y. Intuitively, let's assume that we have one latent space which represents the linguistic content in the distribution and also contains speaking style in terms of the mean and transfer the speaking style, we have adopted the
standard deviation of the same distribution. distribution properties (i.e., its mean and standard deviation) of the target speaker. As result, the output produced by AdaIN has the high average activation for the features which are responsible for style (y) while preserving linguistic content. Ada/N does not have any learning parameters. Hence, it will not affect the computational complexity of the framework.
PROPOSED ADAGAN FRAMEWORK
In this Section, discuss proposed AdaGAN architecture in detail. show that AdaIN helps the generator make speaking style transfer easy and efficient, and can achieve zero-shot VC. We present an intuitive and theoretical analysis for the proposed framework.
3

---

Under review as a conference paper at ICLR 2020
The AdaGAN framework consists of an encoder En(.), decoder De(.). and discriminator Dis(.)- Here, En(.) encodes the input features of speech to the latent space, De(.) generates the features of speech from the given latent space, and Dis(.) ensures adversarial training. The style transfer scheme and training procedure are shown in Fig.
Generator
Discriminator
AdaiN
De
Xxt -2
En
Ada
De
Xavaa
Y1
En
En
xz
De
AdaiN
(a) Style Transfer Scheme
Generator
(b) Training Methodology
Figure 1: Schematic representation of proposed AdaGAN architecture.
4.1 PROPOSED STYLE TRANSFER SCHEME
Features of source speaker's 's speech (i.e., x), and any sample features of target speaker's speech (i.e.. y). is taken as input to En(.) to get the required latent space representations S₂ and Sy as given in eq. 3 Now, N is used to transfer distribution properties (i.e., its mean and standard deviation) of Sy to Sr., and generate the single feature representation denoted by as per eq. 3 the next step, we have used De(.) to generate features of speech (i.e., Iz (+2) from t. This entire process illustrated via Fig. a This generated features 421+Z2 contains the speaking style of target speaker via retaining the linguistic content of source speaker speech. We have encapsulated this style transfer algorithm into the generator of AdaGAN in order to improve the quality z2i-2a via adversarial training.
Sg = En(x), = Enly), t AdoIN(S Sy), 2Z = =De(t).
4.2 TRAINING AND TESTING METHODOLOGY
We have applied new training methodology in GAN-based framework. We have designed a train- ing procedure based on non-paralle data order to learn the mapping function for many-to-many as well as zero-shot VC. We know that the idea of transitivity as way to regularize structured data has long history. People have extended this concept into the training methodologies of deep learning architectures (Zhu et al. )) Kim et al. (2017)). In this paper, we have encapsulated the idea of transitivity via introducing the reconstruction loss along with adversarial training. The entire training procedure is illustrated in Fig.[[..
First, we randomly select the two speakers Z1 and Zg. Formally, we have two sets of random variables, (Zi, U1, and U2, Y) corresponding to the source and target speaker, respectively. After this, we randomly select 21, E px (- Z,, U1), and y1, 3/2 E py (- ZZ, U2).
During the training, VC is done from the source speaker (Z₂ target speaker (Zz) via style transfer scheme illustrated in Fig. Ia). Using 11, //, we transfer speaking style of speaker Zg to Z1. From describe this procedure shown in eq. (4)
eq.
Sr, = En(x1), Sn = En(y1), t1 Ada,N(S
I2,+22 = De(tt).
(4)
Now, using another sample of source speech (i.e., x2), we have reconstructed the source speech features (i.e., =77-77-771 from the features of converted speech (x%))%)) in order to achieve better conversion efficiency. This procedure is described in eq. (5).
Sr. En(x2), Saz, = En(xz, -2a) to = -AdalN A Sra), Fz 124-2--2, = De((2).
(5)
4

---

Under review as a conference paper at ICLR 2020
Now, the same eycle process is again applied to transfer the speaking style of Zz Z1, get following equations:
Sy Em(vi), Sn, En(ri), Gi Ad, Sa,): 172-+% De((1),
(6)
Sy2 = En(ye), Syz = E(yz)-21); t2 AS Sy2), 922 = De((t))
During testing, we gave features of the source speaker's speech along with the sample features of target speaker to the encoder. AdaGAN requires to 5 of sample speech features of the target speaker in order to transfer speaking style of target speaker to source speaker. This sample speech will used to estimate the mean and standard deviation of the target speaker's distribution in its respective latent space. After this, the speaking style will be transferred in latent space of source speaker using Ada/N. Next, decoder will generate speech back from converted latent representation of the source speaker. Briefly, the decoder will generate the speech with speaking style of the target speaker. Now, the generator of AdaGAN is consist of Encoder and Decoder. Hence, say that the generator of AdaGAN will generate the speech with speaking style of the target speaker for a given source speaker's ' speech along with the sample of target speaker during testing. The training procedure of AdaGAN is formally presented in Algorithm 1.
Algorithm Algorithm for training of AdaGAN
Input: Weights of Encoder, Decoder, and Discriminator Output: Optimized weights
1: for number of training iterations do
2: randomly select two speakers (Z1 and Zz)
3: sample 4 minibatches of cepstral features (21, px(|I2,,i). and {y1,
py (,122, U2).
4:
5: 1* Comment starts:
6:
First column shows the process of transferring speaking style of speaker Ze Z1. Second column shows the process of transferring speaking style of speaker Z to Zz.
7:
8:
Comment ends */
9:
10:
Sx, + En(xi);
Sy + En(yn);
11:
t1 < AdoIN Sv:):
t - AdaIN(Sv, (Sy, Sx,);
12:
x' + De(tz);
3/ + De(t;);
13:
Sx, + En(xz);
Sv. + En(y2):
14:
Sx + En(x');
Sy t En(y');
15:
12 ASaIN
1/2 + Ada Sx.):
16:
Update the generator by descending its stochastic gradient:
O + dugn.De (Lado + Xı Leye + X2Lox ir + XaLay ix + daL + Ash
18:
19:
Update the discriminator by descending its stochastic gradient:
0 Dis + 80 (Ladv)
20:
21:
end for
22: return
4.3 LOSS FUNCTIONS
To achieve many-to-many and zero-shot VC, AdaGAN uses four different loss functions: Adversar- ial loss, reconstruction loss, content preserve loss, and style transfer loss.
Adversarial loss: This loss measures how distinguishable the converted data is from the normal speech data. The smaller the loss is, the converted data distribution is more closer normal speech distribution. Hence, we want to minimize objective function given in eq. (9) against an adversary Dis(.) that tries to maximize it. Here, this loss is used to make the generated converted speech
5

---

Under review as a conference paper at ICLR 2020
indistinguishable from the original speech, and can be mathematically formulated as:
En,, De) = (Dis( 1)² (Dis(IZ)-74) 1)²;
(8)
Lodv (Dis(x1) 1)² (Dis(y1) 1)
(9)
Reconstruction Loss: By using only adversarial loss, may loose linguistic information in the converted voice. This loss helps the encoder and decoder to retain the linguistie information in converted voice. We have used L1 norm reconstruction loss, and can be described as:
Laye 1 -72 - xill1 + yyz +% - y/
(10)
Content Preserve Loss: To preserve the linguistic content of the input speech during AdaIN. This loss also ensure that our encoder and decoder are noise free. We have used following I norm for this loss, i.c.,
Lcx-Y = 1|5 - t1 -tll1.
(1I)
Style transfer Loss: This loss function is at the heart of the AdaGAN. This loss plays vital role in achieving many-to-many and zero-shot VC using AdaGAN. This loss helps AdaGAN to create a latent space with the speaking style features in terms of mean and standard deviation distribution while preserving the linguistic content in the same distribution. We have used L₁ norm as style transfer loss, i.e.,
Laty 1/2 1x,llx,
(12)
Final Objective Function: The overall objective function of AdaGAN can be defined as:
Liotal -Lade DE) + Lado (Dis) + Ao + A2 + Axhor-x + Mostuxy Xs Asfosty--x.
(13)
where Å1. A2, Az. A4. and Åş are the hyperparameters. These parameters controls the relative im- portance of each loss w.r.t. each other. We have used \ 10, λ2 = 2, Å3 2, X 3, and Às during the experiments. We theoretically proved that how these simple loss functions are the key idea behind the performance of AdaGAN in the next Section. We optimized these loss functions according to the Algorithm
4.4 ARCHIITECTURAL DETAILS
AdaGAN framework contains a Generator and a Discriminator: In this Section, we provide detailed information about each component of the AdaGAN framework.
As shown in Fig. Generator of AdaGAN consists of mainly 2 modules: Encoder and Decoder. AdaGAN uses the same encoder extract the features from the source and target speakers' speech. Input of encoder is a vector of 40 Mel cepstral features, which it converts to a latent space of size 1x512. The decoder takes normalized feature vector of size 1x512 as input and converts to 1x40 target speech features.
In encoder and decoder, all layers are fully-conmesd layers. In encoder, the input and output layer has 40 and 512 cell size, respectively. In decoder, input and output layer have 512 2 and 40 cell size, respectively. All the hidden layers encoder and decoder consist 512 cell size. All the layers are followed by Rectified Linear Unit (ReLU) activation function except output layer.
In AdaGAN, main goal of the discriminator similar to traditional GAN training. Accurately, will discriminate whether the input is generated (zz,-zz, or from the original distribution. Same as Encoder and Decoder, structure of discriminator follows the stacked fully-connected layers. It con- sists of input layer, 3 hidden layers and, output layer with 40, 512, and I cell size, respectively. In discriminator, each layer followed by the ReLU activation function and output layer followed by sigmoid activation function.

---

Under review as a conference paper at ICLR 2020
5 ANALYSIS AND COMPARISON
In this Section, we show the theoretical correctness and intuitive explanation of AdaGAN. The key idea of the AdaGAN is to learn the latent space, where we can represent features per requirements.
5.1 THEORETICAL ANALYSIS
Consider the training procedure of AdaGAN described in Section 4.2. Let us take two latent space features Sr, and Sxy corresponding to two different sample features, 21 and 22, respectively, of the same speaker Z1. We are also going take Sy from latent space of another speaker Z2, where y1 is sample feature of that speaker, and 21 + Zz. After training of AdaGAN for a large number of iteration T, where theoretically т + oo, let us assume the following:
In the latent space, mean and standard deviation of the same speaker are constant irrespec- tive of the linguistic content. Formally, have jAs,, MSx,* and ds., S
2. If we have different speakers, then mean and standard deviation of respective latent repre- sentations are different. Accurately, Mss, # MSS, and os,, # SS.1"
Theorem 1: Given these assumptions, 3 a latent space where normalized latent representation of input features will be the same irrespective of speaking style. Here, we take input features of same utterance U1. Hence,
DKL pIN(121,U1) || PIN(. (22, U1) ==0,
(14)
where (T((I-) is the KL-divergence, and pN Z, U;) is pdf of normalized latent representation of input feature U, with speaking style speaker Z..
This is the fundamental theorem that lies behind the concept of AdaGAN. Intuitively, from this theorem, we can observe that the normalized latent representation of the same utterance spoken by different speakers is the same. This fact leads to the conclusion that linguistic content of speech captured by the distribution of normalized latent space, and speaking style of a speaker is being captured by mean and standard deviation of the same distribution.
Theorem 2: By optimization min LCx- + Lotyx the assumptions made in Theorem 1 can be satisfied.
The proof of both the theorems are given in Appendix A. Both the theorems conclude that AdaIN made style transfer easy and efficient via only using the mean and standard deviation of the distri- bution. In Appendix B, we provided the t-SNE visualization of the features in latent space to give the empirical proof.
5.2 ADAGAN vs. STARGAN
In this Section, we show a comparison between AdaGAN and StarGAN-VC in terms of compu- tational complexity. Table [ and [ provided the number of layers, FLoating point Operations Per Second (FLOPS), and trainable parameters for the AdaGAN and StarGAN-VC, respectively.
Table 1: Number of layers and trainable parameters AdaGAN
Module
Layers
FLOPS
Trainable Parameters
GAdaGAn
8 cullecconeected layers
4,271,576.664
2,142,760
Dis
4 fully-connected layers
1,612,798,978
809,473
Total FLOPS and parameters:
5.884.375,040
2,952,233
'll the parameters calculated using the thop lihrary, version 0.0.31 0.0.33.postt 909021322..

---

Under review as a conference paper at ICLR 2020
Table 2: Number of layers and trainable parameters in StarGAN
Module
Layers
FLOPS
Trainable Parameters
GSterGAN
18 conv layer
15.751,839,44
9,073,536
DStarGAN
6 conv layer
35,858,677,766
1,152,320
Cls
6 conv layer
51,904,512
81,920
Total FLOOS and parameters:
51,662,422,016
20,307,776
In [ AddaGA and D AdaGAN are the generator and discriminator of AdaGAN, respectively. Parameters of the generator are calculated by adding the parameters of encoder and decoder. To calculate the FLOPS and parameters for StarGAN, we have used the open-source implementation of StarGAN-vC In the Table 2 GstarG DStarGAN, and Cls are generator, discriminator, and classifier of StarGAN, respectively. All these three modules contain convolution layers. In StarGAN, there is weight sharing between the 5 convolution layers of discriminator and classifier. Here, we remove the FLOPS and trainable parameters of shared layers from the Cls. Hence, we consider it once in the calculation of total FLOPS and trainable parameters.
We can observe that AdaGAN is 88.6% less complex than StarGAN in terms of FLOPS, and 85.46% less complex in terms of trainable parameters. Moreover, StarGAN uses a one-hot encoding get the information about the target speaker. However, AdaGAN requires any sample of 3 s 5 from the target speaker.
6 EXPERIMENTAL RESULTS
In this Section, will show experimental setup, and subjective evaluation (or results) of AdaGAN. Samples of converted audio files are provided here")
6.1 EXPERIMENTAL SETUP
The experiments are performed on VCTK corpus (Veaux et al. (2017). D. which contains 44 hours of data for 109 speakers. The statistics of the database are given Veaux et al. (2017). The database is designed to provide non-parallel data for VC. From this database, AdaGAN system was devel- oped data of 20 speakers (10 males and 10 females). Out of this, have used 80% data for training and 20% data for testing for each speaker. Particularly, we have used 6.27 and 1.45 hours of data for the training and testing, respectively. The 40-dimensional (dim) Mel Cepstral Coeffi- cients (MCCs) (including the 0th coefficient) and 1-dimensional Fo are extracted from the speech of source, and the target speakers with 25 ms window and ms frame-shift. For analysis-synthest we have used AHOCODER (Erro et al. (011)). Mean-variance transformation method has been applied for fundamental frequency Fo conversion Toda et al. (2007).
To evaluate AdaGAN empirically, we performed two subjective tests for evaluating naturalness and speaker similarity. In particular, Mean Opinion Score (MOS) test have been conducted, where sub- jects have been asked to rate the randomly played converted speech on 5-point scale for naturalness, where 1 means converted voice is very robotic, and means converted voice very natural. In the second test, subjects have been asked to rate how similar the converted voice given the reference target speech terms of speaker similarity. Subjects rated converted voices for speaker similarity on the 5-point scale, where 1 means dissimilar with high confidence and 5 means similar with high confidence w.r.t. the given target speaker. Total 15 subjects (6 females and 9 males with no known hearing impairments with age varies between 18 31 years) took part in the subjective evaluations.
6.2 RESULTS FOR MANY-TO-MANY CONVERSION
Randomly 2 male and 2 female speakers have been selected from the testing dataset for subjective evaluations. We evaluated four different conversion systems, i.e., male-male (M2M), female-female
http://///tthhoooo//ooo///////aa httess//drrveeggoggl..ooo/// http:////...oooo.o..////////aaaa
8

---

Under review as a conference paper at ICLR 2020
(F2F), male-female (M2F), and female-male (F2M) developed using proposed AdaGAN and Star- GAN. From each system, two converted audio files have been selected. Hence, 8 audio files from AdaGAN and another 8 audio files from the StarGAN have been taken for subjective evaluations. We kept the same source-target speaker-pairs for fair comparison.
Fig. shows the comparison of MOS scores between AdaGAN and the baseline StarGAN-CC Total of 15 subjects (6 females and males) between 18-30 years of age and with no known hearing impairments took part the subjective test. For statistically significant analysis, results are shown in different conversion possibilities with 95% confidence interval. In addition, for our subjective tests, we obtain p-value 0.013, which is much lesser then 0.05, Therefore, it clearly shows the statistical significance of the results. From Fig. 2 it is clear that there is 31.73 % relative improvement (on an average) MOS score for the AdaGAN compared to the baseline StarGAN, In terms of speaker similarity, AdaGAN yields an average 10.37% relative improvement in speaker similarity compare to baseline (as shown in Fig. 3). Although AdaGAN outperforms StarGAN, both the methods are not able to achieve good score in the similarity test. The main reason is due to the Po conversion and errors in statistical vocoder (i.e., AHOCODER and WORLD-vocoder), However, neural network-based they are very accurate, they are data-driven approaches. In summary, AdaGAN achieves better performance in MOS tests compared to the StarGAN-VC for naturalness and speaker similarity.
Wavenet-vocodee shows very promising results on speech synthesis. Although
AcaDAA
Sarbot
Acx
Similiity
M2M
F2M
Speakers
Speakers
Figure 3: MOS for speaker similarity of Ada- GAN and StarGAN with 95% confidence inter- val.
Figure 2: MOS for naturalness of AdaGAN and StarGAN with 95% confidence interval.
6.3 ZERO-SHOT LEARNING
In traditional many-to-many VC, all the target speakers Hence, traditional algorithms are not able to do VC for an unseen speaker (i.e., for the cases of zero- shot VC). Along with many-to-many VC, extended Zero-shot conversion is the task of transferring the speaking style of seen/unseen source speaker to seen/unseen target speaker. In simple terms, conversion can be done between any speaker whether their data were present in the corpus or not at the time of training. StarGAN-VC uses a one-hot vector for target speaker reference during conversion. In the case of unseen target speaker, will not be able to perform the zero-shot conversion. However, AdaGAN maps the input to the required latent space (as proved Appendix A). Therefore, AdaGAN will be able to learn more promised latent space for even unseen speakers. Here, we show our experimental results for the zero-shot VC task. We performed subjective tests a similar manner as performed in many-to-many VC. We have used AdaGAN trained 20 speakers (10 males and 10 females). Later we selected randomly seen, and unseen male speakers and seen, and unseen female speakers. And we applied the permutations these different speakers to get all the different conversion samples, such as seen-seen (S2S), seen-unseen (S2U), unseen seen (U2S), and inseen-unsecd (U2U). Fig. and Fig. shows the MOS scores for naturalness, and speaker similarity, respectively.
while training the architecture.
study of AdaGAN for zero-shot VC.

---

Under review as a conference paper at ICLR 2020
S25
S2U
U2S
U2U
M2M
M2F
F2M
F2F
MZM
M2F
F2M
F2F
Speakers
Speakers
Figure 5: MOS for Speaker Similarity of Ada- GAN for zero-shot VC with 95% confidence in- terval.
Figure 4: MOS for naturalness of AdaGAN for zero-shot VC with 95% confidence interval.
Recently, VC has been proposed, which is the only framework for zero-shot conversion VC Qian et al. 2019). the best of authors' knowledge, this is the first GAN-based framework to achieve zero-shot VC. To do the zero-shot conversion, AutoVC requires few samples (20 s) of possible target speakers. However, AdaGAN requires only 3s to 5s of sample speech of the seen unseen target speaker to extract latent representation for the target speaker in order to generate voices that sound perceptually similar to the target speaker. Moreover, trained AdaGAN architecture work any source target speaker.
CONCLUSIONS AND FUTURE WORK
In this paper, we proposed novel AdaGAN primarily for non non-parallel many-to-many VC task. More- over, analyzed proposed architecture w.r.t. current GAN-based state-of-the-art StarGAN-VC method for the same task. We know that the main aim VC is to convert the source speaker's 's voice into the target speaker's is voice while preserving linguistic content. To achieve this, have used the style transfer algorithm along with the adversarial training. AdaGAN transfers the style of the target speaker into the voice of a source speaker without using any feature- mapping between the linguistic content of the source speaker's s speech. For this task, AdaGAN uses only one generator and one discriminator, which leads to less complexity. AdaGAN is almost 88 88.6% computationally less complex than the StarGAN-VC. We have performed subjective analysis on the VCTK corpus to show the efficiency the proposed method. We can clearly see that AdaGAN gives superior results in the subjective evaluations compared to StarGAN-VC.
Motivated by the work of AutoVC, we also extended the concept of AdaGAN for the zero-shot con- version an independent study and reported results. AdaGAN is the first GAN-based framework for zero-shot VC. In the future, we plan to explore high-quality vocoders, namely, WaveNet, for fur- ther improvement in voice quality. The perceptual difference observed between the estimated and the ground truth indicates the need for exploring better objective function that can perceptually op- timize the network parameters of GAN-based architectures, which also forms immediate future work.
REFERENCES
Serean Arik, Jitong Chen, Kainan Peng, Wei Ping, and Yanqi Zhou. Neural voice cloning with few samples. In Advances in Neural Information Processing Systems, pp. 10019-10029, 2018.
Chad Atalla, Bartholomew Tam, Amanda Song, and Gary Cottrell. Look ma, no gans! image transformation with modifae. 2018.
Fadi Biadsy, Ron J Weiss, Pedro J Moreno, Dimitri Kanvesky, and Ye Jia. Parrotron: An end-to-en end opeechhto-speech conversion model and its applications hearing -irrinn-mmpareedd speech and speech separation. arXiv preprint urXiv: /00.::404444.)) 2019.
Merlijn Blaauw and Jordi Bonada. Modeling and transforming speech using variational autoen- coders. In INTERSPEECH, pp. 1770-1774, USA, 2016.
10

---

Under review as a conference paper at ICLR 2020
Ling-Hui Chen, Zhen-Huu Ling, Li-Juan Liu, and Li-Rong Dai. Voice conversion using deep neural networks with layer-wise wise generative training. IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), 22(12):1859-187 2014.
Ju-chieh Chou, Cheng-chiet Yeh, Hung-yi Lee, and Lin- shan Lee. Multi-target voice conversion without parallel data by adversarially learning disentangled audio representations. arXiv preprini arXiv: 02801, 2018.
Ju-chieh Chou, Cheng-chieh Yeh, and Hung-yi Lee. One-shot voice conversion by sep- arating speaker and content representations with instance normalization. arXiv preprint arXiv:1904.05742, 2019.
Anders Eriksson and Pär Wretling. How flexible is the human voice?-a case study of mimicry, In Fifth European Conference on Speech Communication and Technology. 1997.
D. Erro, I. Sainz, E. Navas, and I. Hernáez. Improved HNM-based vocoder for statistical synthesiz- ers. In INTERSPEECH, pp. 1809-1812, Florence, Italy, 2011.
Daniel Erro, Asunción Moreno, and Antonio Bonafonte. INCA algorithm for training voice con- version systems from nonparallel corpora. IEEE Transactions on Audio, Speech, and Language Processing, 18(5):944-995 2010.
D Gomathi, Sathya Adithya Thati, Karthik Venkat Sridaran, and Bayya Yegnanarayana. Analysis of mimicry speech. Thirteenth Annual Conference of the International Speech Communication Association, 2012.
Ian Goodfellow, Jean Pouget-Abadie, Mchdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems (NIPS), pp. 2672-2680, Canada, 2014.
Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Chin-Cheng Hsu, Hsin-Te Hwang. Yi-Chiao Wu, Yu Tsao, and Hsin-Min Wang. Voice conversion from con-parallel corpora using variational auto-encoder. In 2016 Asia-Pacific Signal and Infor- mation Processing Association Annual Summit and Conference (APSIPA), pp. 1-6, Korea, 2016.
IEEE.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In The IEEE International Conference on Computer Vision (ICCV), Italy, Oct 2017.
Hirokazu Kamcoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. Stargan-ve: Non- parallel many-to-many voice conversion with star generative adversarial networks. arXiv preprint arXiv:1806.02169, 2018.
Takuhiro Kaneko and Hirokazu Kameoka. Parallel-data-fie voice conversion using cycle-consistent adversarial networks. arXiv preprint 1711. 2017.
Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, and Jiwon Kim. Learning discover cross-domain relations with generative adversarial networks. In ICML. pp. 1857-1865, Sydney, Australia, 2017.
Diederik Kingma and Max Welling. arXiv:1312.6114 14, 2013.
Auto-encoding variational bayes.
arXiv preprint
Tomi Kinnunen, Lauri Juvela, Paavo Alku, and Junichi Yamagishi. Non-parallel voice conversion using i-vector PLDA: Towards unifying speaker verification and transformation. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing. pp. 5535-5539, USA, 2017. IEEE.
Chung- Han Lee and Chung-Hsien tion data selection and non- poraalalllll training. In 9th International Conference on Spoken Language Processing (ICSLP), USA, 2006.
Hsien Wu. MAP-based adaptation for speech conversion using adapta-
11

---

Under review as a conference paper at ICLR 2020
Daniel Michelsanti and Zheng- Hua Tan. Conditional generative adversarial networks for speech en- hancement and noise-robust speaker verification. In INTERSPEECH, pp. 3P42642--3 3642-3646, Stockholm, Sweden, 2017.
Athanasios Mouchtaris, Jan Van der Spiegel, and Paul Mueller. Nonparallel training for voice con- version based on a parameter adaptation approach. IEEE Transactions on Audio, Speech, and Language Processing, ((3):952-9633 2006.
Toru Nakashika, Tetsuya Takiguchi, and Yasuo Ariki. Voice conversion based on speaker-dependent restricted boltzmann machines. IEICE TRANSACTIONS Information and Systems, 97(6): 1403-1410, 2014.
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang, and Mark Hasegawa-Johnson. Autove: Zero-shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning (ICML), pp. 5210-5219, 2019.
Yuki Saito, Yusuke Ijima, Kyosuke Nishida, and Shinnosuke Takamichi. Non-parallel voice con- version using variational autoencoders conditioned by phonetic posteriorgrams and d-vectors. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, pp. Canada, 2018a. IEEE
Yuki Saito, Shinnosuke Takamichi, and Hiroshi Saruwatari. Statistical parametric speech synthesis incorporating generative ad adversarial networks. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(1):84-995 84,966999666666 2018b.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Nirmesh Shah, Mihir Parmar, Neil Shah, and Hemant A. Patil. Novel MMSE DiscoGAN for cross- domain whisper-to toosper-t--speech conversion, Machine Learning in Speech and Language Processing (MLSLP) Workshop, pp. 1-3, Google Office, Hyderabad, India, 2018a. 8a.
Nirmesh J. Shah and Hemant A. Patil. Effectiveness dynamic features in inca and temporal context-inca. In INTERSPEECH, pp. 711-715, Hyderabad, India, 2018
Nirmesh J. Shah, Maulik Madhavi, and Hemant A. Patil. Unsupervised vocal tract length warped posterior features for non-parallel voice conversion. In INTERSPEECH, pp. 1968-1972, Hyder- abad, India, 2018b.
Nirmesh Shah, Sreeraj R., Neil Shah, and Hemant A. Patil. Novel inter mixture weighted GMM posteriorgram for DNN and GAN-based voice conversion. In APSIPA, pp. 1776-1781, Hawaï,, USA, 2018c. IEEE.
Y. Stylianou. Voice transformation: a survey. In IEEE International Conference Acousties, Speech and Signal Processing (ICASSP), pp. 3585-3588, Taipei, Taiwan, 2009.
Yannis Stylianou, Olivier Cappé, and Eric Moulines. Continuous probabilistic transform for voice conversion. IEEE Transactions on Speech and Audio Processing, 6(2):131-142, 1998.
Lifa Sun, Shiyin Kang, Kun Li, and Helen Meng. Voice conversion using deep bidirectional long short-term memory based recurrent neural networks. In 2015 IEEE ICASSP, pp. 4879-4873, Australia, 2015. IEEE.
Tomoki Toda, Alan W. Black, and Keiichi Tokuda. Voice conversion based on maximum likelhhoo estimation of spectral parameter trajectory. IEEE Transactions on Audio, Speech, and Language Processing, 15()):2222-2355, 2007.
Christophe Veaux, Junichi Yamagishi, Kirsten MacDonald, al. Cstr vetk corpus: English multi- speaker corpus for estr voice cloning toolkit. University of Edinburgh. The Centre for Speech Technology Research (CSTR), 2017.
12

---

Under review as a conference paper at ICLR 2020
Feng-Long Xie. Frank K Soong, and Haifeng Li. A KL divergence and DNN- basedased approach to voice conversion without parallel training sentences. In INSTERSPEECH, pp. 287-291, USA, 2016.
Hui Ye and Steve Young. Quality-enhanced voice morphing using maximum likelihood transforma- tions. IEEE Transactions on Audio, Speech, and Language Processing, 14(4):1301-1312, 2006.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 2223-2232, 2017.
13

---

Under review as a conference paper at ICLR 2020
A MATHEMATICAL PROOF OF ADAGAN CONCEPT AND DIFFERENT LOSS FUNCTIONS
Here, we will first give the proof of the concept behind AdaGAN and later we will prove how our loss functions help us to satisfy the derived constrained in Theorem 1.
Theorem 1:
Given assumptions in Section 5.1, we can say that there exists latent space where normalized latent representation of input features will be the same irrespective of speaking style.
Proof: From eq. I
we write the goal of AdaGAN following:
DEL Pxz, (IZi,U, 22, U2) II PKz, 122 (.122, 22, ) ) =00. 0.
(15)
From eq. (15). can conclude that the output of AdaGAN after conversion using (Z1, U1) and (Zz, U2) is the same as conversion using (Z1, U1) and (Zz, U2). Because in either way. we are transferring speaking style of speaker Ze to utterance U1. We can say that the output for AdaIN is the same for both the cases in the latent space. Hence, we can write above eq. (I5) as:
= DEL( PAdaIN AAdasN(111,,0 U2) || (.daIN(.122 Un, ) )= 0,
(16)
where PAdaIN (.l.) is the pdf of the latent representation. For given input samples 2ι and yı. can write following term from eq. (6)
PAdaIN,, (.1Z,, U1, Z2, U2)
PAdaIN, vs (11Z2, Z2, U2),
(17)
From Fig. we can write eq. as:
Sz, (T) - (T)
02((T + p2(T) = Syn (T) (T)- d2(T) + PL((T,
(T)
(T)
where 7 represents the training iteration. Now, giving limm both side we assume that (T)
(T) = (2()), and (T((T) = dT(T) = (T).. Therefore,
Sz, S2, (T) (1((T) (2((T) + (2((T) (T(
S. (T) WMFFF
0U(T]
Sz, (T) - (1))-((())
Sy (T) - (2((T)
(18)
(T(
02(T)
At + oc, the assumptions that made in Section 5.1 are true. Hence, from eq. conclude that there exists a latent space where normalized latent representation of input features will be the same irrespective of speaking style.
18 we can
Theorem 2: By optimization min De Lox-v Lyt the assumptions made in Theorem 1 can be satisfied.
Proof: Our objective function is the following:
min Lox_x Lstyx_y En, De.
(19)
Iterate step by step to calculate the term (12) used in loss function Latyx Consider, we have the latent representations Sn, and S,, corresponding to the source and target speech, respectively.
Sm, (T) (T( (2(T) + H2(T)
Step 1:
(Representation of t1),
(T(T)
Enf De So. (T) - (1(
(2((T) + pa(T))}
Step 2&3:
(T(
14

---

Under review as a conference paper at ICLR 2020
After applying decoder and encoder sequentially latent representation, will again get back to the same representation. This is ensured by the loss function Lcx-y. Formally, we want to make Lcx-x - 0. Therefore,
write step as:
Sa, (T) (I(T)] 02((T (2(T)
Step 4:
(i.e., reconstructed (1),
(T)
1
Sz (T) (i(T) golt) + Bett] Heffs
Step 5:
getr)
(T)
(Normalization with its own (i.e., latent representation in Step 4) μ and a during AdaIN),
Sm, (7) (T)
Step 6:
(Final output of Step 5),
(1
Sz, (T) -1 (T) +
Step 7:
(T)
(Output after de-normalization in AdaIN. Representation of (2),
where μ and of are the mean and standard deviations the another input source speech, T2. Now, using the mathematical representation of /2, we write loss function Lstyx x- as:
Lstyx---
[[ S. (T) -T)-) (I(T)
)oi(T)+ + S., (T) (T)
(20)
(T)
According to eq. 199 we want minimize the loss function Cstyx-v Formally, Letyx 0. Therefore, we will get μl fí. and σº of to achieve our goal. Hence, mean and standard deviation of the same speaker are constant, and different for different speakers irrespective of the
linguistic content. We come to the conclusion that our loss function satisfies the necessary con- straints (assumptions) required in proof of Theorem
B T-SNE VISUALIZATION OF LATENT SPACE LEARNED BY ADAGAN
L-SME: Denu2
1-S4E:
-20
B-NE: Dim-1
L-SNE: Dim-1
Figure 6: t-SNE visualization of latent representation of two speakers' speech and its normalized form, where, each point denotes feature extracted from the ms of speech segment.
As we know, Neural Networks (NNs) are hard train and optimize. Even everything has been proven in terms of theoretical proofs, statistical and empirical analysis is required. For this analysis, we have adopted t-SNE visualization. Here, we randomly selected few utterances from two different speakers from the VCTK corpus. Latent representations are extracted for the speech of that speakers, and features are reduced to 2-D using t-SNE. The seatter plot shown in Fig. 6 shows that data points are clustered based the speaking style. After normalized with their respective means and standard deviations, these distribution overlapped. This shows that the distribution latent representation captures linguistic information-basee features irrespective of speaking style as proved in Theorem 1. Therefore, we can say that AdaGAN, and its losses are efficient for practical purposes.
normalized
15


Reviews:
[{"review_id": "r1xjn_YV5r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work describes an efficient voice conversion system that can operate on non-parallel samples and convert from and to multiple voices.  The central element of the methodology is the AdaIn modification.  This is an efficient speaker adaptive technique where features are re-normalized to a particular speaker's domain.  The rest of the machinery is well motivated and well executed, but less novel.  This addition enables the voice conversion between speakers.\n\nSection 4.4 Are all of the utterances the same length?  Based on the architecture description, it appears as though the model generates one output frame for each input frame.  This would suggest that for training input and output need to be synchronized.  If so, make this explicit and include length parameters in Section 6.1 \n\nSection 6.2 states \"For statistically significant analysis, results are shown in different conversion possibilities.\" However, no test of statistical significance is presented.  This pointer may be helpful (https://pdfs.semanticscholar.org/b2b1/d01336323f3794f54de26567335aa0bcac46.pdf)\n\nPresentation Comments:\n\nSection 3.1: I would recommend using different subscripts for Z_i and U_i, since when indexing Z this implies the i-th speaker, and when indexing U it's the i-th utterance.  The formulas in Section 3.1 imply a single index i for both of these which is clearly not intended.\n\nSection 4: Consider using the present tense instead of the perfect tense when describing the results.  \"...we discuss our proposed AdaGAN architecture... We have shown... We have presented...\" can be \"...we discuss our proposed AdaGAN architecture... We show... We present...\"\n\nSection 5.2; Tables 1 and 2: Consider some partition of the FLOPS and Parameters, separation by commas, spaces or even abbreviation e.g. 2952233 -> 2,952,233 or 2 952 233 or 2.9M.  This will make this table much easier to read.\n\nSection 6.2; Figures 2-5: MOS scores have a minimum value of 1.  This should be the axis of the chart, rather than 0.  \n\nIt's pretty bold to star by contextualizing the work with the sentence \"Language is the core of civilization and speech is the most powerful and natural form of communication.\"  :-)", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJxDQYG0FS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a voice conversion approach using GANs based on adaptive instance normalization (AdaIN).  The authors give the mathematical formulation of the problem and provide the implementation of the so-called AdaGAN. Experiments are carried out on VCTK and the proposed AdaGAN is compared with StarGAN.  The idea is ok and the concept of using AdaIN for efficient voice conversion is also good.  But the paper has a lot of issues both technically and grammatically, which makes the paper hard to follow.\n\n1. On writing\nThere are glaring grammar errors in numerous places. e.g.\n  -- \"Although, there are few GAN-based systems that produced state-of-the-art results for non-parallel VC. Among these algorithms, even fewer can be applied for many-to-many VC task. At last, there is the only system available for zero-shot VC proposed by Qian et al. (2019).\"   This is hard to parse.\n -- \"helps generator to make ...\"  -> \"helps the generator make ...\"\n --  \"let assume\" -> \"Let's assume\" \n --  \"We know that the idea of transitivity as a way to regularize structured data has a long history.\"   what does it mean?\n --  \"the generator of AdaGAN is consists of Encoder and Decoder.\"  -> \"consist of\"\n -- \"After training of AdaGAN for large number of iteration of $\\tau$ , where theoretically $\\tau \\rightarrow \\infty$.\" where is the second half of the sentence?\n\n2.  On math notation\n The math notation is messy and there are lots of inaccuracies.  e.g.\n  --  $X_{i} \\in p_{X}(\\cdot|Z_{i},U_{i})$ should be $X_{i} \\sim p_{X}(\\cdot|Z_{i},U_{i})$\n  --  \"generate the distribution denoted by $\\hat{X}_{Z_{1}\\rightarrow Z_{2}}$\"  -> why  $\\hat{X}_{Z_{1}\\rightarrow Z_{2}}$ becomes a distribution? \n  --  \"$p_{N}(\\cdot|Z_{1},U_{1})$, $p_{N}(\\cdot|Z_{2},U_{1})$\" in Eq.14,  $N$ should be replaced by the random variable.\n  --  $S'_{X}$ and $S'_{Y}$ should be $S_{X'}$ and $S_{Y'}$ in line 15 in the algorithm\n\n3. On technical details:\n -- In Fig.1 (b), why is there only one input to the discriminator?  How do you inject the adversarial samples and how do you generate adversarial samples? \n-- In section 4.4, \"in encoder and decoder all layers are Linear layers\".  Are you referring to fully-connected layers? Linear layers are usually referred to those with linear activation functions.  \n-- The experiments are claimed to be zero-shot, but 3-5s of speech is required.  can you explain? \n\nAlthough the samples sound OK, given its current form, the paper needs significant re-work. \n\nP.S. rebuttal read.   I will stay with my score.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BkgNckrRtH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles many-to-many voice conversion task using GAN for style transfer between different speakers. The core idea is adaptive instance normalization (Huang & Belongie, 2017). \n\nDetailed comments:\n\nThere are many typos and wrong notations in the text. Here is an incomplete list:\n- \"it were spoken by target speaker\", should be \"was\". \n- In Section 3.1, \"Here, U_1 and U_2 are spoken by Z_i and Z_2\" should Z_1. Overall, the descriptions in this subsection is confusing. For example, it seems utterance U_i is from speaker Z_i in the dataset, but there are n speakers and m utterances.\n\n- A closely related task is voice cloning, which is arguably more challenging than voice conversion, because the synthesis need generalize to arbitrary text. One may properly discuss the recent advances in this community (e.g., Arik et al., 2018; Nachmani et al., 2018; Jia et al., 2018).\n\nPros:\nThe empirical improvement seems meaningful.\n\nCons:\nThis paper is poorly written and difficult to follow. For example, I could not accurately identify the major contribution & novelty after reading the abstract and introduction. As an application paper, the authors may clearly explain the ideas with a few sentences in the most natural way without \"heavy notations\", e.g., Eq. (5)(6)(7).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper has major presentation issues. The rebuttal clarified some technical ones, but it is clear that the authors need to improve the reading substantially, ,so the paper is not acceptable in its current form.
Decision: Reject