Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work describes an efficient voice conversion system that can operate on non-parallel samples and convert from and to multiple voices.  The central element of the methodology is the AdaIn modification.  This is an efficient speaker adaptive technique where features are re-normalized to a particular speaker's domain.  The rest of the machinery is well motivated and well executed, but less novel.  This addition enables the voice conversion between speakers.

Section 4.4 Are all of the utterances the same length?  Based on the architecture description, it appears as though the model generates one output frame for each input frame.  This would suggest that for training input and output need to be synchronized.  If so, make this explicit and include length parameters in Section 6.1 

Section 6.2 states "For statistically significant analysis, results are shown in different conversion possibilities." However, no test of statistical significance is presented.  This pointer may be helpful (https://pdfs.semanticscholar.org/b2b1/d01336323f3794f54de26567335aa0bcac46.pdf)

Presentation Comments:

Section 3.1: I would recommend using different subscripts for Z_i and U_i, since when indexing Z this implies the i-th speaker, and when indexing U it's the i-th utterance.  The formulas in Section 3.1 imply a single index i for both of these which is clearly not intended.

Section 4: Consider using the present tense instead of the perfect tense when describing the results.  "...we discuss our proposed AdaGAN architecture... We have shown... We have presented..." can be "...we discuss our proposed AdaGAN architecture... We show... We present..."

Section 5.2; Tables 1 and 2: Consider some partition of the FLOPS and Parameters, separation by commas, spaces or even abbreviation e.g. 2952233 -> 2,952,233 or 2 952 233 or 2.9M.  This will make this table much easier to read.

Section 6.2; Figures 2-5: MOS scores have a minimum value of 1.  This should be the axis of the chart, rather than 0.  

It's pretty bold to star by contextualizing the work with the sentence "Language is the core of civilization and speech is the most powerful and natural form of communication."  :-)
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper tackles many-to-many voice conversion task using GAN for style transfer between different speakers. The core idea is adaptive instance normalization (Huang & Belongie, 2017). 

Detailed comments:

There are many typos and wrong notations in the text. Here is an incomplete list:
- "it were spoken by target speaker", should be "was". 
- In Section 3.1, "Here, U_1 and U_2 are spoken by Z_i and Z_2" should Z_1. Overall, the descriptions in this subsection is confusing. For example, it seems utterance U_i is from speaker Z_i in the dataset, but there are n speakers and m utterances.

- A closely related task is voice cloning, which is arguably more challenging than voice conversion, because the synthesis need generalize to arbitrary text. One may properly discuss the recent advances in this community (e.g., Arik et al., 2018; Nachmani et al., 2018; Jia et al., 2018).

Pros:
The empirical improvement seems meaningful.

Cons:
This paper is poorly written and difficult to follow. For example, I could not accurately identify the major contribution & novelty after reading the abstract and introduction. As an application paper, the authors may clearly explain the ideas with a few sentences in the most natural way without "heavy notations", e.g., Eq. (5)(6)(7).
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper presents a voice conversion approach using GANs based on adaptive instance normalization (AdaIN).  The authors give the mathematical formulation of the problem and provide the implementation of the so-called AdaGAN. Experiments are carried out on VCTK and the proposed AdaGAN is compared with StarGAN.  The idea is ok and the concept of using AdaIN for efficient voice conversion is also good.  But the paper has a lot of issues both technically and grammatically, which makes the paper hard to follow.

1. On writing
There are glaring grammar errors in numerous places. e.g.
  -- "Although, there are few GAN-based systems that produced state-of-the-art results for non-parallel VC. Among these algorithms, even fewer can be applied for many-to-many VC task. At last, there is the only system available for zero-shot VC proposed by Qian et al. (2019)."   This is hard to parse.
 -- "helps generator to make ..."  -> "helps the generator make ..."
 --  "let assume" -> "Let's assume" 
 --  "We know that the idea of transitivity as a way to regularize structured data has a long history."   what does it mean?
 --  "the generator of AdaGAN is consists of Encoder and Decoder."  -> "consist of"
 -- "After training of AdaGAN for large number of iteration of $\tau$ , where theoretically $\tau \rightarrow \infty$." where is the second half of the sentence?

2.  On math notation
 The math notation is messy and there are lots of inaccuracies.  e.g.
  --  $X_{i} \in p_{X}(\cdot|Z_{i},U_{i})$ should be $X_{i} \sim p_{X}(\cdot|Z_{i},U_{i})$
  --  "generate the distribution denoted by $\hat{X}_{Z_{1}\rightarrow Z_{2}}$"  -> why  $\hat{X}_{Z_{1}\rightarrow Z_{2}}$ becomes a distribution? 
  --  "$p_{N}(\cdot|Z_{1},U_{1})$, $p_{N}(\cdot|Z_{2},U_{1})$" in Eq.14,  $N$ should be replaced by the random variable.
  --  $S'_{X}$ and $S'_{Y}$ should be $S_{X'}$ and $S_{Y'}$ in line 15 in the algorithm

3. On technical details:
 -- In Fig.1 (b), why is there only one input to the discriminator?  How do you inject the adversarial samples and how do you generate adversarial samples? 
-- In section 4.4, "in encoder and decoder all layers are Linear layers".  Are you referring to fully-connected layers? Linear layers are usually referred to those with linear activation functions.  
-- The experiments are claimed to be zero-shot, but 3-5s of speech is required.  can you explain? 

Although the samples sound OK, given its current form, the paper needs significant re-work. 

P.S. rebuttal read.   I will stay with my score.
review_title: Official Blind Review #2

Meta-review: The paper has major presentation issues. The rebuttal clarified some technical ones, but it is clear that the authors need to improve the reading substantially, ,so the paper is not acceptable in its current form.
Decision: Reject