Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper claims to be the first to tackle unconditional singing voice generation. It is noted that previous singing voice generation approaches leverage explicit pitch information (either of an accompaniment via a score or for the voice itself), and/or specified lyrics the voice should sing. The authors first create their own dataset of singing voice data with accompaniments, then use a GAN to generate singing voice waveforms in three different settings:
1) Free singer - only noise as input, completely unconditional singing sampling
2) Accompanied singer - Providing the accompaniment *waveform* (not symbolic data like a score - the model needs to learn how to transcribe to use this information) as a condition for the singing voice
3) Solo singer - The same setting as 1 but the model first generates an accompaniment then, from that, generates singing voice

Firstly, the authors have done a lot of work - first making their own data, then designing their tasks and evaluating them. The motivation is slightly lacking - it is not clear why we are interested in these three task settings i.e. what we will learn from a difference in their performance, and there is a lack of discussion about which setting makes for better singing voice generation. Also, there is no comparison with other methods: whilst score data is not available it could be estimated, then used for existing models, providing a nice baseline e.g. first a score is extracted with a state of the art AMT method, then a state of the art score to singing voice generation method could be used.

There are existing datasets of clean singing voice and accompaniment, for example MIR-1k (unfortunately I think iKala, another dataset, is now unavailable). It is true that this dataset is small in comparison to the training data the authors generate, but it will certainly be cleaner. I would have liked to see an evaluation performed on this data as opposed to another dataset which was the result of source separation (the authors generate a held out test set on Jazz from Jamendo, on which they perform singing voice separation).

I also had questions about the training data - there is very little information about it included other than it is in-house and covers diverse musical genres (page 6 under 4.1), a second set of 4.5 hours of solo piano, and a third set (?) of jazz singers. This was a bit confusing and could do with clarification. At minimum, I would like to know what genre we are restricting ourselves to - is everything Jazz? Are the accompaniments exclusively piano (it's alluded that the answer to this is no, but it's not clear to me)? Is there any difference between the training and test domain.

On page 6, second to last paragraph when discussing the validation set, I would like the sampling method to be specified - it makes a difference whether the same piece of music will be contained within both the training and validation split, or whether the source piece (from which the 10 second clips are extracted) are in separate splits <- I'd recommend that setting.

The data used to train the model will greatly affect my qualitative assessment of the provided audio samples so, without a clear statement on the training data used, I can't really assess this.

However, with respect to the provided audio samples, I'd first note that these are explicitly specified as randomly sampled, and not cherry picked, which is great, thank you. However, whilst I would admit that the domain is different, when the singing samples are compared with the piano generation unconditional samples of MelNet (https://audio-samples.github.io/#section-3), which I would argue is just as hard to make harmonically valid, they are not as harmonically consistent, even when an accompaniment has been provided. However, samples do sound like human voice, and the pitch is relatively good. The words are unintelligible, but this is explicitly specified as out of scope for this paper, and I agree that this is much harder to achieve.

As an aside, MelNet is not cited in this paper and, given the similarity and relevance, I think it probably should be - https://arxiv.org/abs/1906.01083. It was published this year however so it would be a little harsh to expect it to be there. I would invite the authors to rebut this claim if they think the methods are not comparable.

My main criticism is in relation to the evaluation. For Table 2, without a baseline or the raw data (which would have required no further effort) included in the MOS study, it's very difficult to judge success. If the authors think that comparison with raw data is unfair (as it is an embryonic task) they could include a model which has an unfair advantage from the literature - e.g. uses extracted score information. 

For Table 1, I appreciate the effort that went into the design of 'Vocalness' and 'Matchness' which are 'Inception Score' type metrics leaning on other learned models to return scores. I would like to see discussion which explains the differences in scores for the different model settings (there is a short sentence at the bottom of page 7, but nothing on vocalness).

In summary - this is a hard problem and the authors are the first to tackle it. The different approaches to solve the problem are not well motivated. However, the models are detailed, and well explained. Code is even provided, but data for training is not. If the authors were able to compare with a baseline (like that I describe above), it would go a long way to convincing me that this was good work. As it stands, Tables 1 and 2, and the provided audio samples have no context, so I cant make a conclusion. If this issue and motivation was addressed I would likely vote to accept the paper.

Things to improve the paper that did not impact the score:
1. p2 "we hardly provide any labelled data" specify whether you do or not (I think it's entirely unsupervised since you extract chord progressions and pitch curves using learned models...)
2. p2 "...may suffer from the artifact" -> the artefacts
3. p2 "for the scenario addressed by the accompanied singer" a bit clumsy, may be worth naming your tasks 1, 2 and 3 such that you can easily refer to them
4. p2 "We investigate using conditional GAN ... to address this issue" - which issue do you mean? If it is the issue specified at the top of the paragraph, i.e. that there are many valid melodies for a given harmony (no single ground truth), I don't think using a GAN is a *solution* to this per se. It is a valid model to use, and the solution would be enough varied data (and evaluation to show you're covering your data space and haven't collapsed to a few modes)
5. p2 "the is no established ways" -> there are no established ways
6. p3 "Discriminators in GAN" -> in the GAN
7. p6 "piano playing audio on our own..." -> piano playing on its own (or even just rephrase the sentence - collect 4.5 hours of audio of solo piano)
8. p7 "We apply source separation to the audios divide them into ..." -> we apply source separation to the audio data then divide each track into 20 second...
9. p7 If your piano transcription model was worse than Hawthorne, why didn't you use it? It would have been fine to say you can't reproduce their model if it is not available, but instead you say that 'according to out observation [it] is strong enough' which comes across quite weakly.
10. p8 "in a quiet environment with proper microphone volume" -> headphone volume?
11. p8 "improv" - I think this sentence trailed off prematurely!


review_title: Official Blind Review #6

final_score: rating: 3: Weak Reject
review_content: This paper tries to addresses an interesting problem of generating singing voice track under three different circumstances. Some of the problems that this paper deals with is a new problem and introduced first in this paper, which could be a contribution as well.

Although the paper is fairly well-structured and written, especially in the early sections, I am giving a weak reject due to its weak evaluation. Evaluation is almost always difficult when it comes to generative art, but I am slightly more concerned than that. The literature review can be, although it is nicely done overall, improved, especially on the neural network based singing voice synthesis.

I appreciate the authors tried to find a great problem and provided a good summary of the literature. Successfully training this kind of network itself is already tricky. It is also nice to see some interesting approaches towards objective evaluation. 

Below are my comments. 

> Our conjecture is that, as the output of G(·) is a sequence of vectors of variable length (rather than a fixed-size image), compressing the output of G(·) may have lost information important to the task.

I am rather not convinced. The difficulty to discriminate them doesn't seem to be (strongly) related to their variable length for me because a lot of papers have, at least indirectly, dealt with such a case successfully. 

> For data augmentation, we transpose the chord progressions found in Wikifonia to 24 possible keys

What do you mean by 24 keys? I think there should be only 12 keys. 

> Vocalness measures whether a generated singing voice audio sounds like vocals. We use the singing voice detection tool proposed by Leglaive et al. (2015) and made available by Lee et al.(2018).

Actually, the paper (Lee et al. 2018) suggested that vocal activity detection in the spectrum domain is easily affected by some features such as frequency modulation. I am not sure if this feature is suitable as a measure of the proposed vocalness. The computed vocalness may provide more information if they are computed on other tracks (e.g., guitar, cello, drums, etc).

> Average pitch: We use the state-of-the-art monophonic pitch tracker CREPE (Kim et al., 2018)8 to compute the pitch (in Hz) for each frame. The average pitch is computed by averaging the pitches of all the frames.

I am pretty sure that this is not the right way to evaluate as a metric of generated vocal track. CREPE is a neural network based pitch tracker, which means it is probably biased by the training data, where the pitch values mostly range in that of common musical instruments/voices. This means, when the F0 of input is not really in the right range, CREPE might incorrectly predict somewhat random F0 within THAT range anyway. I'd say the distribution of pitch values can be interesting metric to show and discuss, but not as a metric of vocal track generation.




review_title: Official Blind Review #4

final_score: rating: 1: Reject
review_content: This paper has set a new problem: singing voice synthesis without any score/lyrics supervision. The authors provide a significance of such a problem in section 1. Also, the authors successfully design and implement a novel neural network architecture to solve the problem. It’s also notable that the authors kindly open-source their code to mitigate the reproducibility issue. This paper may serve as baseline results for the proposed problem in the future.

Despite the significance of the problem and the novelty of the solution, this paper aims to solve too many problems at once. Unfortunately, some main ideas were not supported by experimental results or logical arguments with appropriate citations.

The authors seem to overly focus on the task itself, and thus haven’t pay much attention on supporting their choice of neural network architecture. Here are some points regarding this:

1. “We adapt the objective of BEGAN, which is originally for generating images, for generating sequences.”: The original BEGAN paper(Berthelot et al., 2017) did not address sequence modeling. 
2. “As for the loss function of D(·), our pilot experiments show that …”: This hand-wavy argument is unacceptable. The authors should be able to support all of the claims they’ve made, which sometimes require experimental results. “G(·)” of the following sentence should be D(·).
3. “Our conjecture is that, … compressing the output of G(·) may have lost information important to the task.”: PatchGAN (Isola et al., 2017) had already addressed this issue. The authors may want to cite PatchGAN to support their conjecture or compare against PatchGAN to show their own architecture’s strength.
4. “We like to investigate whether such blocks can help generating plausible singing voices in an unsupervised way.”: No ablation studies on GRUs and dilated convolutions are found. If the authors mean that they’re willing to do such studies in the future, “what’s done here” and “what will be done in the future” should be easily distinguished within the text.

Some miscellaneous points worth noting:
1. The readers won’t be able to estimate the strength of the proposed method by looking at table 1 and 2. I suggest doing one of the following: include results from other baselines to compare against or give a brief description of the metrics with typical values. (e.g. values shown in appendix A.3)
2. Are the neural network architecture components described in section 3.1 used for both source separation and the synthesis network?
3. To make readers easily understand the contribution of this paper, there should be a detailed description of the limitation of this work. I suggest to move the details of experiments in section 4 to the appendix, but it may depend on the authors’ writing style.
4. The ‘inner idea’ concept in the “solo singer” setting looks vague and contradicts with the main topic since it uses chord sequences to synthesize singing voice.

Things to improve the paper that did not impact the score:
1. “improv” >> “improvement.”

This paper should be rejected because (1) the paper failed to justify the main idea and results, (2) the amount of literature research was not enough, (3) too many problems were addressed at once, and (4) the writing is not clear enough.
review_title: Official Blind Review #5

final_score: rating: 3: Weak Reject
review_content: In this paper, authors explore the problem of generating singing voice, in the waveform domain. There exists commercial products which can generate high fidelity sounds when conditioned on a score and or lyrics. This paper proposes three different pipelines which can generate singing voices without necessitating to condition on lyrics or score. 

Overall, I think that they do a good job in generating vocal like sounds, but to me it's not entirely clear whether the proposed way of generating melody waveforms is an overkill or not. There is a good amount of literature on generating MIDI representations. One can simply generate MIDI (conditioned or unconditioned), and then give the result to a vocaloid like software. I am voting for a weak rejection as there is no comparison with any baseline. If you can provide a comparison with a MIDI based generation baseline, I can reconsider my decision. Or, explain to me why training on raw waveforms like you do is more preferable. I think in the waveform domain may even be undesirable to work with, as you said you needed to do source separation, before you can even use the training data. This problem does not exist in MIDI for instance. 

review_title: Official Blind Review #1

Meta-review: Main content:

Blind review #1 summarizes it well:

his paper claims to be the first to tackle unconditional singing voice generation. It is noted that previous singing voice generation approaches leverage explicit pitch information (either of an accompaniment via a score or for the voice itself), and/or specified lyrics the voice should sing. The authors first create their own dataset of singing voice data with accompaniments, then use a GAN to generate singing voice waveforms in three different settings:
1) Free singer - only noise as input, completely unconditional singing sampling
2) Accompanied singer - Providing the accompaniment *waveform* (not symbolic data like a score - the model needs to learn how to transcribe to use this information) as a condition for the singing voice
3) Solo singer - The same setting as 1 but the model first generates an accompaniment then, from that, generates singing voice

--

Discussion:

The reviews generally point out that while a lot of new work has been done, this paper bites off too much at once: it tackles many different open problems, in a generative art domain where evaluation is subjective.

--

Recommendation and justification:

This paper is a weak reject, not because it is uninteresting or bad work, but because the ambitious scope is really too large for a single conference paper. In a more specialized conference like ISMIR, it would still have a good chance. The authors should break it down into conference sized chunks, and address more of the reviewer comments in each chunk.
Decision: Reject