Score and Lyrics-Free Singing Voice Generation
Authors: Jen-Yu Liu
Yu-Hua Chen
Yin-Cheng Yeh
Yi-Hsuan Yang
Venue:  (2020)
Keywords: singing voice generation
GAN
generative adversarial network

Abstract: Generative models for singing voice have been mostly concerned with the task of "singing voice synthesis," i.e., to produce singing voice waveforms given musical scores and text lyrics. In this work, we explore a novel yet challenging alternative: singing voice generation without pre-assigned scores and lyrics, in both training and inference time. In particular, we experiment with three different schemes: 1) free singer, where the model generates singing voices without taking any conditions; 2) accompanied singer, where the model generates singing voices over a waveform of instrumental music; and 3) solo singer, where the model improvises a chord sequence first and then uses that to generate voices. We outline the associated challenges and propose a pipeline to tackle these new tasks. This involves the development of source separation and transcription models for data preparation, adversarial networks for audio generation, and customized metrics for evaluation.

Under review as a conference paper at ICLR 2020
SCORE AND LYRICS-FREE SINGING VOICE GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Generative models for singing voice have been mostly concerned with the task of "singing voice synthesis," i... to produce singing voice waveforms given musical scores and text lyrics. In this work, we explore novel yet challenging alter- native: singing voice generation without pre-assigned scores and lyrics, both training and inference time. In particular, we propose three either unconditioned or weakly conditioned singing voice generation schemes. We outline the associ- ated challenges and propose pipeline to tackle these new tasks. This involves the development of source separation and transcription models for data preparation, adversarial networks for audio generation, and customized metries for evaluation.
INTRODUCTION
The task of computationally producing singing voices is usually referred to as singing voice syn- thesis (SVS) in the literature (Cook 1996). Most researchers assume that the note sequence and the lyrics of the waveform to be generated are given as the model input, and aim to build synthesis engines that sound as natural and expressive as a real singer (Blaauw al. 2019 Hono al.) 2019] Kaewtip et al., 2019, Lee et al. 2019a Tamaru et al. 2019). As such, the content of the produced singing voice is largely y determined by the given model input, which usually assigned by human. And, accordingly, progress in SVS has followed closely with that in text-to-speech (TTS) synthesis
(Umbert et al. 2015 Shen et al., 2017 Gibiansky et al. 2017).
However, we argue that singing according to pre-assigned musical score and lyries is only a part of the human singing activities. For human beings, singing can also be a spontaneous activity. We learn to spontancously sing when were children (Dowling) 1984). We do not need score to sing when we are humming on the road or in the bathroom. The voices sung do not have to be intelligible. Jazz vocalists can improvise according
chord progression, an accompaniment,
even nothing.
We aim to explore such a new task in this paper: teaching machine to sing with a training collection of singing voices, but without the corresponding musical scores and lyries of the training data. Moreover, the machine has to sing without pre-assigned score and lyrics as well even in the inference (generation) time. This task is challenging in that, the machine sees no lyrics all, it hardly has any knowledge of the human language to pronounce or articulate either voiced or unvoiced sounds. And, as the machine sees no musical scores all, has to find its own way learning the language of music in creating plausible vocal melodies. It also makes the task different from TTS.
Specifically, we consider three types of such score- and lyrics-free singing voice generation tasks, as shown in Figures [bb (d). A free singer sings with only random noises as the input. An accompa- nied singer learns sing over a piece of instrumental music, which is given as an audio waveform (again without score information). Finally, solo singer also sings with only noises as the input, but uses the noises to firstly generate some kind of 'inner ideas' of what to sing.
From a technical point of view, we can consider SVS as a strongly conditioned task for generating singing voices, as the target output well specified by the input. contrast, the proposed tasks are either unconditioned or weakly conditioned. This work therefore contributes to expanding the "spectrum" (in terms of the strength of conditional signals) of singing voice generation. Doing so has at least two implications. First, while models are more difficult to train than SVS models, they enjoy more freedom in the generation output. Such freedom may be desirable considering the artistic nature of singing. Second, we can more easily use larger training set to train our model-

---

Under review as a conference paper at ICLR 2020
Audio signal
Audio signal
Audio signal
Audio signal
Vocoder
Vocoder
Vocoder
Vocoder
Feature
Feature
Feature
Singer
Feature
Idea
Singer
Singer
Singer
Idea generator
Lyrics
Vocal
Accomp.
Noise
Nolse
Pitch
Noise
(a) Singing synthesis
(b) Accompanied singer
(c) Solo singer
(d) Free singer
Figure 1: Schemes of singing voice generation; the strength of condition decreases from left to right.
due to the difficulty in preparing time-aligned scores and lyrics, the training set employed in existing work on SVS usually consists of tens of songs only (Lee et al. 2019a); in contrast, in our case we do not need labeled and aligned data and can therefore use more than hundreds of songs for training. This may help establish universal model based on which extensions can be made.
The proposed accompanied singer also represents one of the first attempts to produce singing voice given accompaniment. One intuitive approach to achieve this is to first generate score according to an accompaniment in the symbolic domain and then synthesize the singing voices according to the score. The second step of synthesis is relatively well-established, but the first step of generating a score given an accompaniment is not explored yet. Extensive researches have been done in gen- erating scores of one or several instruments (Hadjeres al. 2017] Yang et al. 2017 Huang et al.] 2019 Payne 2019). However, to the best of our knowledge, very few, if any, researches have been done on generating scores of singing voices given an accompaniment. Our approach bypasses the step of generating scores by directly generating the mel-spectrogram representation.
We outline below the challenges associated with the proposed tasks and the solutions we investigate.
First, the tasks are unsupervised as we do not provide any labels (e.g., annotations of phonemes, pitches, or onset times) for the training singing files. The machine has to learn the complex structure of music directly from audio signals. We explore the use of generative adversarial network (GAN) (Goodfellow et al., 2014) to address this issue, for its demonstrated effectiveness for SVS (Hono) let al.) 2019) and pitch-conditioned instrument note synthesis (Engel et al.) 2019). Specifically, design a novel GAN-based architecture to learn to generate the mel-spectrogram of singing voice, and then WaveRNN (Kalchbrenner et al. 2018), single-layer recurrent neural network, as the vocoder to generate the audio waveform. Rather than considering the specttrecss as a fixed- size image as done in recent work on audio generation (Engel al. 2019: Marafioti et al. 2019), we use gated recurrent units (GRUs) (Cho et al.] 2014) and dilated convolutions (van den Oord et al.) 2016) in both the generator and discriminator, to model both the local and sequential patterns in music and to facilitate the generation of variable-length waveforms.
Second, for training the free singer, unaccompanied vocal tracks are needed. As for the accompanied singer, we need additionally an accompaniment track for each vocal track. However, public-domain multi-track music data is hard to find. We choose to implement a vocal source separation model with state-of-the-arl separation quality (Liu & Yang 2019) for data preparation. The proposed pipeline for training and evaluating an accompanied singer is illustrated in Figure The advantage of having vocal separation model is that we can use as many audio files as we have to compile the training data. The downside is that the singing voice generation models may suffer from the artifacts (Cano) cet al. 2018) of the source separation model, which is moderate but not negligible
Third, for the accompanied singer, there is no single "ground truth" and the relationship between the model input and output may be one-to-many. This is because there are plenty of valid ways to
Research on singing voice separation has made remarkable progress in recent years. As reported in (Stöeer et al. 2018), sfate-of the-theert models perform comparably to an oracle approach for many cases. Open-souree implementations of ofttee-----hh-rar--- models are also available (Stöter et al. 2019 Hennequin al. 2019). Visit
https://bit. 1y/ for samples of the separation result of the model we employ.
2

---

Under review as a conference paper at ICLR 2020
A. Data
B. Training
D. Evaluation
Singing
Discriminator
PiSinging|
Accompaniment
Generated
Singing
Singer (Generator)
Matchness Estimator
Source Separation
C. Generation
Accompaniment (Acc.)
Generated
Singing
Singer
Music
(Generator)
Figure 2: A pipeline for building the accompanied singer. We use source separation to get separated singing voice and accompaniment from professionally recorded audio files. Then, we use the sep- arated tracks to train the generators and discriminators in the GAN. In inference time, we feed an unseen accompaniment to the trained singer model and let it "sing."
sing accompaniment track. For diversity and artistic freedom, cannot ask the machine to generate any specific singing voice in response to an accompaniment track, even if we have paired data of vocal and accompaniment tracks. We investigate using conditional GAN (Mirza & Osindero 2014) to retain the possibility generating singing voices with multiple modes.
established ways for performance evaluation. According to our setting, we desire our machine to generate audio waveforms with high quality and diversity, vocal-like timbre, plausible pitch contour, emotion expression, and, for the accompanied singer, that are in harmony with the given accompaniment track. But, the singing does not have to be intelligible. We propose customized objective and subjective metrics evaluate our models in these aspects. For example, we adapt the melody harmonization model proposed by Lim et al. (2017) to measure the matchness between the generated vocal track and the given accompaniment track.
Fourth, the proposed tasks are new, there
Finally, reproducibility is major issue, especially for a subjective task. We intend to use publicly- available copyright-fis instrumental music the conditional signals for testing the accompanied singer, so that other researchers can use the same testing conditions for model comparison in the future. We will also release the testing conditions for the solo singer, the generated singing voices for all our models, well as open source our code through a public git repository [URL removed].
We focus on Jazz music in this work. Samples the generated singing voices can found at https: / /ttpt//bttttt ly/ 2m ImIvoIo vo volc. Our models have many possible use cases. For example, we can use accompanied singer as a backing vocalist. addition, we can use the free singer as a sound source- to demonstrate this, make song by hand in the style of Jazz Hiphop by sampling the output of our free singer. This song can listened to at https: /bt 1y 2QkUJoJ
2 SCHEMES OF SINGING VOICE GENERATION
A free singer takes no conditions at all as the input. We want it to sing freely. The singing voices from a free singer may not even sound good, but they should sound like singing voice. A free singer is like we are freely humming or singing the road walking or in the bathroom taking a shower. We may not even know what we are singing and likely there is no underlying musical score.
From the viewpoint of a P(Y), where Y E I x is matrix representing a sequence K ---dimensiona features and T' is the number of time frames. A free singing is sampled from this distribution without conditions.
generative model, training free singer amounts modcling distribution
An accompanied singer takes as the input a sequence of accompaniment- accompanied singer tries to generate singing voices that match the accompaniment track in some way. It is similar to the case Karaoke, where a backing accompaniment track is played from a speaker, the lyrics and video are displayed screen, and user tries sing according to the
features. An
3

---

Under review as a conference paper at ICLR 2020
lyries and the backing track. The difference is that, this time the user is trained model and do not ask it to follow the lyrics or the exact pitch contour of the accompaniment. The note sequence found in the singing has to be in harmony with, but not duplicate of, that in the backing track.
Training an accompanied singer amounts to modeling a distribution P(Y AA), where Y E R represents a feature sequence of the vocal track, and A #xT is a feature sequence of the given accompaniment track. In our implementation, we use the mel-spectrograms for Y in compliance with the need of the vocoder. For A, different features can be tried, and we investigate using a transcription model (Hawthorne et al. 2018) to extract pitch features. See Section 4.1 for details.
A solo singer is similar to a free singer in that both takes no conditions as the input. However, solo singer would generate an 'inner idea' first, and then sing according to that. In other words, it learns a joint distribution P(Y (I), where I R is a matrix representing the idea sequence. The singer first samples I from a distribution Q, and then uses that to conditionally sample Y from P. The inner idea can take several forms. In this work, we instantiate this scheme with being a chord progression (namely a sequence of chord labels). The distribution Q is modeled by an auto- regressive recurrent network we build for chord progression generation (with network architecture adapted from that in (Waite et al. 2016)), as described in Section 3.3
Alternatively, we can think of a solo singer as a combination of an idea generator and an an accom- panied singer. For an accompanied singer, the information extracted from the given accompaniment track can take several forms such as transcribed pitches and chord progressions. A solo singer learns to generate such information on its own, without reference to actual accompaniment track.
3 MODELS
To account for the absence of supervised data and the highly complicated spatio-temporal patterns in audio spectrograms, we propose a new adversarial net that features heavy use of GRUs (Cho et al. 2014 Chung al., 2014), dilated convolutions (van den Oord et al., 2016;; Sercu & Goel 2016)) and feature grouping to build our singer models. We provide the algorithmic details below.
3.1 BLOCK OF GRU-GROUPED DILATED CONVOLUTION- GROUP NORMALIZATION
Network architectures with stacked blocks of GRUs and dilated convolutions have been used to at- tain state-of-the-art performance in blind musical source separation (Liu & Yang 2019). In source separation task, a model learns decompose, or unmix, different sources (e.g.. vocal, piano, bass, drum) from a mixture signal (Rafii et al. 2018). This requires the abilities to model the relationships between different sources as well as the relationships between neighboring time frames. The output spectrograms are also expected to be distortion-less and of high audio quality. For it has demon- strated its capability in source separation, we adopt it as a building block of the singer models. Especially, we want the singer models to also consider accompaniment information.
Specifically, one such block we adopted our models is a stack of GRU, dilated convolution with feature grouping, and group normalization (Wu & He, 2018). The input to the GRU, the output of the GRU, and the output of the group normalization are summed to form the output of the block. We note that the original 'D2 block' used in (Liu & Yang 2019) uses dilated GRU and uses weight normalization (Salimans& & Kingmal 2016) for the dilated convolutio layers. However, empirically find that it is easier for the singer models to converge by replacing weight normalization with group normalization, and using plain GRUs is as good as using dilated GRUs. We refer to our blocks as GRU-grouped dilated convolution-gros grrupp normalization block ('G3 block').
3.2 SINGER MODELS WITH BEGAN, G3 BLOCKS AND FRAME-WIS WISE NOISES (G3BEGAN)
The accompanied singers and solo singers have to take conditions as part of their input. One desir- able property of the models is the ability to generate voices with arbitrary length, as the conditional signal can be of variable length. Besides, the model has to deal with the one-to-many issue men- tioned in Section [ and the absence of supervisory signals. With these issues in mind, we design a GAN architecture for score and lyrics-free voice generation. In particular, we pay special attention to the following three components: 1) the network architecture, 2) the input noises for GAN, and 3) the loss function of the discriminator.
4

---

Under review as a conference paper at ICLR 2020
Let us first take look at two existing GAN models for audio generation: (Engel al. 2019) and (Donahue et al. 2019). Their generators and discriminators are both based on 2D convolutions, transposed 2D convolutions and dense (linear) layers. The generators take a vector E R as the input noise and use transposed convolutions to expand z so that a temporal dimension emerges in the expanded intermediate matrices. The number of temporal frames in the final output depends on the total strides used in all the transposed convolutions. The discriminators take the output of the generators or the real signal as the input, and compress the input matrix with convolution layers until the output becomes a single value represents the prediction of true (real) or false (generated) data.
A main reason why existing models cannot generate variable-length output the need expand by transposed convolution layers. We remedy this by using an architecture consisting of the pro- posed G3 blocks, and convolutions without strides, for both the generators Gi-) and discriminators D(-). Moreover, instead of using single noise vector, our models take as input a sequence of noise vectors, denoted as Z E RUXT, that has the same temporal length as the desired output Y. Each col- umn of Z is sampled independently from Gaussian distribution Normal 1). At the first glance, might feel unnatural to have one noise vector per frame as that may result in fast oscillations the noises. However, we note that the output of G(.) for the t-th frame depends not only on the f-th col- umn of Z (and C or I), but the entire Z (and the condition matrices), due to the recurrent GRUs in the model. We expect that the GRUs in the discriminator D--) would force G(-) generate consistent consecutive frames. Therefore, the effect of the frame-wise noises might be introducing variations to the generation result (c.g., by adjusting the modes of the generated Frame-wise features).
As for the loss function of DC.), we experiment with the following three options: the vanilla GAN, the LSGAN (Mao et al.] 2017) that adopts the least squares loss function for the discriminator, and the boundary equilibrium GAN (BEGAN) (Berthelot et al. (017) that adopts an "auto-encoder style" discriminator loss. The D(-) in either GAN or LSGAN is implemented as a classifier aiming to distinguish between real and generated samples, whereas the D(-) in BEGAN is an autoencoder aiming to reconstruct its input. Specifically, BEGAN, the loss functions In and lg for the dis- criminator and generator, as in the case of the accompanied singer, are respectively:
In = L(X, C) -
(1)
LG (G)
where X E RxT is the feature sequence of real vocal track sampled from the training data, GiZZ C) E kxT is the feature sequence for the generated vocal track, and L(-) is a function that measures how well the discriminator ((-), implemented as an auto-encoder, reconstructs its input:
1 L(M, C) = WT
Σ 12.
(DM, C) - -aral, for an arbitrary W x T matrix M,
(3)
where we use Ma,, to denote the (wr, 1)-th element a matrix M (and similarly for D(M, ().)))
Moreover, the variable in Eq. (1) is introduced by BEGAN to balance the power of D(.) and G())
during the learning process. It is dynamically set to be T = Ta + X(>L(X, C) C)-L C), C)),
for each training step s. with T2 E [0, 1J. À and y are manually-set hyperparameters? f
Empirical comparison of the performance of GAN, LSGAN and BEGAN can be found in Appendix
D It turns out that the BEGAN-basee one, referred to as G3BEGAN hereafter, works the best.
3.3 CHORD PROGRESSION GENERATOR
We also use auto-regressive RNN to build chord progression generator for implementing the solo singer. Our chord generator is trained on the Wikifonia dataset (http://wwe) /www wikiiiiooo fonia.. org/), ), set of 6,670 songs in the leadsheet format (i.e., with separated symbolic melody and chord tracks). Its chord vocabulary covers 612 different chords. We set the harmonic rhythm of the chord generator such that a chord change may occur every beat. We desire the chord generator to freely generate chord progressions across different tempo values, time signatures, and keys. Moreover, the generated chord progression has be rhythmically correct. In achieving so, we encode the tempo,
We replace C' in the above equations by an empty matrix for the free singer, and by I for the solo singer. For the accompanied singer, note that D)-) has two ways to discriminate true and fake data: from whether G(ZZ C) sounds like real singing voice, and from whether G(Z, C) matches the given condition C.
5

---

Under review as a conference paper at ICLR 2020
time signatures, and key information (which available in the Wikifonia dataset) the initial hidden state of the RNN, and concatenate the chord vector from last time step with the beat position (e.g., 1, 2, 3, 4) of that time step the imput the RNN. For data augmentation, we transpose the chord progressions found in Wikifonia to 12 possible keys. Once trained, we use the chord generator create chord progressions for testing the solo singer. With a one-hot encoding, each column of the condition matrix I would have dimension J = 660.
More details of G3BEGAN and the chord generator can be found in Appendices A.1 and A.2
EXPERIMENTS
4.1 EXPERIMENTAL SETUP AND DATASETS
In our implementation, we use 80-dimensional mel-spectrograas the acoustic features modeled and generated by the singer models (i.e.. K = 80). We use the python package 1 librosa (McFee ct al. 2015), with default settings, to compute the meessccctsoggams from audio. A mel-spectrogram is passed to a WaveRNN vocoder (Kalchbrenner et al. 2018)) to generate an audio signal from mel- spectrograms. Our implementation of the WaveRNN vocoder is based on the code from Fatchord | Instead of using off-the-shelf pre-trained vocoders, which are typically trained for TTS, we train our vocoder from scratch with a set of 3,500 vocal tracks we separate (by separation model) from an in-house collection of music that covers diverse musical genres.
One main difficulty in conducting this research is to get vocal tracks for training our singer models. Existing multitrack datasets that contain clean vocal tracks are usually diverse in musical genre and the singing timbre, making hard for our models learn the relationship between singing and accompaniment. As we set out to focus on Jazz music in this work, we opt for collecting our own vocal tracks from Jazz music only. We therefore implement our source separation model following the architecture proposed by Liu & Yang (2019), which represents the state-of-the-art as evaluated on the MUSDB benchmark (Rafii et al., al.] 2017). We use the whole MUSDB dataset to train our separation model. This dataset contains clean vocal and accompaniment tracks. However, for constructing the condition matrix A of our accompanied singer, we desire to have separated piano tracks well (we will explain why shortly). We therefore collect additionally 4.5 hours of Jazz piano solo audio and use them to augment MUSDB for training our source separation model, which can as a result isolate out not only the vocal track but also the piano track from an arbitrary song.
We collect 17.4 hours of Jazz songs containing female voices and 7.6 hours of Jazz songs with male voices. We use the aforementioned separation model get the vocal tracks. For batched training, we divide the tracks into 10-second sub-clips. Sub-clips that contain less than 40% vocals, as measured from energy, are removed. This leads to 9.9-hour and 5.0-hour training data for female and male Jazz vocals, respectively. 200 and 100 sub-clips are reserved from the training set as the validation set for female singing and male singing, respectively. Each model is trained for 500 epochs. For GAN and LSGAN, we use the models at the 500th epoch for evaluation. For BEGAN, the parameters of the epoch with the best convergence rate (Berthelot al., (017) are used for evaluation.
For the accompanied singer, we experiment with extracting pitch-related information from the ac- companiment track to form the matrix A that conditions the generation of the vocal track. The as- sumption here is that whether the generated vocal track is in harmony with the accompaniment track can be largely determined by pitch-related information. For this purpose, we implement a piano transcription model to transcribe the separated piano track, leading to 88 58-dimensional transcribed frame-wise pitch as the accompaniment condition (i.e., H = 88, as there are 88 piano notes). We implement piano transcription model with the G3 blocks introduced in Section 3.1 following the training procedure of (Hawthorne et al. 2018). We also implement the model of Hawthorne et al. 2018). Under the same training setting, we find that ours is slightly worse than theirs in note Fl score (0.779 vs 0.794) but slightly better in the note precision score (0.834 vs 0.823). We decide to use our model for the better precision, but the difference of using either of them should be small.
The clips in the training set of our singer models may not contain piano playing (see Table B for summary of the datasets). Even if clip contains piano playing, the piano may not play across the entire clip. Hence, the models have to learn to sing either with or without the piano accompaniment.
httpp::////t http://gttttuboooo htt.://///.aacco////eaoeoee

---

Under review as a conference paper at ICLR 2020
For performance evaluation, we collect 5.3 hours of Jazz music from Jamendo (https: : www. jamendo. , com), an online platform for sharing copyright-free music. As said in Section [ this test set is meant to be public. We apply source separation to the audios, divide each track into se-second sub-clips l and remove those that do not contain piano. Piano transeription is also applied to the separated piano track, yielding 402 20-second sub-clips for evaluation. 402 chord progressions generated by the chord generator to evaluate the solo singer and to compute the matchness.
4.2 BASELINES
As this a new task, there is no previous work that we can compare with. Therefore, we establish the baselines by 1) computing the baseline objective metrics (see Section 4.3 from the training data
of the singing models, and 2) using existing SVS systems for synthesizing singing voices.
For the SVS baselines, we employ Sinsy (Oura et al. 2010 Hono et al. 2018) and Synthesizer V (Hua et al. 2019), the two well-known SVS systems that are publicly accessible. For Sinsy, we use the publicly available repositors to query the Sinsy API (http: /tt///sinn// jp/); we use the HMM version (Oura et al. 2010) instead of the deep learning version as the latter cannot generate male voices. For Synthesizer V. we use their software (https: //s:://syttttt http://sttttteeiiteeeeeeeseeeeee com/). ) We use Sinsy for both objective and subjective tests, but Synthesizer for subjective test only, for the latter does not provide functionality to batch process collection of MIDI files and lyrics.
SVS systems have to take lyrics and a melody to synthesize singing voices. For the lyrics, we choose to use multiple 'la,' the default lyrics for Synthesizer For the melodies, we consider two methods:
1. Vocal transcription from singer training data. We use CREPE to transcribe the separated vocals from the singer training data, and convert it to MIDI format.
2. Piano transcription from the Jamendo testing data. As described in Section we have separated and transcribed the piano part of the Jamendo data, Yet, the piano transcription often contains multiple notes at the same time. We use the skyline algorithm (Ozcan et al.) 2005 to the transcription result to get a melody line comprising the highest notes.
4.3 OBJECTIVE METRICS AND OBJECTIVE EVALUATION RESULT
The best way evaluate the performance of the singer models is perhaps by listening the gener- ated results. Therefore, we encourage our readers to listen to the audio files provided in the supple- mentary material. However, objective evaluation remains desirable, either for model development or for gaining insights into the generation result. We propose the following metrics for our tasks.
Vocalness measures whether an audio clip contains singing voices. There are different publicly available tools for detecting singing voices in an audio mixture (e.g., (Lee et al., )018)). We choose the JDC model (Kum & Nam 2019) for it represents the sfate-of-the- art. In this model, the pitch contour is also predicted in addition to the vocal activation. If the pitch at a frame is outside a reasonable human pitch range (73-988 Hz defined by JDC), the pitch is set to 0 at that frame. We consider frame as being vocal if it has a vocal activation 2 0.5 AND has a pitch > O. Morcover, we define the vocalness of an audio clip the proportion of its frames that are vocal. The tool is applied to the non-silence part of an audid the generated singing voices only, excluding the accompaniment.
Average pitch: We estimate the pitch (in Hz) for each frame with two pitch detection models: the state-of-the-at monophonic pitch tracker CREPE (Kim et al. (0188)) and JDC. The average pitch is computed by averaging the pitches across the frames with confidence higher than 0.5 for CREPE, and across the frames that are estimated to be vocal for JDC.
Please note that this is longer than the 10-second sub-clips we used to train the singer models. This is okay as model can generate variable-length output.
'nttps: :ttpps://giii ht//tttttbb com/mat htt///aatiiggg mattiittt..... /miiiivic
"As our models do not contain meaningtul lyrics, be fair the baselines should not contain meaningful lyrics either. We choose 'la" because people do sometimes sing with 'la' and it has no semantic meaning. An alternative way to get the lyrics is hy randomly sampling a number of characters. However, randomly sampling a reasonable sequence of characters is not a trivial task as well.
The non-silence frames derived by using the 1 ibbosa function 'effects. signal frame nonsilent.'
7

---

Under review as a conference paper at ICLR 2020
Average pitch (Hz)
Vocalness
Proposed model
Matchness
CREPE
JDC
JDC
Free singer (female)
288 + 28
292 28
0.48 = 0.09
-13.28 + 3,80
Accompanied singer (female)
313 = 18
316 + 19
0.55 0.11
-9.25 = 3.13
Solo singer (female)
302 - 17
306 - 18
0.56 0.10
-9.30 = 3.11
Free singer (male)
248 + 39
242 + 32
0,44 + 0.16
-13.29 + 3.19
Accompanied singer (male)
207 + 14
200 + 15
0.44 0.13
-9.31 + 3.16
Solo singer (male)
213 = 14
207 + 16
0,46 = 0.12
-9.30 = 3.13
Baseline: Singing voice synthesis
Sinsy (training vocal, female)
305 + 59
308 +: 57
0.71
+: 0.17
-9.20
3.12
Sinsy (training vocal, male)
260 + 86
259
+ 72
0.73
+ 0.14
-9.09
3.14
Sinsy (testing piano skyline, female)
523 + 138
431
+ 62
0.66
0.14
-8.88 + 3.04
Sinsy (testing piano skyline, male)
520 = 137
423
61
0.62
0.15
-8.93
3.02
Baseline: Training data
Wikifonia: real melody-chords Wikifonia: random melody-chords Singer train data (vocals, female) Singer train data (vocals, male) Singer train data (accomp., female) Singer train data (accomp., male) MUSDB clean vocals
-7.04 + 2.91
-13.16 + 3.72
312 + 70
310 + 56
0.60 0.14
-9.24 3.09
263 + 93
258 + 75
0.16
-9.09 3.22
0.05
0.12
271 + 81
283
75
0.59 0.14
Table I: Result of objective evaluation for our singer models and a few baseline methods.
Singing-acoompanimmnt matchness: As detailed in the appendix, objectively measure matchness, we build a melody harmonization RNN by adapting the chord generator de- scribed in Section 3.3 Given pair of melody and chord sequences, the model computes the likelihood of observing that chord sequence as the output when taking the melody se- quence the model input. We use the average of the log likelihood across time frames as the matchness score. As the harmonization model considers symbolic sequences, we use CREPE transcribe the generated voices, and Madmom (Böck et al., 2016 recognize the chord sequence from the accompaniment track.
Several observations can be made from the result shown in Table In terms of the average pitch, we can see that the result of our model is fairly close to that of the singing voices in the training data. Moreover, the average pitch of the generated female voices is higher than that of the generated male voices as expected. We can also see that the Sinsy singing voices tend to have overly high pitches, when the melody line is derived from piano playing (denoted 'testing piano skyline.').
In terms vocalness, our models score in general lower than Sinsy, and the singing voices in the training data. However, the difference is not that far. As reference, also compute the vocalness of the accompaniments in the training set (denoted as 'accomp.') and it is indeed quite low
As for matchness, show in Table T the score computed from the real relody-chorde pairs of Wikifonia (-7.04) and that from random pairs of Wikifonia (-13.16). We can see that the accompa- nied singers score higher than the random baseline and the free singer expected | Moreover, the matchenss scores of the accompanied singers are close to that of the singer training data.
Examples of the generated spectrograms of models can be found in the appendix. From visually inspecting the spectrograms and listening to the result, the models seem to learn the characteristics of the singing melody contour (c.g.. the FO stable over time). Moreover, the female singer models learn better than the male counterparts, possibly because of the larger training set.
WWe note that Sinsy even scores higher vocalness than the training data. This may due to fact that real singing voices are recorded under different conditions and effects. Thhe matchness scores of the free singers are computed by pairing them with the 402 test clips.
8

---

Under review as a conference paper at ICLR 2020
Model (epochs trained)
Sound quality
Vocalness
Expression
Matchness
G3BEGAN (20 epochs)
1.59 = 0.82
1.93 + 0.99
1.98 + 0.88
2.18 - 1.08
G3BEGAN (240 epochs)
2.24 + 0.93
2.66 + 1.01
2.60 + 1.01
2.58 + 1.05
G3BEGAN (final)
2.38 + 0.96
2.98 + 1.02
2.85 + 1.00
2.74 + 1.04
Table 2: Mean opinion scores (MOS) and standard deviations with respect to four evaluation criteria collected from the user study, for three different versions of accompanied singer (female). The scores are in 5-point Likert scale and are from to 5; the higher the better.
Model (epochs trained)
Sound quality
Vocalness
Expression
Matchness
G3BEGAN (final)
1.71 + 0.70
2.39 1.11
2.27 + 1.06
2.34 + 1.16
Sinsy Oura et al. 2010)
3.19 + 1.07
2.90 + 1.01
2.40 + 0.98
2.10 + 0.90
Synthesizer V Hua et al. 2019)
3.57 + 1.07
3.30 + 1.24
3,25 + 1.10
3.35 + 1.15
Table 3: MOS from the second user study, comparing our model and two existing SVS systems.
4.4 USER STUDY AND SUBJECTIVE EVALUATION RESULT
We conduct two online, non-paid user studies evaluate the accompanied singer, the female one. In the first user study, we compare the 'final' model (with the number of epochs selected according to validation set) against two early versions of the model trained with less epochs. In the second one, we compare the proposed accompanied singer with Sinsy and Synthesizer V.
In the first study, we recruit 39 participants to each rate the generated singing for three different accompaniment tracks (each 20 seconds), one accompaniment track per page. The subjects are informed the purpose of our research (i.e., score and lyrics-free singing voice generation) and the user study (to compare three computer models), and are asked to listen in a quiet environment with proper headphone volume. No post-processing (e.g., noise removal, EQ adjustment) is applied to the audio. The ordering of the result of the three models is randomized.
The process of the second study is similar to the first one, but includes five different accom- paniments (randomly chosen from those used in the first user study) and the respective gener- ated/synthesized singing voices. The melodies used for synthesis are those from the piano skyline of the test data, so that model can be compared with the synthesis methods with the same companiment. A separate set of 21 subjects participate in this study. The audio files used in this user study be downloaded from https: //bit.////tt.. 1y/2gyyykkyy
Tables and show the result of the two studies. We can see that the model indeed learns better with more epochs. Among the four evaluation criteria, the Sound Quality rated lower than the other three in both studies, suggesting room for improvement. By comparing the proposed model with the two SVS systems, we see that Synthesizer V performs the best for all the evaluation criteria. Our model achieves better Matchness than Sinsy, and achieves a rating close to Sinsy in Expression To In general, we consider the result as promising considering that our models are trained from scratch with little knowledge of human language.
5 RELATED WORK
While early work on SVS is mainly based on digital signal processing (DSP) techniques such as sampling concatenation (Cook) 1996] [Bonada & Serra (007). machine learning approaches of- fer greater flexibility and have been more widely studied in recent years. Hidden Markov models
We note that Sinsy and Synthesizer V have unfair advantage on matchness because their singing voices are basically synthesized according to the melody lines of the accompaniment. From Table B we see that Synthesizer V does exhibit this advantage, while Sinsy does not. We observe that the Sinsy singing voices do not always align with the provided scores. The fact that Synthesizer V has higher audio quality seem to promote its score in the other criteria. The presence of the result of Synthesizer V seems to also make the subjects in the second study the proposed model lower than the subjects do in the first study.

---

Under review as a conference paper at ICLR 2020
(HMMs), in particular, have been shown to work well for the task (Saino et al. 2006). The Sinsy system, a baseline model in Section 4 is also based on HMMs (Oura et al. 20100. Nishimura et al. 2016 report improved naturalness by using deep neural nets instead of HMMs. Since then, many neural network models have been proposed.
The model presented by Nishimura al. 2016) uses simple fully-connected layers map symbolic features extracted from the user-provided scores and lyrics, to a vector of acoustic features for syn- thesis. The input and output features are time-aligned frame-by-frame beforehand by well-trained HMMs. The input features consist of score-related features ge.g., the key of the current bar and the pitch of the current musical note), and lyrics-related ones (the current phoneme identify, the number of phonemes in the current syllable, and the duration of the current phoneme). The output features consist of spectral and excitation parameters and their dynamic features (Hono et al., 2018) which altogether can then be turned into audio with DSP technique called the MLSA filter (Imai 1983).
The aforementioned model has been extended in many aspects. For instance, using convolutional layers and recurrent layers in replacement of the fully-connected layers for learning the mapping between input and output features has been respectively investigated by NaKaaauua et al. (2019) and
Kim et al. (0188b) Using neural vocoders such as the WaveNet (van den Oord et al. (016) instead
of the MLSA filter has been shown to improve naturalness by Nakamura et al. (2019). 19. Rather than using hand-craften features for the input and output, [ee et al. (2019a) train model to predict the
mel-spectrogram directly from time-aligned lyrics and pitch labels, and then use the Griffin-Lim algorithm (Griffin & Lim 1984) to synthesize the audio. Modern techniques such as adversarial loss and attention module have also been employed (Lee et al.) 2019a). A follow-up work adds a speaker encoder to the network achieve multi-singer SVS (Lee al. 2019b). Synthesizer V (Hua) et al. 2019), the other baseline model we employ in Section is based on a hybrid structure that uses both deep learning and sample- basple-based concatenation
While exciting progress has been made to SVS, the case of score and lyrics-free singing voice generation, to our best knowledge. has not been tackled thus far. Similar to (Lee et al.) 2019a). we do not use hand-crafted features and we train our model to predict the mel-spectrograms.
Using neural nets for score-conditions instrumental audio generation have also been investigated in recent years. However, existing work is mostly concerned with the generation of single notes of, for example, 4-second long (Défossez et al. 2018 Donahue et al., 2019) Engel et al., 2019). A deep neural network that is capable of generating variable-length audio (e.g., "recurrent generator") as the proposed singer models do, to our knowledge, has not been much studied.
6 CONCLUSION
In this paper, we have introduced a novel task of singing voice generation that does not use musical scores and lyrics. Specifically, we proposed three singing schemes with different input conditions: free singer, accompanied singer, and solo singer. We have also proposed a BEGAN based architec- ture that uses GRUs and grouped dilated convolutions to learn generate singing voices in an ad- versarial way. For evaluating such models, we proposed several objective metrics and implemented a model to measure the compatibility between a given accompaniment track and the generated vocal track. The evaluation shows that the audio quality of the generated voices still leave much room for improvement, but in terms of humanness and emotion expression our models work fine.
Score and lyrics-free singing voice generation is a new task, and this work represents only first step tackling it. There are many interesting ideas to pursue. For example, we have chosen to ex- tract pitch-related information only from the accompaniment track for the accompanied singer, but a more interesting way to let the model learns to extract relevant information itself. the near future, we plan to investigate advanced settings that allow for timbre and expression control, and experiment with other network architectures, such as coupling a fine-grainet auto- uuto-regressive model with a multiscale generation procedure as done in MelNet (Vasquez & Lewis 2019), using dis- criminator that examines different chunks of the generated audio as done in PatchGAN for the vision domain (Isola et al.) 2017), using multiple discriminators that evaluate the generated audio based on fulti-frequency random windows as done in GAN-TTS (Bińkowski et al. 2019).
"httpss:/syntt fandom. com/wi ki /F i 1 Synthes zer the herr_ Singi ng_Synth (last accessed: Nov. 12, 2019)
10

---

Under review as a conference paper at ICLR 2020
REFERENCES
David Berthelot, Thomas Schumm, and Luke Metz. BEGAN: Boundary equilibrium generative adversarial networks. arXiv preprint, 2017.
Mikotaj Binikowsi,, Jeff Donahue, Sander Dieleman, Aidan Clark, Erich Elsen, Norman Casagrande, Luis C. Cobo, and Karen Simonyan. High fidelity speech synthesis with adversarial networks. arXiv preprint arXiv; 19t9::1999.1e0. 2019.
Merlijn Blaauw, Jordi Bonada, and Ryunosuke Daido. Data efficient voice cloning for neural singing synthesis. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 0840-6844, 2019.
Scbastian Böck, Filip Korzeniowski, Jan Schlüter, Florian Krebs, and Gerhard Widmer. madmom: new python audio and music signal processing library. arXiv preprint
2016.
Jordi Bonada and Xavier Serra. Synthesis of the singing voice by performance sampling and spectral models. IEEE Signal Processing Magazine, 24(2):67-79, 2007.
Estefanía Cano, Judith Liebetrau, Derry Fitzgerald, and Karlheinz Brandenburg. The dimensions of perceptual quality of sound source separation. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 601-605, 2018.
Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. On the proper- ties of neural machine translation: Encoder-decoder approaches. In Proc. Workshop on Syntax, Semantics and Structure in Statistical Translation, 2014.
Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In Proc. NIPS Workshop on Deep Learning, 2014.
Perry R. Cook. Singing voice synthesis: History, current work, and future directions. Computer Music Journal, 20(3):38-44 46, 1996.
Alexandre Défossez, Neil Zeghidour, Nicolas Usunier, Léon Bottou, and Francis Bach. SING: Symbol-tt-innsrummnn neural generator. Proc. Advances in Neural Information Processing Systems, pp. 9041-9051, 2018.
Chris Donahue, Julian McAuley, and Miller Puckette. Adversarial audio synthesis. In Proc. Inter- national Conference on Learning Representations, 2019.
Walter Jay Dowling. Development of musical schemata in children's 's spontaneous singing. Advances in Psychology, 19:145-163, 1984.
Jesse Engel, Kumar Krishna Agrawal, Shuo Chen, Ishaan Gulrajani, Chris Donahue, and Adam Roberts. GANsynth: Adversarial neural audio synthesis. In Proc. International Conference on Learning Representations, 2019.
Andrew Gibiansky, Sercan Arik, Gregory Diamos, John Miller, Kainan Peng, Wei Ping, Jonathan Raiman, and Yanqi Zhou Deep Voice 2: Multi-speaker neural text-to-speech. In Proc. Advances in Neural Information Processing Systems, pp. 2962-2970. 2017.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. Advances in Neural Information Processing Systems, pp. 2672-2680. 2014.
D. Griffin and J. Lim. Signal estimation from modified short-time fourier transform. IEEE Trans- actions on Acoustics, Speech, and Signal Processing, 32(2):236224 243, 1984.
Gaëtan Hadjeres, François Pachet, and Frank Nielsen. DeepBach: stecrable model for Bach chorales generation. In Proc. International Confernece on Machine Learning, 2017.
Curtis Hawthorne, Erich Elsen. Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. Onsets and frames: Dual-objective piano transcription. In Proc. International Society for Music Information Retrieval Conference, pp. 50-57, 2018.
11

---

Under review as a conference paper at ICLR 2020
Romain Hennequin, Anis Khlif, Felix Voituret, and Manuel Moussalam. Spleeter: A fast and state- of-the art music source separation tool with pre-trained models. In Proc. International Society for Music Information Retrieval Conference, 2019. Late-breaking paper.
Yukiya Hono, Shumma Murata, Kazuhiro Nakamura, Kei Hashimoto, Keiichiro Oura, Yoshihiko Nankaku, and Keiichi Tokuda. Recent development the DNN-based singing voice synthesis systemSinsy. In Proc. Asia-Pacific Signal and Information Processing Association Annual Sum- mit and Conference, pp. 1003-1009, 2018.
Yukiya Hono, Kei Hashimoto, Keicchiro Oura, Yoshihiko Nankaku, and Keiichi Tokuda. Singing voice synthesis based on generative adversarial networks. In Proc. IEEE International Conference Acoustics, Speech and Signal Processing, pp. 6955-6959, 2019.
Kanru Hua et al. Synthesizer V, 2019. /ytppesi/.ytthheieer..oo
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer. In Proc. International Conference on Learning Representations, 2019.
Satochi Imai. Cepstral analysis synthesis the mel frequency scale. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 93-96, 1983.
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. Image-to-image translation with conditional adversarial networks. In Proc. Conference on Computer Vision and Puttern Recogni- lion, 2017.
Kantapon Kaewtip, Fernando Villavicencio, Fang-Yu Yu Kuo, Mark Harvilla, Iris Ouyang, and Pierre Lanchantin. Enhanced virtual singers generation by incorporating singing dynamics to personal- ized eex-to--ppecc--to-iiiine In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6960-6964, 2019.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lock- hart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. Proc. International Conference on Machine Learning, pp. 2410-2419, 2018.
Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. CREPE: A convolutional repre- sentation for pitch estimation. In Proc. IEEE International Conference Acoustics, Speech and Signal Processing, pp. 161-165, 2018a. [Online] https::////. /ttps:///ghhhts com www.ooo/maooooo.ooooooo/o..weooe htt.m/arll/eee/ee //repe
Juntae Kim, Heejin Choi, Jinuk Park, Minsoo Hahn, Sangjin Kim, and Jong-Jin Kim. Korean singing voice synthesis system based LSTM recurrent neural network. In Proc. INTERSPEECH, 2018b.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Sangeun Kum and Juhan Nam. Joint detection and classification of singing voice melody using convolutional recurrent neural networks. Applied Sciences, 9(7), 2019. [Online] https://
github. hwwa///kus///// 1ocy JDC
Juheon Lee, Hyeong-Seok Choi, Chang-Bin Bin Jeon, Junghyun Koo, and Kyogu Lee. Adversarially trained end-to-end Korean singing voice synthesis system. In Proc. INTERSPEECH, 2019a.
Juheon Lee, Flyong-Seok Seok Choi, Chang-Bit -in Jeon, Junghyun Koo, and Kyogu Lee. Multi muntersinggr singing voice synthesis system. In Proc. International Society for Music Information Retrieval Conference, 2019b. Late-breaking paper.
Kyungyun Lee, Keunwoo Choi, and Juhan Nam. Revisiting singing voice detection: quantitative review and the future outlook. In Proc. International Society for Music Information Retrieval Conference, pp. 506-513, 2018.
12

---

Under review as a conference paper at ICLR 2020
Hyungui Lim, Seungyeon Rhyu, and Kyogu Lee. Chord generation from symbolic melody using BLSTM networks. In Proc. International Society for Music Information Retrieval Conference, pp. 621-627, 2017.
Jen- Yu Liu and Yi-Hsuan Yang. Dilated convolution with dilated GRU for music source separation. In Proc. International Joint Conference on Artificial Intelligence, pp.
2019.
Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, and Stephen Paul Smolley. Least squares generative adversarial networks. In Proc. IEEE International Conference on Com- puter Vision, 2017.
Andrés Marafioti, Nicki Holighaus, Nathanaël Perraudin, and Piotr Majdak. Adversarial generation of time-frequency features with application in audio synthesis. In Proc. International Conference Machine Learning, 2019.
Brian McFee, Colin Raffel, Dawen Liang, Daniel P. W Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto. librosa: Audio and music signal analysis in Python. In Proc. Python in Science Conf., pp. 18-25, 2015. [Online] htttps://iibrss:
http://trr....i..........aa btcchub.io/// io/1ibrosa/
Mchdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1884 2014.
Kazuhiro Nakamura. Kei Hashimoto, Keiichiro Oura, and Keiichi Tokuda Yoshihiko Nankaku. Singing voice synthesis based on convolutional neural networks. arXiv preprint arXiv:1904, 06868, 2019.
Masanari Nishimura, Kei Hashimoto, Keiichiro Oura, Yoshihiko Nankaku, and Keiichi Tokuda. Singing voice synthesis based on deep neural networks. In Proc. INTERSPEECH, pp. 2478-
2482,2016
Keiichiro Oura, Ayami Mase, Tomohiko Yamada, Satoru Muto, Yoshihiko Nankaku, and Keiichi Tokuda. Recent development of the 4LMM-baed singing voice synthesis system-sing, In Proc. ISCA Workshop on Speech Synthesis, 2010.
Giyasettin Ozcan, Cihan Isikhan, and Adil Alpkock. Melody extraction on MIDI music files. In Proc. IEEE International Symposium on Multimedia, pp. 414 422, 2005.
Christine Payne. MuseNet, 2019. httpps///oeenaa.ccmmbbbo//////up
Zafar Rafii, Antoine Liutkus, Fabian-Rober Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner. The MUSDB18 corpus for music separation, 2017.
Zafar Rañi, Antoine Liutkus, Robert Stter, Stylianos Ioannis Mimilakis, Derry FitzGerald, and Bryan Pardo. An overview of lead and accompaniment separation in music. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(8):1307-1333 2018.
Keijiro Saino, Heiga Zen, Yoshihiko Nankaku, Akinobu Lee, and Keiichi Tokuda. An HMM-based singing voice synthesis system. Proc. International Conference on Spoken Language Process- ing, 2006.
Tim Salimans and Diederik Kingma. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. Proc. Advances in Neural Information Processing Systems, pp. 901-909, 2016.
Tom Sercu and Vaibhava Goel. Dense prediction sequences with time-dilated convolutions for speech recognition. arXiv preprint arXiv:1611.09288. 2016.
Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, R. J. Ekeery-Ryann Rif A. Saurous, Yannis Agiomyr- giannakis, and Yonghui Wu. Natural TTS synthesis by conditioning wavenet mel spectrogram predictions. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 4779 2017.
13

---

Under review as a conference paper at ICLR 2020
Fabian-Rober Stöter, Antoine Liutkus, and Nobutaka Ito. The 2018 signal separation evaluation campaign. In Proc. Latent Variable Analysis and Signal Separation, pp. 293-305, 2018.
Fabian-Robert Stöter, Stefan Uhlich, Antoine Liutkus, and Yuki Mitsufuji. Open-Unmix ref- erence implementation for music source separation. Journal of Open Source Software, 2019. [Onlin|]https://
unmix. unmix.appp
Hiroki Tamaru, Yuki Saito, Shinnosuke Takamichi, Tomoki Koriyama, and Hiroshi Saruwatari. Gen- erative moment matching network-base based random modulation post-filter for DNN-based singing voice synthesis and neural trackngg. In Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 7070-7074, 2019.
Martí Umbert, Jordi Bonada, Masataka Goto, Tomoyasu Nakano, and Johan Sundberg. Expression control in singing voice synthesis: Features, approaches, evaluation, and challenges. IEEE Signal Processing Magazine, 32(6):55-73, 2015.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint urXiv:160.003499 2016.
Sean Vasquez and Mike Lewis. MelNet: A generative model for audio in the frequency domain. arXiv preprint arXiv: 1tt::/9000 0900699009000 2019.
Elliot Waite, Douglas Bck, Adam Roberts, and Dan Abolafia. Project Magenta: Generating long- term structure in songs and stories, 2016.
tww.srrfow..essooo..oooooooo.m.. org/blog/
Yuxin Wu and Kaiming He. Group normalization. arXiv preprint arXiv:/803.08444, 2018.
2018.
Li-Chia Yang, Szu- Yu Chou, and Yi-Hsuan Yang. MidiNet: A convolutional generative adversarial network for symbolic-domain music generation. In Proc. International Society for Music Infor- mation Retrieval Conference, 2017.
A APPENDIX: TRAINING SETTING AND MODEL DETAILS
A.I G3BEGAN
Singer models with G3BEGAN trained with Adam (Kingma & Ba 2014) with 0.0001 learning rate and mini-bated size being 5. Each model is trained with 500 epochs. Gradient norm clipping with magnitude 3 is used.
The generator in G3BEGAN is implemented with stack of two G3 blocks. Please see Table 4 for details of the network architecture.
A.2 CHORD GENERATOR
The chord generator is aimed generate chord progressions freely under some given conditions. It supports 12 major and 12 minor keys, 10 tempo options from 60 to 240 BPM. 6 time signature options, and 51 chord qualities (612 chords total). The conditions, key, tempo, and time signa- tures, are encoded into one-hot representation and concatenated together
40-dimension vector. The model mainly consists with 3 stacked GRU layers, each with 512 hidden variables. The input of each time step is 524-dimensional vector consisting of chord embedding and beat-related one-hot positional encoding (to encourage the model to follow certain rhythmical pattern. This input array passes through cully-coneected layer to 512-dimension and is used the input of the GRUs. The training data are the leadsheets from the Wikifonia dataset. We augmented the data by rotating the keys, leading to in total 80,040 leadsheets for training.
14

---

Under review as a conference paper at ICLR 2020
Details
Input dim
Output dim
IDConv (kernel-3, dilation-1)
512
Input
Group normalization (group-4)
Leaky ReLU (0.01)
GRU
512
512
Grouped IDConv (kemnel33, dilation-2,, group-4) Group normalization (group-4)
512
512
G3 Block 1
Leaky ReLU (0.01)
GRU
512
512
Grouped IDConv (kernel-3, dilation-2, group-4) Group normalization (group-4) Leaky ReLU (0.01)
512
512
G3 Block 2
Output
IDConv (kemel--3, dilation=
512
80
Table 4: Network architecture of the generator (G) and the discriminator (D) the proposed G3BEGAN model. It uses two G3 blocks introduced in Section 3.1 For the noise input, set the dimension U to 20. For the G and D free singer, the input dimensions are 20 and 80, re- spectively. For the G and D of accompanied singer, the input dimensions 100 20 80 and 160 = 80 + 80, respectively.
Usage
Processing
Quantity
Singer training data (female) Singer training data (male) Jamendo test data Vocoder training data
Tr. female singers
SS, PT
9.9 hrs
Tr. male singers
SS. PT
5.0 hrs
Ev. singers
SS. PT
5.3 hrs
WaveRNN vocoder
SS
3,500 tracks
MAESTRO
Tr. piano transcription
200 hrs
MUSDB 4.5-hr Piano Solo
Tr. source separation model
100 tracks + 4.5 hrs
Table 5: A summary of the datasets employed in this work. The first three datasets and the h.5-hour Piano solo part of the last dataset are Jazz music, whereas the others may not. The PPoocessing' col- umn indicates the music processing models that have been applied to process the respective dataset. (Notation-Tr: training. Ev.: evaluating. SS: source separation. PT: piano transcription)
A.3 MELODY HARMONIZATION MODEL
The melody harmonization model is modified from the chord generator described in Appendix A.2 using additionally the melody tracks found in the Wikifonia dataset. Specifically, the model intends to generate chord sequence given a melody sequence. Such a model can learned by using the pairs of melody and chord tracks in Wikifonia. We add the chroma representation of the melody with window size of a quarter-note to the input vector.
The matchness of the real pairs of melody and chord progression in Wikifonia is -7.04 If pair a melody with a randomly selected chord progression and calculate the matchness, the score becomes -13.16
B APPENDIX: DATASETS
C APPENDIX: EXAMPLES THE SPECTROGRAM THE GENERATED SINGING VOICES
Examples of the spectrograms of the generated singing voices can be found in Figures and
15

---

Under review as a conference paper at ICLR 2020
(a) The given instrumental, accompaniment track
(b) The voices generated by the accompanied singer (female)
(c) The voices generated by the accompanied singer (male)
Figure 3: Samples of spectrograms generated by our accompanied singers; (left) the given accom- paniment tracks, the voices generated by (middle) the female singer and (right) the male singer.
Average pitch (Hz)
Vocalness JDC
Model
CREPE
JDC
BEGAN (female)
288 28
292 28
0.48 0.09
GAN (female)
307 = 7
371 + 25
0.22 + 0.17
LSGAN (female)
1130 = 9
453 14
0.28 0.03
Table 6: Result of using different GAN losses, for free singer (female).
D APPENDIX: ABLATION STUDY ON DIFFERENT GAN LOSSES
We experiment different GANs, including BEGAN (Berthelot al. 2017 vanilla GAN (Goodfel- low et al. 2014). and LSGAN (Mao et al. 2017). for the case of building the free singer. The output of the discriminators in the vanilla GAN and LSGAN is a single real/fake value. To compare the three GANs as fairly as possible. the discriminators used in vanilla GAN and LSGAN are almost the same as the one used in the BEGAN. The only difference that the discriminators used vanilla GAN and LSGAN have an extra average-pooling layer in the output. The validation losses at differ- ent training epochs are shown in Figure and the metrics are shown in Table 6 In Fi we can see that, according to our implementation, only BEGAN converges. In Table we can see that the BEGAN model has much higher Vocalness than other models. By listening to the generated singing voices of the GAN and LSGAN models, we find that they basically noises.
16

---

Under review as a conference paper at ICLR 2020
(a) Free singer (female)
(b) Solo singer (female)
(c) Free singer (male)
(d) Solo singer (male)
Figure 4: Samples of spectrograms generated by our (left) free singers and (right) solo singers. We see salient pitch contour in the spectrograms. Moreover, the pitches sung by the male singers seem on average lower than those sung by the female singers.
043
007
(a) BEGAN: Convergence metric
(b) GAN: discriminator
(c) LSGAN: discriminator loss
Figure 5: Loss trends of different GAN losses. BEGAN comes with convergence metric (Berthelot
et al. 2017 which can be used to examine how well the model converges.
17


Reviews:
[{"review_id": "rkgg0l5eiB", "reviewer": "AnonReviewer6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper claims to be the first to tackle unconditional singing voice generation. It is noted that previous singing voice generation approaches leverage explicit pitch information (either of an accompaniment via a score or for the voice itself), and/or specified lyrics the voice should sing. The authors first create their own dataset of singing voice data with accompaniments, then use a GAN to generate singing voice waveforms in three different settings:\n1) Free singer - only noise as input, completely unconditional singing sampling\n2) Accompanied singer - Providing the accompaniment *waveform* (not symbolic data like a score - the model needs to learn how to transcribe to use this information) as a condition for the singing voice\n3) Solo singer - The same setting as 1 but the model first generates an accompaniment then, from that, generates singing voice\n\nFirstly, the authors have done a lot of work - first making their own data, then designing their tasks and evaluating them. The motivation is slightly lacking - it is not clear why we are interested in these three task settings i.e. what we will learn from a difference in their performance, and there is a lack of discussion about which setting makes for better singing voice generation. Also, there is no comparison with other methods: whilst score data is not available it could be estimated, then used for existing models, providing a nice baseline e.g. first a score is extracted with a state of the art AMT method, then a state of the art score to singing voice generation method could be used.\n\nThere are existing datasets of clean singing voice and accompaniment, for example MIR-1k (unfortunately I think iKala, another dataset, is now unavailable). It is true that this dataset is small in comparison to the training data the authors generate, but it will certainly be cleaner. I would have liked to see an evaluation performed on this data as opposed to another dataset which was the result of source separation (the authors generate a held out test set on Jazz from Jamendo, on which they perform singing voice separation).\n\nI also had questions about the training data - there is very little information about it included other than it is in-house and covers diverse musical genres (page 6 under 4.1), a second set of 4.5 hours of solo piano, and a third set (?) of jazz singers. This was a bit confusing and could do with clarification. At minimum, I would like to know what genre we are restricting ourselves to - is everything Jazz? Are the accompaniments exclusively piano (it's alluded that the answer to this is no, but it's not clear to me)? Is there any difference between the training and test domain.\n\nOn page 6, second to last paragraph when discussing the validation set, I would like the sampling method to be specified - it makes a difference whether the same piece of music will be contained within both the training and validation split, or whether the source piece (from which the 10 second clips are extracted) are in separate splits <- I'd recommend that setting.\n\nThe data used to train the model will greatly affect my qualitative assessment of the provided audio samples so, without a clear statement on the training data used, I can't really assess this.\n\nHowever, with respect to the provided audio samples, I'd first note that these are explicitly specified as randomly sampled, and not cherry picked, which is great, thank you. However, whilst I would admit that the domain is different, when the singing samples are compared with the piano generation unconditional samples of MelNet (https://audio-samples.github.io/#section-3), which I would argue is just as hard to make harmonically valid, they are not as harmonically consistent, even when an accompaniment has been provided. However, samples do sound like human voice, and the pitch is relatively good. The words are unintelligible, but this is explicitly specified as out of scope for this paper, and I agree that this is much harder to achieve.\n\nAs an aside, MelNet is not cited in this paper and, given the similarity and relevance, I think it probably should be - https://arxiv.org/abs/1906.01083. It was published this year however so it would be a little harsh to expect it to be there. I would invite the authors to rebut this claim if they think the methods are not comparable.\n\nMy main criticism is in relation to the evaluation. For Table 2, without a baseline or the raw data (which would have required no further effort) included in the MOS study, it's very difficult to judge success. If the authors think that comparison with raw data is unfair (as it is an embryonic task) they could include a model which has an unfair advantage from the literature - e.g. uses extracted score information. \n\nFor Table 1, I appreciate the effort that went into the design of 'Vocalness' and 'Matchness' which are 'Inception Score' type metrics leaning on other learned models to return scores. I would like to see discussion which explains the differences in scores for the different model settings (there is a short sentence at the bottom of page 7, but nothing on vocalness).\n\nIn summary - this is a hard problem and the authors are the first to tackle it. The different approaches to solve the problem are not well motivated. However, the models are detailed, and well explained. Code is even provided, but data for training is not. If the authors were able to compare with a baseline (like that I describe above), it would go a long way to convincing me that this was good work. As it stands, Tables 1 and 2, and the provided audio samples have no context, so I cant make a conclusion. If this issue and motivation was addressed I would likely vote to accept the paper.\n\nThings to improve the paper that did not impact the score:\n1. p2 \"we hardly provide any labelled data\" specify whether you do or not (I think it's entirely unsupervised since you extract chord progressions and pitch curves using learned models...)\n2. p2 \"...may suffer from the artifact\" -> the artefacts\n3. p2 \"for the scenario addressed by the accompanied singer\" a bit clumsy, may be worth naming your tasks 1, 2 and 3 such that you can easily refer to them\n4. p2 \"We investigate using conditional GAN ... to address this issue\" - which issue do you mean? If it is the issue specified at the top of the paragraph, i.e. that there are many valid melodies for a given harmony (no single ground truth), I don't think using a GAN is a *solution* to this per se. It is a valid model to use, and the solution would be enough varied data (and evaluation to show you're covering your data space and haven't collapsed to a few modes)\n5. p2 \"the is no established ways\" -> there are no established ways\n6. p3 \"Discriminators in GAN\" -> in the GAN\n7. p6 \"piano playing audio on our own...\" -> piano playing on its own (or even just rephrase the sentence - collect 4.5 hours of audio of solo piano)\n8. p7 \"We apply source separation to the audios divide them into ...\" -> we apply source separation to the audio data then divide each track into 20 second...\n9. p7 If your piano transcription model was worse than Hawthorne, why didn't you use it? It would have been fine to say you can't reproduce their model if it is not available, but instead you say that 'according to out observation [it] is strong enough' which comes across quite weakly.\n10. p8 \"in a quiet environment with proper microphone volume\" -> headphone volume?\n11. p8 \"improv\" - I think this sentence trailed off prematurely!\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "r1xCpXLgjr", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper has set a new problem: singing voice synthesis without any score/lyrics supervision. The authors provide a significance of such a problem in section 1. Also, the authors successfully design and implement a novel neural network architecture to solve the problem. It’s also notable that the authors kindly open-source their code to mitigate the reproducibility issue. This paper may serve as baseline results for the proposed problem in the future.\n\nDespite the significance of the problem and the novelty of the solution, this paper aims to solve too many problems at once. Unfortunately, some main ideas were not supported by experimental results or logical arguments with appropriate citations.\n\nThe authors seem to overly focus on the task itself, and thus haven’t pay much attention on supporting their choice of neural network architecture. Here are some points regarding this:\n\n1. “We adapt the objective of BEGAN, which is originally for generating images, for generating sequences.”: The original BEGAN paper(Berthelot et al., 2017) did not address sequence modeling. \n2. “As for the loss function of D(·), our pilot experiments show that …”: This hand-wavy argument is unacceptable. The authors should be able to support all of the claims they’ve made, which sometimes require experimental results. “G(·)” of the following sentence should be D(·).\n3. “Our conjecture is that, … compressing the output of G(·) may have lost information important to the task.”: PatchGAN (Isola et al., 2017) had already addressed this issue. The authors may want to cite PatchGAN to support their conjecture or compare against PatchGAN to show their own architecture’s strength.\n4. “We like to investigate whether such blocks can help generating plausible singing voices in an unsupervised way.”: No ablation studies on GRUs and dilated convolutions are found. If the authors mean that they’re willing to do such studies in the future, “what’s done here” and “what will be done in the future” should be easily distinguished within the text.\n\nSome miscellaneous points worth noting:\n1. The readers won’t be able to estimate the strength of the proposed method by looking at table 1 and 2. I suggest doing one of the following: include results from other baselines to compare against or give a brief description of the metrics with typical values. (e.g. values shown in appendix A.3)\n2. Are the neural network architecture components described in section 3.1 used for both source separation and the synthesis network?\n3. To make readers easily understand the contribution of this paper, there should be a detailed description of the limitation of this work. I suggest to move the details of experiments in section 4 to the appendix, but it may depend on the authors’ writing style.\n4. The ‘inner idea’ concept in the “solo singer” setting looks vague and contradicts with the main topic since it uses chord sequences to synthesize singing voice.\n\nThings to improve the paper that did not impact the score:\n1. “improv” >> “improvement.”\n\nThis paper should be rejected because (1) the paper failed to justify the main idea and results, (2) the amount of literature research was not enough, (3) too many problems were addressed at once, and (4) the writing is not clear enough.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I did not assess the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HklxPZIMcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, authors explore the problem of generating singing voice, in the waveform domain. There exists commercial products which can generate high fidelity sounds when conditioned on a score and or lyrics. This paper proposes three different pipelines which can generate singing voices without necessitating to condition on lyrics or score. \n\nOverall, I think that they do a good job in generating vocal like sounds, but to me it's not entirely clear whether the proposed way of generating melody waveforms is an overkill or not. There is a good amount of literature on generating MIDI representations. One can simply generate MIDI (conditioned or unconditioned), and then give the result to a vocaloid like software. I am voting for a weak rejection as there is no comparison with any baseline. If you can provide a comparison with a MIDI based generation baseline, I can reconsider my decision. Or, explain to me why training on raw waveforms like you do is more preferable. I think in the waveform domain may even be undesirable to work with, as you said you needed to do source separation, before you can even use the training data. This problem does not exist in MIDI for instance. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "B1x9SOtgjH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to addresses an interesting problem of generating singing voice track under three different circumstances. Some of the problems that this paper deals with is a new problem and introduced first in this paper, which could be a contribution as well.\n\nAlthough the paper is fairly well-structured and written, especially in the early sections, I am giving a weak reject due to its weak evaluation. Evaluation is almost always difficult when it comes to generative art, but I am slightly more concerned than that. The literature review can be, although it is nicely done overall, improved, especially on the neural network based singing voice synthesis.\n\nI appreciate the authors tried to find a great problem and provided a good summary of the literature. Successfully training this kind of network itself is already tricky. It is also nice to see some interesting approaches towards objective evaluation. \n\nBelow are my comments. \n\n> Our conjecture is that, as the output of G(·) is a sequence of vectors of variable length (rather than a fixed-size image), compressing the output of G(·) may have lost information important to the task.\n\nI am rather not convinced. The difficulty to discriminate them doesn't seem to be (strongly) related to their variable length for me because a lot of papers have, at least indirectly, dealt with such a case successfully. \n\n> For data augmentation, we transpose the chord progressions found in Wikifonia to 24 possible keys\n\nWhat do you mean by 24 keys? I think there should be only 12 keys. \n\n> Vocalness measures whether a generated singing voice audio sounds like vocals. We use the singing voice detection tool proposed by Leglaive et al. (2015) and made available by Lee et al.(2018).\n\nActually, the paper (Lee et al. 2018) suggested that vocal activity detection in the spectrum domain is easily affected by some features such as frequency modulation. I am not sure if this feature is suitable as a measure of the proposed vocalness. The computed vocalness may provide more information if they are computed on other tracks (e.g., guitar, cello, drums, etc).\n\n> Average pitch: We use the state-of-the-art monophonic pitch tracker CREPE (Kim et al., 2018)8 to compute the pitch (in Hz) for each frame. The average pitch is computed by averaging the pitches of all the frames.\n\nI am pretty sure that this is not the right way to evaluate as a metric of generated vocal track. CREPE is a neural network based pitch tracker, which means it is probably biased by the training data, where the pitch values mostly range in that of common musical instruments/voices. This means, when the F0 of input is not really in the right range, CREPE might incorrectly predict somewhat random F0 within THAT range anyway. I'd say the distribution of pitch values can be interesting metric to show and discuss, but not as a metric of vocal track generation.\n\n\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content:

Blind review #1 summarizes it well:

his paper claims to be the first to tackle unconditional singing voice generation. It is noted that previous singing voice generation approaches leverage explicit pitch information (either of an accompaniment via a score or for the voice itself), and/or specified lyrics the voice should sing. The authors first create their own dataset of singing voice data with accompaniments, then use a GAN to generate singing voice waveforms in three different settings:
1) Free singer - only noise as input, completely unconditional singing sampling
2) Accompanied singer - Providing the accompaniment *waveform* (not symbolic data like a score - the model needs to learn how to transcribe to use this information) as a condition for the singing voice
3) Solo singer - The same setting as 1 but the model first generates an accompaniment then, from that, generates singing voice

--

Discussion:

The reviews generally point out that while a lot of new work has been done, this paper bites off too much at once: it tackles many different open problems, in a generative art domain where evaluation is subjective.

--

Recommendation and justification:

This paper is a weak reject, not because it is uninteresting or bad work, but because the ambitious scope is really too large for a single conference paper. In a more specialized conference like ISMIR, it would still have a good chance. The authors should break it down into conference sized chunks, and address more of the reviewer comments in each chunk.
Decision: Reject