Automatic Music Production Using Generative Adversarial Networks
Authors: Giorgio Barnabò
Giovanni Trappolini
Lorenzo Lastilla
Cesare Campagnano
Angela Fan
Fabio Petroni
Fabrizio Silvestri
Venue:  (2021)
Keywords: music arrangement
generative adversarial networks
music generation

Abstract: When talking about computer-based music generation, two are the main threads of research: the construction of $\textit{autonomous music-making systems}$, and the design of $\textit{computer-based environments to assist musicians}$. However, even though creating accompaniments for melodies is an essential part of every producer's and songwriter's work, little effort has been done in the field of automatic music arrangement in the audio domain. In this contribution, we propose a novel framework for $\textit{automatic music accompaniment}$ $\textit{in the Mel-frequency domain}$. Using several songs converted into Mel-spectrograms, a two-dimensional time-frequency representation of audio signals, we were able to automatically generate original arrangements for both bass and voice lines. Treating music pieces as images (Mel-spectrograms) allowed us to reformulate our problem as an $\textit{unpaired image-to-image translation}$ problem, and to tackle it with CycleGAN, a well-established framework. Moreover, the choice to deploy raw audio and Mel-spectrograms enabled us to more effectively model long-range dependencies, to better represent how humans perceive music, and to potentially draw sounds for new arrangements from the vast collection of music recordings accumulated in the last century. Our approach was tested on two different downstream tasks: given a bass line creating credible and on-time drums, and given an acapella song arranging it to a full song. In absence of an objective way of evaluating the output of music generative systems, we also defined a possible metric for the proposed task, partially based on human (and expert) judgment.

Under review as a conference paper at ICLR 2021
AUTOMATIC MUSIC PRODUCTION USING GENERA- TIVE ADVERSARIAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
When talking about computer- music generation, two are the main threads of research: the construction of autonomous music-making systems, and the design of compuler-based enviromments to assist musicians. However, even though creating accompaniments for melodies is an essential part of every producer's 's and song- writer's work, little effort has been done in the field of automatic music arrange- ment in the audio domain. In this contribution, we propose a novel framework for automatic music accompaniment in the Mel-frequency domain. Using sev- eral songs converted into Mel-spectrograms a two-dimensional time-frequency representation of audio signals we were able automatically generate origi- nal arrangements for both bass and voice lines. Treating music pieces as images -Mel-spectrogrmms) allowed us to reformulate our problem as an unpaired image- to-image translation problem, and to tackle it with CycleGAN, well rell-established framework. Moreover, the choice deploy raw audio and Mel-spectrograms en- abled to more effectively model long-range dependencies, to better represent how humans perceive music, and to potentially draw sounds for new arrangements from the vast collection of music recordings accumulated in the last century. Our approach was tested on two different downstream tasks: given a bass line creat- ing credible and on-time drums, and given acapella song arranging it to full song. In absence of an objective way of evaluating the output of music generative systems, we also defined a possible metric for the proposed task, partially based on human (and expert) judgement.
INTRODUCTION
The development of home music production has brought significant innovations into the process of pop music composition. Software like Pro Tools, Cubase, and Logic- as well as MIDI-based tech- nologies and digital instruments provide a wide set of tools to manipulate recordings and simplify the composition process for artists and producers. After recording a melody, maybe with the aid of guitar or a piano, song writers can now start building up the arrangement one piece at time, some- times not even needing professional musicians or proper music training. As a result, singers and song writers extent even automate, the creation of full songs around their lyrics and melodies. To meet this new demand, the goal of designing computer- based environments assist human musicians has become central in the field of automatic music generation (Briot et al. 2020). IRCAM Open (Assayag al. 1999). Sony CSL-Paris FlowComposer (Papadopoulos
well producers have started asking for tools that could facilitate, or to some
al. 2016). and Logic Pro X Basy Drummer are just some examples. In addition, more solutions based on deep learning techniques, such as RL-Duet (Jiagg al. 2020) deep reinforcement learning algorithm for online accom- paniment generation or PopMAG, a Transformer-basee architecture which relies on multi-track MIDI representation of music (Ren al., 2020), continue to be studied. comprehensive review of the most relevant deep learning techniques applied to music is provided by (Briot et al.. (020). Most of these strategies, however, suffer from the same critical issue, which makes them less appealing in view of music production for commercial purposes: they rely symbolic/MIDI representation of music. The approach proposed in this paper, instead, is a first attempt at automatically generating an euphonic arrangement (two more sound patterns that produce pleasing and harmonious piece of music) in the audio domain, given a musical sample encoded in a -wo-dimensional time- free-freuunccy representation (in partieular, opted for the Mel-spectrogram time-frequency representation). Al-

---

Under review as a conference paper at ICLR 2021
though arrangement generation has been studied in the context of symbolic audio, indeed, switching to -eeestpeess allows us to preserve the sound heritage of other musical pieces (allowing op- erations such sampling) and is more suitable for real-life cases, where voice, for instance, cannot be encoded in MIDI.
We focused attention two different tasks of increasing difficulty: (i) given bass line cre- ate credible and on-time drums, and (ii) given the voice line, to output a new and euphonic musical arrangement. Incidentally, we found out that for training samples our model was able to recon- struct the original arrangement pretty well, even though no pairing among the Mel-spectrograms of the two domains was performed. By means of the Mel-spectrogram representation of music, we can consider the problem of automatically generating an arrangement or accompaniment for spe- cific musical sample equivalent to an image-to-image translation task. For instance, if we have the Mel-spectrogram of an acapella song, we may want to produce the Mel-spectrogram of the same song including suitable arrangement. To solve this task, we tested an unpaired image-to-image translation strategy known as CycleGAN (Zhu et al. 2017). which consists of translating an image from a source domain X to a target domain Y in the absence of paired examples, by training both the mapping from X to Y and from Y to X simultaneously, with the goal of minimizing cycle consistency loss. The aforementioned system was trained on 5s pop music samples (equivalent to 256 x 256 Mel-spectrograms) coming both from the Free Music Archive (FMA) dataset (Defferrard et al. 2017 2018), and from the Demucs dataset (Défossez et al. (019). The short sample duration does not affect the proposed methodology. least with respect to the arrangement task we focus on, and inference can be performed also on full songs. Part of the dataset was pre-processed first, since the FMA songs lack source separated channels (i.e. differentiated vocals, bass, drums, etc.). The required channels were extracted using Demucs (Défossez et al. 2019). The main innovations presented in this contribution are as follows: (i.) treating music pieces as images, we developed a framework to automatically generate music arrangement in the Mel-frequency domain, different from any other previous approach; (ii.) our approach is able to generate arrangements with low computational resourees and limited inference time, if compared to other popular solutions for au- tomatic music generation (Dhariwal et al.) 2020): (ii.. we developed a metric partially based on or correlated to human (and expert) judgement to automatically evaluate the obtained results and the creativity of the proposed system, given the challenges of a quantitative assessment of music. To the best of our knowledge, this is the first work to face the automatic arrangement production task in the audio domain by leveraging a two-dimensional time-frequency representation.
2 RELATED WORKS
The interest surrounding automatic music generation, translation and arrangement has greatly in- creased the last few years, as proven by the high numbers of solutions proposed see (Briot eet al. 2020) for comprehensive and detailed survey. Here we present a brief overview of the key contributions both in symbolic and audio domain.
Music generation & arrangement in the symbolic domain. There is a very large body of research that uses a symbolic representation of music to perform music generation and arrangement. The following contributions used MIDI, piano rolls, chord and note names to feed several deep learning architectures and tackle different aspects of the music generation problem. In (Yang et al.) 2017)
series of MIDI notes either from scratch, by following chord sequence, or by conditioning on the melody of previous bars. In (Mangal et al., [2019] Jaques) let al, 2016 MMgrenn 2016 Makris et al. 2017). LSTM networks are used generate musical notes, melodies, polyphonic music pieces, and long drum sequences, under constraints imposed by metri- cal rhythm information and a given bass sequence. The authors of (Yamshchikov & Tikhonov 2017 Roberts et al. 2018). instead, use VAE networks to generate melodies. In
CNNs are used for generating melody
let al.[ symbolic sequences of polyphonic music are modeled in a completely general piano- roll representation, while the authors of (Hadjeres & Nielsen (017)) propose novel architecture to generate melodies satisfying positional constraints in the style of the soprano parts of the J.S. Bach chorale harmonisations encoded in MIDI. In (Johnson] 2017). RNNs are used for prediction and composition of polyphonic music; in (Hadjeres et al.) 2017). highly convincing chorales in the style of Bach were automatically generated using note names; (Lattner et al. 2018) added higher- structure on generated, polyphonic music, whereas (Mao et al. (2018) designed an end-to-end gener- ative model capable of composing music conditioned specific mixture of composer styles. The
2

---

Under review as a conference paper at ICLR 2021
approach described in (Hawthorne et al. 2018), instead, relies notes intermediate represen- tation to a suite of models - namely, a transcription model based on a CNN and a RNN network (Hawthorne et al.] 2017),
seff-attention-basee music language model (Huang et al. 2018) and WaveNet model (Oord et al., 2016) capable of transcribing, composing, and synthesizing audio waveforms. Finally, (Zhu et al. (018)) proposes an end-to-end melody and arrangement genera- tion framework, called Xiaolce Band, which generates a melody track with several accompaniments played by several types of instruments. As this extensive literature on music generation in the sym- bolic domain shows, promising approach would be to work with symbolic music and then use ofte--of-the-art synthesizers to produce sounds. MIDI, music sheets and piano rolls, however, are not always easy to find or produce. Moreover, many musicians and artists can not read music and would be more comfortable to work in a less formalized setting. Finally, state-of-the-an synthesiz- ers, although increasingly indistinguishable from live recordings, can not yet reproduce the infinite nuances of real voices and instruments. Conversely, raw audio representation could be more appeal- ing for some creators given its flexibility and little music competence required.
Music generation & arrangement in the audio domain. Some of the most relevant approaches proposed far in the field of waveform music generation deal with raw audio representation in the time domain. Many of these approaches draw methods and ideas from the extensive literature on audio and speech synthesis. For instance, in (Prenger al. 2019) flow-based network capable of generating high quality speech from mel-spectrogrrms is proposed, while in (Wang et al.) 2019) the authors present a neural source-filter (NSF) waveform modcling framework that straightforward to train and fast to generate waveforms. In (Zhao et al., (020) recent neural waveform synthesiz- ers such as WaveNet, WaveG-low, and the neural-source- filler (NSF) models are compared. (Mchri et al. 2016) tested a model for unconditional audio generation based on generating one audio sample at a time, and (Bhave et al. 2219) applied Restricted Boltzmann Machine and LSTM architectures to raw audio files in the frequency domain in order to generate music. A fully probabilistic and autore- gressive model, with the predictive distribution for each audio sample conditioned on all previous ones, is used in (Oord et al. 2016) to produce novel and often highly realistic musical fragments. (Manzelli et al. 2018) combined two types of music generation models, namely symbolic and raw audio models, to train raw audio model based on the WaveNet architecture, but that incorporates the notes of the composition as a secondary input to the network. Finally, in (Dhariwal et al. 2020) the authors tackled the long context of raw audio using a multi-seale VQ- (QEV to compress it to discrete codes, and modeled such context through Sparse Transformers, in order to generate music with singing in the raw audio domain. Nonetheless, due to the computational resources required to directly model long-range dependencies in the time domain, either short samples of music can be generated or complex and large architectures and long inference time are required. On the other hand, in (Vasquez & Lewis 2019) novel approach is discussed, which proves that long-range de- pendencies can be more tractably modelled in -wo-dimensional fime-frquency representations such as Mel-spectrograms. More precisely, the authors of this contribution designed highly expressive probabilistic model and a multiscale generation procedure over Mel-spectrograms capable of gener- ating high-fidelity audio samples which capture structure at timescales. It is worth recalling, as well, that treating spectrograms as images is the current standard for many Music Information Retrieval tasks, such music transcription (Sigtia et al. 2016) and chord recognition.
Generative adversarial networks for music generation. Our work is precisely founded on this novel assumption, thus taking the best from the raw audio representation, while tackling the main issues induced by musical signals long-range dependencies thanks to the sovefform-to-specrrogree conversion. Such two-dimensional representation of music paves the way to the application of sev- eral image processing techniques and image-to-imase translation networks to carry out style trans- fer and arrangement generation (Isola et al. 2017 Zhu et al. 2017). It is worth recalling that the application of GANs to music generation tasks is not new: in (Brunner et al. 2018), Generative Adversarial Networks are applied on symbolic music to perform music genre transfer; however, to the best of our knowledge, GANs have never been applied to raw audio in the Mel-frequency do- main for music generation purposes. As to the arrangement generation task, also in this case the large majority of approaches proposed in literature is based on symbolic representation of music: in (Ren et al. 2020). a novel Multi-track MIDI representation (MuMIDI) is presented, which enables simultaneous multi-track generation in single sequence and explicitly models the dependency of the notes from different tracks by means of a Transformer- based architecture; in (Jiang et al.) (2020), a deep reinforcement learning algorithm for online accompaniment generation is described. Coming to the most relevant issues in the development of music generation systems, both the training and
3

---

Under review as a conference paper at ICLR 2021
evaluation of such systems haven proven challenging, mainly because of the following reasons: (i) the available datasets for music generation tasks are challenging due to their inherent high-entropp (Dieleman et al. 2018), and (ii) the definition of an objective metric and loss is a common problem to generative models such as GANs; at now, generative models in the music domain are evaluated based on the subjective response of pool of listeners, and just for the MIDI representation set of simple musically informed objective metrics was proposed (Yang & Lerch 2020).
METHOD
3.1 SOURCE SEPARATION FOR MUSIC
We present novel framework for automatic music arrangement generation using an adversarially trained deep learning model. A key challenge to our approach is the scarce availability of mu- sic data featuring source separated channels (i.c. differentiated vocals, bass, drums, ...). To this end, we leverage Demucs by Défossez et al., freely available tool, which separates music into its generating sources. Demucs features a U-NET encoder-decoder architecture with a bidirectional LSTM middle hidden layer. In particular we used a pre-trained model made available by the original authors, consisting of 6 convolutional encoder and decoder blocks and a middle hidden size of length 3200. Demucs time-equivariant, meaning that shifts in the input mixture will cause congruent shifts in the output. The model does not feature this property naturally, but it is achieved through a workaround (randomized equivariant stabilization) as explained by the original authors. Nonetheless, at times this method produces noisy separations - with watered-down harmonics and traces of other instruments in the vocal segment effectively hindering the ability of later part of the pipeline to properly recognise and reconstruct accompaniment, of which harmonics are a critical part. While better source-separation methods are available [Sota SDR = 5.85, Demucs SDR = 5.67), we chose to use Demucs because it was faster and easier to embed in our pipeline. Moreover, for bass source separation it beats the state of the art [Sota SDR = 5.28, Demues SDR = 6.21J. Finally, were adamant about picking tool using deep learning because this may open the possibility to build an end-to end-to-end trained pipeline in the future. This at least partially solves the challenge of data availability and allows to feed our model with the appropriate signals.
3.2 MUSIC REPRESENTATION - FROM RAW AUDIO TO MEL-SPECTROGRAMS
One of the main features of our method is to choose a two-dimensional time-frequency representa- tion of the audio samples rather than a time representation. The spectrum is common transformed representation for audio, obtained via a Short-Time Fourier transform (STFT). The discrete STFT of given signal x : [0 L ---11 1] = 1, L- L-11 13 + R leads to the kth complex Fourier coefficient for the m th time frame X'm, k) = - x(n + mlf)- w(n) e 2=/²" with m E [0 M] and K E [0= : K]. and where w(n) is sampled window function of length N E N and И E N is the hop size, which determines the step size in which the window is to be shifted across the signal (Mäller 2015). The spectrogram is two wwo-dimensional representation of the squared magnitude of the STFT, i.e. ('mm,)) k)) == X'm k with m E [0- M) and K E [0 K]. Figure | shows a Mel-spectrogram example (Stevens et al. 1937). which is treated single channel image, repre- senting the sound intensity with respect to time- x axis - and frequency -y axis (Briot et al. (020)) This decision allows to better deal with rongerange dependencies typical of such kind of data and to reduce the computational resources and inference time required. Moreover, the Mel-scale is based on a mapping between the actual frequency ff and perceived pitch m = 2595 log1v(1 + 16 ). as the human auditory system does not perceive pitch in a linear manner. Finally, using Mel-spectrograms of pre-existing songs to train our model potentially enables to draw sounds for new arrangements from the vast collection of music recordings accumulated in the last century.
After the source separation task was carried out on our song dataset, each source (and the full song) waveforms were turned into corresponding Mel-spectrograms. This has been done using PyTorch Audica] to take advantage of robust, GPU accelerated conversion. We decided to discard the phase information in this process, to reduce the dimensionality of the representation. To revert back to the time-domain signal, we: (i.) apply a conversion matrix (using triangular filter banks)
Available at: http:////pporroo..o.yy//aaaaa/aa

---

Under review as a conference paper at ICLR 2021
Mel. frequency spectrogram
8192
+10 d8
Reeuancy (Hc)
2048
+0
1024
-30 dB
-40 dB
10
15
20
Figure Example of a Mel-spectrogram
convert the Mel-frequency STFT a linear scale STFT, where the matrix is calculated using gradient-based method (Decorsière et al. (2115) to minimize the euclidean norm between the original Mel-spectrogram and the product between reconstructed spectrogram and filter banks; (ii.) use the Griffin-Lim's algorithm (Griffin & Jae Lim 1984) to reconstruct phase information.
is worth noticing that Mel-scale conversion and the removal of STFT phases respectively discard frequency and temporal information, that results in distortion in the recovered signal. To minimize this problem, we use a high-enough resolution of the Mel-spectrograms (Vasquez Lewis] 2019), whose size can be tweaked with number of mels and STFT hop size parameters. Thus, the optimal parameters we found were the following ones: the sampling rate was initially set to 22050 Hz, the window length N to 2048, the number of Mel-frequency bins to 256 and the hop size H to 512. To fit our model requirements, we cropped out 256 x 256 windows from each Mel-spectrogram with an overlapping of 50 time frames, obtaining multiple samples from each song (each equivalent seconds of music).
3.3 IMAGE TO IMAGE TRANSLATION CYCLEGAN
The automatic arrangement generation task faced through unpaired image-to-imee to-image transla- tion framework, by adapting the CycleGAN model to our purpose. CycleGAN is a framework able translate between domains without paired input-output examples, by assuming some underlying relationship between the domains and trying to learn that relationship. Based on a set of images in domain X and a different set in domain Y. the algorithm learns both a mapping G X Y and mapping F : Y + X, such that the output ğ = G(x) for every x E X, is indistinguishable from images y
and & Ggy) for every y Y. is indistinguishable from images x E X. The other relevant assumption is that, given mapping G X + Y and another mapping F Y + X, then G and F should be inverses of each other, and both mappings should be bijections. This assumption implemented by training both the mapping G and F simultaneously, and adding cycle consistency loss that encourages ((G()) & x and G(F(y)) 2 z. The cycle consistency loss then combined with the adversarial losses domains X and Y (Zhu et al. 2017)
3.4 AUTOMATIC MUSIC PRODUCTION
The method we propose takes as input set of N music songs in the waveform domain X {x; ) where x; is waveform whose number of samples depends on the sampling rate and the audio length. Each waveform is then separated by Demucs into three different sources. Thus, we end up having four different WAV files for each song, which means new set of data of the kind: XNEW = {x4, Vi. d, by [ where Vi, b, d; represents vocal, bass, and drums respectively. Each track is then converted to its Mel-spectrogram representation. Since the CycleGAN model takes 256 256 images as input, each spectrogram is chunked into smaller pieces with an overlapping window of 50 time frames; finally, in order to obtain channel images from the original spectro-
5

---

Under review as a conference paper at ICLR 2021
Dy
D
X
G
Dx
X
Dy
xt
y
X
G
F
X
Y
X
Y
X
gyele-eatsisterng loess
eycls cousistency loss
F
Figure 2: Representation of the CycleGAN model, which consists of two mapping functions G and
F, two discriminators Dx and Dy and two cycle-consistency losses (Zhu et al. 2017))
grams, we performed discretization step in the range [0- 255]. In the final stage of our pipeline, we feed the obtained dataset to the CycleGan model, that has been adapted to the structure this data. Even though the discretization step introduces some distortion original spectrogram values are floats - the impact on the audio quality is negligible.
At training time, the model takes into account two domains X and Y, we considered two different experimental settings of increasing difficulty: (i.) we feed the model with bass and drums lines in order create suitable drums given a bass line; (ii.) take the vocals and the whole song respectively, with the goal of generating an arrangement euphonic to the vocal line. Solving the bass2drums task effectively would represent a first interesting intermediary goal. Drums and bass are usually the first instruments to be recorded when producing a song. Ideally, though, a system the automatically generates full song given voice line as input would be far more ambitious and disruptive because it would allow anyone to express her/himself in music.
EXPERIMENTS
4.1 DATASET
For the quality of the generated music samples, it is important carefully pick the training dataset. To train and test our model we decided to use the Free Music Archive (FMA) and the musdb118 dataset RRaii et al. 2017) that were both made available quite recently. The Free Music Archive (FMA) is the largest publicly available dataset suitable for music information retrieval tasks (Def-) ferrard et al. 2017 2018). In its full form it provides 917 GB and 343 days of Creative Commons- licensed audio from 106,574 tracks from 16.341 artists and 14,854 albums, arranged in a hierarchical taxonomy of 161 unbalanced genres. It provides full-length and high-quality audio, pre-computed features, together with track- and user-level metadata, tags, and free-form text such as biographies. Given the size of FMA, we chose to select only untrimmed songs tagged as either pop. soul-RnB, or indic-rock, for total of approximately 10,000 songs (700 hours of audio). It is possible to read the full list of songs at FMA website, selecting the genre. We discarded all songs that were recorded live by filtering out all albums that contained the word "live" in the title. Finally, in order to better validate and fine-tune our model we decided to also use the full musdb15 88 dataset. This rather small dataset is made up of 100 tracks taken from the DSD100 dataset, 46 tracks from the MedleyDB, tracks kindly provided by Native Instruments, and 2 tracks from the Canadian rock band The Easton Ellises. It represents a unique and precious source of songs delivered in multi-track fashion. Each song comes as 5 audio files vocals, bass, drums, others. full song perfectly separated at the master level. We used the 100 tracks taken from the DSD100 dataset to fine-tune the model ( 6.5 hours), and the remaining 50 songs to test it ( (3.5 hours). It is worth noting that DEMUCS is not a perfect method for source separation: introduces artefacts and noise of the original song in the separated sources output, making the task easier and inducing the model to learn to amplify it. For this reason our training strategy is to pre-train with the artificial FMA dataset then fine-tune with musdb18 18. Intuitively, the former, which is much larger, helps the model to create good representa- tion of musical signal; the latter, which is of higher quality, contributes to reducing bias induced by the noise and further specializing generate base relying only on the (clean) input given.

---

Under review as a conference paper at ICLR 2021
Table 1: Pearson's correlation matrix for all 4 annotators Guitarist
Drummer
Producer 1
Producer 2
Guitarist
na
0.82
0.75
0.77
Drummer
0.82
na
0.76
0.79
Producer 1
0.75
0.76
na
0.85
Producer 2
0.77
0.79
0.85
na
4.2 TRAINING OF THE CYCLE-GAM MODEL
For both the bass2drums and voice2song tasks, we trained our model on 2 Tesla V100 SXM2 GPUs with 32 GB of RAM for 12 epochs (FMA dataset), and fine-tuned it for 20 more epochs (musdb 18 dataset). Each task required days of training. For both the settings, a final step, the spectrograms obtained were converted to the waveform domain, to evaluate the produced music. As to the Cycle- relied the default network available at this GitHub repository
GAN model used for training, As a result, the model uses a resnet 9blocks ResNet generator and a basic 70x70 PatchGAN as a discriminator. The Adam optimizer (Kingma & Ba 2014) was chosen both for the generators and the discriminators, with betas (0.5, 0.999) and learning rate equal to 0.0002. The batch size was set 1. The À weights for cycle losses were both equal 10.
4.3 EXPERIMENTAL SETTING
There is an intrinsic difficulty in objectively evaluating artistic artifacts such as music. As a human construct, there are objective, universal criteria for appreciating music. Nevertheless, in order to establish some form of benchmark and allow comparisons among different approaches, many generative approaches raw audio, such Jukebox (Dhariwal et al.. 2020) or Universal Music Translation Network (Mor et al. 2018). try to overcome this obstacle by having the results manually tagged by human experts. Although this rating may be the best in terms of quality, the result still somehow subjective, thus different people may end up giving different or biased ratings based on their personal taste. Moreover, the computational cost and time required to manually annotate the dataset could become prohibitive even for relatively few samples (over 1000). Aware of the great limits linked to this human-based approach and unable to find a more convincing evaluation proce- metric that highly correlates with human judgment. This could represent
dure, we propose first benchmark for the tasks at hand. The results remain somehow subjective, but at least we were able to automatically replicate evaluators' criteria and grades, saving time and money.
4.4 METRICS
If we consider as general objective for a system the capacity to assist composers and musicians, rather than to autonomously generate music, should also consider evaluation criteria the satisfaction of the composer (notably, the assistance of the computer allowed him to compose and create music that he may consider not having been possible otherwise), rather than the satisfaction of the auditors (who remain too often guided by some conformance to a current musical trend) (Briot al. 2020).
However, as previously stated, an exclusive human evaluation may be unsustainable in terms of computational cost and time required. Thus carried out the following quantitative assessment of our model. We first produced 400 test samples - from as many different songs and authors of artificial arrangements and drum lines starting from voice and bass lines that were not part of the training set. We then asked professional guitarist who has been playing in a pop-rock band for more than 10 years, a professional drum player from the same band, and two pop and indie-rock music producers with more than 4 years of experience to manually annotate these samples, capturing the following musical dimensions: quality, euphony, coherence, intelligibility. More precisely, for each sample, we asked them to rate from 1 to 10 the following aspects: (i) Quality: rating from to 10 of the naturalness and absence of artifacts or noise, (ii) Contanination: a rating from 1 to 10 of the contamination by other sources, (iii) Credibiliry: rating from 1 to 10 of the credibility of the sample, (iv) Time: a rating from to 10 whether the produced drums and arrangements are on time the bass and voice lines. The choice fell these four aspects after asked the evaluators to

---

Under review as a conference paper at ICLR 2021
list and describe the most relevant dimensions in the perceived quality of a piece of pop-rock music. The correlation matrix for all 4 annotators is shown in Table
Ideally, we want to produce some quantitative measure whose outputs when applied to generated samples - highly correlates (i.e. predict) expert average grades. To achieve this goal, we trained logistic regression model with features obtained through a comparison between the original rangement and the model output, as well as the original drums and the artificial drums. Here are the details on how we obtained suitable features:
STOI-like features. We created a procedure - inspired by the STOI (Andersen et al., 2017) whose output vector somehow measures the Mel-frequency bins correlation throughout time between the original sample (arrangement/ruums) and the fake one. The obtained vector can then be used to feed a multi regression model whose independent variable is the human score attributed that sample.
256 (9) (()) g(()) To simplify, Here is the formalisation: H HumanScore = - 256 ai L
to each pair of samples (original and generated onc) a 256 element long vector is associated as follows: V1 ()) = Lt (x! - (r,)) g()). Where: (i.) X' and Y are, respectively,
the Mel-spectrogram matrices of original and generated samples; (ii.) a, is the i-th coefficient for the linear regression; (ii..) z;
(+) and vi the i-th element of the t-th column of matrices X and V. respectively; (iv.) z(9) and 7(() are the means along the I-th column of matrices X' and V. respectively. Each feature of the regression model is sort of Pearson correlation coefficient between row of X' and row of y throughout time.
FID-based features. In the context of GANs result evaluation, the Fréchet Inception distance (FID) is supposed to improve on the Inception Score by actually comparing the statistics of generated samples to real samples (Salimans et al., 2016 Heusel et al.) 2017). In other words, FID mea- sures the probabilistic distance between two multivariate Gaussians, where X, = N NHr,,,, and X, = N (Hg, Eg) are the 4048-dimensional activations of the Inception-v3 pool3 layer - for real and generated samples respectively - modeled as normal distributions. The similarity between the two distributions is measured as follow: FID |,,-- - H + Tr(, + +2,, 2(2, 11/2). Nev- ertheless, since we want to assign a score to each sample, we just estimated the X, = [(r: E,) parameters using different activation layers of the Inception pre-trained network and then we calculated the probability density associated to each fake sample. Finally, we added these scores to the regression model predictors.
4.5 EXPERIMENTAL RESULTS
For the bass2drums task, Figure 3 shows the distribution of grades for the 400 test samples aver- aged among all four independent evaluators and over all the four dimensions. We rounded the results to the closest integer to make the plot more readable. The higher the grade, the better the sample will sound. Additionally, to fully understand what to expect from samples graded similarly, we dis- cussed the model results with the evaluators. We collectively listened to random set of samples and it turned out that all four raters followed similar principles in assigning the grades. Samples with grade 1-3 are generally silent very noisy. In samples graded 4-5 few sounds start to emerge, but they are usually not very pleasant to listen to, nor coherent. Grades 6-7 identify drums that sound good, that are coherent, but that are not continuous: they tend follow the bass line too closely. Finally, samples graded 8 and 9 are almost indistinguishable from real drums, both in terms of sound and timing. In the labeling of non graded samples phase, we therefore assigned 0 those samples whose average grade was between and 5, and I to those between 6 and 10. Finally, we trained a multi- logistie regression model with both the SiOI-like and the FID-based features. The model accuracy on test set was 87%.
Given this pretty good result, we could then used this trained logistic model to label 14000 different 5s fake drums clips, produced from as many real bass lines. Two third of these were labeled as good sounding and on time. Here is private Sound Cloud playlist where you can listen to some of the most interesting results. Regarding instead the voice2song task, results were less encouraging. Even though some nice arrangements were produced, the model failed to properly and euphonically arrange the input voice lines. For this reason, here we limit to report some of the best produced samples, in the hope to improve the model greatly in the following months. As for baselines, ini- tially we thought about comparing our results to three particularly notable works (Dhariwal al.
8

---

Under review as a conference paper at ICLR 2021
0.25
Relative Freq.
0.15
0.05
2
3
7
8
10
Avg. Grade
Figure 3: Grade distribution of generated drums samples
Vasquez & Lewis, 2019 Mor et al. 2018), but after running some experiments eventually realized that they could not be properly used for arrangement purposes. All three model produce very nice music samples, but none of them can take input vocals bass lines and produce complementary arrangement. It is possible though that these models could fine tuned to solve this task. In addition, replicated exactly the same experiments using Pix2Pix by Isola al. a well known paired maag-tto-mmage architecture. Despite long training, results were very poor and quite unpleasant to listen Due space concerns do report more details about this set of experiments.
2020
Finally, with respect the computational resources and time required to generate arrangements, our approach shows several advantages, compared to auto-regressive models (Dhariwal et al. 2020). Since the output prediction can fully parallelised, the inference time amounts a forward pass and Mel-spectrogram- til-speectogrmm-waeefomo waveform inverse conversion, whose duration depends on the input length, but never exceeds few minutes. Indeed, it is worth noting that, at inference time, arbitrary long inputs can be processed and arranged.
CONCLUSIONS AND FUTURE WORK
In this work, we presented a novel approach to automatically produce cuphonic music arrange- ments starting from voice line bass line. We applied Generative Adversarial Networks to real music pieces, treated as grayscale images (Mel-spectrogramss) Given the novelty the problem, proposed reasonable procedure to properly evaluate model outputs. Notwithstanding the promising results, some critical issues need to be addressed before more compelling architecture be developed. First and foremost, larger and cleaner dataset of source separated songs should be created. In fact, manually separated track always contain a big deal of noise. Moreover, the model architecture should be further improved focus on longer dependencies and take into account the actual degradation of high frequencies. Finally, certain degree of interaction and randomness should inserted to make the model less deterministic and give creators some control over the sample generation. Our contribution is nonetheless a first step toward more realistic and useful au- tomatic music arrangement systems and we believe that further significant steps could made to reach the final goal of human-level automatic music arrangement production. Already now software like Melodyne (Neubäcker) 2011 Senior 2009) delivers producers a powerful user interface to di- rectly intervene and restructure vocals, samples and recordings of all kinds. It is not unlikely that the future artists and composers will start creating their music almost like they were drawing.
spectrogram based representation of audio signals to correct, perfect, reshape

---

Under review as a conference paper at ICLR 2021
REFERENCES
Asger Heidemann Andersen, Jan Mark de Haan, Zheng-Hua Tan, and Jesper Jensen. non-intrusive short-time objective intelligibility measure. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5085-5089. IEEE, 2017.
Gérard Assayag, Camilo Rueda, Mikael Laurson, Carlos Agon, and Olivier Delerue. Computer- assisted composition at iream: From patchwork openmusic. Computer music journal, 23(3): 59-72, 1999.
Aishwarya Bhave, Mayank Sharma, and Rekh Ram Janghel. Music generation using deep learning. In Soft Computing and Signal Processing, pp. 203 201-211.22 Springer, 2019.
Nicolas Boulanger-Lewanddowkii Yoshua Bengio, and Pascal Vincent. Modeling temporal depen- dencies in high-dimensional sequences: Application to polyphonic music generation and tran- scription. arXiv preprint arXiv: 1rX.v:/206.6392 2012.
Jean-Pierre Briot, Gaëtan Hadjeres, and François François-David Pachet. Deep learning techniques for music generation. Springer, 2020.
Gino Brunner, Yuyi Wang. Roger Wattenhofer, and Sumu Zhao. Symbolic music genre transfer with cyclegan. In 2018 IEEE 30th International Conference on Tools with Artificial Intelligence (ICTAI). pp. 783.793. IEEE, 2018.
R. Decorsière, L. Søndergaard, E. N. MacDonald, and T. Dau. Inversion of auditory spectrograms, traditional spectrograms, and other envelope representations. JEEE/ACM Transactions Audio, Speech, and Language Processing, 23(1):46-56, 2015. doi: 10.1009TASLP2214233655
Michaël Defferrard, Kirell Benzi, Pierre Vandergheynst, and Xavier Bresson. FMA: dataset for
music analysis. In /8th International Society for Music Information Retrieval Conference (IS- MIR), 2017. URL httlhhtp:://// /atttp::/arxiv..oom wwtp:///////aaaraa.o.o/...ssssra abs/ 111.2111..00 01840
Michaël Defferrard, Sharada P. Mohanty, Sean Carroll, and Marcel Salathé. Learning to recognize musical genre from audio. In The 2018 Web Conference Companion. ACM Press, 2018. ISBN 9781450356404. doi: 014451144558.312333
Utt.httss:///eettt http://axxivg org/ abtp///b//1000
05337
Alexandre Défossez, Nicolas Usunier, Léon Bottou, and Francis Bach. Demues: Deep extractor for music sources with extra unlabeled data remixed. arXiv preprint urXiv:1909.01/ 074, 2019.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: A generative model for music. arXiv preprint 200v:2005500341, 2020.
Sander Dieleman, Aaron van den Oord, and Karen Simonyan. The challenge of realistic music generation: modelling raw audio scale. In Advances in Neural Information Processing Systems, pp. 7989-7999, 2018.
D. Griffin and Jae Lim. Signal estimation from modified short-time fourier transform. IEEE Trans- actions on Acoustics, Speech, and Signal Processing, 32(2):236-243, 1984. doi: 10. 10.TTSP/T
Gaëtan Hadjeres and Frank Nielsen. Interactive music generation with positional constraints using anticipation-mnns. arXiv preprint arXiv: 7709.064944 2017.
Gaëtan Hadjeres, François Pachet, and Frank Nielsen. Deepbach: a steerable model for bach chorales generation. In International Conference on Machine Learning, pp. 1362-1371. PMLR, 2017.
Curtis Hawthorne, Erich Elsen. Jialin Song, Adam Roberts, Ian Simon, Colin Raffel, Jesse Engel, Sageev Oore, and Douglas Eck. Onsets and frames: Dual-objective piano transcription. arXiv preprint arXiv: / 153, 2017.
Curtis Hawthorne, Andriy Stasyuk, Adam Roberts, Ian Simon, Cheng-Zhi Anna Huang, Sander Dieleman, Erich Elsen, Jesse Engel, and Douglas Eck. Enabling factorized piano music modeling and generation with the maestro dataset. arXiv preprint arXiv:18/00 aeti:/0042.2200 /2247, 2018.
10

---

Under review as a conference paper at ICLR 2021
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Ian Simon, Curtis Hawthome, Noam Shazeer, Andrew M Dai, Matthew D Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Generating music with long-term structure. In International Conference on Learning Representations, 2018
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1125-1134, 2017.
Natasha Jaques, Shixiang Gu, Richard E Turner, and Douglas Eck. Tuning recurrent neural networks with reinforcement learning. arXiv preprint arXiv:1611.027900 2016.
Nan Jiang, Sheng Jin, Zhiyao Duan, and Changshui Zhang. RI-duet: Online music accompani- ment generation using deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 710-718, 2020.
Daniel D Johnson. Generating polyphonic music using tied parallel networks. In International conference on evolutionary and biologically inspired music and art, pp. 128-143. Springer, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412 69800 2014.
Stefan Lattner, Maarten Grachten, and Gerhard Widmer. Imposing higher-level structure in poly- phonic music generation using convolutional restricted boltzmann machines and constraints. Jour- nal of Creative Music Systems, 2(2), 2018
Dimos Makris, Maximos Kaliakatsos- Silakaatso-PPapkkostaass
Papakostas, Ioannis Karydis, and Katia Lida Kermanidis. Combining istm and feed forward neural networks for conditional rhythm composition. In In- ternational Conference on Engineering Applications of Neural Networks, pp. 570-582. Springer, 2017.
Sanidhya Mangal, Rahul Modak, and Poorva Joshi. Lstm based music generation system. arXiv 2019.
preprint
Rachel Manzelli, Vijay Thakkar, Ali Siahkamari, and Brian Kulis. An end to end model for auto- matic music generation: Combining deep raw and symbolic audio networks. In Proceedings of the Musical Metacreation Workshop at 9th International Conference on Computational Creativ- ity, Salamanca, Spain, 2018
Huanru Henry Mao, Taylor Shin, and Garrison Cottrell. Deepj: Style-specific music generation. In 2018 IEEE 12th International Conference on Semantic Computing (ICSC), pp. 377-382. IEEE, 2018.
Soroush Mehri, Kundan Kumar, Ishaan Guirajani. Rithesh Kumar, Shubham Jain, Jose Sotelo, Aaron Courville, and Yoshua Bengio. Samplemn: An unconditional end-to-end neural audio generation model. arXiv preprint arXiv: 2016.
Olof Mogren. C-mn-gann Continuous recurrent neural networks with adversarial training. arXiv preprint 161 1.09004, 2016.
Noam Mor, Lior Wolf, Adam Polyak, and Yaniv Taigman. A universal music translation network. arXiv preprint arXiv:/005.00 tete.088455.5 2018.
Meinard Müller. Fundamentals of music processing: Audio, analysis, algorithms, applications. Springer, 2015.
Peter Neubäcker. Sound-object oriented analysis and note-object oriented processing of polyphonic sound recordings, September 20 2011. US Patent 8,022,286.

---

Under review as a conference paper at ICLR 2021
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 2016.
Alexandre Papadopoulos, Pierre Roy, and François Pachet. Assisted lead sheet composition using flowcomposer. In International Conference on Principles and Practice of Constraint Program- ming, pp. 769-785. Springer, 2016.
Ryan Prenger, Rafael Valle, and Bryan Catanzaro. Waveglow: A flow-based generative network for speech synthesis. In ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp- 3617-3621. IEEE, 2019.
Zafar Rafii, Antoine Liutkus, Kabian-Rober Stöter, Stylianos Ioannis Mimilakis, and Rachel Bittner.
The MUSDBI8 corpus for music separation, December 2017. https:t::///o wtgg/....ogg/..o
Yi Ren, Jinzheng He, Xu Tan, Tao Qin, Zhou Zhao, and Tie- Liu. Popmag: Pop music accom- paniment generation. arXiv preprint wxXiv2008.007703 2020.
Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck. hierarchical latent vector model for learning long-term structure in music. arXiv preprint arXiv: 054288528, 2018.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neurul information processing systems, 2016.
pp.
M Senior. Celemony melodyne dna editor. Sound on Sound, 2009.
Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon. An end-to-end neural network for poly- phonic piano music transcription. JEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(5):927-939 2016.
Stanley Smith Stevens, John Volkmann, and Edwin B Newman. scale for the measurement of the psychological magnitude pitch. The Journal of the Acoustical Society of America, 1937.
Sean Vasquez and Mike Lewis. Melnet: A generative model for audio in the frequency domain. arXiv preprint arXiv:1906. 100600030000083,, 2019.
Xin Wang, Shinji Takaki, and Junichi Yamagishi. Neural source-filter waveform models for sta- tistical parametric speech synthesis. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:402-415, 2019.
Ivan P Yamshchikov and Alexey Tikhonov. Music generation with variational recurrent autoencoder supported by history. arXiv preprint arXiv:1705.05455. 2017.
Li-Chia Yang and Alexander Lerch. On the evaluation of generative models in music. Neural Computing and Applications, 3779)447734474, 2020.
Li-Chia Yang, Szu-Yo Yu Chou, and Yi-Hsuan Yang. Midinet: A convolutional generative adversarial
network for symbolic-domain music generation. arXiv preprint /703. 2017.
Yi Zhao, Xin Wang, Lauri Juvela, and Junichi Yamagishi. Transferring neural speech waveform synthesizers to musical instrument sounds generation. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6269-6273. IEEE, 2020.
Hongyuan Zhu, Qi Liu, Nicholas Jing Yuan, Chuan Qin, Jiawei Li, Kun Zhang, Guang Zhou, Furu Wei, Yuanchun Xu, and Enhong Chen. Xiaoice band: A melody and arrangement generation framework for pop music. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2837-2846, 2018
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
12


Reviews:
[{"review_id": "tBjdNZSl8n", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In the paper, the authors adapt CycleGAN, a well-known model for unpaired image-to-image translation, to automatic music arrangement by treating MFCCs extracted from audio recordings as images. Also, the authors propose a novel evaluation metric, which learns how to rate generated audio from the ratings of (some) music experts. The authors make use of two large-scale datasets to train and evaluate the model on two scenarios, namely 1) generating drum accompaniment a given bass line, 2) generating arrangement given a voice line. They report promising results on the first task; however, the model is not as successful on the second (more challenging) task.\n\nThe problem is challenging, and meaningful solutions may bring innovative and creative solutions to music production. The literature is well-covered, with a few missing citations (see below). The approach is built upon existing work, and the experiments are conducted on two relevant, public datasets. On the other hand, the experimental code is not shared, and the dataset section lacks a few details to reproduce the findings easily.\n\nBelow are the shortcomings of the paper:\n\n1. While adapting past music generation work for arrangement generation is not trivial, the authors could have still used variants of CycleGAN and other unpaired image-to-image translation models for comparison.\n2. The sources are primarily limited to bass, drums, and vocals. I do not think the narrow scope is an issue on a paper focusing on an unexplored subject. On the contrary, the experiments could have more variety, e.g. drums2bass, bass&vocals2drums, and other combinations, so that we could examine which settings bring interesting and/or challenging outcomes in arrangement generation.\n4. The evaluation and discussion could have more depth, e.g. inter-annotator agreement, the effect of source separation in the generated audio (separation errors, audible artifacts, ...)\n\nThe paper is novel in its application and brings promising results. However, the authors should extend the experiments, compare relevant models against each other, and discuss the results more in detail. Therefore, I would strongly encourage the authors to build upon their existing work and re-submit the revised paper to ICLR or another conference such as ISMIR.\n\nSpecific comments\n=================\n\n- As mentioned above, the authors should have added more \"experimental settings.\" At least they should have included \"generation of a bass line given the drums\" (reverse of bass2drums) because 1) it would have allowed the readers to contrast the performance with bass2drums, 2) the task would be closer to the real-world use case (drums are typically the first to be recorded in a session followed by bass).\n\n- The method works on music strictly with drums, bass and vocals, which is not mentioned until Section 3.4. This limitation/condition should be specified clearly and earlier in the Introduction and/or in Section 3.1.\n\n- \"Nevertheless, only raw audio representation can produce, at least in the long run, appealing results in view of music production for artistic and commercial purpose.\"\n\n  Even if we restrict ourselves to popular music, this argument is too ambitious if not misleading. Many artists (performers, composers, conductors, etc.) are not only well fledged but - by profession - required to appreciate music by reading sheet music. Countless programmable interfaces and software, which make use of symbolic/hybrid music representations but do not generate raw audio directly, have been used extensively as part of music performances and production in a both artistic and commercial setting. While audio - without any doubt - is the essence of music, we can never disregard other representations.\n\n- Citing the two papers below could improve the literature review:\n\n  >Hawthorne, Stasyuk, Roberts, Simon, Huang, Dieleman, Elsen, Engel and Eck, \"Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset\", International Conference on Learning Representations, 2019. => similar to the authors' design decision, this paper uses a cheaper intermediate representation (music scores) for efficiency\n\n  >Donahue et al. LakhNES: Improving multi-instrumental music generation with cross-domain pre-training => the paper involves mapping (\"arranging\") the instrumentation in MIDI files to NES sound channels.\n\n- Please cite `FMA` and `MusDB18` datasets following the instructions in the respective online sources.\n\n- Section 3.1. \"While showing nice properties,\"\n\n  The authors only mention that Demucs solve audio source separation (for the data the authors use) and the algorithm is time equivariant. However, the text reads like the authors would like to state other properties as well. If there are others, they should be stated explicitly.\n\n- Section 3.2.\n\n  The authors should mention and cite the library they have used to extract MFCCs.\n\n- Section 4.1 \"we chose to select only pop music and its sub-genres for a total of approximately 10,000 songs\"\n\n  It would be beneficial to share IDs of the songs in the subset for reproducibility purposes. Also, the authors do not state whether they use the untrimmed or trimmed versions of the tracks in the FMA dataset, which is a crucial detail for model training as well as experimental reproducibility.\n\n- The authors should state:\n\n  1. number of songs used from the MusDB18 dataset (i.e. have they used both the train and test splits?)\n  2. Total duration and number of samples in training, test and fine-tuning\n\n- In the test set, instead, we chose only a few samples for each song due to the relative uniformity of its content: in other word, we expect our model to perform in similar ways on different parts of the same song.\n\n  I find this assumption a bit unrealistic. In what sense, is the content uniform across the song? Is it uniformity in mixing, structure, arrangement, melody, tempo, or rhythm? Even if the authors use trimmed audio excerpts for training/testing, these characteristics can vary substantially within seconds (even if they use trimmed tracks). \n\n  The authors should clearly state how they define content uniformity, provide a more informed argument around this assumption and experimentally show that the assumption holds for the test set.\n\n- Section 4.2: \"the result is somehow subjective thus different people may end up giving different or biased ratings based on their personal taste\"\n\n  The authors portrait subjectivity as unfavourable. However, - as a human construct - there are no objective, universal criteria for appreciating music. Likewise, the evaluation metric, which the authors are proposing, is based on the subjective responses from music experts. I think the justification needs rephrasing.\n\n- Section 4.3: In the paper, the authors do not state the cultural background or the genre(s) of the focus of the music experts. The inter-agreement between the experts are not presented either. Due to lack of information and the small number of subjects, it is difficult to assess whether the (trained) evaluation metric has positive/negative/desired biases based on the experience, knowledge, personal taste etc. of the experts. Therefore, the claim about the proposed \"metric correlating with human judgment\" is a bit weak.\n\n- What is the distribution of scores for bass and voice?\n\n- How much do the artifacts (due to imperfections in source separation) affect the judgements?\n\nMinor comments\n==============\n\n- Introduction, Paragraph 1: \"allow artists and producers to easily manipulate recordings and create high quality songs directly from home.\"\n\n  The phrasing somewhat disregards the music studios.\n\n- Page 2, top row: \"given a musical sample encoded in a two-dimensional time-frequency representation (known as Mel-spectrogram)\"\n\n  It reads like all two-dimensional time-frequency representations are called \"Mel-spectrogram\"s, instead of the authors using Mel-spectrograms, which is one type of two-dimensional time-frequency representations. \n\n- The text should explain the relevance of the selected experimental settings to the music production: e.g. drums and bass are usually the first \"sessions\" to be recorded; a demo typically consists of the melodic prototype/idea with minimal accompaniment, which is later arranged by many collaborators...\n\n- \"Figure 1 shows a Mel-spectrogram example, a visual representation of a spectrum, where the x axis represents time, the y axis represents the Mel bins of frequencies and the third gray tone axis represents the intensity of the sound measured in decibel (Briot et al., 2020).\"\n\n  I do not understand what the authors mean by \"third gray tone axis.\" Is it because the MFCCs are treated as a single channel image, hence \"gray\"? If yes, it is better to state that the \"MFCCs are treated as a single channel image\" without resorting to image processing jargon.\n\n- \"Mel-frequency cepstral coefficients are the dominant features used in speech recognition, as well as in some music modeling tasks (Logan & Robinson, 2001)\"\n\n  It may be better to introduce this sentence earlier in the paragraph.\n\n- Section 3.4: \"On the one hand, ... On the other hand\"\n\n  It might be easier to read if the setting is enumerated for readability.\n\n- Section 4.1: \"To train and test our model We decide\"\n\n  Lowercase \"We\" -> \"we\"\n\n- MusDB18 URL is broken\n\n- Section 4.3: \"Time: a rating from 1 to 10 of whether the produced drums and arrangements are on time the the bass and voice lines\"\n\n  Double \"the the\" -> \"with the\"\n  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "pwtRn2VKyPb", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes an approach to what is often termed \"automatic accompaniment\" generation in music.\nGiven an input signal of one source (e.g., vocal or bass), the system is trained to generate one or more accompanying signals (drums, full arrangement).\nThe authors propose a CycleGAN model to learn transformations between source and accompaniment domains.\nThe model is trained on a combination of a large collection of automatically separate signals (FMA) and a small collection of isolated stem recordings (musdb).\nThe models for two tasks (bass->drum and vocal->full) were evaluated by a combination of human listener testing and automatic offline scoring, to somewhat mixed results.\n\nOverall, I found this paper interesting and generally well-written.\nThe combination of subjective and offline evaluation was nice to see, and given the inherent difficulty of the problem, I don't consider the mixed results to be a total negative here.\nThat said, I do think there are areas in which this paper could be significantly improved, both in terms of experimental design and exposition.\n\n\nThe experiments presented here make use of both pre-separated stems (MusDB) and automatically separated signals produced by DEMUCS on the FMA dataset.\nGiven the size of available stem datasets, I understand the motivation for going this route.\nHowever, I think there needs to be some quantitative evaluation of the impact of each part here, for several reasons:\n\n1. DEMUCS is by no means perfect, and we should expect some bleed-through of the target signal (eg drums) into the separated signal (eg bass).  If this happens, the task becomes significantly easier than if the system was presented with clean stems.\n2. We can't rely on previously reported BSS-EVAL metrics to give a sense of DEMUCS' performance on FMA for generating the training data.  The FMA dataset is quite different from MusDB in terms of production quality and instrumentation, and given the small size of MusDB, the reported metrics are almost certainly an over-estimate of quality we should expect on FMA.\n3. It is not demonstrated that including the FMA data is necessary or beneficial for this task (though it's not unreasonable to expect that this is indeed true).  An experiment showing how the system performs if only trained end-to-end on musdb would make the existing results easier to interpret and place in context.\n\n\nIn terms of exposition, as stated above, I find the paper mostly clear and easy to follow.\nHowever, many technical details are omitted that make it both impossible to reproduce and difficult to interpret.\nThe biggest omission here is the specific method for recovering the waveform from the generated Mel spectrograms.\nPhase information is discarded early on in the process, but is critical to the perceptual quality of generated audio.\nIn listening to the included examples, it's pretty clear that there's a great deal of phase distortion in the results of both tasks.\n(It's less perceptible in the drum synthesis task because the target signal does not generally consist of sustained tones, but it's still audible.)\nThis left me wondering how exactly the phase retrieval is done, and to a lesser extent, how the Mel spectrogram inversion is done.\n\n\nMinor comments:\n\n- The authors claim that the source separation model (DEMUCS) is time-equivariant (section 3.1), but I don't see how this is justified.  DEMUCS uses a U-net architecture with a bidirectional LSTM middle layer, which is not generally time-equivariant.\n\n- Why are the spectrograms quantized to 256 values?  I agree that this probably doesn't introduce much distortion, but it seems unnecessary.  Point of clarification: are these spectrograms using linear magnitude or logarithmic (decibel) magnitude?  This decision would have a significant effect on how quantization is performed, but it's not clearly articulated in the paper.  Figure 1 suggests a log scaling, but does not provide details.  An equation would go a long way here.\n\n- Is there any windowing applied in the short-time Fourier transform (eg Hann or Hamming)?  I would expect so based on the lack of transient artifacts in Figure 1, but it's not explicitly stated.  I ask because having listened to the provided examples, it sounds like there could be some modulation artifacts in the reconstruction that could be traced to the choice of window function.  Aside: if you're using an existing software package to implement your Mel spectrogram, it should be cited.\n\n- I like the approach of mapping automatic scores to human judgments, but I'm confused as to why the targets were binarized.  Why not do an ordinary least squares or isotonic regression, that would discard less of the information?", "extra_scores": {}, "extra_text": {}}, {"review_id": "OUqoXfqliy", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a method for automatically generating accompaniments using Mel-spectrograms as inputs to a CycleGAN. Overall I think the paper requires significant revision and additional work before it can be accepted as a conference publication. \n\nTitle: \n\n-The title is misleading. The title claims that the proposed model is for \"Automatic Music Production\". However the actual task considered is more restrictive. The authors propose a model for automatic accompaniment. Music Production involves many other tasks like mixing, mastering and so on, none of which are a part  of this study. The title should therefore be updated to be more specific. \n\nAbstract: \n\n-\"Despite consistent demands from producers and artists...\": I think this sentence should be rephrased to motivate the need for automatic accompaniment from a different angle. If not, the authors should present some justification for the demand for this technology from artists and producers. \n\n-\"Automatic music arrangement from raw audio in the frequency domain\": why not simply say automatic music arrangement/accompaniment in the Mel-frequency domain? I find the raw audio part of the description unnecessary and confusing. \n\n-The authors claim that the they are the first to treat music audio as images and then apply techniques from computer vision. However, treating spectrograms as images is the current standard for many MIR tasks like music transcription, chord recognition and so on e.g. \"An end-to-end Neural Network for Automatic Music Transcription\": https://ieeexplore.ieee.org/abstract/document/7416164/. There are hundreds of other publications that are similar to this approach. \n\nIntroduction: \n\n-The authors claim that automatic accompaniment in the waveform/frequency domain has many advantages. However they fail to motivate the short-comings of this approach. Namely the lack of source separated training data and the extreme difficulty in source separation for music recordings. It  would also be useful to cite a review paper or some of the many publications on automatic accompaniment generation in the symbolic domain so that the reader can find references to this problem which has an extensive literature already. \n\n-The authors mention that they use the Demucs algorithm for source separation. However they do not provide any details whatsoever about this approach, especially the downsides. A quick scan of the paper reveals that the algorithm introduces severe artefacts under various conditions. \n\n-The authors mention the low-computational cost of their proposed method, however they do not satisfactorily quantify this claim. Firstly, is computational cost an issue? Does this algorithm have to run on a mobile device? Will it be run in a streaming setting? These questions are not answered in the paper. \n\nRelated Works:\n\n-The authors cite many papers on music generation in the waveform domain however they do not cite any of the extensive literature on music generation in the symbolic domain. This literature is extremely relevant to the work presented in this paper. \n\n-\"Nevertheless, only raw audio representation can produce, at least in the long run, appealing results in view of music production for artistic and commercial purposes.\" Why is this the case? Why is generating music in the symbolic domain and then using state-of-the-art synthesisers not an appealing direction? This point isn't made clear in the paper. \n\nMethod:\n\n-There are no details provided about the Demucs algorithm used to separate the source training data into various channels like vocal, bass, drums etc. How big was the model? Did the authors train the model themselves? Did they use a pre-trained model? Were there any artefacts present in the source separated tracks? Are there any downsides to this algorithm? Are there any alternatives to this algorithm? Do the artefacts not interfere with the  downstream task? \n\n-A reference/citation about the Mel scale would be useful. \n\n-There are no details about the CycleGAN used in the paper. How big is the model? What is the architecture? How was it trained? What flavour of gradient descent was used for training? What are the hyper-parameters? Was the model trained on a single GPU? \n\nExperiments:\n\n-How was the subset of pop music selected? How was the metadata filtered to obtain the 10000 tracks used for training? If the filtering algorithm cannot be outlined, then it would be useful to provide a list of the 10000 tracks used for training, for the purpose of reproducibility. \n\n-How did the authors arrive on the 4 attributes quality, euphony, coherence and intelligibility? Is there some theory that suggests that these 4 attributes would be useful in determining whether the accompaniment is somehow good? These attributes have been presented without justifications and citations. \n\n-The features (STOI, FID) used to compare the automatically generated accompaniment have also been presented without much justification. Why is it that these features  are an adequate representation of the generated audio? \n\n-I found the description of the grades and the subsequent comparison in Figure 3 difficult to follow. I think the description needs to be significantly more rigorous. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All Reviewers point out that the paper, although having some strong points, does not meet the bar for a highly-selective machine learning conference like ICLR. Hence, my recommendation is to REJECT the paper. As a brief summary, I highlight below some pros and cons that arose during the review and meta-review processes.

Pros:
- Well-written paper.
- Ambitious task.
- Code will be released.

Cons:
- Unclear task terminology (music production; misleading title).
- Mixed results.
- Experimental design could be improved.
- Exposition could be improved (technical details missing).
- Lack of comparison (for instance with other CycleGAN variants; more experimental setups).
- Lack of discussion on the use of a source algorithm for pre-processing data.
Decision: Reject