Non-Sequential Melody Generation
Authors: Mitchell Billard
Robert Bishop
Moustafa Elsisy
Laura Graves
Antonina Kolokolova
Vineel Nagisetty
Zachary Northcott
Heather Patey
Venue:  (2020)
Keywords: melody generation
DCGAN
dilated convolutions

Abstract: In this paper we present a method for algorithmic melody generation using a generative adversarial network without recurrent components. Music generation has been successfully done using recurrent neural networks, where the model learns sequence information that can help create authentic sounding melodies.  Here, we use DCGAN architecture with dilated convolutions and towers to capture sequential information as spatial image information, and learn long-range dependencies in fixed-length melody forms such as Irish traditional reel. 

Under review as a conference paper at ICLR 2020
NON-SEQUENTIAL MELODY GENERATION
Anonymous authors Paper under double-bling blind review
ABSTRACT
In this paper we present method for algorithmic melody generation using gen- erative adversarial network without recurrent components. Music generation has been successfully done using recurrent neural networks, where the model learns sequence information that can help create authentic sounding melodies. Here, we use DCGAN architecture with dilated convolutions and towers capture sequen- tial information as spatial image information, and learn long-range dependencies in fixed-length melody forms such as Irish traditional reel.
INTRODUCTION
Algorithmic music composition almost as old as computers themselves, dating back to the 1957 "Illiac suite" (Hiller Jr & Isaacson, 1958). Since then, automated music composition evolved with technology, progressing from the first randdmessss based methods to the sophisticated tools made possible by modem-day machine learning (see Fernández & Vico (2013) and Briot et al. 2017) for detailed surveys on history and state of the art of algorithmic music composition).
One of the first machine learning (ML) approaches to music generation was Conklin & Witten 1995), who used the common notion of entropy as a measurement to build what they termed a multiple viewpoint system. Standard feedforward neural networks have difficulties with sequence based information such as music. Predicting the next note of a piece, when only based on the current note, does not account for long-range context or structure (such key and musical sections) which help give coherence to compositions. As music is traditionally represented as sequences of notes, recurrent neural networks are natural tool for music (especially melody) generation, and multiple groups used RNNs fairly successfully for variety of types of music. Todd (1989) used sequential model for composition in 1989, and [Ek & Schmidhuberi (2002) used the adapted L.TTM structure to successfully generate music that had both short-term musical structure and contained the higher-level context and structure needed. Subsequently, there have been a number of RNN-based melody generators (Simon & Oore 2017 Lee et al. 2017 Eck & Lapalme 20088] (Sturm al. 2016 Hadjeres et al. 2017]Chen & Mükkkulainen 2001 Boulanger-Lewanddowst
Lewandowski et al. 2012] 2 Yu et al, 2017). Other approaches such as MidiNet by Yang et al. (2017). though not RNNs, also leveraged the sequential representation of music.
Using RNN architecture provides lot of flexibility when generating music, ability to generate pieces of varying length. However, in some styles of music this is not as desired. This is of traditional Irish music and especially their jigs and reels. These pieces have more rigid format where the varying length can prevent capturing the interplay between the phrases of the Finding jigs and reels train on was made easy by an excellent database of Irish traditional melodies in ABC notation (a text based format), publicly available at TheSessiorKeeth Several RNN-based generators were trained on the melodies from TheSession, most notably Sturm et al.
RNN has the
(turm et al. 2016 Sturm & Ben- Tal 2018), well as Eek & Lapalme (2088).
natural to view music, and particular melodies, as sequential data. However, to better rep- resent long-term dependencies it can be useful to present music as two-dimensional form, where related parts and occurrences of long patterns end up aligned. This benefit is especially apparent in forms of music where a piece consists of well-defined, fixed-length components, such as reels in Irish music. These components are often variations on the same theme, with specific rules on where repeats changes should be introduced. Aligning them allows use vertical spatial proximity to capture these dependencies, while still representing the local structure in the sequence by horizontal proximity.

---

Under review as a conference paper at ICLR 2020
In this project, we leverage such two-dimensional representation of melodies for non-sequential melody generation. We focus on melody generation using deep convolutional generative adversarial networks (DCGANs) without recurrent components for fixed-format music such as reels. This ap- proach is intended to capture higher-level structures in the pieces (like sections), and better mimic interplay between smaller parts (musical motifs). More specifically, we use dilations of several semantically meaningful lengths (a bar or phrase) to further capture the dependencies.
Dilated convolutions, introduced by Yu & Koltun (2015) have been used in number of applications over the last several years to capture long-range dependencies, notably in WaveNet (Oord et al.) 2016). However, they are usually combined with some recurrent component even when used for CAN-based generation such as in Zhang et al. (2019) or Liu & Yang (2019).
Not all techniques applicable to images can be used for music, however: pooling isn't effective, as the average of two pitches can create notes which fall outside of the 12-semitone scale (which is the basis the major and minor scale as well as various modes). This is reflected in the architecture of our discriminator, with dilations and towers as the main ingredients.
2 STRUCTURE OF THE DATA: TRADITIONAL IRISH TUNES
In spite of emigration and well-developed connection to music influences from its neighbouring cultures- especially Scotland Irish traditional music has kept many of its elements and itself in- fluenced many forms of music. Its influences can be found in the folk music of Newfoundland and Quebec and in Bluegrass from the Appalachian region of United States.
A significant part of the traditional Irish music is instrumental dance music, where a tune following a set format is played on a solo instrument, such as a fiddle, tin whistle, accordion or flute. The core of the piece is usually monophonic, so while harmonies and embellishments can be introduced during performances, they are considered extra and not usually encoded when piece is transcribed. Generally the melody consists of two distinct 8-bar parts, which in turn consist of two 4-bar phrases each. Usually, phrases represent pattern which repeats with variations from phrase to phrase, with one (most often third) phrase deviating most from the pattern, and the final phrase returning to it. When performing, each part is repeated twice; sometimes the second repeat of part is different enough that it is transcribed separately, making the transcription from 16 to 32 bars long. The whole piece is usually performed three (or more) times, especially when accompanying dance.
The two most common types of dances are reels and jigs: the reels have time signature of 4/4 (that is, each bar contains 4 units, where each unit is a quarter note), whereas jig has time signature 6/8 (that is, each bar contains 6 units, where each unit is an eighth note).
Many Irish tunes are in major key, with D major being most common. However, number of tunes use the Dorian and Mixolydian modes (starting with 2nd or 5th note with respect to the relative major scale, respectively). The natural minor key (Aeolian mode) is less common. Overall, it is uncommon for a tune to incorporate notes not in its key and to go outside of a 2-octave range (either of these would make it hard to play on tin whistle).
Traditionally, musicians learned tunes by ear. However, the relative simplicity of the structure of Irish tunes has lent itself well text-based representation, culminating in the ABC notation de- veloped by Chris Walshaw in the 1970s. There, the key and default length of a note are specified in header (together with the title, time signature, etc), and the melody is then encoded by letters: A-G where C represents middle C, and a-g, which represents the notes an octave higher than A-G. Additionally there are modifiers which raise and lower notes by an octave, as well as charaters which represent sharps and flats. Like in the staff notation, sharps and flats which are implied by the key are not explicitly written into the piece. These letters are also appended with duration modifiers: for example, if ABC header sets the default note length as 1/8, then D represents an eighth note (which is a D), D2 represents a quarter note, and D/2 represents a sixteenth note. For readability, bar separa- tors and spacing are often included: for example, the sequence "d2dA BAFA BAFAJABB ABde f2ed Beed egfe" encodes the first phrase of a reel called "Merry Blacksmith", where the key was specified to be D major and the default note length is set to 1/8.
2

---

Under review as a conference paper at ICLR 2020
Nowadays, ABC notation is widely used, especially for Irish folk music, with additional features for representing ornaments, repetitions, and so on. In particular, this is the default representation for tunes in databases such TheSession which we used for this project.
2.1 PREPROCESSING THESESSION DATASET
TheSession dataset contains over 31,000 tunes, which contain 10,000 reels major key. However, after removing improperly formatted tunes, tunes with features we did not want to generate (such as triplets), and tunes that had more than bars we were left with 820 reels, which we used as our training data.
The samples in ABC notation were then converted into numerical vectors, with 16 numbers per bar (so 256 numbers total), corresponding to 16 possible notes (normalized midi values). This encoding did not preserve information about the duration each note, only their (normalized) midi value. However, found that with simple postprocessing (converting a sequence of occurrences of the same note into a note of a longer duration, starting with an occurrence on beat) we were able to recover the tunes with not much more variation than would be introduced in performances.
One of the main ideas behind our encoding was representing the resulting vectors of 256 values as a 64x4 image, with midi values corresponding pixel values. The resulting image had a line for each the phrase of the tune, and the vertical alignment of the corresponding notes and bars let us exploit the long-range dependencies among the phrases using their spatial proximity. Intuitively, this is akin to building a mental map of a space explored by touch: though the touch information is sequential, the resulting map can be two or dineesilnnsion,l, with significant amount of information encoded by proximity in the second and third dimensions.
3 GAN ARCHITECTURE
Traditionally, sequential data is learned using some variation of RNNs. Simple implementations of RNNs (standalone GRU and LSTM cells) tend to struggle with capturing long-distance depen- dencies; a popular solution to this issue is to use bi-directional RNNs (that is, pair of layers that run in opposite directions) with activation mechanism (using an auxiliary trainable network to better relate elements of the output sequence to those of the input sequence) so that information for multiple sections of the sequence can be captured(Zhou et al., 2016).
A similar effort at capturing long-dis din-distance dependencies has been applied to Convolutional Neural Networks (CNNs) too. Yu al. have proposed dilated convolutions as solution the problem: in dilated convolution, the convolution kernel is not contiguous block, but rather subsections of the kernel are separated by some number of pixels from each other. As result, the convolution operation is not limited to the immediate neighbourhood, and long-distance dependencies are made available to the kernel (Yu & Koltun| 2015). While attention-basee and bi-directional RNNs have proven be quite successful at capturing long-d long-distance dependencies, believe that there are few notable benefits for being able to use a DCGAN for generating sequential data, as opposed to an RNN:
DCGANs that incorporate dilations allow for domain-specifer heuristics to be passed to the neural network, by means of the dilation rates and the convolution kernel dimensions. This allows for whole new dimension of flexibility and fine-tuning that is not available through RNNs.
A DCGAN yields both a discriminator and a generator, as opposed to just generator.
Unlike an RNN, GAN does not require meaningful domain-specific seed to be passed as an input to the generator; a vector of random noise can be used instead, making it easier to generate novel outputs from the generator.
GANs have traditionally worked well with pictures, but to our knowledge existing music generation with GANs either relies on RNNs in the discriminator, as in SeqGAN (Yu et al. 2017) or generates the melody in sequential bar-after-bar process in MidiNet YYang al. 2017).
3

---

Under review as a conference paper at ICLR 2020
Figure I:
Highlighted pixels show convolution patterns for each tower, with the grey pixel being the center of each kernel.
3.1 MODEL
3.1.1 DISCRIMINATOR
The discriminator starts off with a 6-tower convolution network (that is, convolutions run side- by-side, as opposed being dependent each other). We chose use dilated filters 5 of the towers to capture relative information between bars and phrases (we shall refer to this as the global context), with the remaining layer being a contiguous 2x9 convolution also capture the structure of the immediate neighbourhood (local context). Having multiple towers each learning different aspects gives us versatility and helps avoid the "tunnel vision" nature of other generative music models. This can be viewed in Figure
All towers use convolution filters, and use zero-padding for edge pixels that might not have sufficient neighbourhood to be covered by each filter. The outputs of the convolutions from each tower are then stacked horizontally, and passed to a second (regular) layer convolution made of 64 filters, using 3x3 kernel, 2x2 stride and with the convolution being only applied to pixels that have a sufficient neighbourhood for the convolution filter. The output of this layer is then flattened, and passed to dense layer consisting of 1024 neurons, which is in turn followed by sigmoid neuron that generates the prediction of the discriminator. No batch normalization is applied to any of the layers the discriminator.
3.1.2 GENERATOR
Unlike the discriminator. the generator does not make use of dilations or towers. We start off with dense layer made of 32x2x256 (where the last dimension the number of filters). The dense layer is then reshaped into 256 filters of size 32x2, and passed through two layers of deconvolution, each with halving the number of filters, while using a stride of 1x1 and a kernel size of 2x5. A last deconvolution layer (also 2x5 in kernel size) is used to bring down the filters into one image, with stride of 2x2 to return the original image dimension of 64x4; this layer also uses a tanh activation to place the output values in the same range as the training dataset. All layers in the generator use batch normalization.
3.2 POST-PROCESSING
After our GAN has generated music in the form of 64x4 matrix, we map the real values to notes in ABC notation, rounding to the nearest note in the D major key. We then take each beat and merge sequence of four 16th notes C starting from 1st, 5th, 9th
notes with the same pitch together, or 13th note in a bar becomes a quarter note C instead. The resulting ABC notated music can then be converted into sheet music played with midi generators.
4

---

Under review as a conference paper at ICLR 2020
4 COMPARISON WITH OTHER MELODY GENERATORS
Our approach is similar to the model by Sturrm al. (2016) as both train on text-based musical notation and aim to learn Irish traditional instrumental music that has distinct structure. The models in Magenta are considered be state of the art in music generation at the moment. For this reason, we compared our samples with the Folk-RNN and Magenta models specifically the MelodyRNN model Magenta- training using the same dataset 820 curated reels. We generated an equivalent number of samples from each model and compared the samples. Since our goal is for the GAN to learn the global patterns, we use measures that highlight similarity in structure.
The first metric we chose the normalized mean Fréchet distance. Fréchet distance is measure of similarity between vectors which prioritizes the ordering and magnitude of the values in those vectors. Since the order and distribution of notes are important for tune to be considered struc- turally similar to another, this metric is ideal for us. We normalize this Fréchet distance so we can view the changes between the phrases better.
The results of this comparison are found in Figure 2 Each group records the normalized Fréchet distance between corresponding combinations of phrases i.e the first group highlights the normal- ized mean Fréchet distance between phrase and 2 for all the distributions. We can see that our generated samples compare favorably with the Folk-RNN model and both exhibit similar structure to the training set. It is important to note that the MelodyRNN used here is the default model without the attention mechanism.
Normalized Mean Frechet Distance
00000
Model
0:775
FolidtNN
Magenta
0150
0.325
0.00
0675
0.00
0.000
2/3
Phrases
Figure 2: Average normalized Fréchet distance between the phrases of the distributions
Another measure to visualize similarity between distributions is the t-distributed stochastic neigh- bourhood embedding (t-SNE algorithm developed byMaaten & Hinton (208).. This algorithm groups points that are similar in higher dimension closer together in low dimensions making it useful visualize similarity between the tunes generated by both models along with the training data. Here, we set the tunable parameter perplexity to 50. This increases the importance of global similarity between vectors.
Figure 3 ssows distributions of the training data and samples generated by our model, FolkRNN and Magenta trained the same data. Since t-SNE is stochastic and can generate different visualizations for each run, we display visualization produced by two runs of t-SNE algorithm. It can be observed that most of the time (> 75%), the tunes generated by our model seem lie within the distribution of the training data, although they only cover a subset of the training data. Both RNN models seem to mimic a slightly different distribution, where we can think of the tunes generated by Magenta as subset of the FolkRNN ones, with FolkRNN generating by far the most diverse set of samples.
A third metric is looking at the distribution of notes in the samples. Figure 4 shows the frequency of distribution of notes as midi key values in the samples. As music theory would suggest, the tonic of the key (in this case, the midi value corresponding to D) is the most common, with the 3rd and 5th of the key more frequent than the rest of the scale. Both our model and FolkRNN created set of samples which mimics this distribution of the notes in the key, however Magenta seems to have a more uniform distribution of note frequencies.
5

---

Under review as a conference paper at ICLR 2020
The Session
Model
FolkRNN
-20
-40
-20
X
-20
Session Our Model
Magenta
-60
-20
x
Figure 3: I-SNE visualization of the distributions of tunes
Overall, these metrics show that music generated by our DCGAN is comparable to melodies gener- ated by RNNs on the same symbolic data. The use of dilations allows for our model to learn global tune structure.
0.75
The Session
Model
0.50
FolkRNN
Magenta
0.25
Frequency
0.000
0.075
0.50
0025
0.000
60
Notes (midi key value)
Figure 4: Distribution of notes
5 CONCLUSION
Converting sequential data into format which implicitly encodes temporal information spatial information is an effective way of generating samples such data as whole pieces. Here, we explored this approach for melody generation of fixed-length music forms, such as an Irish reel, using non-recurrent architecture for the discriminator CNN with towers and dilations, as well as CNN for the GAN itself.

---

Under review as a conference paper at ICLR 2020
One advantage of this approach is that the model learns global and contextual information simul- taneously, even with small model. LSTMs and other approaches need a much larger model to be able to learn both the contextual neighboring note sequences and global melody structure.
In future work, we would like to introduce boosting in order to capture the structure the distri- bution more faithfully, and increase the range of pieces model generate. Natural extensions of the model would be to introduce multiple channels to capture durations better (for example, as in Colombo al. (2016)), and add polyphony (ie, using some form of piano roll representation). An- other direction could be to experiment with higher-dimeenional representation of the sequence data, better capture several types of dependencies simultaneously. Additionally, would be interesting to apply it to other kinds of fixed-length sequential data with long-range patterns.
REFERENCES
Nicolas Koulanger-Lewaaddoskki Yoshua Bengio, and Pascal Vincent. Modeling temporal depen- dencies in high-dimensional sequences: Application to polyphonic music generation and tran- scription. arXiv preprint arXiv:1206.6392. 2012.
Jean-Pierre Briot, Gaëtan Hadjeres, and François Pachet. Deep learning techniques for music generation-a survey. arXiv preprint arXiv: arXiv:/709.01620 2017.
C-CJ Chen and Risto Miikkulainen. Creating melodies with evolving recurrent neural net- works. In IJCNN'01. International Joint Conference Neural Networks. Proceedings (Cat. No. 01CH37222), volume 3, pp. 2241-22444 IEEE, 2001.
Florian Colombo, Samuel P Muscinelli, Alexander Secholzer, Johanni Brea, and Wulfram Gerst- ner. Algorithmic composition of melodies with deep recurrent neural networks. arXiv preprint 1rXiv:1606.07721,, 2016.
Darrell Conklin and Ian H Witten. Multiple viewpoint systems for music prediction. Journal of New Music Research, 24(1):51-3 1995.
Douglas Eck and Jasmin Lapalme. Learning musical structure directly from sequences of music.
Technical report, Universite de Montreal DIRO, 2008
Douglas Eck and Juergen Schmidhuber. first look at music composition using LSTM recurrent
neural networks. Istituto Dalle Molle Di Studi Sull Intelligenza Artificiale, 103, 2002.
Jose D Fernández and Francisco Vico. Ai methods algorithmic composition: A comprehensive survey, Journal of Artificial Intelligence Research, 48:513-582, 2013.
Gaëtan Hadjeres, François Pachet, and Frank Nielsen. Deephach: a steerable model for bach chorales generation. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1362-1371. JMLR. org, 2017.
Lejaren A Hiller Jr and Leonard M Isaacson. Musical composition with a high-speed digital com- puter. Journal of the Audio Engineering Society, 6(3):154-160, 1958.
Jeremy Keith. Traditional Irish music on The Session. URL
Sang-gil Lee, Uiwon Hwang, Seonwoo Min, and Sungroh Yoon. Polyphonic music generation with sequence generative adversarial networks. arXiv preprint arXiv:1710.11412 2017.
Jen-Yu Liu and Yi-Hsuan Yang. Dilated convolution with dilated gru for music source separation. arXiv preprini arXiv: /90660/2060. 2019.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 97No))2559-26605, 2008.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609.33999 2016.

---

Under review as a conference paper at ICLR 2020
Ian Simon and Sageev Oore. Performance RNN: Generating music with expressive timing and dynamics. Magenta Blog: https://magenta. tensorflow. org/performancernn, 2017. URLbttpss http:///wwa///araaaaaaaaaeeee
Bob Sturm and Oded Ben-Tal. Lets have another gan ainm: An experimental album of irish tradi- tional music and computer-generatee tunes. Technical report, 2018.
Bob L. Sturm, Joao Felipe Santos, Oded Ben-Tal, and Iryna Korshunova. Music transcription mod- elling and composition using deep learning. arXiv preprint arXiv: 160.00870 2016.
Peter M Todd. connectionist approach to algorithmic composition. Computer Music Journal, 13 (4):27-43, 1989.
Li-Chia Yang, Szu-Yu Chou, and Yi-Hsuan Yang. Midinet: A convolutional generative adversarial network for symbolic-domain music generation. arXiv preprint arXiv:1703.10885 2017.
Fisher Yu and Vladlen Koltun, Multi-scale context aggregation by dilated convolutions. arXiv
preprint arXiv:1511.071111 2015.
Lantao Yu, Weinan Zhang, Jun Wang, and Yong Yu. SeqGAN: Sequence generative adversarial nets with policy gradient. In Thirty-Fire First AAAI Conference Artificial Intelligence, 2017.
Yujia Zhang, Michael Kampffmeyer, Xiaoguang Zhao, and Min Tan. Dtr-gan: Dilated temporal relational adversarial network for video summarization. In Proceedings of the ACM Turing Cele- bration Conference-China, pp. 89. ACM, 2019.
Peng Zhou, Wei Shi, Jun Tian, Zhenyu Qi, Bingchen Li, Hongwei Hao, and Bo Xu. Attention-bart -based bidirectional long short-term memory networks for relation classification. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume Short Papers), pp. 207-212, 2016.
8


Reviews:
[{"review_id": "SkxwB_uvKB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a DCGAN for generating Irish folk music. The fixed length of Irish reels (in terms of bars) means that a piece can be represented as a 2-D image with fixed height and fixed width. This study represents each performance as a 4x64 image, where each successive row represents a phrase and pixels in neighbouring rows represent corresponding notes in successive phrases. A DCGAN architecture is then used to generate novel melodies using this representation. Each pixel has a normalised value between [-1,1] which correspond to the notes A-G, a-g in the vocabulary. \n\nThis study to me appears incomplete and at an early stage and requires further work before it can be accepted as a conference publication. Firstly, I believe the title is misleading. The title encourages the reader to expect a general model for generating melodies with GANs however the application is specifically to generate monophonic Irish folk music only, with a fixed number of bars. Secondly, the main novelty in the paper, the DCGAN architecture has not been described in enough detail, for instance the optimiser and hyper parameters are not mentioned, making it difficult for this work to be reproduced. \n\nThe authors claim 3 major motivations for  the DCGAN; 1) Dilated convolutions for introducing music related inductive priors, however the model description does not provide any intuition or insight for why the exact filters used were chosen. 2) The authors claim the GAN also yields a discriminator in addition to a melody generator, but they do not provide any explanation for why a discriminator might be useful. 3) They claim using random noise instead of a musically meaningful seed is better, but they do not describe the input noise distribution that the proposed generator is conditioned on.\n\nFor the evaluations, the Frechet distance is introduced without a reference/citation and the “models in Magenta” are referenced without a citation.  One downside of using the GAN is that there is no way to calculate the log-likelihood of the generated samples under the model distribution, which makes evaluating/comparing models a difficult problem. I think the distribution of notes in the generated samples is not a very informative metric for comparing the quality of generated samples. A few audio examples would have been extremely useful in getting a sense of how this model performs. \n\nI think this work should be resubmitted with major revisions. \n\nMinor Comments\n\n1. Section 2 does a good job of providing background for Irish folk music and the ABC notation. \n2. “normalised MIDI value” should be elaborated. I inferred it to mean the MIDI values are normalised to the range [-1,1] after reading the whole paper, however this is not clear in Section 2.1 \n3. “resulting vectors of 256 values as a 64x4 image”, I think the image is 4x64 (Figure 1). \n4. “..to use bidirectional RNNs with an activation mechanism”, this should be attention mechanism. \n5. There should be some comment on the motivation behind the specific choice of filters in the generator and discriminator. \n6. There should be some details about the optimisation algorithm, batch sizes, learning rate schedules etc used to train the network. \n7. The magenta model requires a citation as it is not obvious what is being referred to. \n8. The Frechet distance should be introduced with a citation. \n\n\n\n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "S1eEwK92Fr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper describes a (DC)GAN architecture for modeling folk song melodies (Irish reels).\nThe main idea of this work is to exploit the rigid form of this type of music --- bar structure and repetition --- to enable 2-dimensional convolutional modeling rather than the purely sequential modeling that is commonly used in recent (melodic) music generation architectures.\nThe ideas in this paper seem sound, though they primarily consist of recombining known techniques for a specific application.\nThe main weaknesses of this paper are in the evaluation (see below), and while I understand that evaluating generate models for creative applications is difficult and fraught territory, I don't think the efforts taken here are sufficiently convincing.\n\n\n\nStrengths:\n\n- The paper is clearly written, and the authors have taken great care to describe the unique structures of the data they are modeling.\n- The proposed architecture seems well motivated, and matches to the structure of the data.\n\n\nWeaknesses:\n\nThere are three components to the evaluation, and each of them are problematic:\n\n- The first evaluation (Fig 2) compares the average Frechet distance between phrases generated by different models, and within the original dataset.  Some brief argument is given for why Frechet is a good choice here, but it still seems quite tenuous: what does this distance intuitively mean in terms of the data?  How should the scale of these distances be interpreted / what's a meaningfully large difference?  How concentrated are these average distances (ie, please show error bars, variance estimates, or some notion of spread)?\n\n- The second evaluation (Fig 3) uses t-SNE to embed the generated melodies into a 2D space to allow visual inspection of the differences between distributions produced by each model.  While this might be a reasonable qualitative gut-check, t-SNE is by no means an appropriate tool for quantitative evaluation.  The authors at least did multiple runs of t-SNE, but this hardly amounts to compelling evidence.  Moreover, combining all data sources into one sample prior to running t-SNE induces dependencies between the point-wise neighbor selection distributions, which seems undesirable if the eventual goal is to determine how similar each model's distribution is to the source data.  A better approach might be to create independent plots for each model's output (with the original data), but I'd generally advise against using t-SNE for this kind of analysis altogether.\n\n- The third evaluation (Fig 4) measures the amount of divergence from the key (D) in terms of note unigrams.  This evaluation is done qualitatively, and the histogram is difficult to read --- it may be easier to read if the octave content was collapsed out to produce pitch classes rather than pitches.  If, however, the goal is to actually measure distance from the target key, one could do this quantitatively by comparing histograms to a probe tone profile (or otherwise constructed unigram note model) to more clearly characterise the behaviors of the various models in question.\n\n\nAt a higher level, there is no error analysis provided for the model, nor any ablation study to measure the impact of the various design choices taken here (eg dilation patterns in Figure 1). \nThe authors seem to argue that these choices are the main contribution of this work, so they should be explicitly evaluated in a controlled setting.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HygOPyIfqr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is interested in music generation, leveraging a 2D representation of the music data. \n\nThe idea of using a 2D representation is quite good, though here very specific of the considered type of music. The authors might want to discuss related approaches , considering 2D general representations of music [1, 2]. The originality of the proposed approach is to use dilated convolutions to capture the long range dependencies in a GAN framework. \n\nThe paper however looks premature for publication at ICLR, for several reasons:\n\n* The thorough discussion of the dataset might be put in supplementary material. Instead, the reader would like to know whether the authors considered data augmentation of their relatively small dataset (820 tunes). Likewise, the description of the architecture could be put in a supplementary section, or on github for reproducibility.\n\n* The authors did not justify the choice of the specifics of the architecture, such as the number of filters, layers or the presence / absence of batch normalization (part 3.1). The reviewer would like to see how the results vary with respect to those parameters. It would also be interesting to see what the generated images look like before the tanh (part 3.1.2).\n\n* The assessment of the results is hard to interpret; the reasons why the Frechet distance varies depending on the models and the phrases combinations should be discussed.\n\n* The gold standard for evaluating music is based on the human assessment of the generated music (involving naive, advanced and expert people), as done in [3], cited; having a human being evaluate and compare the tunes would help to gain insight into the generation. In the same perspective, it would be appreciated to put the results on a website for the reader to assess the quality of the generated music. \n\n* The contraction of the support (as displayed in Fig. 3) suggests that there might be some mode dropping with the GAN; this should be studied in depth. \n\n* The authors consider Magenta and FolkRNN as baselines; the reviewer suggests to also consider e.g. [4]  (although not not exploiting the specifics of the style), to comparatively assess the proposed approach.\n\n[1] \"Onsets and Frames: Dual-Objective Piano Transcription\", Hawthorne et al., 2017.\n[2] \"TimbreTron: A WaveNet(CycleGAN(CQT(Audio))) Pipeline for Musical Timbre Transfer\", Huang et al., 2018.\n[3] \"DeepBach: a Steerable Model for Bach Chorales Generation\", Hadjeres et al., 2017\n[4] \"Music Transformer\", Huang et al., 2018.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers pointed out issues with the experiments, which the rebuttal did not address. The paper seems interesting, and the authors are encouraged to improve it.
Decision: Reject