Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The last paragraph of section 5 discusses an experiment to assess to which extent the model is able to generate new music. This is done by generating a bunch of sequences and checking for the longest subsequences that also occur in the training data.

In the context of generative models of images, it has been established that searching for generated examples in the training data is not a great way to determine whether or not the model is overfitting. To what extent do the authors think these concerns also apply here?

Additionally, such an experiment does not demonstrate diversity of the samples, so the model might still be overfitting to a particular subset of modes of the distribution. For all we know, it could have generated the same sample 100 times. Could you comment on the diversity of the samples generated by the model and perhaps also how this could be demonstrated (or even measured quantitatively)?
review_title: diversity

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents a recurrent neural network (RNN) for generating pop music.  The model is trained on 100 hours of user composed pop songs and video game music and the resulting music is evaluated in user studies against songs produced by the Magenta framework.

Overall, I find the paper to be well written and clear.  I appreciate the review early on of music theory concepts.  I think the paper provides a reasonable support for the connection between how pop music is composed and the hierarchical model for generating melody accompanied with chords and drums.  With some post-processing, the model appears to generate pleasant sounding music as judged by users and from a personal perspective of listening to the examples available on the web.

While the generated examples on the web sound pleasant, they also sound quite similar and make it hard to judge what the model has learned.  There are some open questions regarding evaluation of the model.  The paper would benefit from improvements in both user and metric evaluations.

*  The Magenta system serves as a lower baseline for evaluation.  The study would benefit from an upper baseline by also evaluating against human composed songs.  This would help contextualize the findings for both this and future work.

*  The user study could be improved by examining other dimensions of appeal, perhaps to gauge diversity through "interestingness" over a collection of samples. 

*  I think a paired/side-by-side design for the user study seems limited (examples on http://www.cs.toronto.edu/songfrompi/eval/eval.html).  A simpler design with rating one sample at a time may have been more appropriate because there is no natural way to pair the songs.  The examples from each system used in the experiment should be provided with labels or an answer key so that readers can compare the merits of each of the systems' compositions themselves.

*  The authors propose specific metrics for insight into the diversity (longest subsequence and number of repeats).  These would be more meaningful with some context, e.g. by comparison with baseline Magenta samples and the training data (as an upper baseline).

*  Details of the baseline Magenta system would also benefit the paper.

*  No guidance is provided on how to judge the applications of neural singing, dancing and karaoke.  

review_title: The paper is well written, clear and proposes a reasonable model for generating melody accompanied with chords and drums.  The evaluation of the model requires clarification or improvements.

final_score: rating: null
review_content: Could you provide more detail about the Magenta system that you compared against?  I know that it is being improved fairly regularly, and you provide a citation for the website in general.  Is it possible to reference the specific version you were using?  Did you use it to generate the data yourself, or did you use pre-generated music that they distributed?  What parameters did you use?  Was it trained on their data or yours?  etc.  Even providing the comparison Magenta signals in the example webpage would be helpful.  

Also, how did you decide which Magenta outputs to pair with which outputs of your system?  Are both systems completely unguided in their synthesis?  If you had a way of matching up Magenta outputs and your outputs, could you also use it to match up your outputs with real midi examples from the training data for a more direct evaluation of realism?

review_title: Comparison system(s)

final_score: rating: 7: Good paper, accept
review_content: The paper describes a recurrent neural network model for generating pop music in the symbolic domain (i.e. MIDI). The layers of the model each generate part of the output, with the first few layers responsible for generating the melody, and further layers generating drums and chords conditioned on the generated melody. The authors argue that this matches how pop music is usually composed. The model is trained on 100+ hours of pop music in MIDI format. The resulting generated music is compared against that produced by another system using human evaluation, which is probably the only way in which such a system can be fairly evaluated. I appreciate that the authors went through the trouble of setting up these experiments.

The RNNs generating the different outputs (i.e. key, duration, chord, melody) are trained in sequence, conditioned on the output of the previous step(s).I found the text a bit confusing at times as it initially seems to describe a single end-to-end trained model (even if this is never stated explicitly). It only becomes clear later on that the layers are trained in sequence, with additional supervision provided at each stage. This may simply be a personal bias because recent work on hierarchical RNNs that I've read has focused on end-to-end training, but nevertheless it might be useful to mention this more clearly from the get-go.

The post-processing of model samples described in the 2nd paragraph of Section 4.5 seems to affect results quite dramatically (based on the results in Table 1). It seems equally applicable to the outputs of the Magenta system, so it might be interesting to compare this version to Magenta as well, to get an idea of how much it contributes to the improvement over the Magenta system. It would be somewhat disappointing if it ends up accounting for most of the gain.

I am still unconvinced by the experiment described in the last paragraph of Section 5, where subsequences of generated music fragments are searched for in the training data. While I agree with the authors that minor differences in note choice can have profound effects on how the melody is perceived, I still think this is not particularly convincing, and I think drawing the unambiguous conclusion that "our model is able to generate new music" from this experiment is a bit premature.

The additional applications described in Section 6 feel a bit like an afterthought and the datasets used are probably too small for the results to be meaningful. Instead I would have preferred to read about how to reduce the importance of prior knowledge in the design of the model. Considering the venue this work was submitted to, moving towards a more "end-to-end learning" approach (rather than incorporating even more prior knowledge, as the conclusion seems to imply) seems like an interesting direction for future research.

Minor remark: giving the formulas for LSTM is probably a bit of a waste of space, especially if you're not explaining the semantics. A reference is sufficient, and in fact adding a reference to the original LSTM paper is probably a good idea regardless.
review_title: null

final_score: rating: null
review_content: A nice paper and nice idea!

I have two clarification questions:

1) Why is the rhythm generation not the lowest layer? Most musicians have an explicit or implicit idea of the metre their composition is based upon.
Further, rhythm is the foundation of melody in many ways. So what made you think, melody should be the base layer?

2) How do you make sure that your outputs are not mere artifacts of the models and priors you use?
I don't see any meaningful relationship between dancing, text, and music in the karaoke video. The music changes, but the same sentence is repeated and the stick person moves in ways that are not clearly related to the music, other than in trivial ways.
And similarly, the rhythm patterns the network learns do not add anything meaningful to the chord/melody progressions.
It seems even though the musical results sound coherent in terms of the scale (but that one you fix manually anyways), the evolution in the pieces doesn't seem to be meaningful.

It would be great if you can elaborate on the two points, so it is easier for me to interpret the results in the proper way.
review_title: How meaningful is the learned part?

final_score: rating: 4: Ok but not good enough - rejection
review_content: In this paper, the authors build music-theoretical structure directly into a music generating LSTM. Even though such simple rules should be learnable from data, this surely is a neat idea for the limited-data regime. Further, it seems like a desirable model when thinking about musicians, for instance, wanting to train on own (and thus limited) source material. 

They consider a dataset of 100 hours of midi and add multiple priors, drawn from basic music theory. The priors as such are OK, but some of them seem rather heuristic and should, in my opinion, be learned from data and it should be discussed how the performance changes if you remove them. 
Further, the authors evaluate their study on an artificial benchmark, consisting of a behavioral experiment where 27 subjects judge songs generated by Magenta and their approach in a questionable side-by-side evaluation. Using this as a performance criterion is problematic, as no details about the subjects are given away and no attempt is made to assess the statistical significance of such results, let alone discussing the difficulty of pairing the songs. Further, I assume there are standard behavioral batteries, concerned with assessing music preferences that should have been used or at least addressed. Introducing the neural Karaoke and dancing is fun, but does not have much scientific value at this point, as it does not seem to work in a meaningful way. I would recommend to either improve results of the latter drastically or add it as an extra to a blog post and remove it from the paper.

It is good that the authors make an attempt to encode general prior knowledge into their architecture, but I am not convinced by the results and the heuristic choices being made. Further, it is still not 100% clear to me how the weighted probability distribution is constructed for the scales and how strong the prior it effectively incorporates is. If it is very strong, it is not surprising to me, that the songs sound relatively coherent, as in scale playing with rejected outliers have to sound somewhat coherent. I am not familiar enough with the Magenta baseline system and it is problematic that the baseline is not explained well. If the baseline does not take explicit scale priors into account, it does make sense that it sounds less coherent by definition. This has to be discussed and the effect of the introduced priors has to be evaluated. Finally, the question remains if this will generalize to datasets with more than 4 dominant scales and why the authors chose their thresholds the way they did. Did the model perform worse if one chooses to include more scales? How do you know, that the heavy tail of such a distribution is not desirable and important for natural sounding music, did you investigate this?

The multitrack idea is great. However, I am not convinced it works in this case. The results sound more like assigning a couple of notes in the melodies to rhythmic sounds, but they do not interact with the melody, they just move along, as if they were part of the melody. This is not how rhythm works in music in most cases.

Pro:
+ Incorporating general musical knowledge into the learned network is a good idea and non-trivial.
+ Idea to introduce a behavioral measure for the quality of the generated samples is useful, as music is very subjective.
+ The multitrack idea seems useful and a clear step beyond Magenta as far as I understand.
Con:
- However, the multitrack part of the architecture does not seem to work properly, rhythm does not seem to behave differently from melodic movement.
- The music excerpts sound very simplistic and similar.
- The pair-wise evaluation metric with 27 supposedly "random" subjects is not very meaningful and very likely, not significant.
- Evaluation of generative models is difficult, but the authors could have done better.
- The 4 bins of the random scale variable seem ad-hoc.

I have to emphasize that I like the ideas introduced in this paper, but I am not convinced by the way they are presented and evaluated. I would like to suggest this paper for workshop publication.
review_title: Nice idea, lacking thorough evaluation.

Meta-review:  This paper provides some interesting ideas for integrating music theoretic structure into an LSTM based music generation model. The work also provides a human study of results (albeit one of limited size).
 
 The posted revision goes part of the way necessary to improve the paper, but a larger evaluation and some improvements to the model architecture might allow this work to have even higher impact in an eventual peer reviewed venue/format.
 We therefore recommend this paper for presentation at the workshop track.
 
 The most negative reviewer notes the following pros and cons:
 
 Pro: 
 + Incorporating general musical knowledge into the learned network is a good idea and non-trivial. 
 + Idea to introduce a behavioral measure for the quality of the generated samples is useful, as music is very subjective. 
 + The multitrack idea seems useful and a clear step beyond Magenta as far as I understand. 
 
 Con: 
 - multitrack part of the architecture does not seem to work properly, rhythm does not seem to behave differently from melodic movement. 
 - The music excerpts sound very simplistic and similar. 
 - The pair-wise evaluation metric with 27 supposedly "random" subjects is not very meaningful and very likely, not significant. 
 - Evaluation of generative models is difficult, but the authors could have done better. 
 - The 4 bins of the random scale variable seem ad-hoc.
Decision: Invite to Workshop Track