This paper proposes a model for generating pop music melodies with a recurrent neural network conditioned on chord and part (song section) information. They train their model on a small dataset and compare it to a few existing models in a human evaluation. I think this paper has many issues, which I describe below. As a broad overview, the use of what the authors call "word" representations of notes is not novel (appearing first in BachBot and the PerformanceRNN); I suspect the model may be outputting sequences from the training set; and the dataset is heavily constrained in a way that will make producing pleasing melodies easily but heavily limits the possible outputs of the model. The paper is also missing important references and is confusingly laid out (e.g. introducing a GAN model in a few paragraphs in the experiments). Specific criticism: - "often producing works that are indistinguishable from human works (Goodfellow et al. (2014); Radford et al. (2016); Potash et al. (2015))." I would definitely not say that any of the cited papers produce anything that could be confused as "real"; e.g. the early GAN papers you cite were not even close (maybe somewhat close for images of bedrooms, which is a limited domain and certainly cannot be considered a "work"). - There are many unsubstantiated claims in the second paragraph. E.g. "there is yet a certain aspect about it that makes it sound like (or not sound like) human-written music." What is it? What evidence do we have that this is true? " notes in the chorus part generally tend to be more high-pitched" Really? Where was this measured? "music is not merely a series of notes, but entails an overall structure of its own" Sure, but natural images are not merely a series of pixels either, and they certainly have structure, but we are making lots of good progress modeling them.  Etc. - Your related work section is lacking. For example, Eck & Schmidhuber in 2002 proposed using LSTMs for music composition, which is not much later than works from "the early days" despite having not "employed rule or template based approach". Your note/time offset/duration representation is very similar to that of BachBot (by Liang) and Magenta's PerformanceRNN. GANs were also previously applied to piano roll generation, see MuseGAN (Dong et al), MidiNet (Yang et al), etc. Your critique of Jaques et al. is misleading; "they defined a number of music-theory based rules to set up the reward function" is the whole point - this is an optional step which improves results, and there is no reason a priori to think that hand-designing regularizers is better than hand-designing RL objectives. - The analogy to image captioning models is interesting, but this type of image captioning model is not only model which is effectively a conditional language model - any sequence-to-sequence model can be looked at this way. I don't think that these image captioning models are even the most commonly known example, so I'm not sure why the proposed approach is being proposed in analogy to image captioning. - I don't think you need to reproduce the LSTM equations in your text, they are well-known. - You should define early on what you mean by "part", I think you mean the song's section (verse, chorus, etc) but I have also heard this used to refer to the different instruments in a song. I don't think you should expect this term to be known outside of musical communities (e.g. the ICLR community). - It seems simpler (and more in keeping with the current zeigeist, e.g. the image captioning models you refer to) to replace your HMM with a model that - The regularization is interesting, but a simpler way to enforce this constraint would be to just only allow the model to produce notes within that predefined range. Since you effectively constrain it to an octave, it would be simple to wrap all notes in your training data into this octave. This baseline is probably worth comparing to since it is substantially simpler than your regularizer. - You write that the softmax cost should have \frac{\partial E}{\partial p_i} \mu added to it for the regularizer. First, you don't define E anywhere, you only introduce it in its derivative (and of course you can't "define" the derivative of an expression, it's an analytically computed quantity). Second, are you sure you mean that the partial derivative should be added, and not the cost C itself? - Your results showing that human raters preferred your models are impressive, but you have made the task easier for yourself in various ways: 1) Constraining the training data to pop music 2) Making all of the training data in a single (major) key 3) Effectively limiting the melody range to within a single octave. - It sounds very much like your model is repeating bars, e.g. it generates a melody of length N bars, then repeats this melody. Is this something you hard-coded into the model? It would be very surprising if it learned to exhibit this behavior on its own. If you hard-coded it into the model, I would expect it to sound better to human raters, but this is a strong heuristic. - I'd suggest you provide example melodies from your model in isolation (more like the "varying number of bars" examples) rather than as part of a full music mix - this makes it easier to judge the quality of the model's output. - The GAN experiments are interesting but come as a big surprise and are largely orthogonal to the other model; why not include this in your model description section? The model and training details are not adequately described and I don't think it adds much to the paper to include it. Furthermore it's quite similar to the MidiNet and MuseGAN, so maybe it should be introduced as a baseline instead. - How did you order the notes for chords? If three notes occur simultaneously (in a chord), there's no a priory correct way to list them sequentially (two with an interval of length zero between notes). - "Generated instruments sound fairly in tune individually, confirming that our proposed model is applicable to other instruments as well" Assuming you are still using C-major-only melodies, it's not surprising that the generations sound in tune! - It is not surprising that your model ends up overfitting because your dataset is very small, your model is very powerful, and your regularizer does not really limit the model's capacity much. I suspect that your model is overfitting even earlier than you think. You should check that none of the sequences output by your model appear in the training set. You could easily compute n-gram overlap of the generated sequences vs. the training set. At what point did you stop training before running the human evaluation? If you let your model overfit, then of course it will generate very human-sounding melodies, but this is not a terribly interesting generative model.