FLAGNet : Feature Label based Automatic Generation Network for symbolic music

Abstract: The technology for automatic music generation has been very actively studied in recent years. However, almost in these studies, handling domain knowledge of music was omitted or considered a difficult task. In particular, research that analyzes and utilizes the characteristics of each bar of music is very rare, even though it is essential in the human composition. We propose a model that generate music with musical characteristics of bars by conditional generative adversarial network, and analyze the good combination of the sequence of which characterized bars  for symbolic-domain music generation by Recurrent Neural Network with Long short term memory layer. Also, by analyzing symbolic music data as image-like based on relational pitch approach, it increases the utilization of the data set with arbitrary chord scales and enables the use of generational results extensively. The resulting model FLAGNet generates music with the understanding of musical domain knowledge while handling inputs like minimum unit of note, length of music, chart scales, and chord condition.

Reviews and author responses:

Thank you for your response. I'm not sure my concerns here are the kind which can be resolved by minor changes. I recommend you think more about how your work builds on existing music generation schemes before submitting again. I also recommend you reach out to a music theorist or musicologist and have them look over your paper before resubmitting. While I wouldn't go so far as to call this paper musically ignorant, the lack of domain expertise is a major limitation.

With respect to 2.4, my personal preference is to avoid math notation in a paper unless it is either novel or essential for understanding an important point. This isn't a major issue.

Summary:
In this paper the authors propose a method for generating music using a GAN. They claim to have improved performance by incorporating domain knowledge. Unfortunately, the evidence of this, even by their own metrics, is somewhat lacking. Despite this, their contribution could have been notable if they had detailed their feature extraction and evaluated the impact/benefit of each feature upon performance, but this was also not done. Overall, even by the metrics provided in the paper, the contribution is very unclear. Without clarification of this, and addressing the above comments, I would recommend this paper is rejected.

The good things:
* Code is released with this paper which helped me to understand the feature generation and modelling process
* Figure 2: explains the modelling process relatively well on a high level
* The authors make use of a publicly available dataset
* The authors provide examples for comparison with another method

Things for improvement:
* Descriptions could be much clearer: for instance - I have interpreted the meaning of "musical skill"/"skill labels" to mean "bar level feature". There are other terms used which do not match well with the english meanings.
* Details of these "skill labels" should be provided: this seem to be the main contribution of the work, but there is no explicit discussion of the features derived from the bars of the music.
* The data being used for modelling needs clarifying: In 3.1 Dataset, the authors state they are using the monophonic data from PPDD, but modelling is done for monophonic and monophonic + chords. In the final para of 3.3 it is implied that these chords are added to each bar randomly but the process is very unclear. If this is a random process, a justification is required as to why the learned representations are interesting.
* The evaluation metrics are not well grounded: the authors pick 4 terms by which participants should give a score of max 5 (is 1 the minimum or 0?). There is no link to the literature that these are good/accepted terms to use. At minimum, there should be some discussion as to why these specific terms were selected and a comment made about. Also, on page 7, the authors begin to refer to "stability" is such a way that it makes me think it was a previously evaluated but now omitted term.
* Additional justification is required for some claims: for instance, in the abstract the authors state "However, almost in [sic] these studies, handling domain knowledge of music was omitted or considered a difficult task." but there is no justification of this in the text (one approach from the literature is given in the introduction, but there have been many other approaches).
* Figure/table captions should be more descriptive: for example, Table 1' caption reads "Human evaluation result". I would propose something more like: "Human survey comparison of model performance against true data: 15 participants were surveyed and asked to score examples on a scale of 0(?)-5 for the qualities listed. The rows listed as human music are real examples taken from the Lakh midi dataset." (I would also: add the standard deviations to the results, since they are calculable and would help establish significance, and round all values to a consistent and appropriate level)

Thank you for directing the reader to the appendix regarding the skill labels, this helps.

This project is a good start and I look forward to seeing it published in future.

I believe that this paper is addressing the problem of incorporating domain knowledge into the generation of symbolic music. I find this problem very interesting.
\newline
The primary problem with this paper is its lack of clarity. Unfortunately, this is so severe, that I cannot tell what is being done. 

In particular, when reading the PDF, I highlighted every sentence that had minor or major writing issues (ranging from sentences that were nevertheless understandable to others that were impossible for me to decode), and by the end I found that I had highlighted nearly every sentence in the paper. It is the cumulative effect that becomes quite problematic.

As an example, I will provide the second half of the paragraph given at the end of the Introduction (p1-2), in which I believe that the authors give an overview of their system (as far as I can tell):

"Thus, based on a given music dataset, we combine how the sequence of musical skills can be attractive and how the bars created by using the simple RNN model, i.e. handling about flow of music. In addition, to make image size can be properly reduced and the Musical Skill can be maintained while processing MIDI Bar with Image we utilize image processing based on Relational Pitch Change. This approach allows the use of the music in the train without relying on the chord scale of given music in dataset, and provides a wider range of possibilities for the music produced by matching the music to the 12 basic chart scale (C, C#, D, D#, E, F, F#, G, G#, A, A#, B). Also, to distinguish major scale and minor scale, we can generate music with chord conditions with major scale. The resulting model, FLAGNet can use the musical skill contained in the music bar to understand the musical domain knowledge, analyze the sequence of the musical skills to control the overall flow of music, and process the generated images to make symbolic music, as well as to utilize all 12 basic chart scales and handling chord condition to distinguish major/minor scale."

I find there to be several problems here that make this paragraph hard to follow. First, "musical skills" are not defined or explained, although they are frequently referred throughout the paper. I believe that they relate to domain knowledge, but I cannot figure out exactly what they mean, and I think that this actually matters in this paper. (Note: on a subsequent read, I believe they relate to the heuristics described in Appendix A. If so, then I would say that, given the paper's current form, Appendix A is absolutely essential for having a chance of understanding the paper.) Second, the term "Relational Pitch Change" may refer to intervals, or something to do with transposition, but the following sentence, "This approach [...] (...A, A#, B)." doesn't make much sense to me. I believe that it refers to transposition-invariance, but how is the music "matched" to the "basic chart scales"? (Note that the latter are not clearly defined, but again, I can guess). The next sentence refers to generating music with "chord conditions with major scale" but it is not clear how this is done-- is the music generated conditionally based on a chord? or based on a scale-treated-as-though-it-were-a-chord? or some other possibility? Finally, the last sentence puts some of these ideas together and suggests that the system analyzes the "musical skills" (heuristically-determined features) in a bar and then somehow uses those to synthesize new music that presumably has some of these features. Exactly how this happens is not at all clear to me, even by the end of the paper.

Many paragraphs have this sort of opacity, and the cumulative effect is a paper which I find to be completely opaque.

On p2, musical terms are defined. E.g. "Time signature 3/4 means 3 of 4th note construct 1 bar. So, if one bar at 4/4 beats is divided into 16 number of parts, one minimum unit is divided into 16th notes and the triplet note of 16th note is the smallest unit in the case of 24 number of parts. our study use these 2 minimum units."  Fortunately time signatures are familiar, so a bit of confusion here was no problem. Later, the authors write: "However, we only uses MIDI only with 4/4 time signature. The reason is that we judging that handling all kinds of time signature can reduce model’s performance a lot." I will note that in many MIDI datasets, 4/4 is a default time signature in the MIDI file, even if the piece itself is not actually in 4/4 (i.e. the default time signature does not actually affect the playback, and so it is left uncorrected).  This is not necessarily a problem, but if they were counting on the pieces being in 4/4, then the authors could indicate what they did to check or ensure that their MIDI files were indeed in 4/4 (not necessarily an easy task).

For better or worse, Section 2.5 is missing entirely, other than a section title that suggests it was going to provide background on LSTMs. On its own, a missing background section might not be a problem, but again the cumulative effect is that of a sloppiness that runs throughout. This extends not only to missing words, but also to the logic of the writing itself, e.g. the last sentence of the first paragraph begins with the phrase "In other words", but I don't actually see the rest of the sentence as summarizing what preceded it in any way. Small and picky, I realize, but the pervasiveness of this logical sloppiness makes it hard to follow.

In terms of quality, I think that the general intuition of incorporating musical heuristics is a sensible approach. But it is simply unclear to me exactly how it is being done here. Some key published references are missing, e.g. a few starting points for references include: "Sequence Tutor..." by Jaques et al (2017) which also aims to incorporate domain knowledge, and perhaps the transformer-based approaches by Huang et al and Payne et al as examples of recent methods. 

The evaluation is somewhat problematic as well. Table 1 is a bit unclear, and the user-study categories are confusing and I am not sure that they are measuring what they are intending to measure, nor am I even sure what they are intending to measure. E.g. "Creativity" was explained to the users as: "If the possibility of using a given rhythm is enormous, or if you think it has not existed before, it has highly creativity points."  

Finally, the samples that I listened to were not very convincing either.

In summary: I would be glad to read a clear version of this paper, and would not hesitate to change my score if appropriate. The samples I heard do not sound particularly effective, so unless I understand them, too, in a new light, I am unlikely to change my score by much, but I would certainly be open. I am very interested in this direction of work. I cannot accept the current version because I am often unsure of what is being done, and therefore I cannot assess if it is reasonable or not. 

In terms of my confidence score: I am absolutely certain that the paper is too unclear to be accepted, and I am very familiar with the relevant literature. However, I am not absolutely certain that the evaluation is correct, because in fact I can't evaluate the system itself very well based on the description given in the paper. 

Thanks for your response. 

I have taken a quick look at some sections of the updated paper. Some problematic paragraphs---e.g. the one that I long-quoted in my first review---are verbatim identical to how they appeared in the first version (other than s/"In addition"/"Also"). The few highly relevant references I mentioned have not been added yet, etc. As this is a work in progress, that is no problem; I understand that one would not be able to fix everything in one rebuttal period!

My understanding is also that you plan to keep working on all aspects of the project: the writing, the evaluation, and the model itself. Best of luck with all this!



Thanks for your comments and information for our paper, and sorry for the late reply.

The evaluation method for our paper has many problems, so we're working on the improvement of the evaluation method by using more samples, and more participants. And also find new evaluation methods.

2.1 Section is for explanation musical words, We areconsidering moving this part to the appendix, but in the current revised version, I have only modified the words first.

2.2 To avoid overfitting in cGAN(Case of all of the generated value has close to 0 or 1), We use a Gaussian filter. Although some cases of using Gaussian blurs for GAN have been found, it is likely to be difficult to citation the use of them to clearly avoid overfitting. It seems possible to provide evidence-based on experimental results.

2.4 We're really sorry about this but We don't understand what you mean. Should we delete the cGAN equation form if we cited other's paper? or how we modify this section specifically?

2.5  This empty section is all of our faults. We decided to delete the LSTM section because this model is used simply as a layer for performance, not as a key model used in FLAGNet. We deleted it in the revised version.

3.1 Our model doesn't consider the whole following melody but only considers what skill to generate, so It may not perfectly match with the melody continuation task.(But We'll try that task with our model.)But considering the following melody task can improve our generation model, We think that we can add it later. 

4. Of course, This is not generated by the Bernoulli distribution, but GAN-based music generators often sound unstable, depending on the scope of the pitch and how the minimum note is set (in fact, in the MIDINet paper, which is a GAN-based music generator, only 24 Pitch and 16th note are used). In this paper, I think we can supplement this by utilizing conditions such as Pitch, Minimum Note, Chart scale, etc. And also explained in 3.1, our model doesn't consider the following melody, the generated music can be hard to remember. Certainly, it would be necessary to consider the following model.

There is a revised version with some correction of minor writing issues and some adding explanations for contents in papers.

Thanks for your comments!

We appreciate all your detailed feedback, and sorry for the late reply.


1.We recognize that there are many major/minor problems in this whole paper. we are now spending time to supplement them.

2~3.With these comments, we realized how the evaluation method should be changed. We'll consider increasing the cases for each test condition and Using the platforms for larger participants.

There is a revised version with some correction of awkward languages and more explanation for the paper's contents.
I'm working on the effective usage of the FLAGNet model, and the evaluation method.

 There is an environment in which cases can be manufactured using the unconditional GAN, So we can set more test conditions and cases.
 We had no information about platforms such as Mechanical Turk, thanks for your specification! I plan to actively review the above contents and reflect them in the evaluation.



We appreciate all your detailed feedback on my paper's contents, and sorry for the late reply. 

There is a revised version with some correction of minor writing issues and some adding explanations for contents in papers.

In order to solve the major writing issues, it seems that the only way to revise our thesis is to constantly read it. We're spending more time on that. I will do as much as I can by referring to what you told us in this comment.

Section 2.5 is all of our faults. We decided to delete the LSTM section because this model is used simply as a layer for performance, not as a key model used in FLAGNet. We deleted it in the revised version.

I'm working on the effective usage of the FLAGNet model, and the evaluation method.

Thanks for your comments!



We appreciate all your detailed feedback, and sorry for the late reply. Since your review contains our (relatively) good things and things for improvement, we were able to figure out what problems we had easily.

There is a revised version with some correction of awkward languages and more explanation for the paper's contents. We tried to resolve minor problems like description problem, clarifying problem, and grammar problem what you mentioned.

Details of "skill labels" are given in appendix A, but reviewers, including you, argue that there is a lack of explanation of skill labels. So, at the beginning of the model's explanation, we tried to add some explanation of "skill labels" and mention for appendix A.

 Our results contain monophonic data + chord, but the model(cGAN) which trained with monophonic data generates only monophonic rhythm with binary matrix form. The chord is selected in major chords based on the given chart scale(C, C#, D...) randomly. Major chords for the given chart scale has 5 formal cases(There can be 1 more sub-case, but We didn't consider it), and we thought that all combinations are 'generally' available for generating music.
 But of course, defining good combinations with chords is better than a random process, we think that we should consider this chord definition task.

I know we have critical problems you said: Our result has a lack of proof to be successful or have good impacts. It would be best to find a way to develop the model as much as possible in the remaining time and to find a clear evaluation method.

Thanks for your comments!




This paper tries to take a GAN-based approach to monophonic music composition. They introduce a concept of "skill labeling" which they use to condition their GAN so that the generated music has certain bar-by-bar characteristics. They evaluate the generated music by comparing it with human and computer baselines in a survey.

Before getting into specific issues, there has been a lot of research into symbolic music generation over the past 60 years, and the issues encountered are usually related to generating structurally coherent music. A sophisticated hand-engineered markov chain can incorporate domain knowledge and generate music which is locally coherent, but even contemporary methods like MIDINet which model longer-term structure struggle to create globally coherent music. The model presented here doesn't address this limitation in the existing literature and, arguably, doesn't generate locally coherent music either.

More details on your survey methodology would help strengthen your case. There is a lot of evidence that surveys are a poor way to evaluate creativity in music composition (e.g. http://ccg.doc.gold.ac.uk/ccg_old/events/ecai06/proceedings/Moffat.pdf) because it is easy to unintentionally mislead or bias your participants. Your method also only scores the best based on novelty, but fails on your other questions. That's not very strong evidence in favor of it over MIDINet.

Specifically:

2.1 This section is both unnecessary and unusual. You use the term "polyphonic rhythm" when you may mean "homophonic texture." You shouldn't introduce new terms for concepts which are already well understood in music theory.

2.2 Why do you use a gaussian filter over your binary matrix? Can you provide a citation or experimental evidence for that decision?

2.4 You cite Mirza and Osindero, no need to restate these equations.

2.5 Empty section?

3.1 If you're using a dataset from MIREX, why don't you provide results versus their benchmarks? The melody continuation task has clear evaluation measures, which are more reliable than a survey.

4 The music you show here is rhythmically novel, but looks like it was sampled from a Bernoulli distribution. It would be difficult to perform and harder to remember.

Generally, there are also a lot of spelling and grammatical errors. Please spend more time editing your writing before submitting.

Despite these issues, the idea of conditioning your GAN on descriptors is good! If there was some way to describe musical structure at multiple levels, you may be able to generate something really musical.

This paper proposes a music generation system consisting of A) an RNN over sequences of hand-engineered musical features, which are fed into B) a conditional GAN to generate pianoroll images which are then post-processed to be monophonic melodies.

The paper has many issues that need to be addressed:

1) The language of the paper needs a lot of editing and is quite difficult to comprehend.  I'm pretty sure I understand the proposed system at a high level, but a lot of the details are unclear to me due to the awkward language and at times non-standard vocabulary.

2) The main "idea" in the paper is to use hand-engineered musical features as an intermediate representation for generation.  However, there's no experiment that tests whether or not this is even helpful; I would expect to see a comparison between the GAN conditioned on musical features and an unconditional GAN.  Using hand-engineered features makes sense for human control or to get around technical limitations, but in some sense the whole point of deep learning is that we don't need to use such features and can train end-to-end.  If the authors would like to continue to pursue this line of research, as a very first step I suggest performing the above experiment.

3) The evaluation setup is insufficient for comparing the various experimental conditions.  With only 1-2 examples chosen per condition (and how were they chosen?), the human ratings provide very little information about the quality of the model's output in general.  It's easy to generate a large number of samples from such a model, and with platforms such as Mechanical Turk it's fairly inexpensive to run a large listening study with hundreds or even thousands of participants.

4) Even if the model conditioned on hand-engineered features were clearly superior to an unconditioned model, this result would probably not be of broad interest to the ICLR community.  If the authors choose to continue this line of research, I recommend submitting to a music-specific conference.

Meta-review: All Reviewers and myself agree that the paper presents several major issues that require important rethinking of the research done, as well as a full rewriting of the manuscript. Hence, my recommendation is to REJECT the paper. As a brief summary, I highlight below some pros and cons that arose during the review and meta-review processes.

Pros:
- Code is available and clarifies parts of the approach.
- Use of publicly-available data sets.
- Samples are provided.
- Interesting problem.

Cons:
- There are several problems with language and writing. Sometimes there is also incorrect terminology.
- There are several problems with the description of the approach, which makes it opaque and hard to understand.
- Proposed model not addressing basic limitations in the existing literature.
- Insufficient evaluation.
- No clear indication of successful results.
- Potential lack of broad impact/interest to the ICLR community.
- Unclear contribution.
- Unconvincing samples.

Decision: withdrawn-rejected-submissions