FLAGNet : Feature Label based Automatic Generation Network for symbolic music
Authors: SeongHyeon Go
Venue:  (2021)
Keywords: cGAN
RNN
MIDI generation
music

Abstract: The technology for automatic music generation has been very actively studied in recent years. However, almost in these studies, handling domain knowledge of music was omitted or considered a difficult task. In particular, research that analyzes and utilizes the characteristics of each bar of music is very rare, even though it is essential in the human composition. We propose a model that generate music with musical characteristics of bars by conditional generative adversarial network, and analyze the good combination of the sequence of which characterized bars  for symbolic-domain music generation by Recurrent Neural Network with Long short term memory layer. Also, by analyzing symbolic music data as image-like based on relational pitch approach, it increases the utilization of the data set with arbitrary chord scales and enables the use of generational results extensively. The resulting model FLAGNet generates music with the understanding of musical domain knowledge while handling inputs like minimum unit of note, length of music, chart scales, and chord condition.


Reviews:
[{"review_id": "utfmOZjct9p", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tries to take a GAN-based approach to monophonic music composition. They introduce a concept of \"skill labeling\" which they use to condition their GAN so that the generated music has certain bar-by-bar characteristics. They evaluate the generated music by comparing it with human and computer baselines in a survey.\n\nBefore getting into specific issues, there has been a lot of research into symbolic music generation over the past 60 years, and the issues encountered are usually related to generating structurally coherent music. A sophisticated hand-engineered markov chain can incorporate domain knowledge and generate music which is locally coherent, but even contemporary methods like MIDINet which model longer-term structure struggle to create globally coherent music. The model presented here doesn't address this limitation in the existing literature and, arguably, doesn't generate locally coherent music either.\n\nMore details on your survey methodology would help strengthen your case. There is a lot of evidence that surveys are a poor way to evaluate creativity in music composition (e.g. http://ccg.doc.gold.ac.uk/ccg_old/events/ecai06/proceedings/Moffat.pdf) because it is easy to unintentionally mislead or bias your participants. Your method also only scores the best based on novelty, but fails on your other questions. That's not very strong evidence in favor of it over MIDINet.\n\nSpecifically:\n\n2.1 This section is both unnecessary and unusual. You use the term \"polyphonic rhythm\" when you may mean \"homophonic texture.\" You shouldn't introduce new terms for concepts which are already well understood in music theory.\n\n2.2 Why do you use a gaussian filter over your binary matrix? Can you provide a citation or experimental evidence for that decision?\n\n2.4 You cite Mirza and Osindero, no need to restate these equations.\n\n2.5 Empty section?\n\n3.1 If you're using a dataset from MIREX, why don't you provide results versus their benchmarks? The melody continuation task has clear evaluation measures, which are more reliable than a survey.\n\n4 The music you show here is rhythmically novel, but looks like it was sampled from a Bernoulli distribution. It would be difficult to perform and harder to remember.\n\nGenerally, there are also a lot of spelling and grammatical errors. Please spend more time editing your writing before submitting.\n\nDespite these issues, the idea of conditioning your GAN on descriptors is good! If there was some way to describe musical structure at multiple levels, you may be able to generate something really musical.", "extra_scores": {}, "extra_text": {}}, {"review_id": "t4Hl9Y5iz8h", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nIn this paper the authors propose a method for generating music using a GAN. They claim to have improved performance by incorporating domain knowledge. Unfortunately, the evidence of this, even by their own metrics, is somewhat lacking. Despite this, their contribution could have been notable if they had detailed their feature extraction and evaluated the impact/benefit of each feature upon performance, but this was also not done. Overall, even by the metrics provided in the paper, the contribution is very unclear. Without clarification of this, and addressing the above comments, I would recommend this paper is rejected.\n\nThe good things:\n* Code is released with this paper which helped me to understand the feature generation and modelling process\n* Figure 2: explains the modelling process relatively well on a high level\n* The authors make use of a publicly available dataset\n* The authors provide examples for comparison with another method\n\nThings for improvement:\n* Descriptions could be much clearer: for instance - I have interpreted the meaning of \"musical skill\"/\"skill labels\" to mean \"bar level feature\". There are other terms used which do not match well with the english meanings.\n* Details of these \"skill labels\" should be provided: this seem to be the main contribution of the work, but there is no explicit discussion of the features derived from the bars of the music.\n* The data being used for modelling needs clarifying: In 3.1 Dataset, the authors state they are using the monophonic data from PPDD, but modelling is done for monophonic and monophonic + chords. In the final para of 3.3 it is implied that these chords are added to each bar randomly but the process is very unclear. If this is a random process, a justification is required as to why the learned representations are interesting.\n* The evaluation metrics are not well grounded: the authors pick 4 terms by which participants should give a score of max 5 (is 1 the minimum or 0?). There is no link to the literature that these are good/accepted terms to use. At minimum, there should be some discussion as to why these specific terms were selected and a comment made about. Also, on page 7, the authors begin to refer to \"stability\" is such a way that it makes me think it was a previously evaluated but now omitted term.\n* Additional justification is required for some claims: for instance, in the abstract the authors state \"However, almost in [sic] these studies, handling domain knowledge of music was omitted or considered a difficult task.\" but there is no justification of this in the text (one approach from the literature is given in the introduction, but there have been many other approaches).\n* Figure/table captions should be more descriptive: for example, Table 1' caption reads \"Human evaluation result\". I would propose something more like: \"Human survey comparison of model performance against true data: 15 participants were surveyed and asked to score examples on a scale of 0(?)-5 for the qualities listed. The rows listed as human music are real examples taken from the Lakh midi dataset.\" (I would also: add the standard deviations to the results, since they are calculable and would help establish significance, and round all values to a consistent and appropriate level)", "extra_scores": {}, "extra_text": {}}, {"review_id": "gmZ43F0JFmF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I believe that this paper is addressing the problem of incorporating domain knowledge into the generation of symbolic music. I find this problem very interesting.\n\\newline\nThe primary problem with this paper is its lack of clarity. Unfortunately, this is so severe, that I cannot tell what is being done. \n\nIn particular, when reading the PDF, I highlighted every sentence that had minor or major writing issues (ranging from sentences that were nevertheless understandable to others that were impossible for me to decode), and by the end I found that I had highlighted nearly every sentence in the paper. It is the cumulative effect that becomes quite problematic.\n\nAs an example, I will provide the second half of the paragraph given at the end of the Introduction (p1-2), in which I believe that the authors give an overview of their system (as far as I can tell):\n\n\"Thus, based on a given music dataset, we combine how the sequence of musical skills can be attractive and how the bars created by using the simple RNN model, i.e. handling about flow of music. In addition, to make image size can be properly reduced and the Musical Skill can be maintained while processing MIDI Bar with Image we utilize image processing based on Relational Pitch Change. This approach allows the use of the music in the train without relying on the chord scale of given music in dataset, and provides a wider range of possibilities for the music produced by matching the music to the 12 basic chart scale (C, C#, D, D#, E, F, F#, G, G#, A, A#, B). Also, to distinguish major scale and minor scale, we can generate music with chord conditions with major scale. The resulting model, FLAGNet can use the musical skill contained in the music bar to understand the musical domain knowledge, analyze the sequence of the musical skills to control the overall flow of music, and process the generated images to make symbolic music, as well as to utilize all 12 basic chart scales and handling chord condition to distinguish major/minor scale.\"\n\nI find there to be several problems here that make this paragraph hard to follow. First, \"musical skills\" are not defined or explained, although they are frequently referred throughout the paper. I believe that they relate to domain knowledge, but I cannot figure out exactly what they mean, and I think that this actually matters in this paper. (Note: on a subsequent read, I believe they relate to the heuristics described in Appendix A. If so, then I would say that, given the paper's current form, Appendix A is absolutely essential for having a chance of understanding the paper.) Second, the term \"Relational Pitch Change\" may refer to intervals, or something to do with transposition, but the following sentence, \"This approach [...] (...A, A#, B).\" doesn't make much sense to me. I believe that it refers to transposition-invariance, but how is the music \"matched\" to the \"basic chart scales\"? (Note that the latter are not clearly defined, but again, I can guess). The next sentence refers to generating music with \"chord conditions with major scale\" but it is not clear how this is done-- is the music generated conditionally based on a chord? or based on a scale-treated-as-though-it-were-a-chord? or some other possibility? Finally, the last sentence puts some of these ideas together and suggests that the system analyzes the \"musical skills\" (heuristically-determined features) in a bar and then somehow uses those to synthesize new music that presumably has some of these features. Exactly how this happens is not at all clear to me, even by the end of the paper.\n\nMany paragraphs have this sort of opacity, and the cumulative effect is a paper which I find to be completely opaque.\n\nOn p2, musical terms are defined. E.g. \"Time signature 3/4 means 3 of 4th note construct 1 bar. So, if one bar at 4/4 beats is divided into 16 number of parts, one minimum unit is divided into 16th notes and the triplet note of 16th note is the smallest unit in the case of 24 number of parts. our study use these 2 minimum units.\"  Fortunately time signatures are familiar, so a bit of confusion here was no problem. Later, the authors write: \"However, we only uses MIDI only with 4/4 time signature. The reason is that we judging that handling all kinds of time signature can reduce model’s performance a lot.\" I will note that in many MIDI datasets, 4/4 is a default time signature in the MIDI file, even if the piece itself is not actually in 4/4 (i.e. the default time signature does not actually affect the playback, and so it is left uncorrected).  This is not necessarily a problem, but if they were counting on the pieces being in 4/4, then the authors could indicate what they did to check or ensure that their MIDI files were indeed in 4/4 (not necessarily an easy task).\n\nFor better or worse, Section 2.5 is missing entirely, other than a section title that suggests it was going to provide background on LSTMs. On its own, a missing background section might not be a problem, but again the cumulative effect is that of a sloppiness that runs throughout. This extends not only to missing words, but also to the logic of the writing itself, e.g. the last sentence of the first paragraph begins with the phrase \"In other words\", but I don't actually see the rest of the sentence as summarizing what preceded it in any way. Small and picky, I realize, but the pervasiveness of this logical sloppiness makes it hard to follow.\n\nIn terms of quality, I think that the general intuition of incorporating musical heuristics is a sensible approach. But it is simply unclear to me exactly how it is being done here. Some key published references are missing, e.g. a few starting points for references include: \"Sequence Tutor...\" by Jaques et al (2017) which also aims to incorporate domain knowledge, and perhaps the transformer-based approaches by Huang et al and Payne et al as examples of recent methods. \n\nThe evaluation is somewhat problematic as well. Table 1 is a bit unclear, and the user-study categories are confusing and I am not sure that they are measuring what they are intending to measure, nor am I even sure what they are intending to measure. E.g. \"Creativity\" was explained to the users as: \"If the possibility of using a given rhythm is enormous, or if you think it has not existed before, it has highly creativity points.\"  \n\nFinally, the samples that I listened to were not very convincing either.\n\nIn summary: I would be glad to read a clear version of this paper, and would not hesitate to change my score if appropriate. The samples I heard do not sound particularly effective, so unless I understand them, too, in a new light, I am unlikely to change my score by much, but I would certainly be open. I am very interested in this direction of work. I cannot accept the current version because I am often unsure of what is being done, and therefore I cannot assess if it is reasonable or not. \n\nIn terms of my confidence score: I am absolutely certain that the paper is too unclear to be accepted, and I am very familiar with the relevant literature. However, I am not absolutely certain that the evaluation is correct, because in fact I can't evaluate the system itself very well based on the description given in the paper. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "JK4-dX4Akkm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a music generation system consisting of A) an RNN over sequences of hand-engineered musical features, which are fed into B) a conditional GAN to generate pianoroll images which are then post-processed to be monophonic melodies.\n\nThe paper has many issues that need to be addressed:\n\n1) The language of the paper needs a lot of editing and is quite difficult to comprehend.  I'm pretty sure I understand the proposed system at a high level, but a lot of the details are unclear to me due to the awkward language and at times non-standard vocabulary.\n\n2) The main \"idea\" in the paper is to use hand-engineered musical features as an intermediate representation for generation.  However, there's no experiment that tests whether or not this is even helpful; I would expect to see a comparison between the GAN conditioned on musical features and an unconditional GAN.  Using hand-engineered features makes sense for human control or to get around technical limitations, but in some sense the whole point of deep learning is that we don't need to use such features and can train end-to-end.  If the authors would like to continue to pursue this line of research, as a very first step I suggest performing the above experiment.\n\n3) The evaluation setup is insufficient for comparing the various experimental conditions.  With only 1-2 examples chosen per condition (and how were they chosen?), the human ratings provide very little information about the quality of the model's output in general.  It's easy to generate a large number of samples from such a model, and with platforms such as Mechanical Turk it's fairly inexpensive to run a large listening study with hundreds or even thousands of participants.\n\n4) Even if the model conditioned on hand-engineered features were clearly superior to an unconditioned model, this result would probably not be of broad interest to the ICLR community.  If the authors choose to continue this line of research, I recommend submitting to a music-specific conference.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All Reviewers and myself agree that the paper presents several major issues that require important rethinking of the research done, as well as a full rewriting of the manuscript. Hence, my recommendation is to REJECT the paper. As a brief summary, I highlight below some pros and cons that arose during the review and meta-review processes.

Pros:
- Code is available and clarifies parts of the approach.
- Use of publicly-available data sets.
- Samples are provided.
- Interesting problem.

Cons:
- There are several problems with language and writing. Sometimes there is also incorrect terminology.
- There are several problems with the description of the approach, which makes it opaque and hard to understand.
- Proposed model not addressing basic limitations in the existing literature.
- Insufficient evaluation.
- No clear indication of successful results.
- Potential lack of broad impact/interest to the ICLR community.
- Unclear contribution.
- Unconvincing samples.
Decision: Reject