Title: Towards Deep Interpretability (MUS-ROVER II): Learning Hierarchical Representations of Tonal Music
Authors: Haizi Yu
Lav R. Varshney
Venue: ICLR 2017

Abstract: Music theory studies the regularity of patterns in music to capture concepts underlying music styles and composers' decisions. This paper continues the study of building \emph{automatic theorists} (rovers) to learn and represent music concepts that lead to human interpretable knowledge and further lead to materials for educating people. Our previous work took a first step in algorithmic concept learning of tonal music, studying high-level representations (concepts) of symbolic music (scores) and extracting interpretable rules for composition. This paper further studies the representation \emph{hierarchy} through the learning process, and supports \emph{adaptive} 2D memory selection in the resulting language model. This leads to a deeper-level interpretability that expands from individual rules to a dynamic system of rules, making the entire rule learning process more cognitive. The outcome is a new rover, MUS-ROVER \RN{2}, trained on Bach's chorales, which outputs customizable syllabi for learning compositional rules. We demonstrate comparable results to our music pedagogy, while also presenting the differences and variations. In addition, we point out the rover's potential usages in style recognition and synthesis, as well as applications beyond music.


Review discussion:
In the paper, I didn't see detailed introductions about the student optimization (right part of Fig. 1). What is S_q? Could the authors elaborate on it? I assume this is part of the author's previous system. It would be great to mention it briefly in the text.
Also, what is the meaning of mod_12, diff, order, etc (in Table. 1). The introduction in Appendix A.1 is a bit abstract and might not be helpful for people who are not familiar to music.
Thanks for the comments! We slightly revised the paper to include some brief explanations, and hopefully it clears the confusions. Here, I'll quickly address your questions directly:
1) Yeah, we had detailed descriptions of the student's optimization problem in our previous papers (Yu et al., 2016a;b), and we didn’t make any changes to this optimization problem. In the revision, we have made this crystal clear in the sub-section “What is Inherited from MUS-ROVER I” (the 3rd paragraph in particular), and also briefly mentioned it in the caption of Figure 1.
S_q(p), in its most generic form, is a Tsallis entropy of a probability distribution p, which achieves a maximum when p is uniform and a minimum when p is deterministic.
Thus the student's optimization problem disperses probability mass across all the rule-satisfying possibilities and encourages creativity from randomness. Since each constraint is a linear equality constraint, when q=2, S_q(p) = 1-||p||_2^2, which explains why the student's optimization problem is a linear least-squares problem (Figure 1).
2) mod_12, diff, order, etc are atomic arithmetic operators. For a vector x, mod_12(x) takes modulo 12 element-wisely to x, diff takes the difference between successive vector values, order is the ordering of the vector elements (similar to argsort). We have added the precise formulae for these operators in the Appendix A.4 now.
The paper only expects the reader to understand these operators in terms of their mathematical meanings, NOT their music meanings. And you are absolutely right that Table 3 (in Appendix A.1) is not useful for people who are non-musicians, and that’s why we put it in the appendix. The purpose of Appendix A.1 is just pointing out that sometimes the compositions of these math operators will coincide with some pre-defined notions in music, and accordingly, one is potentially learning the same music concept in terms of math operators.
Would any rule recognized in the self-learning loop be consistent with the music theory? If not, how did you interpret and address the exception?
Thanks for the question!
The rules extracted by the self-learning loop have a great deal of overlap with our music theory, and they are consistent. We have shown 4 of them in Table 2. No rules are inconsistent with the music theory.
But there are some rules in music theory that are not covered by MUS-ROVER II, and there are some rules from MUS-ROVER II (see some of the asterisks in Table 2) that are not covered by existing music theory. The former shows the incompleteness of MUS-ROVER II, and the latter shows the incompleteness of the current theory.
So one of the use-cases of MUS-ROVER II is to complement the study of music theory, acting as a good assistant of theorists.
One thing I'd like to clarify is that one should not judge the correctness of the extracted rules, or in some sense, they are always "correct". Since both MUS-ROVER and music theory are objective revealings of the underlying patterns. Those patterns are already there! It is a matter of finding it or not, but not a matter of right or wrong. That being said, if both MUS-ROVER and music theory find a particular rule describing the same high-level concept (feature), they must be consistent.
Just to clarify: would the described system be able to handle voice leading such as “resolving a suspension downwards”, and “avoid parallel fifths” (even though a *fifth* is fine), which depend on multiple voices as they move through time? The only place I saw that might address this is the end of Section 6, but it was not clear to me how well such characteristics are actually handled, if at all.
If this *can* be handled in principle, then the next question is whether the resulting rules be in a form that would make any intuitive sense to a beginning student? (i.e. "resolve the 4th down to the 3rd" is a fairly intuitive single phrase that is easy to for a student to understand, but i could imagine this same concept being expressed as a conjunction of various rules involving interval counting, etc, that would not be nearly as clear.) Given that the system is intended for students, and given that a big part of generating Bach chorales is understanding voice leading, this is an important consideration.
Thanks for these two great questions regarding music theory and teaching!
First question first. Not just in principle, but also in reality, MUS-ROVER is indeed able to recover many well-known compositional rules that one will typically learn in a theory class. Those include not only voice leading rules but also rules on harmonic progression and harmonic function. More specially, in our earlier work, MUS-ROVER I already recovered rules like “Parallel fifths/octaves are rare” and “resolving a tritone outward or inward”. These two were actually the examples we used in our previous publications (Yu et al., 2016a;b). In this paper, MUS-ROVER II has a much more expressive and structured feature universe, further recovering harmonic rules such as resolving a V/V7 chord to a I/i chord, and even rules that involve a short phrase model (i.e. T-PD-D-T) due to the longer dependencies introduced in this paper.
In light of the audiences of ICLR (as opposed to MUME, which is more music related), we don’t intend to make the readers be overwhelmed by music terminologies and concepts that they are not familiar with. For readers with extra interest and expertise in music, I would recommend referring to our earlier publications as external resources, which is definitely NOT required for understanding the model and algorithms in this paper.
For the 2nd question, the resulting rules are indeed in a form that would make intuitive sense to a student. We held a discussion in our previous publications on why it is absolutely necessary to learn rules from A RANGE of n-grams instead of just ONE SINGLE n-gram with the largest possible n. Each time when the self-learning loop extracts a rule, it could be either from a new rule feature, or from an existing rule feature but in a different context (i.
e. same feature, different n-gram). In the latter case, the rule with the same rule feature will replace the rule that was learned earlier, and this is exactly when “Parallel P5s” is making sense. Here is a concrete example: when a student first learned fundamentals (1-gram) from MUS-ROVER, she will learn one rule, featured on interval class, which talks about consonant and dissonant intervals, and tells her that P5 is consonant and encouraged.
However, later on, in learning part writing which is context dependent (higher-order n-grams), a 2-gram rule with the same feature (i.e. interval class) will be extracted if the previous sonority exposes a P5. The 2-gram rule distribution is very different from the 1-gram case, which shows a great drop for P5’s probability mass. And this is exactly how the student learns the concept that “try avoiding parallel P5s”.
The importance of keeping a range of n-grams, and keeping track of the entire rule learning process are further emphasized in this paper, since now we are enabling much longer dependencies (Music is highly context dependent). And when we talk about DEEPER INTERPRETABILITY in the title, we mean something more DYNAMIC. As opposed to most of the machine learning algorithms, where people try interpreting end results (e.g., visualizing the LEARNED intermediate layers of a neural net), MUS-ROVER emphasizes the interpretability through the entire learning process, not just the final results.
So students are expected to be highly involved in the learning loop, keeping track of not just new rules, but also how an old rule is modified due to the new context (e.g. a single fifth to parallel fifths).
I am curious how well the described approach might scale. Certain aspects (seem to me to) depend on exhaustive searches/enumerations, but this was not entirely clear. Suppose, for example, that rather than Bach chorales, the system was optimized to help students learn orchestration, where there can easily be 10-20 instruments. In this case, the family of windows in Eq(1) would therefore correspond to a significantly larger power set.
Furthermore, the pieces are much longer, so counting occurrences of arbitrary compositions of atomic operators could be combinatorially non-trivial, etc.
Yet, conceptually organizing the rules of orchestration hierarchically might actually be quite a helpful application. Any comment on this would be appreciated.
Again a great great question! We indeed started the problem as an exhaustive search problem in our previous work. And part of the motivation of this paper is to alleviate such computation burden by leveraging a feature hierarchy (see those hierarchal filters in eq. (3)).
Also when generating the feature hierarchy, we have exploited some heuristics on both windows and descriptors (see Appendix A.3), so together with Algorithm 1 in Appendix A.2, we are not enumerating all possibilities in a brute-force manner.
However, MUS-ROVER II is, by no means, the perfect search engine in exploring the feature universe, and there is much work to be done in the future to make it more intelligent. Currently for Bach’s 4-part chorales, the complete learning process is doable on a 16GB RAM Macbook Pro (from 1-gram up to 10-gram).
However, I don’t think the current framework is applicable for pieces with 10-20 instruments on the same machine. Since we are in the process of building future generations of MUS-ROVERs, which will extend the dataset from Bach’s chorales to other music styles, we’ll further exploit the hierarchy which would have more compact factorizations, or alternatively design algorithms to reduce pieces with more than 4 parts to 4 parts as part of the data pre-processing task. Any further suggestions on this open-ended research problem will be much appreciated!
Do you have any intuition of what is happening in the bottom row (“5-gram”) of the top part of Figure 4?
The top heat map in Figure 4 illustrates how the number of relevant examples varies across different features and different n-grams.
When I say the word “relevant”, it means the example (found from Bach’s chorales) matches the corresponding context. More specifically, given a feature, and an n-gram, a relevant example comprises n consecutive sonorities: the features of the first (n-1) sonorities in this example should match the features of the previous (n-1) sonorities written down by the student.
Then in this heat map, a redder region denotes a larger number of (relevant) examples, while a bluer region denotes a smaller number of (relevant) examples. Given a feature (i.e. fixing a column), one will typically have a smaller number of examples associated with a higher order n-gram, because it is harder to find an example that matches more features (longer context). This is indeed the tradeoff between matching a longer pattern, and at the same time, being less confident.
So the color is getting colder when you look at a particular column from top down. You can ignore the color changes horizontally, since for better visualization, I manually ordered the columns in a way such that the left part looks colder, and the right part looks warmer.
This paper proposes an interesting framework (as a follow-up work of the author's previous paper) to learn compositional rules used to compose better music. The system consists of two components, a generative component (student) and a discriminative component (teacher). The generative component is a Probabilistic Graphical Models, generating the music following learned rules. The teacher compares the generated music with the empirical distribution of exemplar music (e.g, Bach’s chorales) and propose new rules for the student to learn so that it could improve.
The framework is different from GANs that the both the generative and discriminative components are interpretable. From the paper, it seems that the system can indeed learn sensible rules from the composed music and apply them in the next iteration, if trained in a curriculum manner.
However, there is no comparison between the proposed system and its previous version, nor comparison between the proposed system and other simple baselines, e.g., an LSTM generative model. This might pose a concern here.
I found this paper a bit hard to read, partly due to (1) lots of music terms (e.g, Tbl. 1 does not make sense to me) that hinders understanding of how the system performs, and (2) over-complicated math symbols and concept.
For example, In Page 4, the concept of raw/high-level feature, Feature-Induced Partition and Conceptual Hierarchy, all means a non-overlapping hierarchical clustering on the 4-dimensional feature space.
Also, there seems to be no hierarchy in Informational Hierarchy, but a list of rules. It would be much clearer if the authors write the paper in a plain way.
Overall, the paper proposes a working system that seems to be interesting.
But I am not confident enough to give strong conclusions.
Summary:
The paper presents an advanced self-learning model that extracts compositional rules from Bach's chorales, which extends their previous work in: 1) the rule hierarchy in both conceptual and informational dimensions; 2) adaptive 2-D memory selection which assumes the features follow Dirichlet Distribution. Sonority (column of 4 MIDI numbers) acts as word in language model: unigram statistics have been used to learn the fundamental rules in music theory, while n-grams with higher order help characterize part writing. Sonorities have been clustered together based on feature functions through iterations. The partition induced by the features is recognized as a rule if it is sufficiently significant.
As a result, two sample syllabi with different difficulty strides and "satisfactory gaps" have been generated in terms of sets of learned rules.
1. Quality:
a) Strengths: In the paper, the exploration of hierarchies in two dimensions makes the learning process more cognitive and interpretable. The authors also demonstrate an effective memory selection to speed up the learning.
b) Flaws: The paper only discussed N<=5, which might limit the learning and interpretation capacities of the proposed model, failing to capture long-distance dependence of music. (In the replies to questions, the authors mentioned they had experimented with max N=10, but I'm not sure why related results were not included in the paper).
Besides the elaborated interpretation of results, a survey seeking the opinions of students in a music department might make the evaluation of system performance more persuasive.
2. Clarity:
a) Pros: The paper clearly delivers an improved automatic theorist system which learns and represents music concepts as well as thoroughly interprets and compares the learned rules with music theory. Proper analogies and examples help the reader perceive the ideas more easily.
b) Cons: Although detailed definitions can be found in the authors' previous MUS-ROVER I papers, it would be great if they had described the optimization more clearly (in Figure 1. and related parts).  The "(Conceptual-Hierarchy Filter)" row in equations (3): the prime symbol should appear in the subscript.
3. Originality:
The representation of music concepts and rules is still an open area, the paper investigate the topic in a novel way. It illustrates an alternative besides other interpretable feature learning methods such as autoencoders, GAN, etc.
4. Significance:
It is good to see some corresponding interpretations for the learned rules from music theory. The authors mentioned students in music could and should be involved in the self-learning loop to interact, which is very interesting. I hope their advantages can be combined in the practice of music theory teaching and learning.
After the discussion below, I looked at previous work by the authors (MUS-ROVER) on which this paper was based. On one hand, this was very helpful for me to better understand the current paper.
On the other hand, this was very needed for me to better understand the current paper.
Overall, while I think that I like this work, and while I am familiar with the JSB chorales, with probabilistic approaches, with n-grams, etc, I did find the paper quite hard to follow at various parts. The extensive use of notation did not help the clarity.
I think the ideas and approaches are good, and certainly worth publishing and worth pursuing. I am not sure that, in the paper's current form, ICLR is an appropriate venue. (Incidentally, the issue is not the application as I think that music applications can be very appropriate, nor is the problem necessarily with the approach. see my next suggestion. ). I get the sense that a long-form journal publication would actually give the authors the space necessary to fully explain these ideas, provide clearer running examples where needed, provide the necessary background for the appropriate readership, provide the necessary background on the previous system, perhaps demonstrating results on a second dataset to show generality of the approach, etc.
A short conference paper just seems to me to be too dense a format for giving this project the description it merits. If it were possible to focus on just one aspect of this system, then that might work, but I do not have good suggestions for exactly how to do that.
If the paper were revised substantially (though I cannot suggest details for how to do this within the appropriate page count), I would consider raising my score. I do think that the effort would be better invested in turning this into a long (and clearer) journal submission.
[Addendum: based on discussions here & revisions, I have revised my score]
We thank all the reviewers for their helpful comments and questions!  Below we clarify a few points raised in the reviews and describe the revisions we have made to the paper.
All three reviewers, from different perspectives, suggested better positioning of the paper relative to our own prior work and other existing work. We have taken these comments to heart and revised the paper accordingly.
In particular, we have further emphasized the following two connections.
MUS-ROVER II vs. MUS-ROVER I: The purpose of the “MUS-ROVER Overview” section was to clarify the connection between the current rover and its earlier version. We have revised it to make the connection clearer. More specifically, we have updated Figure 1’s caption as well as the “(Rule Representation)” paragraph to include a brief explanation of the student’s optimization problem whose formulation is unchanged in the current rover (Reviewer 1).
Also, although we had compared the two rovers in terms of their models, and made it clear what is inherited from our prior work and what is new in our current work, we realize that we did not explicitly compare the two rovers in terms of results and/or contributions.
So in the revision, we first state the contribution of our prior work, making it clear that the earlier rover was already able to extract basic voice-leading rules such as “Parallel perfect intervals are rare” (Reviewer 2); and then we emphasize that the current rover not only extracts more rules with longer-term dependencies, but more importantly, studies the hierarchies of the extracted rules.
Furthermore, we have added a visualization subsection in the end of the “Experiments” section to further compare the results from the two rovers, i.e. hierarchical rule families and subfamilies vs. unstructured rule sequences (Reviewer 3).
In addition, we want to clarify that the purpose of this paper is indeed to focus on one aspect of the system (Reviewer 2) rather than to try to explain everything in details for the entire MUS-ROVER system. The aspect that we emphasize in this paper, is the deeper interpretability that is achieved by rule hierarchies and adaptive 2D memory selection. We totally agree with Reviewer 2 that at some point in the future, a comprehensive journal paper will be the best place to fully explain the entire MUS-ROVER system.
However, it is apparent that MUS-ROVER II is not the end of this line of research, and we are making progress in continuing the study of building new versions of rovers, each of which emphasizes a different aspect of the system. It is quite common in our area to publish several conference papers with distinct focuses and then to synthesize them in a longer journal paper to unify the framework. This is the path that we plan to follow.
MUS-ROVER vs. Generative models (e.g. LSTMs): Although MUS-ROVER has a generative component that can be used to generate sample pieces, we do not evaluate the rover’s performance based on its generating power. The reasons are twofold.
First, the quality (good or bad) of a music piece is not easy to quantify, though there is emerging work in psychology that suggests the Consensual Assessment Technique (CAT) of evaluation by many experts may be useful.
Secondly and more importantly, MUS-ROVER is not targeted as an automatic composer, but as an automatic theorist or pedagogue, so the goal is not to generate pieces, but to explain what has been learned through the entire learning process. We have revised several places in the “Introduction” section to try to make this distinction clear: MUS-ROVER, as a pathfinder to Mount Parnassus, cares about the path towards the destination a lot more than the destination per se. Given that the outputs of generative models (such as LSTMs) are generated samples, they are not comparable to outputs of MUS-ROVER, which are instead ways of generating samples (Reviewer 3).
Reviewer 3 mentioned that the math section on features, feature-induced partitions, and conceptual hierarchy is an over-complicated way of describing non-overlapping hierarchical clustering. We’d like to clarify that it is absolutely necessary to have this multi-step process: features → partitions → hierarchy, and a simple hierarchical clustering wouldn’t work to achieve our goal. The reasons are twofold. 1) Algorithmically, hierarchical clustering will lose many inter-connections due to its greedy algorithm and its tree structure (conceptual hierarchy on the other hand is a DAG). 2) More importantly, hierarchical clustering will lose the interpretability of the resulting partitions: as we pointed out in the “Feature-Induced Partitions” subsection, “We use a partition to refer to the essence of a concept, and the inducing feature function as a mathematical name to interpret the concept”.
So without features, having partitions alone will miss the goal of achieving deeper interpretability. To make this clear, we added a small paragraph in the end of the “Conceptual Hierarchy” subsection, emphasizing the necessity of our approach.
Reviewer 3 also mentioned difficulty in understanding the music terms. We apologize if any of the music symbols in the paper cause difficulty for non-musician readers.
However, as mentioned above and mentioned in the earlier responses, we do not require readers of this paper to have any music background, and moreover, we do not require the users of MUS-ROVER to have any prior knowledge on music theory either, since the whole purpose of MUS-ROVER is to teach music theory from scratch. The focus of this paper, as opposed to our prior work published in a music venue, is on hierarchies and adaptive 2D memory selection. Taking Table 1 as an example, we do not expect readers to be aware of any underlying music concept so there are no music terms there: all notations are functions, or more precisely, descriptors and windows that are introduced in the “Interpretable Features” subsection. We tried our best to restrict any music-related terms and symbols from the main body of the paper, and put them in the Appendix whenever possible.
Lastly, we strongly agree with Reviewer 1, that deploying MUS-ROVER into a real educational environment will be one of the most exciting things to try next. We are actively collaborating with professors from our music department, to make a live and personalized teaching system in the near future.

Meta-review:
role: PC
data: comment: Given that all reviewers were positive aobut this paper and given the unusual application domain, we recommend to accept this paper for poster presentation at the main conference.

Decision: Accept (Poster)