Title: COUPLED RECURRENT MODELS FOR POLYPHONIC MUSIC COMPOSITION Anonymous

Abstract: This work describes a novel recurrent model for music composition, which accounts for the rich statistical structure of polyphonic music. There are many ways to factor the probability distribution over musical scores; we consider the merits of various approaches and propose a new factorization that decomposes a score into a collection of concurrent, coupled time series: "parts." The model we propose borrows ideas from both convolutional neural models and recurrent neural models; we argue that these ideas are natural for capturing music's pitch invariances, temporal structure, and polyphony.
We train generative models for homophonic and polyphonic composition on the KernScores dataset (Sapp, 2005) , a collection of 2,300 musical scores comprised of around 2.8 million notes spanning time from the Renaissance to the early 20th century. While evaluation of generative models is known to be hard (Theis et al., 2016) , we present careful quantitative results using a unit-adjusted cross entropy metric that is independent of how we factor the distribution over scores. We also present qualitative results using a blind discrimination test.
The composition of music using statistical models has been strongly influenced by developments in deep learning; see Briot et al. (2017) for a recent survey of this field. Previous work in this field mostly focuses on either monophonic scores (Sturm et al.

2. RELATED WORKS:

Early efforts to build statistical models of music focus on single part, monophonic sequences (melodies). Possibly the first statistical model for music generation was proposed by Pinkerton (1956) . This work was followed concretely by Brooks et al. (1956) , who built a Markov transition model estimated on small music corpora. A proliferation of work on computer-generated music and data-driven musicology followed these pioneering works in the 1960's and 1970's; see Roads (1980) for a survey. An important development during this era was the application of Chomsky-inspired grammatical analysis to music, exemplified by Kohonen (1989) ; this latter work contemplates the generation of two concurrent musical parts, one of the earliest examples of polyphonic generation.

The first application of neural networks to algorithmic melody composition was proposed by Todd (1989) . This work prompted followup by Mozer (1994) , who altered the representation of the input to Todd's model using pitch geometry ideas inspired by Shepard (1982) ; the relative pitch and note-embedding schemes considered in the present paper can be seen as a data-driven approach to capturing some of these geometric concepts. Neural melody generation was revisited by Eck & Schmidhuber (2002) , using long short-term memory models. More recent work on melodic composition experiments with techniques to capturing longer-term structure than classic recurrent models provide. Jaques et al. (2017) explore reinforcement learning as a tool for eliciting long-term structure, expanding on ideas first considered by Franklin (2001) . Roberts et al. (2018) also attempt to capture long-term structure, proposing a variational auto-encoder for this purpose.

The work on polyphonic music is considerably younger. The aforementioned work of Kohonen (1989) considers two-part composition. Another early precursor to polyphonic models was introduced by Ebcioglu (1988) , who proposed an expert system to harmonize 4-part Bach chorales. The harmonization task became popular, along with the Bach chorales dataset. See Allan & Williams (2006) for a classic discussion of this problem. Lavrenko & Pickens (2003) directly address multipart polyphony, albeit using a simplified preprocessed encoding of scores that throws away duration information.

Maybe the first work with a fair claim to consider polyphonic music in full generality is Boulanger-Lewandowski et al. (2012) . This paper proposed a coarse discrete temporal factorization of musical scores (for a discussion of this and other factorizations, see Section 4) and explored a variety of neural architectures several music datasets (including the Bach chorales). Many works on polyphonic models since 2012 have focused on the dataset and encoding introduced in Boulanger- Lewandowski et al. (2012) , notably Vohra et al. (2015) and Johnson (2017) . These works typically focus on quantitative log-likelihood improvements, and the degree to which these quantitative improvements correlate with quality is less clear.

For the Bach dataset, qualitative success is more definitive. Recently, concurrent work by Liang et al. (2017) and Hadjeres et al. (2017) proposed models of the Bach chorales with large-sample discrimination tests demonstrating the convincing quality of their results. In this line of work, quantitative results are lacking; Liang et al. (2017) learn a generative model and could in principle report cross entropies, although their work focuses on the qualitative study. The system proposed in Hadjeres et al. (2017) optimizes a pseudo-likelihood, so its losses cannot be easily compared to generative models. Quantitative metrics are reintroduced for the chorales in Huang et al. (2017) . Both the latter papers propose non-sequential Gibbs-sampling schemes for generation, in contrast to the ancestral sampler used by Liang et al. (2017) . Huang et al. (2017) make the case that a non-sequential sampling scheme is important for generating plausible compositions.

3. DATASET AND EVALUATION:

The models presented in this paper are trained on data from the KernScores library (Sapp, 2005) , a collection of early modern, classical, and romantic era digital scores assembled by musicologists and researchers associated with Stanford's CCARH.1 This dataset consists of over 2,300 scores containing approximately 2.8 million note labels. Tables 1 and 2 give a sense of the contents of the dataset.

Bach Beethoven Chopin Scarlatti:

Early Joplin Mozart Hummel Haydn 191, 374 476, 989 57, 096 58, 222 1, 325, 660 43, 707 269, 513 3, 389 392, 998 Table 1 : Notes in the KernScores dataset, partitioned by composer. The "Early" collection consists of Renaissance vocal music; a plurality of this collection is composed by Josquin.

We contrast this dataset and its Humdrum encoding with the MIDI encoded datasets used by most of the works we have discussed in this paper (a notable exception is Lavrenko & Pickens (2003) , who used data derived from the same KernScores collection consided here). MIDI is an operational format in the sense that it consists of a stream of instructions that describe a musical performance. Indeed it was designed as a protocol for communicating digital performances, rather than digital scores. For example, it cannot explicitly represent concepts such as quarter-notes or eighth-notes or rests, only notes of a certain duration or the absense of notes. Heuristics are necessary to display a MIDI file as a visual score and, if the MIDI wasn't prepared specifically for this purpose, these heuristics are liable to fail badly. For example, a MIDI file might encode "stacatto" articulations by shortening the length of certain notes; the heuristics to determine the value of a note (quarter-note, eighth-note, etc.) based on its length become exceedingly complicated in such cases. 2012) is too coarsely preprocessed to continue to serve this purpose. And the Bach chorales dataset is too small to sustain much further research. The KernScores collection considered here is readily available, reasonably large, and is structurally guaranteed have high quality.

3.1. EVALUATION:

Let p denote the unknown distribution over musical scores S, and let q be our model of p. We want to measure the quality of q by its cross-entropy to p. Because the entropy of a score grows with its length T , we will consider a cross-entropy rate.2 By convention, we measure time in units of beats, so our cross-entropy rate will have units of bits per beat. Defining cross-entropy for a continuous process generally requires some care. But in the case of music, we observe that notation occurs at rational points in time, and for rational durations. We can therefore quantize time using the finest denominator ∆ of times that appear in the dataset and define

EQUATION

)

This quantity is invariant to further refinement of the discretization. Suppose we quantize S at a rate ∆/2; then

q(S 0 , S ∆/2 , . . . , S 2T ∆/2 ) = 2T k=0 log q(S k |S 0 , . . . , S (k-1)∆/2 ) = 2T k even log q(S k |S 0 , . . . , S (k-1)∆/2 ) = q(S 0 , S ∆ , . . . , S T ∆ ).

The odd terms vanish under our assumption that ∆ was the finest denominator of notation in the dataset. We can think of ∆ as the resolution of the score process.

Observe that Definition 1 is independent of any choice about how we factor p: it is a cross entropy measure of the joint distribution. As we will discuss in Section 4, there are many ways to construct a generative model of scores. These choices lend themselves to different natural cross-entropies with their own units, depending on how we factor. By converting to units of bits per beat given by Definition 1, we can compare results under different factorizations.

4. FACTORING THE DISTRIBUTION OVER SCORES:

Polyphonic scores consist of features (notes and other notation) of variable length that overlap each other in quasi-continuous time. To factor the probability distribution over scores, we must somehow impose a sequential order upon the data. There is a loose partial order on scores implied by time but, in contrast to language, this order is not total. This slack admits many reasonable ways to factor the distribution over scores.

Most previous work factors a score by discretizing time. As we discussed in the previous section, there is a time resolution to all musical processes and we can discretize at this resolution without losing information. From this perspective, music looks like a large binary matrix of notes crossed with time; entry (t, n) in this matrix indicates whether note n is on at time t. 3 We can then generate music one slice of time at a time, generating a slice of time with a vector-valued prediction as in Boulanger-Lewandowski et al. (2012) or imposing an order (e.g. low to high) on notes and further factoring into binary predictions as in Liang et al. (2017) .

The above factorization, while popular, is computationally difficult for rhythmically complex music.

The process resolution ∆ required to discretize scores without loss of information is the common denominator of all rhythmic events in the corpus. A corpus that contains both triplets and thirtysecond notes, for example, would require a discretization of 48 positions per a beat. The datasets considered in Boulanger-Lewandowski et al. ( 2012) are discretized at either 1 or 2 positions per beat; as discussed by Huang et al. (2017) , this downsampling is quite destructive to the structure of music, more analogous to dropping words from a sentence than downsampling an image. Hardware capabilities may eventually overcome the computational obstruction to discretized score modeling, but we take a different approach in this paper that scales better with rhythmic complexity.

One alternative to discretizing time and predicting notes at each time step is instead to operationalize scores. From this perspective, a score becomes a long sequence of instructions: start playing C, start playing E, advance to the next time step, stop playing C, etc. We can think of this approach as a run-length encoding of the discrete factorization. It has been considered recently for the related task of modeling expressive performances by Oore et al. (2018) . A similar factorization was proposed by Walder (2016) , although that work does not implement a complete model. Operationalized runlength encodings greatly reduce the computational costs of learning and generation, at the expense of segmenting a score into highly non-linear segments. The number of items in the sequence between the beginning and the end of a note depends on how many other notes begin or end in the interim. Contrast this with the discrete factorization, for which every quarter-note (for example) lasts for exactly 1/∆ time slices.

In this paper, we adopt a factorization inspired by the operationalized perspectives. First, we decompose a polyphonic score into a collection of parts. We can loosely think of a part as the set of notes in a score assigned to a particular voice or instrument. 4 Each part is homophonic and therefore run-lengths in a part correspond to the duration of notes (in contrast to operationalized full scores, where run-lengths have no musical interpretation). We learn to predict the next homophony (note, chord, or rest) in a part, making a prediction in the part that has advanced the least far in time, and breaking ties between parts by an arbitrary (fixed) order that we impose on the parts in each score.

To predict the homophony, we impose an order pitches from low to high frequency, and make a sequence of binary predictions of whether each pitch is conditioned on lower-frequency pitches.

5. MODELS AND WEIGHT-SHARING:

Homophonic models Factoring the score as discussed in Section 4 allows us to think of the polyphonic composition problem as a collection of coupled homophonic composition problems. We therefore consider a homophonic composition task on the KernScores dataset's parts: this is a singlepart prediction task that generalizes classic monophonic prediction tasks to allow for chords. We explore a variety of fully connected, convolutional, and recurrent models for this task and find that the recurrent architecture works quite well. Table 5 summarizes these experiments; the third block of Experiments (15-21) compare the most interesting architectures over a reasonable amount of history.

The remaining question is: how do we encode the history of a polyphonic score, and how do we model correlations between parts in this history? We encode the history of a score as an order-3 binary tensor x ∈ {0, 1} T ×P ×(N +D) , indexed by a time axis of length T , a part axis of length P , and an (N + D)-dimensional feature axis consisting of N -dimensional multi-hot vector of notes, and a D-dimensional one-hot vector of durations. The note bits x t,p,n for n ∈ {0, . . . , N -1} indicate whether note n is on at time t in part p. The duration bits x t,p,d for d ∈ {N, N + D -1} indicate the duration of a homophonic event in part p beginning at time t or, if the previous state of part p continues at time t, a special continuation duration ' * ' is indicated. In this way, we interlace the events in each part between each other: see Figure 2 for a visual example of this encoding.

Our hope is that a recurrent network designed for the single-part task would be relatively unhampered when retrained on part data interspersed with these continuations; Experiment 22 in Table 5 suggests that this is the case; performance degrades slightly in comparison to Experiment 21, but this is to be expected because a length-10 history interspersed with continuations is effectively a slightly shorter history. This encourages us to consider the coupled recurrent models described below.

Polyphonic models We now consider the full polyphonic task; results for this task are summarized in Table 3 . We must now model correlations between parts in the history tensor, which we achieve coupling the representations of the individual parts. One natural extension of a recurrent neural network part model to multiple concurrent parts is a hierarchical architecture: a global network that integrates the states of the part (weights W hp ) with the previous global state (weights W h ) to build a coupled global state h t at time t. Because the order of the parts is arbitrary, we sum over their states before feeding them into the global network. At each time step, we can use the learned state of each part together with the global state to predict what follows (see Figure 1 ). Another natural extension of a recurrent part model is to directly integrate the state of the other parts' states into each individual part's state, resulting in a distributed state architecture:

EQUATION

EQUATION

)

EQUATION

We find that the distributed architecture underperforms the hierarchical architecture (see Table 3 ; Experiments 2 and 3) although this comparison is not conclusive: for example, the hierarchical model's implementation has more parameters than the distributed model. For the hierarchical model, we can also consider whether the global state representation is as sensitive to history-length as the parts. Could we make successful predictions using only the final state of each part, rather than coupling the states at each step? Experiments (4,5,6) in Table 3 suggest that this is not the case.

In the remainder of this section, we explore a variety of weight-sharing ideas that are somewhat orthogonal to our methods for factoring and modeling scores. These ideas may be of general interest for both monophonic and polyphonic composition, beyond the specific models under consideration.

5.1. AUTOREGRESSIVE MODELING:

To build a generative model over sequential data x = (x 1 , . . . , x t ), rather than directly model the distribution p(x), it often makes sense to factor the joint distribution into conditionals p(x t |x 1:t ) and make an autoregressive assumption p(x t |x 1:t ) = p(x s |x 1:s ) for all s, t ∈ N. We can then learn a single conditional distribution p(x t |x 1:t ) and share model parameters across all time translations. If the data is conditionally stationary (or nearly so) this approach is extremely effective (analogous to convolution for vision problems).

Scores are not quite conditionally stationary; their distribution varies substantially depending on the position within the beat. For example, the distribution has quite a lot of variance on the beat and new notes are frequently introduced. In contrast, notes almost never begin an ε-fraction of time after the beat and the distribution is quite peaked on the notes initiated in the previous time-step. To address this non-stationarity, we follow the lead of Johnson (2017) and Hadjeres et al. (2017) and augment our history tensor with a one-hot location feature vector ( in Table 5 ) that indicates the subdivision of the beat for which we are presently making predictions. 5 Compare the loss of duration models (Loss t ) with and without these features in Experiment pairs (3,4), (6,7), (10,11), (12,13), and (15,16)

5.2. PART DECOMPOSITION:

We have previously discussed decomposing a score into multiple parts. This presents us with an opportunity to share weights between part models by imposing the assumption p(x t,p |x t,1:p , x 1:t ) = p(x t,q |x t,1:q , x 1:t ) for all parts p, q. This corresponds to learning a single set of weights W p in equations ( 2) and ( 4), rather than learning unique part-indexed weights W pi for each part p i . Indeed, because the index of a part is arbitrary, the weights W pi should converge to the same values for all i; sharing a single set of weights W p accelerates learning by enforcing this property.

5.3. RELATIVE PITCH:

With a little effort, we can perform a similar weight-sharing scheme over the notes as we did over time and parts. This idea was first proposed in Johnson ( 2017). Recall that we factor the vector of pitches in part p at time t into a sequence of binary predictions, from lowest to highest pitch. Instead of building an individual predictor for each pitch conditioned on the notes in the history tensor, we can build a single predictor that conditions on a shifted version of the history tensor centered around the note we want to predict. By convolving this predictor over the pitch axis of the history tensor, we can make a prediction at each note location based on a relativized view of the history: see Figure 3 for a visualization of this transformation. Right: a relative pitch predictor, that learns a single classifier and translates the data along the frequency axis to center it around the pitch to be predicted. Whereas the absolute predictor decides whether C5 is on given the previous note was A4, the relative predictor decides whether the note under consideration is on given the previous note was 3 steps below it.

Like with the time axis, we observe that the distribution over notes is not quite conditionally stationary along the note-class axis. For example, a truly relative predictor would generate notes uniformly across the note-class axis, whereas the actual distribution of notes concentrates around middle C. Therefore we augment our history tensor with a one-hot pitch-class feature vector 1 n that indicates the note n for which we are presently making a prediction. This allows us to take full advantage of all available information when making a prediction, while borrowing strength from shared harmonic patterns in different keys or octaves. We compare absolute pitch-indexed classifiers (lin n ) to a single, relative pitch classifier (lin) in Table 5 : compare the loss of pitch models (Loss n ) in Experiments (2,3,4), (5,6,7), (8,9,10), (11, 12, 13) , and (15,16).

5.4. PITCH EMBEDDINGS:

Borrowing the concept of a word embedding from natural language processing, we consider learned embeddings of pitch vectors, denoted by c in Table 5 . For recurrent models, we do not observe substantial performance benefits to learning these embeddings: compare Experiments (20,21) in Table 5 . However, we do find that we can learn quite compact embeddings (16 dimensions for the experiments presented in this paper) without sacrificing performance, and working on these compact embeddings speeds up processing time for learning and generation. We also find that a simple 12 dimensional fixed embedding of pitches f, in which we quotient each pitch class by octave, reduces overfitting for the rhythmic model while preserving performance.

6. EXPERIMENTAL RESULTS:

Quantitative Results. The single-part (homophonic) and multi-part (polyphonic) prediction tasks are presented in tables 5 and 3 respectively. We advise caution in thinking about these results. Small differences in the log-loss can have large effects on the quality of output, especially if the differences are attributable to missing information. For example, failing to include a pitch-class feature vector in the relative models (described in Section 5.3) has a catastrophic impact on generated sequences, even though the log loss gap between is not always large. Detailed discussions of the individual results in these tables are presented in context in Section 5. 

# History:



Qualitative results:

We asked twenty study participants to listen to a variety of audio clips, synthesized from either a real composition or from the output of one of our models: Experiment 4 in Table 3 . For each clip, participants were asked to rate whether they thought the clip was written by a computer or by a human composer. Participants were presented with clips of varying length, from 10 frames of data (2-3 seconds; the length of the model's Markov window) to 50 frames of data (10 or more seconds). We expected that participant success would improve with the length of the clips, but we did not find this to be the case; indeed, even among the longest clips (around 20 seconds) participants occasionally identified an artificial clip as a human composition. Results are presented in Table 4 As with the quantitative results, we again urge caution in interpreting these qualitative results. Our study results superficially suggest that we have done well in modeling the short-term structure of our dataset (we make no claims to have captured long-term structure; indeed, the Markov windows of our models preclude this). But it is not clear that humans are good (or should be good) at the task of identifying plausible or implausible local structure in music. It is also unclear how to use such studies to compare between models, where differences would be less pronounced. Indeed, it is not even clear how to prompt a user to discriminate in such a setting.

7. CONCLUSION:

Given the difficulties evaluation generative models, it may be important to pay further attention to downstream tasks. One important downstream task is music transcription, which is considered together with polyphonic composition in Boulanger-Lewandowski et al. ( 2012) and more recently in Sigtia et al. (2016) . Both these systems operate on performance-aligned label sequences: a warping of a score to an expressive performance. More work would be necessary to generate an actual score that correctly identifies the value of notes (e.g. quarter note, or half-note) and not just the durations of notes in the audio.

The authors would also like to point out the oddity of training generative models on such a diverse set of music: from Josquin to Joplin. Music is inherently a low-resource learning problem; for comparison, modern language models are regularly trained on datasets larger than the entire classical music canon (Chelba et al., 2014) . Fortunately, music has a much lower entropy rate than language. But we may need new tools to learn properly to compose in the style of Mozart. 

# Params Model:



A PIANO MUSIC:

For piano music, we need to draw a distinction between an instrument and a part. Consider the piano score given in Figure 4 . This single piano part is more comparable to a complete score than the individual parts of, for example, a string quartet (compare the piano score in Figure 4 to the quartet score in Figure 2 ). Indeed, an educated musician would read this score in four distinct parts: a high sequence of quarter and eighth notes, two middle sequences of sixteenth notes, and a low sequence of quarter notes. In measure 12, the lowest two parts combine into a single bass line of sixteenth notes.

These part divisions are indicated in score through a combination of beams, slurs, and other visual queues. We do not model these visual indicators; instead we rely on part annotations provided by the KernScores dataset. The provision of these annotations is a strong point in favor of the KernScores dataset's Humdrum format; although in principle formats like MIDI can encode this information, in practice they typically collect all notes for a single instrument into a single track, or possibly two tracks (for the treble and bass staves, as seen in the figure) in the case of piano music.

In extremely rare cases, this distinction between instrument and part must also be made for stringed instruments; a notable example is Beethoven's string quartet number 14, in the fourth movement in measures 165 and 173, where the four instruments each separate into two distinct parts creating brief moments of 8-part harmony. The physical constraints of stringed instruments discourage more widespread use of these polyphonies. For vocal music, of course, physical constraints prevent intrainstrument polyphony entirely.

As Figure 4 illustrates, these more abstract parts can weave in and out of existence. Two parts can merge with each other; a single part can split in two; new parts can emerge spontaneously. The KernScores data provides annotations that describe this behavior. We can represent these dynamics of parts as a P × P flow matrix at each time step (where P is an upper bound on the number of parts; in our corpus P = 6) that describes where each part moves in the next step. At most time steps, this flow matrix is the identity matrix.

The state-based models discussed in this paper can easily be adjusted to accommodate these flows. If two parts merge, sum their states; if a part splits in two, duplicate its state. These operations amount to hitting the vector of state estimates for the parts with the flow matrix at each time step. However, we do not currently model the flow matrix. Because the flow matrix for piano music contains some (small) amount of entropy, we therefore exclude piano music from the results reported in Table 3 . We do however include the piano music in training. We would like to factor the distribution over scores, to take advantage of its autoregressive nature and other structure. We are immediately faced with the question of how to handle continuous time.

B.1 DISCRETE "RASTER" FACTORIZATION:

One approach to making predictions in a continuous domain is to discretize the domain. As discussed in Section 5, we can choose a discrete time step ∆ > 0 such that s i = k i ∆ and d i = j i ∆ with k ∈ N for all times s i and durations d i in all scores. In other words, we can discretize a score without losing information: we just need to specify its contents at each point on a discrete grid of resolution ∆. To this end we define a discretized score x ∈ {0, 1} T ×P ×2N where, at each time step t ∈ {1, . . . , T }, for each part p ∈ {1, . . . , P } and each note n ∈ {1, . . . , N }, we set

x t,p,n = 1 iff note n is on at time t in part p, x t,p,2n = 1 iff note n begins at time t in part p.

Observe that both a "note" bit and an "onset" bit are required to faithfully represent the score S; with only a single bit to indicate the occurrence of a note, both the scores in Figure 7 would be confused with the score in Figure 5 .

We can then factor the distribution over scores into a binary-valued conditional distributions:

p(x) = T t=1 p(x t |x 1:t ) = T t=1 P p=1 p(x t,p |x 1:t , x t,1:p ) = T t=1 P p=1 2N n=1 p(x t,p,n |x 1:t , x t,1:p , x t,p,1:n ).

Discretization is the approach taken by Boulanger-Lewandowski et al. ( 2012), with some caveats. First, their work doesn't model parts (this can be recovered in our exposition by setting P = 1). Second, their work doesn't take the final step of factoring p(x t,p |x 1:t , x t,1:p ) into binary-valued distributions; they choose rather to directly target the vector x t with NADE. Third, their work using a single bit to indicate notes, thus losing expressivity in the manner discussed in Figure 7 . Fourth, rather than work with the process resolution ∆, their work discretizes at a coarse resolution of eighth notes (quarter notes in the case of the JSB chorales segment of their corpus) thus losing substantial information. Figure 6 illustrates the effect of discretizing the score in Figure 5 at eighth-note resolution.

B.2 RUN-LENGTH FACTORIZATION:

Training and sampling from a model over a discrete factorization of scores at the process resolution ∆ can be expensive, prompting some earlier works to discretize at a coarser resolution (as discussed 2017) in which they are referred to as "Tie" bits).

in the previous section). One approach to preserve fine rhythmic structure (e.g. triplets and thirtysecond notes) without committing to a fine discretization is to factor a score into run-lengths. To this end, we define a run-length encoded score x ∈ N ⊕ {0, 1} P ×2N T where, at each time index t ∈ {1, . . . , T }, we set

x t,0 = 1 dt , where d t is the duration of the event at time index t, x t,1,p,n = 1 iff note n is on at time t in part p, x t,1,p,2n = 1 iff note n begins at time t in part p.

Crucially, the sequence x t is non-linear in the index t: entry x t+1 occurs d t beats after entry x t , in contrast to the raster where x t+1 always occurs a constant interval ∆ after x t .

We can then factor the distribution over scores into conditional distributions over binary note values and natural-number duration values:

p(S) = T t=1 p(x t |x 1:t ) = T t=1 p(x t,0 |x 1:t ) P p=1 2N n=1 p(x t,p,n |x 1:t , x t,0 , x t,1,1:p , x t,1,p,1:n ).

Because music typically doesn't evolve at the finest possible resolution ∆, we save a substantial amount of computation by predicting run-lengths x t,0 ∈ N rather than re-iterating the predictions x t,p,n at successive time steps.

One criticism of the run-length factorization is that, when notes of different durations overlap in a score, the longer notes are chopped up along the boundaries of the short notes as illustrated in Figure 8 . Instead of predicting musically meaningful quantities like note values (quarter, eighth, dotted-eighth, etc.) instead we predict run-length chunks.

B.3 OPERATIONALIZED FACTORIZATION:

Both the raster and run-length factorizations discussed above don't directly recognize the concept of note-values: quarter-notes, half-notes, eighth-notes, etc.. One approach to factorization that explicitly models note-values is to describe a score as a sequence of operations that produce the score, and then factor this sequence. Formally, we can describe an operationalized score as a sequence x ∈ (N ⊕ (N ⊕ {0, 1} P ×N )) L of length L, where each index x i is three-hot with the following semantics:

x i,0,d = 1

x i is a time-shift operation: time advances by d steps, x i,1,v = 1 and x i,2,n = 1

x i is a note operation: note n with note-value v begins playing.

This approach explicitly learns to predict note values v.

We remark that there are other operational factorizations that capture note values implicitly, like the raster or run-length encodings. Another factorization, similar to Oore et al. (2018) and inspired by the popular MIDI format, operationalizes a score as a sequence x ∈ (N⊕{0, 1} P ×N ⊕{0, 1} P ×N ) L of length L, where each index x i is one-hot with the following semantics:

x i,0,d = 1

x i is a time-shift operation: time advances by d steps, x i,1,p,n = 1

x i is a note-on operation: note n begins playing in part p, x i,2,p,n = 1

x i is a note-off operation: note n stops playing in part p.

In either case, we can then factor the distribution over these sequences:

p(S) = L i=1 p(x i |x 1:i ).

While the first operational approach describes above explicitly predicts note values, both operational factorizations require us to predict time-shifts, which are analogous to the durations predicting in the run-length factorization described in Appendix B.2. We will now turn our attention to a single-part setting where run-lengths are equivalent to note values, thus eliminating these strange intermediate quantities.

B.4 RUN-LENGTH FACTORIZATION FOR PARTS:

Observe in Figure 8 that if we were to run-length encode or operationalize the top and bottom staff individually, the run-lengths or time-shifts respectively would correspond to note-values. Because these parts are individually homophonic-notes within a part move in rhythmic lock-step-we do not observe the phenomenon where short notes overlap long notes in time, resulting in chopped up run-lengths. Applying the run-length factorization described in Appendix B.2 we see that, in the context of homophonic parts, run-lengths are equivalent to note-values and therefore the run-length factorization explicitly models these note values. The KernScores dataset takes this homophony property as the definition of a part (they call parts "voices") and therefore encode the score in Figure 5 as two parts; in cases of polyphony as in Figure 4 , the KernScores dataset decomposes the score into additional parts to maintain the homophony property.

Therefore, we can construct a run-length factorization of the KernScores parts that explicitly models note-values. We describe a part by x ∈ (N ⊕ {0, 1} N ) T where, at each time index t ∈ {1, . . . , T }, we set

x t,0 = 1 dt , where d t is the duration of the note (or chord) at time index t, x t,1,n = 1 iff note n is on at time t.

In contrast to the run-length encoding described in Appendix B.2, we no longer need a second bit to distinguish onsets from continuations; because notes don't get chopped up, we will never need a bit to indicating that we are re-iterating a note that was initiated in a previous run. We can then proceed to factor the part as:

p(x) = T t=1

p(x t,0 |x 1:t ) N n=1 p(x t,n |x 1:t , x t,0 , x t,1,1:n ).

Learning the conditional distributions of pitches p(x t,n |x 1:t , x t,0 , x t,1,1:n ) and note-values p(x t,0 |x 1:t ) is the homophonic modeling task considered in Section 5.

B.5 INTERLACING PARTS TO FACTOR A SCORE:

Let S be a score consisting of P parts. The run-lengths that comprise each part can be a totally ordered by the timestamp each run-length begins. We extend this to a total order across parts using these same timestamps, breaking ties (where run-lengths begin at the same time in multiple parts) using the arbitrary given by the indices of the parts. For an index i in this total order, let P i be the part associated with this index and T i be the time (by definition, T i is a non-decreasing sequence). Finally, let L p be the number of run-lengths in part p and define L ≡ P p=1 L p . For example, consider the score in Figure 5 . In this case L 0 = 6 (the treble part) and L 1 = 8 (the bass part). The time T and part P index sequences are given by T = [0, 0, .5, 1, 1, 1.5, 1.75, 2, 2, 2.5, 3, 3, 3.5, 3.75], P = [0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 1,

We describe a score by x ∈ (N ⊕ {0, 1} N ) L where, at each index x i , we set

x i,0 = 1 di , where d i is the duration of the note (or chord) at time T i in part P i , x i,1,n = 1 iff note n is on at time T i in part P i .

We then proceed to factor this distribution in the same manner as the homophonic parts model:

p(x) = L i=1 p(x i,0 |x 1:i ) N n=1

p(x i,n |x 1:i , x i,0 , x i,1,1:n ).

This factorization models the next run in part P(i) as we did in the homophonic model, but additionally condition on runs in other parts that were initiated prior to the run we are currently predicting (and also on runs initiated at the same time in parts with indices less than p). Note that we do not need to predict T and P; they are completely determined by the sequential ordering given on the x i 's. This is the polyphonic modeling task considered in Section 5.

:

http://kern.ccarh.org/

This is analogous to the adjustment of cross-entropy by sentence length used in language modeling.

We actually need two bits at each (t, n) entry to distinguish between repeated notes and held notes: see Appendix B, Figure7.

For polyphonic instruments like the piano, we must adopt a more refined definition of a part than "notes assigned to a particular instrument;" see Appendix A for details.

Note that this location can always be computed from the full history tensor. But in practice we will truncate the history, effectively imposing a Markov assumption on our models and losing this information.

Figures and Tables:

Figure 1: Coupled state estimation of Mozart's string quartet number 2 in D Major, K155, movement 1, from measure 1, rendered by the Verovio Humdrum Viewer. A representation (blue) of the state of each part is built at each time step, based on the previous state's representation and the current content of the part. A representation of the global state of the score is built from the previous global state and a sum (red) of the current states of each part. For each part, new notes (purple) are predicted using features of the global representation and the representation of the relevant part.

Figure 2: Left: a flattened description of the history tensor x t,p,n of features for a Haydn string quartet: opus 55 number 3, first movement, from measure 16. Parts are indicated by columns. A frame of time is indicated by a row. Each event in a part is denoted by a pair of duration and note(s), separated by a colon. Durations are denominated in beats. An asterisk indicated continuation of the previous note(s). Right: the score corresponding to the history tensor (in a score, the time and part axes are transposed).

Figure 3: Left: an absolute pitch predictor that learns individual classifiers for each pitch-class.Right: a relative pitch predictor, that learns a single classifier and translates the data along the frequency axis to center it around the pitch to be predicted. Whereas the absolute predictor decides whether C5 is on given the previous note was A4, the relative predictor decides whether the note under consideration is on given the previous note was 3 steps below it.

). The inclusion of location features discussed in Section 5.1 is indicated by . lin indicates the relative pitch log-linear classifier described in Section 5.3 and 1 n indicates the inclusion of pitch-class features. fc indicates a fully connected layer. f and c indicates the pitch embeddings described in Section 5.4. conv k indicates 1d convolution of width k. rnn indicates a recurrent layer. All hidden layers are parameterized with 300 nodes. Models were regularized with early stopping when necessary. The subscript k on the history tensor x k indicates the number of frames of history used in each experiment (either 1, 5, or 10 frames). The history tensor xk is modified to include continuation symbols '*' as it would in the polyphonic prediction task; see the discussion at the top of section 5.

Figure 4: Beethoven's piano sonata number 8 (Pathetique) movement 2, from measure 9, rendered by the Verovio Humdrum Viewer. Although visually rendered on two staves, this sonata consists of four parts: a high sequence of quarter and eighth notes, two middle sequences of sixteenth notes, and a low sequence of quarter notes.

Figure 5: Mozart's piano sonata number 8 in A minor, movement 1, from measure 1, rendered by the Verovio Humdrum Viewer.

Figure 6: A visualization of the score from Figure 5, discretized at eighth-note resolution.

Figure 7: Two scores with the same raster representation as the score in Figure 5, if only a single bit is used to indicate the occurrence of note n at time t. Other variants of these scores with the same raster encoding are also possible. The popular dataset introduced by Boulanger-Lewandowski et al. (2012) uses a single-bit raster. A second bit is used in some more recent work, for exampleLiang et al. (2017) in which they are referred to as "Tie" bits).

Figure 8: The Mozart from Figure5, with red lines that indicate the boundaries of events under a run-length factorization of the score. Notes in the treble staff are chopped up into eight-note runs, so instead of predicting note durations (quarter, dotted-eighth, sixteenth, etc.) we instead predict fragments of notes (eighth, continue eighth, continue eighth, etc.).

Notes in the KernScores dataset, partitioned by ensemble type.It is not impossible to construct a high-quality dataset of MIDI scores; but the Humdrum format, designed consciously by musicologists to encode scores, helps to ensure the quality of the data by enforcing constraints that are absent from MIDI. Polyphonic music needs a new benchmark dataset. AsHuang et al. (2017) point out, the dataset introduced by Boulanger-Lewandowski et al. (

Table content: <table/>

Multi-part results. The "hierarchical" architecture is defined by equations (2) and (3), and the "distributed" architecture is defined by equation (4): see the polyphonic models discussion in Section 5. Part and global history refer to the number of time steps used to construct the part states h p,t and global states h

Table content: <table><tr><td/><td>Architecture</td><td colspan="2">Loss Loss t</td><td>Loss n</td></tr><tr><td>(part/global)</td><td/><td colspan="3">(total) (time) (notes)</td></tr><tr><td>1 3 / 3</td><td>hierarchical</td><td>14.05</td><td>5.65</td><td>8.40</td></tr><tr><td>2 5 / 5</td><td>hierarchical</td><td>13.40</td><td>5.35</td><td>8.04</td></tr><tr><td>3 5</td><td>distributed</td><td>13.82</td><td>5.41</td><td>8.41</td></tr><tr><td>4 10 / 1</td><td>hierarchical</td><td>13.20</td><td>5.22</td><td>7.98</td></tr><tr><td>5 10 / 5</td><td>hierarchical</td><td>12.94</td><td>5.13</td><td>7.81</td></tr><tr><td>6 10 / 10</td><td>hierarchical</td><td>12.87</td><td>5.12</td><td>7.75</td></tr><tr><td>7 20 / 20</td><td>hierarchical</td><td>12.78</td><td>5.01</td><td>7.76</td></tr><tr><td>8 10</td><td>independent</td><td>18.63</td><td>6.56</td><td>12.08</td></tr></table>

. Qualitative evaluation of the 10-frame hierarchical model: Experiment 4 in Table3. Twenty participant were asked to judge 50 audio clips each of varying length. The scores indicate participants' average correct discriminations out of 10 (5.0 would indicate random guessing; 10.0 would indicate perfect discrimination).

Table content: <table><tr><td>Clip Length</td><td>10</td><td>20</td><td>30</td><td>40</td><td>50</td></tr><tr><td>Average</td><td colspan="5">5.78 6.89 6.61 6.72 6.78</td></tr></table>

Loss Loss t Loss n Single-part results. Loss is the cross-entropy described in Section 3.1. Loss t and Loss n are decompositions of the loss into component losses for duration ŷt and pitch ŷn predictions respectively. lin n indicates a log-linear classifier (sigmoid for ŷn and softmax for ŷt

Table content: <table><tr><td>1</td><td>112 ŷt = bias t , ŷn = bias n</td><td>10.07</td><td>2.92</td><td>7.15</td></tr><tr><td>2</td><td>21k ŷt = lin(x 1 ), ŷn = lin n (x 1 , y t , y 1:n )</td><td>8.05</td><td>2.00</td><td>6.05</td></tr><tr><td>3</td><td>9k ŷt = lin(x 1 ), ŷn = lin(x 1 , y t , y 1:n )</td><td>6.29</td><td>2.00</td><td>4.29</td></tr><tr><td>4</td><td>11k ŷt = lin(x 1 , ), ŷn = lin(x 1 , y t , y 1:n , 1 n )</td><td>6.12</td><td>1.83</td><td>4.29</td></tr><tr><td>5</td><td>149k ŷt = lin • fc(x 1 ), ŷn = lin n • fc(x 1 , y t , y 1:n )</td><td>5.92</td><td>1.99</td><td>3.93</td></tr><tr><td>6</td><td>135k ŷt = lin • fc(x 1 ), ŷn = lin • fc(x 1 , y t , y 1:n )</td><td>6.05</td><td>1.99</td><td>4.07</td></tr><tr><td>7</td><td colspan="2">172k ŷt = lin • fc(x 5.70</td><td>1.80</td><td>3.90</td></tr><tr><td>8</td><td>72k ŷt = lin(x 5 ), ŷn = lin n (x 5 , y t , y 1:n )</td><td>7.91</td><td>1.86</td><td>6.05</td></tr><tr><td>9</td><td>36k ŷt = lin(x 5 ), ŷn = lin(x 5 , y t , y 1:n )</td><td>5.76</td><td>1.86</td><td>3.91</td></tr><tr><td>10</td><td>38k ŷt = lin(x 5 , ), ŷn = lin(x 5 , y t , y 1:n , 1 n )</td><td>5.63</td><td>1.73</td><td>3.91</td></tr><tr><td>11</td><td colspan="2">418k ŷt = lin • fc(x 4.90</td><td>1.64</td><td>3.26</td></tr><tr><td>12</td><td colspan="2">497k ŷt = lin • fc(x 4.80</td><td>1.64</td><td>3.16</td></tr><tr><td>13</td><td colspan="2">535k ŷt = lin • fc(x 4.68</td><td>1.59</td><td>3.10</td></tr><tr><td>14</td><td colspan="2">228k ŷt 4.63</td><td>1.58</td><td>3.05</td></tr><tr><td>15</td><td>134k ŷt = lin(x 10 ), ŷn = lin n (x 10 , y t , y 1:n )</td><td>7.88</td><td>1.83</td><td>6.05</td></tr><tr><td>16</td><td>71k ŷt = lin(x 10 , ), ŷn = lin(x 10 , y t , y 1:n , 1 n )</td><td>5.53</td><td>1.71</td><td>3.83</td></tr><tr><td>17</td><td>372k ŷt = lin • fc(f(x 10 ), )</td><td/><td/><td/></tr><tr><td/><td>ŷn = lin • fc(c(x 10 ), y t , y 1:n , 1 n )</td><td>4.55</td><td>1.55</td><td>3.00</td></tr><tr><td>18</td><td>250k ŷt = lin • conv 5 (f(x 10 ), )</td><td/><td/><td/></tr><tr><td/><td>ŷn = lin • conv 5 (c(x 10 ), y t , y, 1 n )</td><td>4.56</td><td>1.55</td><td>3.01</td></tr><tr><td>19</td><td colspan="2">769k ŷt = lin • conv 4.42</td><td>1.50</td><td>2.92</td></tr><tr><td>20</td><td>342k ŷt = lin • rnn(x 10 , )</td><td/><td/><td/></tr><tr><td/><td>ŷn = lin • rnn(x 10 , y t , y 1:n , 1 n ))</td><td>4.37</td><td>1.48</td><td>2.89</td></tr><tr><td>21</td><td>283k ŷt = lin • rnn(f(x 10 ), )</td><td/><td/><td/></tr><tr><td/><td>ŷn = lin • rnn(c(x 10 ), y t , y 1:n , 1 n ))</td><td>4.36</td><td>1.48</td><td>2.88</td></tr><tr><td>22</td><td>301k ŷt = lin • rnn(f(x 10 ), )</td><td/><td/><td/></tr><tr><td/><td>ŷn = lin • rnn(c(x 10 ), y t , y 1:n , 1 n ))</td><td>4.52</td><td>1.59</td><td>2.93</td></tr></table>

Peer review (iclr2019):
 PROs -seemingly reasonable approach to polyphonic music generation: figuring out a way to splitting the parts, share parameters appropriately, measuring entropy per time, all make sense -the resulting outputs tend to have very short-term harmonic coherence (e.g. often a ‘standard chord’ with some resolving suspensions, etc), with individual parts often making very small stepwise motion (i.e. reasonable local voice leading) -extensive comparison of architectural variations -positive results from listening experiments CONs -musical outputs are *not* clearly better than some of the polyphonic systems described; despite the often small melodic steps, the individual lines are quite random sounding; this is perhaps a direct result of the short history -I do not hear the rhythmic complexity that is described in the introduction -the work by Johnson (2015) (ref. provided below) should be looked at and listened to; it too uses coupled networks, albeit in a different way but with a related motivation, and has rhythmic and polyphonic complexity and sounds quite good (better, in my opinion) -some unclear sections (fixable, especially with an appendix; more detail below) -despite the extensive architectural comparisons, I was not always clear about rationale behind certain choices, eg. if using recurrent nets, why not try LSTM or GRU? (more questions below) -would like to have heard the listening tests; or at least read more about how samples were selected (again, perhaps in an appendix and additional sample files) quality, clarity, originality and significance of this work, including a list of its pros and cons (max 200000 characters). Quality -- In this work, various good/reasonable choices are made. The quality of the actual output is fine. It is comparable to-- and to my ears not better than-- existing polyphonic systems such as the ones below (links to sample audio are provided here): -Bachbot - https://soundcloud.com/bachbot (Liang et al 2017) - tied parallel nets - http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/ (Johnson 2015, ref below) -performanceRNN - https://magenta.tensorflow.org/performance-rnn - (Simon & Oore 2017) ..others as well.. Clarity -- Some of the writing is "locally" clear, but one large, poorly-organized section makes the whole thing confusing (details below). It is very helpful that the authors subsequently added a comment with a link to some sample scores; without that, it had been utterly impossible to evaluate the quality. There are a few points that could be better clarified: -p5”a multi-hot vector of notes N”. It sounds like N will be used to denote note-numbers, but in fact it seems like N is the total number of notes, i.e. the length of the vector, right? What value of N is used? -p5 “a one-hot vector of durations D”. It sounds like D will be used to denote durations, but actually I think D is the length of the 1-hot vector encoding durations right? What value of D is used, and what durations do the elements of this vector represent? -similarly, does T represent the size of the history? This should really be clarified. -p5 Polyphonic models. -Eq (2), (3), (4): Presumably the h’s are the hidden activations layers? -the networks here correspond to the blue circles in Fig 1, right? If so, make the relationship clear and explicit -Note that most variables in most equations are left undefined -actually defining the W’s in Eq(2-4) would allow the authors to refer to the W’s later (e.g. in Section 5.2) when describing weight-sharing ideas. Otherwise, it’s all rather confusing. For example, the authors could write, “Thus, we can set W_p1 = W_p2 = W_p3 = W_p4” (or whatever is appropriate). -Generally, I found that pages 5-7 describe many ideas, and some of them are individually fairly clearly described, but it is not always clear when one idea is beginning, and one idea is ending, and which ideas can be combined or not. On my first readings, I thought that I was basically following it, until I got to Table 5, which then convinced me that I was in fact *not* quite following it. For example, I had been certain that all the networks described are recurrent (perhaps due to Fig1?), but then it turned out that many are in fact *not* recurrent, which made a lot more sense given the continual reference to the history and the length of the model’s Markov window etc. But the reader should not have had to deduce this. For example, one could write, “We will consider 3 types of architectures: convolutional, recurrent, .... In each architecture, we will have [...] modules, and we will try a variety of combinations of these modules. The modules/components are as follows:”. It’s a bit prosaic, but it can really help the reader. -Appendices, presented well, could be immensely helpful in clarifying the exact architectures; obviously not all 22 architectures from Table 5 need to be shown, but at least a few of them shown explicitly would help clarify. For example, in Fig1, the purple boxes seem to represent notes (according to the caption), but do they actually represent networks? If they really do represent notes, then how can “notes” receive inputs from both the part-networks and the global network? Also, I was not entirely clear on the relationship of the architecture of the individual nets (for the parts) to that of the global integrating network. E.g. for experiment #20, the part-net is an RNN (with how many layers?? with regular or LSTM cells?) followed by a log-linear predictor (with one hidden layer of 300 units right? or are there multiple layers sometimes?), but then what is the global network? Why does the longest part-history vector appear to have length 10 based on Table 5, but according to Table 3 the best-performing history length was 20? Though, I am not sure the meaning of the “bottom/top” column was explained anywhere, so maybe I am completely misunderstanding that aspect of the table? Etc. -Many piano scores do not easily deconstruct into clean 4-part polyphony; the example in Appendix A is an exception. It was not clear to me how piano scores were handled during training. -Terminology: it is not entirely clear to me why one section is entitled “homophonic models”, instead of just “monophonic models”. Homophonic music usually involves a melody line that is supported by other voices, i.e. a sort of asymmetry in the part-wise structure. Here, the outputs are quite the opposite of that: the voices are independent, they generally function well together harmonically, and there is usually no sense of one voice containing a melody. If there’s some reason to call it homophonic, that would be fine, but otherwise it doesn’t really serve to clarify anything. However, the authors do say that the homophonic composition tasks are a “minor generalization of classic monophonic composition tasks”, so this suggests to me that there is something here that I am not quite understanding. The last sentence of Section 5.3 is very confusing-- I don’t understand what lin_n is, or 1_n is, or how to read the corresponding entries of the table. The first part of the paragraph is fairly clear. Table 4: “The first row” actually seems like it is referring to the second row. I know what the authors mean, but it is unnecessarily confusing to refer to it in this way. One might as well refer to “the zeroth row” as listing the duration of the clip :) The experimental evaluation: I would like to hear some of the paired samples that were played for subjects. Were classical score excerpts chosen starting at random locations in the score, or at the beginning of the score? It is known that listening to a 10-second excerpt without context can sometimes not make sense. I would be curious to see the false positives versus the false negatives. Nevertheless, I certainly appreciate the authors’ warning to interpret the listening results with caution. Originality & Significance -- So far, based both on the techniques and the output, I am not entirely convinced of the originality or significance of this particular system. The authors refer to “rhythmically simple polyphonic scores” such as Bachbot, but I cannot see what is rhythmically fundamentally more sophisticated about the scores being generated by the present system. One nice characteristic of the present system is the true and audible independence of the voices. One of the contributions appears to be the construction of models that explicitly leverage with shared weights some of the patterns that occur in different “places” (pitch-wise and temporally) in music. This is both very reasonable, and also not an entirely novel idea; see for example the excellent work by Daniel Johnson, “Generating Polyphonic Music Using Tied Parallel Networks” (paper published 2017, first shared online, as far as I know, in 2015: links to all materials available at http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/ ) Another now common (and non exclusive) way to handle some of this is by augmenting the data with transposition. It seems that the authors are not doing this here. Why not? It usually helps. Another contribution appears to be the use of a per-time measure of loss. This is reasonable, and I believe others have done this as well. I certainly appreciated the explicit justification for it, however. Note that the idea of using a vector to indicate metric subdivision was also used in (Johnson 2015). Playing through some of the scores, it is clear that melodies themselves are often quite unusual (check user studies), but the voices do stay closely connected harmonically, which is what gives the system a certain aural coherence. I would be interested to hear (and look at) what is generated in two-part harmony, and even what is generated-- as a sort of baseline-- with just a single part. I encourage the authors to look at and listen to the work by Johnson: -listening samples: http://www.hexahedria.com/2015/08/03/composing-music-with-recurrent-neural-networks/ -associated publication: http://www.hexahedria.com/files/2017generatingpolyphonic.pdf Overall, I think that the problem of generating rhythmically and polyphonically complex music is a good one, the approaches seem to generally be reasonable, although they do not appear to be particularly novel, and the musical results are not particularly impressive. The architectural choices are not always clearly presented.