Title: COUNTERPOINT BY CONVOLUTION

Abstract: Machine learning models of music typically break down the task of composition into a chronological process, composing a piece of music in a single pass from beginning to end. On the contrary, human composers write music in a nonlinear fashion, scribbling motifs here and there, often revisiting choices previously made. We explore the use of blocked Gibbs sampling as an analogue to the human approach, and introduce COCONET, a convolutional neural network in the NADE family of generative models (Uria et al., 2016) . Despite ostensibly sampling from the same distribution as the NADE ancestral sampling procedure, we find that a blocked Gibbs approach significantly improves sample quality. We provide evidence that this is due to some conditional distributions being poorly modeled. Moreover, we show that even the cheap approximate blocked Gibbs procedure from Yao et al. ( 2014) yields better samples than ancestral sampling. We demonstrate the versatility of our method on unconditioned polyphonic music generation.

:

Step 0

Step 1

Step 4

Step 16

Step 64 Ground Truth Step 0 shows the corrupted Bach chorale.

Step 64 shows the result.

1. INTRODUCTION:

Machine learning can be used to create compelling art. This was shown recently by Deep-Dream (Mordvintsev et al., 2015) , an optimization process that created psychedelic transformations of images. A similar idea underlies a variety of style transfer algorithms (Gatys et al., 2015) , which impose textures and colors from one image onto another. More recently, the multistyle pastiche generator (Dumoulin et al., 2016) exposes adjustable knobs that allow users of the system fine-grained control over style transfers. Neural doodle (Champandard, 2016) further closes the feedback loop between algorithm and artist.

We wish to bring similar artistic tools to the domain of music. Whereas previous work in music has relied mainly on sequence models such as Hidden Markov Models (HMMs, Baum & Petrie (1966) ) and Recurrent Neural Networks (RNNs, Rumelhart et al. (1988) ), we instead employ convolutional neural networks due to their emphasis on capturing local structure and their invariance properties. Moreover, convolutional neural networks have shown to be extremely versatile once trained, as shown by a variety of creative uses in the literature (Mordvintsev et al., 2015; Gatys et al., 2015; Almahairi et al., 2015; Lamb et al., 2016) .

We introduce COCONET, a deep convolutional model trained to reconstruct partial scores. Once trained, COCONET provides direct access to all conditionals of the form p(x i | x C ) where x C is a fragment of a musical score x and i / ∈ C is in its complement. Figure 1 shows an example of such conditionals used in completing a partial score.

COCONET is an instance of deep orderless NADE Uria et al. (2014) , and thus learns an ensemble of factorizations of the joint p(x). However, the sampling procedure for orderless NADE is not orderless. Sampling from an orderless NADE involves (randomly) choosing an ordering, and sampling ancestrally according to the chosen ordering. We have found that this produces poor results for the highly structured and complex domain of musical counterpoint.

Instead, we propose to use blocked-Gibbs sampling, essentially improving sample quality through rewriting. An instance of this was previously explored by Yao et al. (2014) who employed a NADE in the transition operator for a Markov Chain, yielding a Generative Stochastic Network (GSN). The transition consists of a corruption process that masks out a subset x ¬C of variables, followed by a process that independently resamples variables x i , i / ∈ C according to the distribution p θ (x i | x C ) emitted by the NADE. Crucially, the effects of independent sampling are amortized by annealing the probability with which variables are masked out. Whereas Yao et al. (2014) treat their procedure as a cheap approximation to ancestral sampling, we find that it produces superior samples.

We show the versatility of our method on unconditioned polyphonic music generation.

Section 2 discusses previous work in the area of automatic musical composition. The details of our model and training procedure are laid out in Section 3. In Section 4 we show that our approach is equivalent to that of deep and orderless NADE Uria et al. (2014) . We discuss sampling from our model in Section 5. Results of quantitative and qualitative evaluations are reported in Section 6. Finally, Section 7 concludes.

2. RELATED WORK:

Sequence models such as HMMs and RNNs are a natural choice for modeling music. However, one of the challenges in adapting such models to music is that music generally consists of multiple interdependent streams of events. This can be most clearly seen in the notion of counterpoint, which refers to the relationships between the movement of individual instruments in a musical work. Compare this to typical sequence domains such as speech and language, which involve modeling a single stream of events: a single speaker or a single stream of words.

Successful application of sequence models to music hence requires serializing or otherwise rerepresenting the music to fit the sequence paradigm. For instance, Liang (2016) serialize four-part Bach chorales by interleaving the parts, while Allan & Williams (2005) construct a chord vocabulary. Boulanger-Lewandowski et al. (2012) adopt a piano roll representation, which is a binary matrix x such that x it is hot if some instrument is playing pitch i at time t. To model the joint probability distribution of the multi-hot pitch vector x t , they employ a Restricted Boltzmann Ma-chine (RBM (Smolensky, 1986; Hinton et al., 2006)) or Neural Autoregressive Distribution Estimator (Uria et al., 2016) at each time step.

Moreover, the behavior of human composers does not fit the chronological mold assumed by previous authors. A human composer might start his work with a coarse chord progression and iteratively refine it, revisiting choices previously made. Sampling according to x t ∼ p(x t |x <t ), as is common, cannot account for the kinds of timeless dependencies that composers employ. Hadjeres et al. (2016) sidestep the choice of causal factorization and instead employ an undirected Markov model to learn pairwise relationships between neighboring notes up to a specified number of steps away in a score. Sampling involves Markov Chain Monte Carlo (MCMC) using the model as a Metropolis-Hastings (MH) objective. The model permits constraints on the state space to support tasks such as melody harmonization. However, the Markov assumption severely limits the expressivity of the model.

We opt instead for a convolutional approach that avoids many of these issues and naturally captures both relationships across time and interactions between instruments.

3. COUNTERPOINT BY CONVOLUTION:

We approach the task of music composition with a deep convolutional neural network (Krizhevsky et al., 2012) . This choice is motivated by the locality of contrapuntal rules and their near-invariance to translation, both in time and in the frequency spectrum.

We represent the music as a stack of piano rolls encoded in a binary three-tensor x ∈ {0, 1} I×T ×P . Here I denotes the number of instruments, T the number of time steps, P the number of pitches, and x i,t,p = 1 iff the ith instrument plays pitch p at time t. We will assume each instrument plays exactly one pitch at a time, that is, p x i,t,p = 1 for all i, t.

For the present work we will restrict ourselves to the study of four-part Bach chorales as used in prior work (Allan & Williams, 2005; Boulanger-Lewandowski et al., 2012; Goel et al., 2014; Liang, 2016; Hadjeres et al., 2016) . Hence we assume I = 4 throughout. We discretize pitch according to equal temperament, but constrain ourselves to only the range that appears in our training data (MIDI pitches 36 through 88). Time is discretized at the level of 16th notes for similar reasons. To curb memory requirements, we enforce T = 64 by randomly cropping the training examples.

Given a training example x ∼ p(x), we present the model with the values of only a strict subset of its elements x C = {x (i,t) | (i, t) ∈ C} and ask it to reconstruct its complement x ¬C . The input to the model is obtained by masking the piano rolls x to obtain the context x C and concatenating this with the corresponding mask:

h 0 i,t,p = 1 (i,t)∈C x i,t,p (1) h 0 I+i,t,p = 1 (i,t)∈C (2)

where the first dimension ranges over channels and the time and pitch dimensions are convolved over.

a l = BN(W l * h l-1 ; γ l , β l )

(3)

h l = ReLU(a l + h l-2 ) for 3 < l < L -1 and l = 0 mod 2 (4) h L = a L (5)

With the exception of the first and final layers, all of our convolutions preserve the size of the input. That is, we use "same" padding throughout and all activations h l , 1 < l < L have 128 channels.

The network consists of 64 layers with 3 × 3 filters on each layer. After each convolution we apply batch normalization Ioffe & Szegedy (2015) (denoted by BN(•)) with statistics tied across time and pitch. After every second convolution, we introduce a skip connection from the hidden state two levels below to reap the benefits of residual learning He et al. (2015) .

Finally, we obtain predictions for the pitch at each instrument/time pair:

EQUATION

The loss function is given by

EQUATION

= -

EQUATION

where p θ denotes the probability under the model with parameters

θ = W 1 , γ 1 , β 1 , . . . , W L-1 , γ L-1 , β L-1 .

To minimize the expected loss

EQUATION

with respect to θ, we sample piano rolls x from the training set and contexts C ∼ p(C) and optimize by stochastic gradient descent with step size determined by Adam (Kingma & Ba, 2014) .

4. RELATIONSHIP TO NADE:

Our approach is an instance of orderless and deep Neural Autoregressive Distribution Estimators (Uria et al., 2016) . NADE models a d-variate distribution p(x) through a factorization

EQUATION

)

where o is a permutation, and the parameters θ are shared among the conditionals. NADE can be trained for all orderings o simultaneously using the orderless NADE (Uria et al., 2014) 

EQUATION

Letting o <d = C, we obtain our loss from Equation 7L COCONET (x; C, θ) = -

EQUATION

For any one sample (x, C), this loss consists of |¬C| terms of the form log p θ (x i,t | x C , C). We let p(C) be uniform in the size of the mask |C| and reweight the sample losses according to

EQUATION

)

This correction, due to Uria et al. (2014) , ensures consistent estimation of the negative log-likelihood of the joint p θ (x).

However, we might wish to increase the difficulty by choosing p(C) so as to frequently mask out large contiguous regions, as otherwise the model might learn only superficial local relationships. This is discussed in Pathak et al. (2016) for the case of images, where a model might learn only that pixels are similar to their neighbors. Similar low-level relationships hold in our case, as our piano roll representation is binary and very sparse. For instance, if we mask out only a single sixteenth step in the middle of a long-held note, reconstructing the masked out step does not require any deep understanding of music. To this end we also consider choosing the context C by independent Bernoulli samples, such that each variable has a low probability of being included in the context.

5. SAMPLING:

We can sample from the model using the NADE ancestral ordering procedure. However, we find that this yields poor samples, and we propose instead to use Gibbs sampling.

5.1. NADE SAMPLING:

To sample according to NADE, we start with an empty (zero everywhere) piano roll x 0 and context C 0 and populate them iteratively by the following process. We feed the piano roll x s and context C s into the model to obtain a set of categorical distributions p θ (x i,t |x s C s , C s ) for (i, t) / ∈ C s . As the x i,t are not conditionally independent, we cannot simply sample from these distributions independently. However, if we sample from one of them, we can compute new conditional distributions for the others. Hence we randomly choose one (i, t) s+1 to sample from, and let x s+1 i,t equal the one-hot realization. Augment the context with C s+1 = C s ∪ (i, t) and repeat until the piano roll is populated. This procedure is easily generalized to tasks such as melody harmonization and partial score completion by starting with a nonempty piano roll.

Unfortunately, samples thus generated are of low quality, which we surmise is due to accumulation of errors. While the model provides conditionals p θ (x i,t |x C , C) for all (i, t) / ∈ C, some of these conditionals may be better modeled than others. We suspect in particular those conditionals used early on in the procedure, for which the context C consists of very few variables. Moreover, although the model is trained to be order-agnostic, different orderings invoke different distributions, which is another indication that some conditionals are poorly learned. We test this hypothesis in Section 6.2.

5.2. GIBBS SAMPLING:

To remedy this, we allow the model to revisit its choices: we repeatedly mask out some part of the piano roll and then repopulate it. This is a form of blocked Gibbs sampling (Liu, 1994) . Blocked sampling is crucial for mixing, as the high temporal resolution of our representation causes strong correlations between consecutive notes. For instance, without blocked sampling, it would take many steps to snap out of a long-held note. Similar observations hold for the Ising model from statistical mechanics, leading to the development of the Swendsen-Wang algorithm (Swendsen & Wang, 1987) in which large clusters of variables are resampled at once. We consider two strategies for resampling a given block of variables: ancestral sampling and independent sampling. Ancestral sampling invokes the orderless NADE sampling procedure described in Section 5.1 on the masked-out portion of the piano roll. Independent sampling simply treats the masked-out variables x ¬C as independent given the context x C .

Using independent blocked Gibbs to sample from a NADE model has been studied by Yao et al. (2014) , who propose to use an annealed masking probability given by

α n = max α min , α max - n ηN (α max -α min )

for some minimum and maximum probabilities α min , α max , number of Gibbs steps N and fraction η of time spent before settling onto the minimum probability α min . This scheme ensures the Gibbs process with independent resampling produces samples from the model distribution p θ (x). Initially, when the masking probability is high, the chain mixes fast but samples are poor due to independent sampling. As the masking probability reduces, fewer variables are sampled at a time, until finally variables are sampled one at a time and conditioned on all the others. Yao et al. (2014) treat independent blocked Gibbs as a cheap approximation to ancestral sampling. Indeed, per Gibbs step, independent sampling requires only a single model evaluation, whereas ancestral sampling requires as many model evaluations as there are variables to sample. Moreover, we find that independent blocked Gibbs sampling in fact yields better samples than the NADE procedure from Section 5.1. Samples can be heard here: https:// soundcloud.com/czhuang/sets/coconet-nade and https://soundcloud.com/ czhuang/sets/coconet-independent-gibbs.

6. EVALUATION:

We evaluate our approach on a corpus of four-part Bach chorales. The literature features many variants of this dataset (Allan & Williams, 2005; Boulanger-Lewandowski et al., 2012; Liang, 2016; Hadjeres et al., 2016) , and we follow the unfortunate tradition of introducing our own adaptation. Although this complicates comparisons against earlier work, we feel justified in doing so as our approach requires instruments to be separated, and other authors' eighth-note temporal resolution is too coarse to accurately convey counterpoint.

We rebuilt our dataset from the Bach chorale musicXML scores readily available through (Cuthbert & Ariza, 2010), which was also the basis for the dataset used in (Liang, 2016) . The scores included 357 four-part Bach chorales. We excluded scores that included note durations less than sixteenth notes, resulting in 354 pieces. These pieces were split into train/valid/test in 60/20/20% ratios.

We compare with Liang (2016) based on note-level likelihood and Boulanger-Lewandowski et al.

(2012) based on frame-level likelihood. Note that train/valid/test differs among both prior work and also with our work, and that Liang (2016) uses a 80/10/10% split instead.

However, evaluation of generative models is hard (Theis et al., 2015) . The gold standard for evaluation is qualitative comparison by humans, and we therefore report human evaluation results.

6.1. EVALUATING LOG-LIKELIHOOD:

To estimate the log-likelihood of a datapoint x, we follow the orderless NADE approach. That is, we sample a random ordering (i 1 , t 1 ), (i 2 , t 2 ), . . . (i IT , t IT ), and compute the notewise log-likelihood

EQUATION

where

C d = d c=1 {(i c , t c )}.

Note that we randomly crop each datapoint to be T time steps long before processing it, as this facilitates batch processing.

We repeat this procedure k times and average across all point estimates. The numbers for our models in Table 1 were obtained with k = 5.

The process for computing the notewise log-likelihood is akin to teacher-forcing, where at each step of the way the model observes the ground truth for all its previous predictions. To compute the framewise log-likelihood, we instead let the model run free within each frame t. This results in a more representative measure of the model's quality as it is sensitive to accumulation of error.

Table 1 lists notewise and framewise likelihoods of the validation data under variants of our model, as well as comparable results from other authors. We include four variants of COCONET that differ in the choice of the distribution p(C) over contexts during training. By importance sampling we refer to the orderless NADE strategy discussed in Section 4, in which p(C) is uniform over |C| and the sampled losses are reweighted by 1/|¬C|. We also evaluate three variants where the contexts are chosen by biased coin flips, that is, Pr((i, t) ∈ C) = ρ, for ρ ∈ 0.5, 0.25, 0.1. The framewise log-likelihood for ρ = 0.5 is listed as ∞ as its estimation repeatedly overflowed.

Overall, COCONET seems to underperform in terms of notewise likelihood, yet perform well in terms of framewise likelihood. Estimating the loss by importance sampling appears to work significantly better than determining the context using independent Bernoulli variables, as one might expect. However, the choice of Bernoulli probability ρ strongly affects the resulting loss, which suggests that some of the conditionals benefit from more training.

Table 2 : Mean (± SEM) negative log-likelihood under the model of unconditioned samples generated from the model by various procedures.

Sampling scheme:

Notewise NLL Framewise NLL Ancestral Gibbs, ρ = 0.00 (NADE) 0.565 ± 0.011 3.872 ± 0.052 Ancestral Gibbs, ρ = 0.01 0.560 ± 0.010 3.824 ± 0.052 Ancestral Gibbs, ρ = 0.25 0.444 ± 0.008 3.276 ± 0.036 Ancestral Gibbs, ρ = 0.50 0.438 ± 0.007 3.332 ± 0.040 Contiguous Gibbs, ρ = 0.50 0.447 ± 0.008 3.476 ± 0.048 Independent Gibbs (Yao et al., 2014) 0.440 ± 0.008 3.348 ± 0.040

6.2. SAMPLE QUALITY:

In Section 5 we conjectured that the low quality of NADE samples is due to poorly modeled conditionals p θ (x i,t | x C , C) where C is small. We test this hypothesis by evaluating the likelihood under the model of samples generated by the ancestral blocked Gibbs procedure with C chosen according to independent Bernoulli variables. When we set the inclusion probability ρ to 0, we obtain NADE. Increasing ρ increases the expected context size |C|, which should yield better samples if our hypothesis is true. The results shown in Table 2 confirm that this is the case. For these experiments, we used sample length T = 32 time steps and number of Gibbs steps N = 100.

Figure 2 shows the convergence behavior of the various Gibbs procedures, averaged over 100 runs. We see that for low values of ρ (small C), the chains hardly make progress beyond NADE in terms of likelihood. Higher values of ρ (large C) enable the model to bootstrap and reach significantly better likelihood. However, high values of ρ cause the chain to mix slowly, as can be seen in the case where ρ = 0.50. For each MTurk hit, users are asked to rate on a Likert scale which of two random samples they perceive as more musical. The study resulted in 192 ratings, where each source was involved in 92 pairwise comparisons. Figure 6 .3 reports for each source the number of times it was rated as more musical. We see that although ancestral sampling on NADE performs poorly compared to Bach, both ancestral and independent Gibbs Yao et al. (2014) were considered at least as musical as fragments from Bach, with independent Gibbs Yao et al. (2014) outperforming ancestral sampling (NADE) by a large margin. Pairwise comparisons are listed in Appendix A.

7. CONCLUSION:

We introduced a convolutional approach to modeling musical scores based on the NADE (Uria et al., 2016) framework. Our experiments show that the NADE ancestral sampling procedure yields poor samples for our domain, which we have argued is because some conditionals are not captured well by the model. We have shown that sample quality improves significantly when we use blocked Gibbs sampling to iteratively rewrite parts of the score. Moreover, annealed independent blocked Gibbs sampling as proposed by Yao et al. (2014) is not only faster but in fact produces better samples.

A PAIRWISE HUMAN EVALUATION RESULTS:

This appendix supplements Section 6.3 on the evaluation of samples by human subjects. 3 (a) shows the number of times the source corresponding to the row was preferred over the source corresponding to the column. Table 3 (b) shows the number of ties. Table 3 (c) shows the number of losses and is the transpose of Table 3 (a). Source legend: I denotes Independent Gibbs (Yao et al., 2014) , C denotes Contiguous Gibbs, N denotes NADE and B denotes Bach.



Figures and Tables:

Figure 1: Ancestral inpainting of a corrupted Bach chorale by COCONET. Colors are used to distinguish the four voices. Grayscale heatmaps show predictions p(x i | x C ). The pitch sampled in the current step is indicated by a rectangular outline. The original Bach chorale is shown in the bottom right. Step 0 shows the corrupted Bach chorale. Step 64 shows the result.

Figure 3: Human evaluations from MTurk on comparing sampling schemes.

Figure 3: Pairwise human evaluation results. Each element of Table3(a)  shows the number of times the source corresponding to the row was preferred over the source corresponding to the column. Table3(b) shows the number of ties. Table3(c) shows the number of losses and is the transpose of Table3(a). Source legend: I denotes Independent Gibbs(Yao et al., 2014), C denotes Contiguous Gibbs, N denotes NADE and B denotes Bach.

Negative log-likelihood on the test set for the Bach corpus. As discussed in the text, our numbers are not directly comparable to those of other authors due to the use of different splits. Results fromBoulanger-Lewandowski et al. (2012) were based on an eighth-note temporal resolution (our resolution is sixteenth notes). Please note that our results are preliminary validation likelihoods.

Table content: <table><tr><td>Model</td><td colspan="2">Notewise NLL Framewise NLL</td></tr><tr><td>Bachbot (Liang, 2016)</td><td>0.477</td><td>-</td></tr><tr><td>NADE (Boulanger-Lewandowski et al., 2012)</td><td>-</td><td>7.19</td></tr><tr><td>RNN-RBM (Boulanger-Lewandowski et al., 2012)</td><td>-</td><td>6.27</td></tr><tr><td>RNN-NADE (Boulanger-Lewandowski et al., 2012)</td><td>-</td><td>5.56</td></tr><tr><td>COCONET, i.i.d Bernoulli(0.50)</td><td>0.924</td><td>∞</td></tr><tr><td>COCONET, i.i.d Bernoulli(0.25)</td><td>0.655</td><td>4.48</td></tr><tr><td>COCONET, i.i.d Bernoulli(0.10)</td><td>0.812</td><td>4.66</td></tr><tr><td>COCONET, importance sampling</td><td>0.569</td><td>3.73</td></tr></table>

Peer review (iclr2017):
The paper tackles the task of music generation. They use an orderless NADE model for the task of *fill in the notes*. Given a roll of T timesteps of pitches, they randomly mask out some pitches, and the model is trained to predict the missing notes. This follows how the orderless NADE model can be trained. During sampling, one normally follows an ancestral sampling procedure. For this, an ordering is defined over outputs, and one runs the model on the current input, samples one of the outputs according to the order, adds this output to the next input, and continues this procedure until all outputs have been sampled. The key point of the paper is that this is a bad sampling strategy. Instead, they suggest the strategy of Yao et al. 2014, which uses a blocked Gibbs sampling approach. The blocked Gibbs strategy instead masks N inputs randomly and independently, samples them, and repeats this procedure. The point of this strategy is the make sure the sampling chain mixes well, which will happen for large N. However, since the samples are independent, having a large N gives incoherent samples. Thus, the authors follow an annealed schedule for N, making it smaller over time, which will eventually reduce to ancestral sampling (giving global structure to the sample). They conduct a variety of experiments involving both normal metrics and human evaluations, and find that this blocked Gibbs sampling outperforms other sampling procedures. This is a well written paper - great job. My main problem with the paper is that having read Uria and Yao, I don*t know how much I have learned from this work in the context of this being an ICLR submission. If this was submitted to some computational music / art conference, this paper would be a clear accept. However, for ICLR, I don*t see enough novelty compared with previous works this builds upon. Orderless NADE is an established model. The blocked Gibbs sampling and annealing scheme are basically the exact same one used in Yao. Thus, the main novelty of this paper is its application to the music domain, and finding that Yao*s method works better for sampling music. This is a good contribution, but more tailored to those working in the music domain. If the authors found that these results also hold for other domains like images (e.g. on CIFAR / tiny Imagenet) and text (e.g. document generation), then I would change my mind and accept this paper for ICLR. Even just trying musical domains other than Bach chorales would be useful. However, as it stands, the experiments are not convincing enough.