Transformer-NADE for Piano Performances
Curtis Hawthorne∗
fjord@google.com

Anna Huang∗annahuang@google.com

Daphne Ippolito
dei@google.com

Douglas Eck
deck@google.com

Abstract
Self-attention based Transformers are compelling sequence models because they
can capture relatively long-term dependencies by having direct access to the past.
However, their memory requirements grow quadratically with sequence length,
making it prohibitive to model long sequences with global attention. In contrast to
previous representations of music that “flatten” a note’s performance attributes suchas velocity, note on, and note off into a single sequence, we reduce sequence lengthby proposing a new representation, NoteTuple, which groups a note’s attributes asone event. This makes it natural to factorize a musical performance into a sequenceof notes and model a note as a NADE on its attributes. The resulting models require
fewer parameters and have faster generation. NoteTuples is a promising extension
to Music Transformer (Huang et al., 2018b), enabling rich downstream tasks suchas infilling of piano performances, e.g., Ippolito et al. (2018). Samples can be heard
at https://goo.gl/magenta/notetuple-examples.

1Introduction

In contrast to musical scores, MIDI recordings of piano performances include timing and dynamics.
To capture this expressiveness, Oore et al. (2018) proposed a Performance representation that serializes
all notes and their attributes into a sequence of events. This causes already lengthy musical pieces tobe on average 3-4x longer, thus requiring >10x memory when using a Transformer model (Vaswaniet al., 2017). Furthermore, a note’s attributes, such as note on and note off, may be far apart in thelist of events. To address these challenges, we propose a new representation, NoteTuple, whichgroups a note’s attributes as one event. Similar motivations are seen in image modeling. For example,PixelCNN++ (Salimans et al., 2017) conditions on whole pixels rather than R/G/B sub-pixels tosimplify the causal structure of the model. Our proposed representation makes it natural to factorize
a piano performance into a sequence of notes and then model a note as a NADE (Larochelle &Murray, 2011) on its attributes. Our approach is inspired by Boulanger-Lewandowski et al. (2012)’sRNN-NADE where 2D pianorolls are factorized as a sequence of discretized timesteps (columns) andat each timestep any number of the notes (rows) can be turned on, with its joint probability modeledby a NADE. This formulation enables rich downstream tasks such as chunk-wise infilling of pianoperformances (Ippolito et al., 2018), and potentially OrderlessNADE (Uria et al., 2014, 2016) styleinpainting such as in Coconet (Huang et al., 2017, 2018a).

2NoteTuple Representation

Encoding virtuosic piano performances at a temporal resolution of 10ms, a pianoroll requires asequence length of 100 to represent a second regardless of how many events happen, while thePerformance representation only uses a token when an event occurs. The latter uses four event types
to represent a single note: time shift to when it is played, velocity, pitched note on, and pitched note
off. It does not explicitly represent duration and instead relies on time shifts to a note-off event, which
can be far away in the sequence due to interleaving notes, as shown by the circled events in Figure 1.
∗Equal contribution.

32nd Conference on Neural Information Processing Systems (NIPS 2018), Montréal, Canada.

Input Music

Performance Representation[450, 62, 67, 300, 195, 260,386, 69, 300, 197, 260, 67,71, 345, 190, 195, 199]

Note Tuple Representation[(0, 0, 62, 94, 4, 29), (0, 0, 67, 94, 1, 4), (0, 50, 69, 30, 1, 4),(0, 50, 67, 30, 2, 9), (0, 0, 71, 30, 2, 9)]

Figure 1: In the performance representation, the tokens (circled) encoding the velocity, pitch, and end
time of the low D are far apart in the sequence. In the NoteTuple representation, all information forthe low D is encoded in the first tuple.
To address the challenges in existing representations, we created the NoteTuple representation, wherea note’s attributes are represented by a single tuple that contains the pitch, velocity, and duration of
the note, as well as a time offset from the previous note in the sequence. Rather than one large shared
vocabulary, each element in the note tuple has its own vocabulary. Since time shifts and duration canboth be potentially very long, we avoid large vocabulary sizes for these fields by separating theminto major and minor tick fields. Time shift has 13 major ticks and 77 minor ticks, representing 0through 10 seconds. Duration has 25 major tick values and 40 minor ticks. Therefore, each note tuple
contains six elements: time shift major ticks, time shift minor ticks, pitch, velocity, duration major
ticks, and duration minor ticks. Each element has a vocabulary size of 100 or less. In cases wheremultiple notes start on the same tick (e.g., chords), the note tuples are ordered from lowest pitch tohighest pitch. This can be seen in the first and second tuples in Figure 1, where the tuple for the D(pitch 62) whole note precedes the tuple for the G (pitch 67) quarter note.

3

Transformer-NADE

To save space, we refer the reader to Vaswani et al. (2017) on the Transformer architecture. We
replace their output softmax with a NADE to autoregressively predict a note’s attributes xi , each witha smaller softmax. Wi and c correspond to the hidden embedding weights and biases that projectprevious x<i into the hiddens hi , and Vi and bi the attribute-specific output weights and biases.
p(xi |x<i ) = softmax(Vi hi + bi )hi = sigm(W<i x<i + c)

4(1)
(2)

Experiments

We compare Transformer-NADE (using NoteTuple representation) to prior work on Music Transformer (using Performance representation), on the Piano-e-Competition dataset 2 . As the representations are different, losses (in NLL) are not directly comparable, 1.43 (ours) and 1.84 (prior work).
Comparing model sizes, ours need only 4 layers and 512 filters, in contrast to 6 layers and 2048
filters in prior work. For the latter, the attention mechanism operates directly on the performance
representation which is a lower level sub-note level representation, which may explain why themodel requires more filters. In our small listening test of 5 participants and a total of 25 pairwise
comparisons, our model was rated as more preferable in 5 pairwise comparisons and the prior work
as more preferable in the remaining 20. However, the differences were not always very noticeable, as
indicated by several comments from participants, including, “Very hard time comparing these two,and it could go either way."
In preliminary analysis, our model seems to struggle with note durations, causing consecutive notes
to overlap and resulting in undesirable dissonances. It performs better with relatively equal durations,and is therefore better at generating pieces that are more fugue-like rather than virtuosic. Note durationis challenging to model explicitly because it involves modeling the effects of the sustain pedal, which
extends the duration of all notes under the pedal until the release of the pedal. This requires learning
to compute an arithmetic so that all note offs align. In the previous Performance representation this iscircumvented by not explicitly representing duration and using consecutive note-off events to turn offa sustained chord at the same time. For future work, we propose using a discretized logistic mixture
likelihood (Salimans et al., 2017) to smooth over the note-off alignments. We invite readers to listen
to our current samples at https://goo.gl/magenta/notetuple-examples.
2Piano-e-Competition dataset (competition history tab): http://www.piano-e-competition.com/

2References
Nicolas Boulanger-Lewandowski, Yoshua Bengio, and Pascal Vincent. Modeling temporal dependencies inhigh-dimensional sequences: Application to polyphonic music generation and transcription. International
Conference on Machine Learning, 2012.
Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville, and Doug Eck. Counterpoint byconvolution. In International Conference on Music Information Retrieval, 2017.
Cheng-Zhi Anna Huang, Sherol Chen, Mark Nelson, and Douglas Eck. Towards mixed-initiative generation of multi-channel sequential structure. In Workshop track at International Conference on Learning
Representations, 2018a.
Cheng-Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, Noam Shazeer, Curtis Hawthorne, Andrew M Dai,Matthew D Hoffman, and Douglas Eck. Music Transformer. arXiv preprint arXiv:1809.04281, 2018b.
Daphne Ippolito, Anna Huang, Curtis Hawthorne, and Douglas Eck. Infilling music performance. In submission
to NIPS Workhop on Machine Learning for Creativity and Design, 2018.
Hugo Larochelle and Iain Murray. The neural autoregressive distribution estimator. In AISTATS, volume 1, pp.
2, 2011.
Sageev Oore, Ian Simon, Sander Dieleman, Douglas Eck, and Karen Simonyan. This time with feeling: Learning
expressive musical performance. arXiv preprint arXiv:1808.03715, 2018.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelcnn++: Improving the pixelcnn withdiscretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017.
Benigno Uria, Iain Murray, and Hugo Larochelle. A deep and tractable density estimator. In International
Conference on Machine Learning, pp. 467–475, 2014.
Benigno Uria, Marc-Alexandre Côté, Karol Gregor, Iain Murray, and Hugo Larochelle. Neural autoregressive
distribution estimation. arXiv preprint arXiv:1605.02226, 2016.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser,
and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp.
6000–6010, 2017.

3