Anticipatory Music Transformer
Authors: John Thickstun
David Leo Wright Hall
Chris Donahue
Percy Liang
Venue: Accepted by TMLR ()

Abstract: We introduce anticipation: a method for constructing a controllable generative model of a temporal point process (the event process) conditioned asynchronously on realizations of a second, correlated process (the control process). We achieve this by interleaving sequences of events and controls, such that controls appear following stopping times in the event sequence. This work is motivated by problems arising in the control of symbolic music generation. We focus on infilling control tasks, whereby the controls are a subset of the events themselves, and conditional generation completes a sequence of events given the fixed control events. We train anticipatory infilling models using the large and diverse Lakh MIDI music dataset. These models match the performance of autoregressive models for prompted generation, with the additional capability to perform infilling control tasks, including accompaniment. Human evaluators report that an anticipatory model produces accompaniments with similar musicality to even music composed by humans over a 20-second clip.

Published in Transactions on Machine Learning Research (04/2024)
Anticipatory Music Transformer
John Thickstun
Phhickssun@@cssttar//pa edu
Department of Computer Science
Stanford University
David Hall
dlwh@stanfort edu
Deportment of Computer Science
Stanford University
Chris Donahue
chrisdonahuue@omuu.dd
Google DeepMind and Carnegic Mellon University
Percy Liang
pliang@cs.stanve edu
Department of Computer Science
Stanford University
Reviewed on OpenReview: https: // // openrevie.. pormfo id- EBNJ33Fer1
Abstract
We introduce anticipation: method for constructing controllable generative model of a temporal point process (the event process) conditioned asynchronously on realizations a second, correlated process (the control process). We achieve this by interleaving sequences of events and controls, such that controls appear following stopping times in the event sequence. This work is motivated by problems arising in the control of symbolic music generation. We focus infilling control tasks, whereby the controls and conditional generation completes sequence of events given the fixed control events. We train anticipatory infilling models using the large and diverse Lakh MIDI music dataset. These models match the performance autoregressive models for prompted generation, with the additional capability to perform infilling control tasks, including accompaniment. Human evaluators report that an anticipatory model produces accompaniments with similar musicality to even music composed by humans over 20-second clip.
subset of the events themselves,
1 Introduction
Imagine you are given melody, and asked to compose a harmonizing accompaniment. This melody can be modeled by a temporal point process: a probability distribution over (musical) events that arrive stochastically at points in time. An accompaniment to this melody is also rcalization of temporal point process. The events in the accompaniment are tightly correlated but often asynchronous with the events in the melody. Generating accompaniment to given melody is example of a control task: seek the ability to generate an accompaniment (the events) conditioned on given melody (the controls). Models that generate symbolic music (i.e., compose) subject to user-specified controls are of broad interest tools for music co- creation (Louie et. al., 2020).
Motivated by this example, (the event process) that can be conditioned on realizations of second, correlated point process (the control process). Generating an accompaninent to a melody is an instance of a more general infilling lask, whereby we generate a complete realization of a temporal point process given partial observation a subset of its events. Infilling a powerful control mochanism for music generation: previous work on musical infill- ing (Huang al., 2017) powered the J.S. Bach Google Doodle (Huang al., 2019), an interactive music experience with broad popular appeal.
interested in constructing generative models of a temporal point process

---

Published in Transactions on Machine Learning Research (04/2024)
The dynamics of temporal point process can be captured by neural autoregressive model trained to predict the next event in a time-ordered sequence (Du et al., 2016). A natural extension of this paradigm to conditional distributions is sequence- -oreeueee----euuencee modeling (Seq2Seq) (Sutskever et al., 2014), whereby the control sequence is prepended to the sequence events. For long sequences, Seq2Seq places time-localized controls lar from the events they describe. While there substantial recent work on long-contexl modeling (Child et al., 2019; Dao et al., 2022; Gu et al., 2022; Hawthorne al., 2022), rather than brute-force the learning of artificial long-range dependencies, we propose to structure conditional generation so that control time t is located close to events near time Our premise is that the most relevant context for predicting the next event is the recent event. history (unidirectional context) and both recent and near-future controls (bidirectional context).
Standard practice to efficiently train an autoregressive model relies on the observation that context for prediction at one index in the sequence a prefix of the context for predictions at future indices. This allows us to update the model based on - predictions for each sequence of length M. Conditioning on asynchronous controls by constructing an ad-hoc context (e.g., the M/2 previous events and M/2 nearest controls) to predict each event would be computationally wasteful: interleaved sequence of events and controls. This is straightforward if the events and controls are synchronous: to condition on bidirectional control context [t - 5, 6] at time t, simply shift the control sequence by a time offset 5 and construct a joint sequence by interleaving events and controls at alternating sequence positions. Or alternatively, construct an encoder-decoder model that ingests the paired control tokens through a separate encoder (Raffel et al., 2020).
want to define single, coherent
When events and controls are asynchronous, simple approaches to interleaving these sequences make sampling from the ensuing joint model intractable. This includes the natural sort order, whereby we interleave control on time t as if it were at time t - 8. For tractable sampling, we will see in Section 3 that the index in the interleaved sequence that immediately precedes a control must be a stopping time (Billingsley, 1995). We propose a mothod for interleaving asynchronous events and controls such that a control on time t appears in the interleaved sequence al a stopping time close to events near time I d. d. We call this method anlicipation. The interval 5 is hyperparameter chosen be long enough to give the model time to account for (i.e. anticipate) upcoming controls, but short enough to maintain proximity of events and controls (if 8 = co, we recover Seq2Seq modeling). The interleaved structure of anticipation is visualized in Figure 1.
Contributions. We define arrival-time encoding of events and controls that is amenable to expres- sive auloregressive sequence modeling and facilitates anticipation (Section 2). We describe the interleaved structure of an anticipatory autoregressive model, and how to train and sample from this model (Section 3). We apply anticipation to construct anticipatory infilling models for music, trained on the Lakh MIDI music dataset (Raffel, 2016). These models unlock new control capabilities for music generation without sacrificing the performance of unconditional generation (Section 4). We release all code for reproducing these models, along with pre-trained model weights. 1
2 Music as a Temporal Point Process
A marked temporal point process is a probability distribution over sparse events situated at points in continuous time (Daley & Vere-Jones, 2007).
Definition 2.1. A marked temporal point process is a probability distribution over events e; = (4,, m;), where 4 E R (4 S Ly if j) a point in time and m, E a mark from a finite vocabulary V.
Given controls un: provided by a user, we say that we can control generation of the events e1N with respect to ui:k if we can sample from ((l::)) We focus on infilling control, whereby the controls u1:K share a vocabulary with the events CI.N- Given a user-specified set of K events ui:K, we would like to generate a complete realization of the process e1.V such that uI:k S C1:N- This generalizes the span- infilling task-whie) asks to generate missing contiguous span of events previously studied in the music literature (Ippolito et al., 2018; Chang et al., 2021; Tan et al., 2022a;b).
1 For assets and supplemental material, see: https://jjontte wwstun..ccktcccc. centttuunnccm/atttteeeeee www..ntiiiiaaaiii.n.....m
2

---

Published in Transactions on Machine Learning Research (04/2024)
Next Event Prediction: p(e Model Context)
Controls
U
u
Up434k+1
1k45
+
Time
Events
ej
e
0/+2
C
ej+7
Time - 6
Sk +
86+1
8k-2
8k- 3 S6+-
8k+5
Time
Events
e--1
ej
0jt1
0j42
ej-3
e ej45
0jfe
Serialized Sequence Structure ej e,41, e,22 ejt3, e/44, eyt5, 0546, 0j47,
Unconditional:
e1,
Seq2Seq:
u1, Ug Uk+5 e1, ej e,41, e,z2, ejs3, e/44, e,45, ej+6, ejt7,
Sort Order:
e1, .... Us Ug+1, Cj U 1 ej, eit3, Us+3, U6+4 @j,4 ejts, Up45, 0446 ej+7,
Anticipation:
e,, ... ej UA, U Bj+1, 2 2jf2, 0j+3, 0j44, 3 Up+4, 0jtō, Gj4c, 5k-5 0j+7,
We construct generative models for sequences of events e1:,, conditioned on controls U1:K- We scrialize these paired sequences to define an autorogrossive factorization of the joint distribution over events and controls. Anticipation interleaves event and control sequences so that a control uı on time 8 appears in the recent history when predicting events near time Sk. An anticipated control ug on time 8k appears as if it were at approximately time s/2 = 8 - 8. For example, when prodicting Cj+7 above, the recent context of the anticipation sequence contains contains past events and controls, as well as the future control Ug+5i we say that model predicting 0j+7 given this context anticipates the control Uk-5, approximately 5 seconds in advance. Crucially, to be able to condition on controls, the index that immediately preceeds each control in the serialized sequence must be stopping time, property that naively interleaving events and controls using the sort order of times s does not satisfy.
Figure 1:
Music can be described by a marked temporal point process, where the vocabulary of marks consists of musical notes and other musical events. In this work, we restrict the vocabulary of musical events to notes consisting of pitch p, instrument class k, and duralion d. Following the MIDI standard (International MIDI Association, 1988), we quantize pitch p {O..... 127} according to the 12-tone Western scale (p 60 denotes "middle C", or 261Hz); we represent instrument classes k E {0..... 128} using standard MIDI instrument codes, including drums (code 128); we represent duration in units of seconds, quantized to 10ms intervals with a maximum duration of 10 seconds. We represent combined pitch and instrument information using a single value n = 128k + p. mark thus a note m, = (d,, n:) V from a vocabulary of 17,512 marks, and music consists of these notes situated points in time.
Older work music generation typically rasterizes time, encoding music uniformly sampled matrix tensor, i.e., piano-roll SBoulangerLLewandooskk et al., 2012; Dong et al., 2018). In the piano-roll setting, there are strong solutions to accompaniment and infilling tasks that avoid the complexities of asynchronicity that we address in this paper (Hadjeres et al., 2017; Huang et al., 2017; Pati et al., 2019; Chen et al., 2020). However, for diverse and rhythmically intricate music data (including the Lakh MIDI datasct) piano-roll rasterization comes at a cost: the rasterization is coarse then rhythmic detail is lost, and the rasterization is fine then the piano-rolls are hign-dimensional, presenting computational challenges. We instead chose to model music as temporal point process, an approach inspired by Huang et al. (2018); Oore et al. (2020).
Modeling Temporal Point Processes. Much of the machine learning literature on temporal point processes focuses on modeling conditional intensity functions (Du et al., 2016; Mei & Eisner, 2017; Omi et al., 2019; Zuo el al., 2020). In contrast, we will model the probability distribution over the next event in a time-ordered sequence. In this regard, our work is most comparable to Shchur et al. (2020); but whereas that work models continuous densitics, we model discrete distributions over quantized time values. This allows us to directly apply the full modern machinery of causal autoregressive transformers and large language models to modeling point processes.
3

---

Published in Transactions on Machine Learning Research (04/2024)
Modeling Arrival Times. Stochastic arrival times t, are the defining characteristic of a temporal point process. In Section 2.1 we propose arrival-time lokenization that represents realization of a marked point process as a sequence of (discretized) arrival times and marks. In Section 3 we exploit an invariance of this sequence representation (context-free subsoquences) to create anticipation. We also consider an interorrival- lime lokenizalion that is comparable lo encodings used in recent work on music generation (Huang et al., 2018; Donahue et al., 2019; Payne, 2019; Oore et al., 2020). Both of these encodings reduce point process modeling to sequence density estimation. However, only the proposed arrival-time encoding facilitates the construction of anticipation.
2.1 Encoding Music as Sequences
We represent arrival times t using finite vocabulary of 10,000 values, quantizing time in 10ms intervals (the quantization proposed by Music Transformer Huang et al. (2018)) with maximum time of 100 seconds. While many musical performances of the model context. The models described in this paper have context length M = 1024. Differences between M successive arrival limes that exceed 100 seconds appear in less than 0.1% of Lakh MIDI training examples; we discard these examples during preprocessing. By quantizing time we reduce realizations of temporal point processes discrete soquences composed of successive pairs of times t, quantize(t,) and marks m, or, in the music application, triplets of time, duration, and note (t,, n.).))
longer than 100 seconds,
relativize arrival times to the beginning
Definition 2.2. The arrival-time tokenization events C1:N is a sequence X1aN defined by
X3i- tip
X3 d,
X3 n..
(1)
The vocabulary has size |V| 27,512: 10,000 quantized times, 1000 quantized durations, and 16,512 instrument-picch pairs.
Crucially, the triplets (X3--2, 100-,,X33:11 X3 of these sequences are contert-free: we re-order the triplets, the semantics of the sequence do not change. We can recover the original ordering by sorting these triplets according to their arrival times X3:-2- We exploit the re-orderability of arrival time tokenized sequences to construct anticipatory autoregressive infilling models in Section 3. A similar encoding has been used by Gardner et al. (2021), but their work does not exploit this ro-orderability.
The more common practice in symbolic music modeling encodes music as a sequence onset and offset events, separated by interarrival times A IR.. For each event e = (t, m), we define onset on = (t, n) and offsct off = (t + n). We bound the interarrival time between events by 10 seconds, corresponding to a vocabulary of 1000 possible (10ms quantized) interarrival times.
Definition 2.3. Given events eI,,, let X1.2N denote an interleaved sequence of onsets on and offsets offiNN, ordered by time, where values x = (t% n²) have interarrival times A; t t. The interarrivll time tokenization of events e1:N is a sequence X1MN defined by
X
X2 A,
(2)
Following standard practice, we omit interarrival tokens when A, = O. The vocabulary has size | 16,512 note onsets, 16,512 note offsets, and 1000 quantized interarrival times.
34,024:
In contrast to Definition 2.2, the interarrival-time tokenization described by Definition 2.3 and other com- mon music encodings including including REMI (Huang & Yang, 2020) and OctupleMidi (Zeng et al., 2021) are contert sensilive: the limings of lokens are determined contextually by their positions in the sequence. Interarrival-time tokenization may appear less compact than arrival-time tokenization: each event is encoded as an onset and an offset-cach with an associated interarrival time-totaling 4 tokens, versus a triplet of arrival time, duration, and note totaling 3 tokens. However, omitting zero-duration interrarival times results in sequences of comparable length under either encoding of the Lakh MIDI dataset.
4

---

Published in Transactions on Machine Learning Research (04/2024)
3 Anticipation
Given an anticipation interval 5 > O, we construct sequence a1:N+K = interIeaves(e1.)) u1:k) that interleaves events e1.N and controls ui:K such that à control ug on time appears close to events near time sk o;; i.e, we anticipate ug seconds in advance. We can then construct a standard autoregressive sequence model over interleaved sequences a1:N+K- Model predictions under the anticipatory ordering a1:N+K combine filtering (i.e., causal) estimate based on the local history of events with a smoothing (i.e., bidirectional) estimate based on local controls (Wiener et al., 1949). The value of is hyperparameter that cont the degree of smoothing. If 8 is too short, then the model will be blind to upcoming control context. If 8 is too long, then ticallocality of the sequence a1:N+K is broken.
The natural sort order that interleaves events and controls by merging a control on time 8k between events at times t,,1 such that tÚ S 5k access to prefixes of aI,NKK and so the criterion that determines whether to condition on u2 at index must be a function of the history a1:--1- However, the sort order placement of control ug depends on both the event that precedes it and the event that follows it in the merged sequence, which requires information that is unavailable at inference time. We formalize this problem using the concept of stopping time (Definition 3.3). Definition 3.1 describes a version of anticipation that interleaves controls uy at indices Tup E (1,...,NN K} after a stopping time Tu: 1.
tj+1 makes inference intractable. During inference, we only have
Definition 3.1. (Anticipation) Let C1:N be events with vocabulary and let ui:K be controls with vocab- ulary U. Given > 0, we define a combined sequence a = interleave,(e1.N; ui:k) with vocabulary u U V that interleaves these two sequences. We write fy indicate the arrival times of event ey and control u, respectively. To simplify notation define to 80 -co In the combined sequence a1:N+K: event ey and control up appear (respectively) at indices
Te, =j j + arg max max 11 88 - 5),
(3)
OSkSK
Jup = k + arg min (t, 8h 0}.
(4)
OSiSN
Unpacking this definition, Equation (3) says that event ej appears sequence a after the first events (term) j in the sum) and after any controls that appear earlier in sequence a1:N+K (the arg max term, mirroring Equation (4)). Equation (4) says that control uı appears after the first k --1 1 controls and after the first event ej exceeding time 8& - 6. The indices 7 of events and controls in the sequence a are random variables, determined by stochastic realizations e1:w and ul:k-
An anticipatory autoregressive model is an autoregressive model defined over sequences of events and controls a1:N+K interleaved according to Equations (3) and (4):
N+K II im1
p(aLN4K)
p(a.aan=1))
(5)
Example 3.2. Suppose f = 1, 12 = 3, t3 = 5, 81 = 7, and 5. Then 81 -
and
Tej = 1 + arg max 00 2 8 - =1,
(6)
Tea = 2 + arg max (1 > s 8) 2,
(7)
Teg = 3 + arg max (3 2 88 8) = 4,
(8)
Tuy = + arg min (1, 2 2) = 3.
Therefore a14 (c1, C2, in 03). Contrast this order with the more natural sort order that interleaves u as if it were an event at time 8 - 2: (e, un eg 0z). Under the sort order, un appears before the first event ej exceeding time 81 5, i.e, the event e2 appearing index 3. This rule for placing un cannot be
5

---

Published in Transactions on Machine Learning Research (04/2024)
applied during autoregressive inference, it requires to place i at index 2 with foreknowledge of the event e2 that has yet to be generated. Under the anticipatory order (Definition 3.1), appears after the event 02 appearing at index 2. This rule for placing i can be applied during the process of autoregressive inference: upon observing 2₂ at index 2, we place u1 at index 3. This distinction between the anticipatory order (which admits autoregressive inference) and the sort order (which does not) can be formalized using the concept of a stopping time, which we describe in Section 3.1.
3.1 Stopping Times
Informally, a stopping time of a stochastic process is a random index T for which the occurrence of an event {r i} can be determined based only the information observed in the prefix X1: (modeled by sigma algebra F.). A classic example of stopping time is first hit time: the first index at which stochastic process x attains a value v. This is a stopping time because the condition {r = i) can be determined at time i simply by inspecting whether x; t. An example of a random time that is not a stopping time is a last exit time: the last index i at which x; = v. In contrast to the first hit time, the last exit. time can only be determined after observing the entire process x.
Definition 3.3. (Stopping Times) Let 7 be an ordered index set, let (2, (Fi)ier) be filtered measurable space, and let r 12 + I be a random index defined on this space. We say that T is a stopping time if [T= E for all E I.
In our case, -= f1.................... N K), and is random index into the sequence a The filtration (Fi) consists of the sigma algebras F, generated by the prefix sequences a1s- While adopt the conventional terminology of stopping times, in this case it might be better to think of r a stopping inder of a1:++K: not to be confused with a (continuous) random time in the underlying point process.
To condition on a control uy, it essential that Tu - is a stopping time: inference relies on a criterion computed at cach index i-1 1 to determine whether to (temporarily) stop sampling and insert a control a; = ug, or to continue sampling events a; p(e.jan--)) (see Section 3.4). During inference, we only have access to the prefix a1;-1 and therefore the criterion that determines whether to condition on uN at index must be a function of this history. Whereas Tuk - 1 is stopping time, examples of random indices that do not appear after stopping times include:
1. ou k + arg min, (tz+1 2 8t 8}. This is where uy would appear in sort order, as if u were time 8t (naively anticipating ug, 8 seconds in advance).
2. Jux = k + arg min, (ts+10 2 sx). This is 10 indices before where uN would appear in sort order, as Ug were an event at time 8& (naively anticipating ug, 10 indices in advance).
In both cases, fou. = i} cannot be determined based on observation of the prefix a1s-1. Each random index depends on unobserved future events and therefore ou - is not a stopping time.
3.2 Sparse Sequences
The interleaving rule proposed in Definition 3.1 provides no guarantee that a control will appear some number of seconds (or number of indices) in advance of the time that it controls. For very sparse sequences (relative to 5) control can even appear after the time that it controls.
Example 3.4. Suppose 4 = 1, le 2, lg = 5, 81 4.5, and 8 8= =2 2. Then 81 - 5 == 2.5 and
Te, = 1 + arg max - 00 2 8 - 5) = 1,
(10)
Te: = 2 + arg m |1 2 s 87 = 2,
(11)
Tog = 3 + arg max (3 > 8 8} = 3,
(12)
Tu: = 1 + arg min(t, 2 2.5} = 4.
(13)
j
6

---

Published in Transactions on Machine Learning Research (04/2024)
Therefore a1:4 = (e1, C2: C3. u1). And in particular, the control u1 on time $1 4.5 appears after event e3: which occurs at time 13 = 5.
Definition 3.5. Given a sequence of events e1:N that occur at times ti:,, let Amax denote the maximum distance between adjacent events, icc
maxx max - 1
(14)
We say that the sequence e1:V is dense.
For a dense sequence e1:N, we can guarantee that. a control on time appears at or before time -- + Amax using the sequence order given by Definition 3.1. If 5 - Amax > 0 is small, the model may have little time to plan for anticipated controls; if Amax > then we risk anticipating some controls after the times that they are supposed to control, as in Example 3.4. To ensure dense sequences, we insert special REST events into inter-event intervals that exceed a target density A nA < t,+1 t, (n + 1)A then we insert REST events at times t; + A +2 ..... t, +n* This is analogous to how a musician counts out rests in a musical score.
Example 3.6. Continuing Example 3.4, we see that Amax = =3 5. Inserting REST events to ensure A A--density for ===1, the event. sequence becomes e/.5, adding REST tokens at times 3 and 4 such that
ej el:
e =2 eg,
e/ = (3, REST),
e (4, REST),
e/s e3
(15)
In this case, the anticipatory interleaving of un with e/is is a' e/ 832 u1, c e(). We anticipate the control i on time $1 = 4.5 between the event e at time 3 and the event e/ time 4. All the anticipatory models in this paper are trained with -' second.
3.3 Training Anticipatory Models
We train anticipatory autoregressive models using standard maximum likelihood estimation of the sequences A1:N+K (Definition 3.1). We tokenize these sequences according to the encoding described in Definition 2.2: (),(+KK)) = (okenize(aa:n-aa of Brown et al. (2020)) to construct training examples of fixed length M (the model context) from variable- length sequences X1:3(N+K) using special event SEP as the sequence separator. We prepend each training example with single global control code z {0, that indicates whether the example contains local 0 (no controls) facilitates comparisons between anticipatory and autoregressive models. If the training example spans multiple sequences, then 2 describes the sequence preceding the first SEP event. We randomly shuffle and mini-batch training examples for stochastic gradient training.
We follow a standard sequence packing procedure (see, e.g, Appendix B
controls ug: setting
We can choose to either predict all of aI:+KK learning a joint generative model over events and controls, or alternatively just predict the events e1.N (by zeroing out the training losses at indices corresponding controls) and learn a conditional generative model over the events, given controls. In this paper, we predict controls in addition to the events: this maximizes the number of predictions for each example (M-1 predictions for a training example of length M). For the infilling application (Section 3.5) predicting events and predicting control are similar enough tasks that improvements in the two tasks ought to reinforce each other; this reinforcement has been observed empirically in the language modeling domain (see Donahue et al. (2020), Appendix C).
Dividing up a dataset into training examples of length M introduces boundary effects. For general autore- gressive models, this procedure results in an "early token curse," " whereby predictions early in a training example must be made with limited context (Press et al., 2021). For anticipatory autoregressive models, more subtle boundary effects arise. For an anticipation interval 8, controls on first á seconds of the training example do not appear in the context: these controls appear at the end of the previous training example. If 5 is large relative to M then the boundary effects become severe: many event predictions will be made without the relevant contextual controls, and vice-versa. This tempers the value of making predictions for all M - sequence indices, imposing a drag on training efficiency for large values relative to M. Like the early token curse, this effect is mitigated with larger contexts M relative to 8; in practice, the maximum practical anticipation interval 8 is thus coupled with the context size of the model that we plan to train.
7

---

Published in Transactions on Machine Learning Research (04/2024)
Algorithm 1: Anticipatory Autoregressive Sampling
Parameters: Anticipatory autoregressive model p with context length M Anticipation interval Time-localized controls u1:K (monotone increasing in time)
Input
Non-localized controls z (global control codes)
Output
A generated sequence a1:N+K
ao SEP
// special sequence separator event // Index i tracks position in the generated sequence // Index h tracks position in the control sequence
++1
k + 0
do
f time((a--))
// Get the time of the previous event a;-1
while time(u:) St + do
// While there are controls before time t 8
a; + Ug i - 1 1 k - k +
// Anticipate control ur index // Advance to index i ++1 1 // Consume control u
end
Sample ~ i+i+1
// Sample an event from the model
// Advance to index it
while a SEP return a1:-1
// The value N+X i-1
3.4 Anticipatory Inference
We draw conditional samples from an anticipatory autoregressive model p according to the procedure de- scribed in Algorithm 1. If there are controls (K * 0) we set z = 1 (anticipatory sampling mode). In the outer loop, we sample a; ~ p(-la1:s-1). We impose three constraints upon the sampling distribution, all implemented by logit masking: (1) each arrival time must equal or exceed the previous arrival time (mono- tonicity) (2) a time token must follow a note token, a duration token must follow a time token, and a note token must follow duration token (proper ordering) and (3) the model must not generate control token (controls pro-specified by the user).
Each time we draw a sample a, from the model, we note its time t; = time(a,) and check for controls satisfying the condition given by Equation (4) (leveraging the fact that this condition is stopping time). We anticipate these controls by appending them to the generated output following a, The sequence a1:N+K returned by Algorithm contains N events interleaved with K labels. We can postprocess this sequence by stripping out the controls to recover the sequence of generated events e1:N.
3.5 Anticipatory Infilling Models
We apply the anticipatory modeling framework to infilling control, whereby the labels ui: consist of a subset of the events ei.v We duplicate the event vocabulary to distinguish between regular events e/, E V and control events u, E U;; let 9 V + U denote the (bijective) map between the event and control vocabularies. In this case, the combined sequence aI=(N-K)KK = aiN (Definition 3.1) is a re-ordering of the event sequence CI.N, consisting of events cj S C1.N and controls uI:k, with & (u,kk S C1:N and re-order
eINN-K K n n ( Ø. Using an arrival-time tokenization of events (Definition 2.2) allows e1.N while preserving the semantics of the sequence. We can recover the original sequence by converting the control events back to the event vocabulary and sorting all the events according to their arrival times:
e K. U1:K split(an,N)
(16)
C1:N sort (c/ Up {(u1:))-
(17)
We describe the precise encoding of 4 U V that we for music infilling models in Appendix C.
8

---

Published in Transactions on Machine Learning Research (04/2024)
Previously we assumed that the controls U1:K are given to us, separate from the the events eI:-- For the infilling task, there is no distinction between events and controls in the training data; a user could specify an arbitrary set of events to condition on at inference time. To train an infilling model, we must impose a prior on the distribution over subsets of control events ui:K S CI.N- This prior should simulate common infilling patterns, and generalize to accommodale patterns we did not foresee during training. To that end, we propose a distribution over control events consisting of a mixture of random spans of time, random subgroups of instrument, and uniformly random events. We describe specifics of this prior in Appendix D.
We caution that the conditionals learned by an anticipatory infilling model are not consistent with a unique joint distribution over sequences C1:V- Given controls u1:K, the map between the sequences C1:N and a1:N defined by Equation (17) is a bijection. The probability distribution over sequences a1: therefore also defines a distribution over sequences e1:N. Every subset o (ui:k) S @1:N can be used as infilling controls, each resulting a distinct anticipatory sequence aI:N But the probability distributions p(e).)) implied by soquences a1:V will not be the same. We rely on the model's ability to well-approximate the data distribution to ensure approximate consistency between these families of learned distributions.
4 Anticipatory Infilling Models of Music
We train and release anticipatory infilling models on the Lakh MIDI dataset (Raffel, 2016). See Appendix B for licensing information and Section A for a discussion of copyright considerations regarding models trained on Lakh MIDI. The Lakh MIDI dataset consists of 178,561 MIDI files (event sequences) that preprocess into 663,555,310 events (1,990,665,930 tokens using arrival-time encoding) encompassing 8943 hours of music. This dataset is orders of magnitude larger than other common music datasets (Dong et al., 2020), but orders of magnitude smaller than the datasots used to train large models in other domains, c.g. language (Gao et al., 2020) and vision (Schuhmann et al., 2022). For reference, the OpenWebText corpus that reproduces the training set used to train GPT-2 contains approximately 10 billion tokens (Gokaslan & Cohen, 2019). We describe training splits and additional information about Lakh MIDI and preprocessing in Appendix E,
All models trained in this paper are parameterized using causal masked transformers (Vaswani et al., 2017) (decoder-only models) with a context length of M = 1024 tokens (defined in Section 3.3). We train models at three scales, following GPT-2 naming conventions (Radford et al., 2019): Small (128M parameters), Medium (360M parameters), and Large (780M parameters). Because anticipatory models are trained like sequence models on the augmented sequences aI:N+K (see Section 3.3) we are able to use standard libraries for training anticipatory music transformers; in this work, we use the Levanter library for training. 2 For additional details of models and training procedures, see Appendix G.
Anticipatory models are trained with an anticipation interval of d ==5 5 seconds; this allows the models to look ahcad (i.c. anticipate) controls up to 5 scconds before their arrival. This interval is chosen to maximize anticipation, while not anticipating controls so far in advance that a control on time 8 might. appear more than M tokens earlier than events near time s (outside of the model context M). Music in the Lakh MIDI dataset is represented using an average of 68 tokens per second, with a standard deviation of 51 tokens per second (see Appendix E). Therefore, setting 5 seconds ensures that controls on time s appear within M 1024 tokens of events near time 8, unless tokens are generated at sustained rate of more than 2.67 standard deviations above the mean (i.e., more than 204 x 68 + 2.67 * 51 tokens/second 2 M/0).
4.1 Automatic Metrics
See Table for a summary of configurations and performance metrics derived from the log-loss for antic- ipatory infilling models trained Lakh MIDI. Arrival-time encoding appears more effective for training autoregressive transformers than interarrival-timee encoding (rows 1 and 2). Anticipatory training incurs a small tax on performance compared to a baseline autoregressive model for short training schedules (compare rows 2 and 3) however this gap disappears with a longer training schedule (compare rows 4 and 5). In this sense, anticipation unlocks infilling control capabilitics in autoregressive music models "for free." Larger models achieve significantly better perplexity (compare rows 3, 6, and 9) as do longer training schedules (compare rows 2 and 4; 3 and 5; 6, 7, and 8).
/httpe://github. com/ wwwsssad-ddddrreo cr arfn//eeaattrr
9

---

Published in Transactions on Machine Learning Research (04/2024)
Table I: Evaluation Loss. All losses are reported on the Lakh MIDI test set sequences ei:N (without anticipa- tion). For the arrival-time models, we define a per-event loss Le summed across event triples (Definition 2.2) exp (Le). This loss decomposes into Le = Le La + La
and report the per-event perplexity ppl(e) with corresponding perplexities for onsets t, durations d and notes The time-normalized bits per second metric (bps) (Thickstun et al., 2019) is defined in Appendix F. Parameter counts differ between arrival and interarrival models, due to the difference in vocabulary size. Anticipatory models are indicated by the 'AM' flag. For anticipatory models, we evaluate with z = 0 (see Section 3.3).
#
Config
Params
Steps
Encoding
AM
bps
ppl(e)
ppl(t) ppl(d)
ppl(n)
1
Small
112M
100k
interarrival
x
85.9
2
Small
128M
100k
arrival
X
80.4
14.9
1.59
3.90
2.40
3
Small
128M
100k
arrival
/
80.7
15.0
1.58
3.98
2.40
4
Small
128M
800k
arrival
X
75.7
12.7
1.53
3.65
2.27
5
Small
128M
800k
arrival
75.0
12.4
1.52
3.64
2.24
6
Medium
360M
100k
arrival
74.4
12.1
1.54
3.55
2.22
7
Medium
360M
200k
arrival
71.5
11.1
1.51
3.39
2.16
8
Medium
360M
800k
arrival
69.7
10.4
1.49
3.29
2.12
9
Large
780M
100k
arrival
73.2
11.7
1.52
3.44
2.23
4.2 Human Evaluation
We solicited human evaluation of generated music to ground the performance of these models in human assessments of quality, to compare these models to other music generation systems, and to evaluate the anticipatory capabilities of these models that are not captured by the predictive log-loss. We evaluate generated outputs for two tasks described below folow-following a similar procedure to Huang et al. (2018), whereby we ask workers to identify which of two 20-socond synthesized audio clips more conventionally musical. We recruited crowd workers on the Amazon Mechanical Turk platform to perform these tasks. We paid workers $0.75 US dollars for each pairwise evaluation. Assuming that workers listen to each clip twice and spend an additional 40 seconds to make their decision and overhead time between tasks this amounts to two minutes of time per task, or $22.50 hourly rate. We pre-qualified workers for by asking them to distinguish between five pairs of human compositions versus melodies accompanied by the random retrieval baseline (described below for the accompaniment task).
All samples from our models used for human evaluation are generated using nucleus sampling (Holtzman et al., 2020) with top-p probability p = 0.95. We chose this threshold by manually inspecting music generated from the 800k-step Medium model (Row 8 in Table 1) conditioned on prompts from the validation set and using values of E {0.9, 0.95, 0.98, 1.0]. We chose the value that most consistently produced high quality music (in the authors' judgement). Sampling with < 1.0 generates less diverse music; anecdotally we observe that music generated at = 0.95 is more conservative than both music generated at = 1.0 and music composed by humans (c.g., sampling at = 0.95 is less likely to introduce a new instrumental part). It is possible that our instruction for workers to identify the most conventionally musical clip favors this more conservative generated music.
Prompt Continuation. Workers evaluated 50 clips, each consisting of a three-bar -ranoyyy selected from the beginning of tracks in the Lakh MIDI test sol-followed by a continuation generated by a model or from the original human composition. See Table 2 for human evaluation of music generated from prompts, compared to human compositions; for full pairwise comparison results, see Appendix J. We use this task to compare our model in the standard autoregressive setting to baseline Music Transformer (Huang et al., 2018) as implemented by von Rütte et al. (2023) and trained on the Lakh MIDI dataset; we refer to this as the FIGARO baseline. While continuing a prompl does not require anticipation, evaluators find that music generated by anticipatory models given a prompt considerably more musical than music generated by the bascline FIGARO model (this should not be surprising: the FIGARO Music Transformer 30M parameter model). Human evaluators prefer the Small arrival-time model (Row 3; Table 1) over the Small interarrival-tine model (Row 1; Table 1) consistent with the superior log-loss of the arrival-time model.
smaller,
10

---

Published in Transactions on Machine Learning Research (04/2024)
Table 2: Human evaluation of generated continuations three-bat musical prompts versus human compo-
sitions. P-values are reported using Wilcoxon signed rank test. Row numbers reference Table 1.
Model
Wins
Ties
Losses
p-value
Medium (Row 8)
29
77
0.0027
Music Transformer (von Rütte et al., 2023)
18
119
2.806
10-20
Small (Row 3)
43
23
84
0.0002
Small (Row 1)
31
16
103
4.976
10 10-10
Table 3: Human evaluation of 15-second accompaniments versus human compositions. P-values are reported using& a Wilcoxon signed rank test.
Algorithm B
Wins
Ties
Losses
p-value
Anticipatory
18
31
11
0.194
Autoregressive
5
10
45
1.542 x 10
Retricval
2
6
52
1.017 x 10-11
Accompaniment. Workers evaluated 20 clips, each consisting of five-second prompt (randomly selected from tracks in the Lakh MIDI test set) followed by 15-second accompaniment (i.e. infilling) conditioned the prompt and the full (20 seconds) melodic line. See Table 3 for human evaluation of generated music accompaniments, compared to human compositions; for full pairwise comparison results, see Appendix J. For this task, we crudely define melodic line to be the instrumental part with the highest pitch. This task allows us to probe the infilling capabilities of our anticipatory autoregressive models. We compare anticipation to two baseline accompaniment algorithuss: random retrieval and autoregressive accompaniment. Random retrieval is simple baseline whereby we accompany the melody with a random 15-second clip retrieved from elsewhere in the track. Autoregressive accompaniment is algorithmic attempt to autoregressive generation (without anticipation) solve the accompaniment task; see Appendix I for details. Both anticipatory and autoregressive accompaniments are generated using a Medium anticipatory model (Row 8; Table 1). Evaluators express a mild preference for anticipatory accompaniments over the human composition skyline, While this result is not statistically significant, it points to the effectiveness of anticipation, espocially in the more coistrained accompaniment setling.
5 Related Work
Controllable Generative Modeling. Anticipatory infilling modcls are motivated in part by control capabilities of recent lext. infilling models (Zhu et al., 2019; Du al., 2022; Aghajanyan et al., 2022) and the growing empirical evidence that autoregressive models can be augmented with infilling objectives without sacrificing unconditional modeling performance (Donahue et al., 2020; Bavarian et al., 2022). An analogous approach to ipanninfilling for music using Seq conditioning was proposed by Ippolito et al. (2018). Our discussion of anticipation draws inspiration from the asynchronous control setting considered by Hassibi et al. (1999). Anticipatory infilling models also draw inspiration from orderless NADE (Uria et al., 2014) and XLNet (Yang et al., 2019), which learn ensembles of models over different autoregressive factorizations. Anticipatory infilling models differ from these methods by (1) applying local rather than global permutations of the factorization order and (2) achieving this by permuting the sequence itsolf (facilitatod by context-free arrival-time encoding; Section 2.1) rather than masking it.
Markov-chaie Monte Carlo (MCMC) samplers are a natural candidate for conditional sampling tasks, and in particular infilling. Diffusion models have proven highly effective as generative models of images with infilling capabilities (Ho et al., 2020; Lugmayr et al., 2022). Masked language models can also be repurposed as MCMC samplers for infilling (Goyal et al., 2021; Mireshghallah et al., 2022; Wang & Xia, 2021). These models are not easily applied to conditional point process generation, which requires the generation of variable numbers of events within specified region of space or time; for work developing MCMC infilling methods in restricted families of point processes, see Shelton al. (2018). Applications of MCMC methods for symbolic
11

---

Published in Transactions on Machine Learning Research (04/2024)
music infilling have compromised by either discretizing time with piano-rolls (Hadjeres et al., 2017; Huang et al., 2017) or modeling coarse, fixed-rate latent variables, without the ability to control gramular details of the variable-length sequence (Mittal et al., 2021).
Sequential Monte Carlo (SMC) samplers have been considered for infilling of both contintous temporal point processes (Mei et al., 2019) and sequence models (Lin & Fisner, 2018); the later more directly relates to our setting because discretizing time (Section 2.1) reduces point process modeling to sequence modeling. These SMC methods approximate samples from the conditional distribution over events given all past and future controls, using importance sampling to select among candidate samples offered by proposal model. In contrast to SMC, anticipatory inference is more efficient, requiring just one model call to generate an event. The limitations of anticipation are also more explicit: conditional independence of controls more than seconds in the future. In contrast, the limitations of SMC approximations are implicitly determined by the number of candidates and the effectiveness of the proposal model.
Musical Accompaniment. In contrast to harmonization tasks, which seek to accompany a melody with simple chords (Simon et al., 2008; Ych et al., 2021; Chen et al., 2021; Wu et al., 2024; Rhyu et al., 2022), here we seek to generate complete, asynchronous musical accompaniments to a given melody. Huang et al. (2018) previously considered a solo-piano accompaniment generation task using Seq2Seq models; anticipatory models generalize the Seq2Soq approach, motivated by the long iulti-instrument sequences found in the Lakh MIDI dataset. Dong et. al. (2018) considered an accompaniment task using piano-roll encodings of the Lakh MIDI dataset; see Section for discussion of the limitations of piano-roll encodings. Both Zhu et al. (2018) and Ren et al. (2020) propose encoder-decoder architectures for conditional generation of accompaniments given a melody: we discuss obstructions to efficiently training mncoder-decodee models of music at scale in Section 1. Shih et al. (2022) consider a more abstract form of conditioning, using a melody as thematic material to generate an arrangement rather than a literal accompaniment of the given melody. Our accompaniment lask is also loosely analogous to recent work on vocal accompaniment in the audio domain (Donahue et al., 2023).
6 Conclusion
While the focus of this work has been the music domain, the anticipatory modeling techniques developed in Section 3 are generally applicable to the controllable generation of temporal point processes. Temporal point processos appear in any sctting where data is associated with timestamps, including ccommerce activity (Du et al., 2015), social media logs (Farajtabar et al., 2017), healthcare records (Euguehard et. al., 2020), and neuroscience data (Williams et al., 2020). Using analogs to the encoding developed for music in Section 2, anticipatory modeling could be used to facilitate the controllable simulation of data in these and other modalities. Furthermore, while the experiments in this paper focus on infilling control a special case where the controls and events belong to the same to simulate events in one modality, guided asynchronous by controls in different modality.
modeling could be applied more generally
Provided that locality contimues to be important inductive bias for learning, we believe that anticipa- tion will be useful for modeling conditional temporal point processes. Locality is a perennial theme in the machine learning literature. Popular model architectures including ConvNets (Forsyth et al., 1999) and LSTM (Hochreiter & Schmidluber, 1997) exploit locality as an inductive bias. Many long context Trans- formers adopt local sparsity as an approximation to dense attention (Child et al., 2019; Beltagy et al., 2020; Zaheer al., 2020). Locality can be exploited to improve training efficiency via staged training (Press et al., 2021; 2022). The local structure of anticipatory sequences is conducive to the broad class of methods that exploit depend upon locality. Because anticipation only intervenes to modify the data, it can be seamlessly combined with other modeling innovations, e.g., in the music space: the relative transformer (Huang et al., 2018), the compound word transformer (Hsiao et al., 2021), and RIPO attention (Guo et al., 2022).
Generative music models have not yet reached a broad audience alongside generative models of language (Brown et al., 2020) or images (Ramosh et al., 2022). Slow adoption of music models within creative communtiiss partly attributable to the difficulty of controlling these models (Briot & Pachet, 2020). Users value agency in human-AT collaborations, preferring to take an active role over more automated solutions (Oh
12

---

Published in Transactions on Machine Learning Research (04/2024)
al., 2018; Roy et al., 2019). We note the proliferation of recent work music generation (primarily in the audio domain) controlled by text (Forsgren & Martiros, 2022; Kreuk et al., 2023; Agostinelli et al., 2023; Schneider et al., 2023; Huang et al., 2023a;b); we are intrigued by the possibility of applying anticipation to generate symbolic music conditioned on localized text labels (c.g., lyrics). More broadly, we view symbolic music generation and audio music synthesis as complementary, analogous to text generation and speech synthesis. We hope that the Anticipatory Music Transformer may fill a role that currently underdeveloped in the construction of controllable music generation systems that support the human creative process.
Acknowledgements
We thank Jennifer Brennan, Niladri Chatterji, Peter Henderson, Cheng-Zhi Anna Huang, Sidd Karamcheti, Mina Lee, Mark A. Lemley, Nelson Liu, Michael C. Mozer, Joon Sung Park, Ofir Press, and Dimitri von Rütte for providing invaluable advice, discussion, and support for various aspects of this project. We also thank the crowdworkers who provided impartial human ovaluation of the music generated by our models, including: Oliver Crangle, Dare, Razr-Dylan, Bryan Haskins, and Tammy.
This work was done the Center for Research on Foundation Models (CRFM) Stanford University. We thank the CRFM and the Stanford Institute for Human-Centered Artificial Intelligence (HAI) for supporting this work. Toyola Research Institute (TRI) and Schmidt Futures provided funds to support this work. The experiments discussed in this paper were conducted on Cloud TPU VMs, provided by the Google TPU Research Cloud (TRC).
References
Circuit. Authors Guild, inc. v. Google, inc. 721 F.3d 132, 2015. A.2
Sajjad Abdoli. Iranian traditional music dastgah classification. In International Society for Music Informa- tion Retrieval, 2011. A.3
Daron Acemoglu and Pascual Restrepo. Artificial intelligence, aulomation, and work. In The economics of artificial intelligence: An agenda. University of Chicago Press, 2018. A.1
Armen Aghajanyan, Bernie Huang, Candace Ross, Vladimir Karpukhin, Hu Xu, Naman Goyal, Dmytro Okhonko, Mandar Joshi, Gargi Ghosh, Mike Lewis, et al. CM3: A causal masked multimodal model of the internet. arXiv preprint arx::22001.07520 2022.
Andrea Agostinelli, Timo I Denk, Zalán Borsos, Jesse Engel, Mauro Verzetti, Antoine Caillon, Qingging Huang, Aren Jansen, Adam Roberts, Marco Tagliasacchi, et al. MusicLM: Generating music from text. arXiv preprint ar Xic:2300 1 1325, 2023. 6
Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. Efficient training of language models to fill in the middle. arXiv preprint 2XXi:22077.1255 2022.1
Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint (rXiv:2004.05150, 2020.
Emmanouil Benetos and Andre Holzapfel. Automatic transcription of turkish makam music. In International Society for Music Information Retrieval, 2013. A.3
Patrick Billingsley. Probobility and Measure (3nd Edition). John Wilcy & Sons, 1995.
Nicolas Boullagee-TTewwandowski Yoshua Bengio, and Pascal Vincent. Modeling temporal dependencies in high-dimensional sequences: application to polyphonic music generation and transcription. In Interna- tional Conference on Machine Learning, 2012.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne and Qiao Zhang. JAX: compos- G
able transformations of Python+ programs, 2018. URL http://github.cce
13

---

Published in Transactions on Machine Learning Research (04/2024)
Jean-Pierre Briot and François Pachet. Deep learning for music generation: challenges and directions. Neural Computing and Applications, 2020. 6
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. 3.3, 6
Nicholas Carlini, Daphne Ippolito, Matthew Jagiclski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In International Conference on Learning Rep- resentations, 2023. A.2
Chin-Jui Chang, Chun-Yi Lee, and Yi-Hsuan Yang. Variable-length music score infilling via XLNet and musically specialized positional encoding, In International Society for Music Information Retrieval, 2021.
2
Ke Chen, Cheng-i Wang, Taylor Berg- Kerg-tirkp,, and Shlomo Dubnov. Music SketchNet: Controllable music generation via factorized representations of pitch and rhythm. In International Society for Music Information Retrieval, 2020. 2
Yi-Wei Chen, Hung-Shin Lee, Yen-Hsing Chen, and Hsin-Min Wang. SurpriseNet: Mclody harmonization conditioning on user-contrroled surprise conlours. In International Sociely for Music Information Retrieval, 2021.
CHESScom. Chess is booming! and our servers are struggling. https://www.dd. http://wcwsss.cmmllogoeeeeoo///o www.ccooo/c./aieeee.c leess-ii-boonig---e wew---rreer--rreeeeeeeeee-----ee
2023. Accessed on 22 May 2023. A.4
Rewon Child, Scott Gray, Alec Radford, and Πya Sutskever. Generating long sequences with sparse trans- formers. arXiv preprint (tt::1990.100099 2019. 1, 6
Daryl J Daley and David Vere-Jones. An introduction to the theory of point processes: volume II: general theory and structure. Springer Science & Business Media, 2007. 2
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory- efficient exact attention with 10-awareness. In Advances in Neural Information Processing Systems, 2022. 1
Chris Donahue, Huanru Henry Mao, Yiting Ethan Li, Garrison W Cottrell, and Julian McAuley. LakhNES: Improving inntimiisttuu music generation with cross-domain pre- pre-training. In International Society for Music Information Retrievol, 2019. 2
Chris Donahue, Mina Lee, and Percy Liang Enabling language models to fill in the blanks. In Annual Meeting of the Association for Computational Linguistics, 2020. 3.3, 5
Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, Andrea Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour, et al. SingSong: Generating musical accompaniments from singing. arxiv preprint vrXi::2301.12662 2023.
Hao- Wen Dong, Win-Yi Hsiao, Li-Chia Yang, and Yi-Hsuan Yang. Musegan: Multi-track sequential genera- tive adversarial networks for symbolic music generation and accompaniment. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018. 2, 5
Hao-Wen Dong, Ke Chen, Julian McAnley, and Taylor Berg- Kirkpatrick. MusPy: A toolkit for symbolic music generation. In International Society for Music Information Retrieval, 2020. 4
Nan Du, Yichen Wang, Niao He, Jimeng Sun, and Le Song. Time-sensitive recommendation from recurrent user activities. In Advances in Neural Information Processing Systems, 2015. 6
Nan Du, Hanjun Dai, Rakshit Trivedi, Utkarsh Upadhyay, Manuel Gomez-Rodriguez, and Le Song. Re- current marked temporal point processes: Embedding event history to vector. In SIGKDD International Conference Knowledge Discovery and Dala Mining, 2016. 1,
14

---

Published in Transactions on Machine Learning Research (04/2024)
Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GLM: General language model pretraining with autoregressive blank infilling. In Annual Meeting of the Association for Computational Linguistics (Vo (Volume 1: Long Papers), 2022.
Joseph Enguehard, Dan Busbridge, Adam Bozson, Claire Woodcock, and Nils Hammerla. Neural temporal point processes for modelling electronic health records. In Machine Learning for Health, pp. 85 113.
PMLR, 2020. 6
Mehrdad Farajtabar, Yichen Wang, Manuel Gomez-Rodrriguez, Shuang Li, Hongyuan Zha, and Le Song, COEVOLVE: A joint point process model for information diffusion and network evolution. Journal of Machine Learning Research, 2017. 6
Seth Forsgren and Hayk Martiros. Riffusion Stable diffusion for real-time music generation, 2022. URL
https://riffusi.nt com/about. 6
David A Forsyth, Joseph Mundy, Vito di Gesú, Roberto Cipolla, Yann LeCun, Patrick Haffner, Léon Bot- tou, and Yoshua Bengio. Object recognition with gradient-based learning. Shape, Contour and Grouping in Compuler Vision, pp. 319 345, 1999.
Lco Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa. Nabeshima, et al. The Pile: An 800gb datasel of diverse text for language modeling. arXiv preprint 21x1:21011.10 01.00027, 2020.
Joshua P Gardner, Tan Simon, Ethan Manilow, Curtis Hawthorne, and Jesse Engel. MT3: Multi-task multitrack music transcription. In International Conference on Learning Representations, 2021. 2.1
Aaron Gokaslan and Vanya Cohen. Openwebtext corpus, 2019. URL. http://Skkllono007 gi github. io/
OpenVebTextCCrpus. 4
Kartik Goyal, Chris Dyer, and Taylor Berg-Kirkpatrick. Exposing the implicit energy networks behind masked language models via metropolis hastings. In International Conference on Learning Representa- lions, 2021. 5
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference Learning Representations, 2022. 1
Zixun Guo, Jaeyong Kang, and Dorien Herremans. A <domainnkuuolllgge-inniieed music embedding space and a novel attention mechanism for symbolic music modeling. In AAAI Conference on Artificial Intelligence, 2022.
Gaätan Hadjeres, François Pachet, and Frank Nielsen. DeepBach: a steerable model for bach chorales generation. International Conference on Machine Learning, 2017. 2,
Arash Hassibi, Stephen P Boyd, and Jonathan P How. Control of asynchronous dynamical systems with rate constraints on events. In Conference on Decision and Control. IEEE, 1999. 5
Curtis Hawthorne, Andrew Jaegle, Câtälina Cangea, Sebastian Borgeaud, Charlie Nash, Mateusz Mali- nowski, Sander Dieleman, Oriol Vinyals, Matthew Botvinick, Ian Simon, et al. General-purpose, long- context autoregressive modeling with Perceiver AR. In International Conference on Machine Learning, 2022.1
Michael A Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. A survey recent approaches for natural language processing in low-resource scenarios. In Proceedings of the 2021 Confer- ence of the North American Chapler of the Association for Compulational Linguistios: Human Language Technologies, 2021. A.3
Peter Henderson, Xuechen Li, Dan Jurafsky, Tatsunori Hashimoto, Mark A. Lemley, and Percy Liang, Foundation models and fair use. arXiv preprint Xxx::2230..15715, 2023. A.2
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units. orXiv preprint ar 160:x:606.08.... 2016. G
15

---

Published in Transactions on Machine Learning Research (04/2024)
Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information. Processing Systems, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 1997. 6
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optinal large language models. In Advonces in Neural Information Processing Systems, 2022. G
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text. degener- ation. In International Conference Learning Representations, 2020. 4.2
Wen-Yi Yi Hsiao, Jen-Yu Liu, Yin-Cheng Ych, and Yi-Hsuan Yang. Compound word transformer: Learning to compose full-song music over dynamic directed hypergraphs. In AAAI Conference on Artificial Intelligence, 2021.
Cheng-Zhi Anna Huang, David Duvenaud, and Krzysztof 7. Gajos. ChordRipple: Recommending chords help novice composers go beyond the ordinary. In Proceedings of the 21st international conference on intelligent user interfaces, 2016. A.4
Cheng-Zhi Anna Huang, Tim Cooijmans, Adam Roberts, Aaron Courville, and Douglas Eck. Counterpoint by convolution. In International Society for Music Information Retrieval, 2017. 1, 2,
Cheng-Zht Zhi Anna Huang, Ashish Vaswani, Jakob Uszkoreit, lan Simon, Curtis Hawthorne, Noam Shazeer, Andrew M Dai, Matthew Hoffman, Monica Dinculescu, and Douglas Eck. Music transformer: Gener- aling music with long-lerm structure. In International Conference on Learning Represenlations, 2018. 2, 2, 2.1, 4.2, 4.2, 5, 6, H. 8,
Cheng-Zhi Anna Huang, Curtis Hawthorne, Adam Roberts, Monica Dinculescu, James Wexler, Leon Hong, and Jacob Howcroft. The Bach Doodle: Approachable music composition with machine learning at scale. In International Sociely for Music Informalion Retricual, 2019. 1
Qingqing Huang, Daniel S Park, Tao Wang, Timo I Denk, Andy Ly, Nanxin Chen, Zhengdong Zhang, Zhishuai Zhang, Jiahui Yu, Christian Frank, et al. Noise2Music: Text-conditioned music generation with diffusson models. Xiv preprint or Xrxx:22302. 039177, 2023a.
Rongjic Huang, Jiawei Huang, Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusson models. In International Conference Machine Learning, 2023b. 6
Yu-Siang Huang and Yi-Hsuan Yang. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In ACM International Conference on Multimedia, 2020. 2.1, H
International MIDI Association. Standard midi files, 1988.
Daphne Ippolito, Anna Huang, Curtis Hawthorne, and Douglas Eck. Infilling piano performances. In NIPS Workshop on Machine Learning for Creativity and Design, 2018.
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh,
Yaniv Taigman, and Yossi Adi. AudioGen: Textually guided audio generation. In International Conference on Learning Represenlations, 2023. 6
Mina Lee, Percy Liang, and Qian Yang. CoAuthor: Designing a human-ai collaborative writing dataset. for exploring language model capabilities. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, 2022. A.5
Jordan Lenchitz. Reconsidering quantization in MIR. In International Society for Music Information Re- trieval, 2021. A.3
16

---

Published in Transactions on Machine Learning Research (04/2024)
Robert Levinson. The case for computer chess engines in the classroom. International Journal of Technology, Knowledge and Sociely, 2011. A.4
Chu Mau-Cheng Lin and Jason Eisner. Neural particle smoothing for sampling from conditional sequence models. In Proceedings of the 2018 Conference of the North Americon Chapter of the Association, for Computational Linguistics: Human Language Technologies, 2018.
llya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. 6, G
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2018. 6, G
Ryan Louie, Andy Coenen, Cheng Zhi Anna Huang, Michael Terry, and Carrie J Cai. Novice-ai music co-crcation via ai-stecring tools for deep generative models. In CHI Conference on Human Factors in Computing Systems, 2020.
Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffuuson probabilistic models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Sancho McCann. Copyright thronghout a creative ai pipeline. Canadian Journal of Law & Technology. 2021. A.2
Hongyuan Mei and Jason M Eisner. The neural hawkes process: A neurally self-modulating multivariate point process. In Advances in Neural Information Processing Systems, 2017. 2
Hongyuan Mei, Guanghui Qin, and Jason Fisner. Imputing missing events in innoustute event streams. In International Conference on Machine Learning, 2019. 5
Fatemehsadat Mireshghalahh, Kartik Goyal, and Taylor Berg- Bong-Kirkpatrick. Mix and match: Learning-free controllable text gencrationusing energy language models. In Annual Meeting of the Association for Com- pulational Linguisties (Volume 1: Long Papers), 2022. 5
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting. In Conference on Fairness, Accountability, and Transparency, 2019. 12
Gautam Mittal, Jesse Engel, Curtis Hawthorne, and Ian Simon. Symbolic music generation with diffussin models. In International Society for Music Information Retrieval, 2021.
Changhoon Oh, Jungwoo Song, Jinhan Choi, Seonghyeon Kim, Sungwoo Lee, and Bongwon Suh. I lead, you help but only with enough details: Understanding user experience of co-creation with artificial intelligence. CHI Conference on Human Factors in Computing Systems, 2018.
Takahiro Omi, Kazuyuki Aihara, al. Fully neural network based model for general temporal point pro- cesses. In Advances in Neural Information Processing Systems, 2019. 2
Sageev Oore, Ian Simon, Sander Dieleman, Douglas Eck, and Karen Simonyan. This time with feeling: Learning expressive musical performance. Neural Computing and Applications, 2020. 2, 2
Ashis Pati, Alexander Lerch, and Gaëtan Hadjeres. Learning to traverse latent spaces for musical score inpainting. In International Socicty for Music Information Retricval, 2019. 2
Christine Payne. Musenet, 2019. URL openai. matiaaiiccombooommmeeeeeee 2
Ofir Press, Noah A Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs. In Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), 2021. 3.3,
17

---

Published in Transactions on Machine Learning Research (04/2024)
Ofir Press, Noah Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables imput length extrapolation. In International Conference on Learning Represenlations, 2022. 6
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervisod multitask learners. OpenAT blog, 2019.
Jack W Rae, Scbastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint 1rXic:2112.1144 2021. 6
Colin Raffel. saarnnng-bosed methods for comparing sequences, with applications to audio-to tisio-tt--iid midi alignment and matching. Columbia University, 2016. 1,
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rX1v:2204.06125, 2022. 6
Yi Ren, Jinzheng He, Xu Tan, Tao Qin, Zhou Zhao, and Tie-Ya Liu. Popmag: Pop music accompaniment generation. In ACM International Conference on Multimedia, 2020. 5
Seungyeon Rhyu, Hyeonseok Choi, Sarah Kim, and Kyogu Lee. Translating melody to chord: Structured
and fiexible harmonization of melody with transformer. IEEE Access, 2022. 5
Quentin Roy, Futian Zhang, and Daniel Vogel. Automation accuracy is good, but high controllability may be better. In CHI Conference on Human Factors in Compuling Systems, 2019. 6
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. PixeICN+++: Improving the pixel- cnn with discretized logistic mixture likelihood and other modifications. In International Conference on Learning Representations, 2017. F
Flavio Schneider, Zhijing Jin, and Bernhard Schölkopf. Moñsai: Text-to-music generation with long-context latent diffusion. arXiv preprint arXhv:2301.11757, 2023. 6
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. LATON-5B: An open largo-scale dataset for training next generation image-text models. In Neural Information Processing Systems Track on Datasets and Benchmarks, 2022. 4
Oleksandr Shchur, Marin Biloš, and Stephan Günnemann. Intensity-free learning of temporal point processes. In International Conference on Learning Representations, 2020. 2
Christian Shelton, Zhen Qin, and Chandini Shetty. Hawkes process inference with missing data. In AAAI Conference on Artificial Intelligence, 2018.
Yi-Jen Shih, Shih-Lun Wu, Frank Zalkow, Mcinard Muller, and Yi-Hsuan Yang. Theme transformer: Sym- bolic music generation with thermecoonditioned transformer. IEEE Transactions on Multimediaa, 2022. 5
Ian Simon, Dan Morris, and Sumit Basu. MySong: automatic accompaniment generation for vocal melodies. In CHI Conference on Human Factors in Computing Systems, 2008.
Benjamin LW Sobel. Artificial intelligence's fair use crisis. Columbia Journal of Law & the Arts, 2017. A.2
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art or digital forgery? investigating data replication diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. A.2
18

---

Published in Transactions on Machine Learning Research (04/2024)
Ily Sutskever, Oriol Vinyals, and Quoe V Le. Sequence to sequence learning with neural networks. In Aduances in Newral Information Processing Systems, 2014. 1
Chih-Pin Tan, Chin-Jui Chang, Alvin WY Su, and Yi-Hsuan Yang. Music score expansion with variable- length infilling. In Estended Abstracts for the Late- Bele-Breainng Demo Session of the International Society for Music Information Retrieval, 2022a. 2
Chih-Pin Tan, Alvin WY Su, and Yi-Hsuan Yang. Melody infilling with user-provided structural context. In International Sociely for Music Information Retrievol, 2022b. 2
John Thickstun, Zaid Harchaoui, Dean Foster, and Sham M Kakade. Coupled recurrent models for polyphonic music composition. In International Society for Music Information Retrieval, 2019. 1, F
Benigno Uria, lain Murray, and Hugo Larochelle. A deep and tractable density estimator. In International Conference on Machine Learning, 2014. 5
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Newral Information Processing Systems, 2017. 4
Venkata Subramanian Viraraghavan, Rangarajan Aravind, and Hema A Murthy. Precision of sung notes in carnatic music. In International Society for Music Information Retrieval, 2018. A.3
Dimitri von Rutte, Luca Biggio, Yannic Kilcher, and Thomas Hoffman. FIGARO: Generating symbolic music with fine-grained artistic control. In International Conference on Learning Representations, 2023. 4.2, 2, H. 8,
Nikhil Vyas, Sham Kakade, and Boaz Barak. Provable copyright protection for generative models. International Conference on Machine Learning, 2023. A.2
Ziyu Wang and Gus Xia. Musebert: Pre-training music representation for music understanding and control- lable generation. In International Society for Music Information Retrieval, 2021. 5
Norbert Wiener, Norbert Wiener, Cyberneticist. Mathemalician, Norbert Wiener, Norbert Wiener, and Cybernéticien Mathématicien. Extrapolation, interpolation, and smoothing of stationary time series: with engineering applications. MIT press Cambridge, MA, 1949. 3
Alex Williams, Anthony Degleris, Yixin Wang, and Scott Linderman. Point process models for sequence detection in high-dimensional neural spike trains. In Advances in Newral Information Processing Systems. 2020. 6
Shangda Wu, Yue Yang, Zhaowen Wang, Xiaobing Li, and Maosong Sun. Generating chord progression from melody with flexible harmonic rhythm and controllable harmonic density. EURASIP Journal on Audio, Speech, and Music Processing, 2024. 5
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoe V Le. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Newral Information Processing Systems, 2019.
Yin-Cheng Yeh, Wen-Y Yi Hsiao, Satoru Fukayama, Tetsuro Kitahara, Benjamin Genchel, Hao-Min Liu, Hao- Wen Dong, Yian Chen, Terence Leong, and Yi-Hsuan Yang. Automatic melody harmonization with triad chords: A comparative study. Journal of New Music Research, 2021. 5
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, al. Big Bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems, 2020. 6
Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, and Tic- Yan Liu. MusicBERT: Symbolic music understanding with large-scale pre-training. In Findings of the Association for Computational Linguistics, 2021. 2.1
19

---

Published in Transactions on Machine Learning Research (04/2024)
Hongyuan Zhu, Qi Liu, Nicholas Jing Yuan, Chuan Qin, Jiawei Li, Kun Zhang, Guang Zhou, Furn Wei, Yuanchun Xu, and Enhong Chen. Xiaolce Band: A melody and arrangement. generation framework for pop music. In SIGKDD International Conference on Knowledge Discovery 8 Data Mining, 2018. 5
Wanrong Zhu, Zhiting Hu, and Eric Xing. Text infilling. arXiv preprint or arXi::1901.0013 2019.
Simiao Zuo, Haoming Jiang, Zichong Li, Tuo Zhao, and Hongyuan Zha. Transformer hawkes process. In International Conference on Machine Learning, 2020.
A Ethical Considerations for Generative Models of Music
We find it likely that generative models of music will become widely deployed and used within the next several years (if not sooner). We are particularly concerned about the economic implications of this technology: how will these models affect labor markets for creative work (Section A.1)? We are uncertain about the legal status of these models: how will music created using models be treated by intellectual property and copyright law (Section A.2)? We are also curious how the advent of these models will change our perceptions and experience of art and creativity. Will widespread deployment of models trained on Western music reinforce existing Western hegemony in music culture (Section A.3)? Could the use of these models lead to creative stagnation, or loss of interest in music as an art form (Section A.4)? Analogous questions and concerns have been raised in the broader context of research on generative models, but the focus of this discussion is often grounded in generation of language or images. These discussions are worth revisiting in the specific context of music generation.
A.1 The Creative Economy and Labor Markets
Music composition has cconomic value. What happens to the music cconomy-and the people who rely on it for their livelihoods- if its creative product can be automatically generated by a machine? Economic research suggests that new technologies exert two countervailing effects on the labor market: displacement effect whereby automation reduces demand for labor, and productivity effect whereby greater productivity increases demand for labor (Acemoglu & Restrepo, 2018). It is unclear to us what the net effect of these two forces will be on the size of the labor market for music composition. If latent demand for new music is high, then docrcasing the cost of music production could lead to more employment opportunities for composers. That said, this reconfiguration of the music economy might be disruptive, and success the new music economy could require a new skillset that is difficult (or unappealing) for current workers to acquire. We suspect that generative music technology will affect the current music cconomy, and may displace some workers.
One response to concerns about labor displacement is that researchers should focus on building productivity- enhancing systems that augment human capabilities, rather than automating them. This is an appealing perspective for researchers, because it suggests that we can actively steer the development of generative technologies towards pro-social outcomes. Indeed, the methods developed in this paper facilitate the creation of controllable generative models that support the human creative process, rather than automating it. That said, augmentative and productivity-enhancii that augments human capabilities can be used to replace high-paid, high-skill worker with a low-paid, low-skill worker; or to replace team of workers with a single highly-productive worker. Augmentation can also change the nature of creative work, potentially devaluing or automating aspects of the job that bring a worker joy.
technologies can still cause labor displacement. A system
A.2 Copyright
While the use of copyrighted data to train models is often defensible under fair use doctrine (2d Circuit, 2015), the use of such data for generalive modeling is an open legal question (Sobel, 2017; Henderson el al., 2023). To evaluate the copyright status of generative music models, it may be helpful to distinguish between the model itself (i.c. the parameters) and outputs generated by the model (MeCann, 2021). It unclear how the law will ultimately view (the parameters of) generative model. Is the model a transformative fair use of the training data? Is the model itself creative construct subject to copyright?
20

---

Published in Transactions on Machine Learning Research (04/2024)
Regarding the outputs of a generative music model: generative models have been shown to plagiarize train- ing data in both the language (Carlini et al., 2023) and vision (Somepalli et al., 2023) domains. Based on this evidence, we are certain that generative models of music are similarly capable of plagiarism. Techni- cal mitigation strategies to prevent this behavior based on differential privacy or near access-freenes are nascent. (Vyas et al., 2023), as are socio-technical stralegies, such as providing copyright holders an opt-out mechanism to exclude their content from training datasets (Henderson et al., 2023). The models trained in this paper provide no technical guarantees against plagiarism. Therefore, any outputs of this model should be presumed to risk infringing on copyrighted material.
The Lakh MIDI dataset used to train models for this paper is licensed under the Creative Commons CC-BY 4.0 terms, which ostensibly allows free redistribution, reuse, remixing, and transformation of its content for any purpose. However, the contents of this dataset consists of MIDI files aggregated from a variety of sources and subject to their own (heterogencous, mostly undocumented) copyright terms. In many cases, these MIDI files are derivative work, transcribed from source material (c.g., pop music) which itsolf subject to copyright. Therefore, we presume that the copyright status of Lakh MIDI is more restrictive than its license would suggest.
A.3 Western Bias
While the methods developed in this paper are broadly applicable to modeling diverse musical traditions, the MIDI data format and the Lakh MIDI dataset used to train the anticipatory music transformer both impose a strong bias towards the Western music tradition. In Section 2 we adopted a musical vocabulary that describes musical pitch according to the 12-tone chromatic scale, which we inherit. from the MIDI format of our training data. Models trained using this vocabulary are incapable of expressing music outside the 12-tone scale. This excludes, for example, the Koron and Sori quarter tones of Persian Dastgah music (Abdoli, 2011), the 53 Koma of Turkish makam (Benetos & Holzapfel, 2013), and the continuous-piee gamaka appearing in Carnatic music (Viraraghavan et al., 2018). For further discussion of the consequences of modeling using 12-tone discrete scale, see Lenchitz (2021).
The Lakh MIDI dataset used to train anticipatory music transformers in Section 4 consists almost exclusively of Western music. This incvitably biases the model's predictions towards infilling completions that follow Western rules of composition. We encourage work lowards building models for other music traditions. We believe that the primary obstruction this work will be the limited availability of data. In this sense, extending the work presented in this paper to other music traditions may be analogous to work in the natural language processing community on low-resource language modeling (Hedderich et al., 2021).
A.4 Models and Art
If generated music becomes too good, might people lose interest in creating music altogether? We find this unlikely. Our appreciation and emotional attachment to art is closely connected to its provenance, the story behind its creation, and the story of its creator. Machine-generated music is unlikely to supplant high art, although we expect that many artists may incorporate generative tools into their creative processes. Economic pressures aside, artists are free to use generative tools or not. We expect many artists to embrace these tools, but surely others will eschew them in favor of more traditional creative processes.
From & humanist perspective, we are optimistic for future with generative models. Due to the high skill required to compose music, most people are currently unable to create music at all. Lowering the barrier to music crcation is more likely to increase interest in music than to decrease it. A strong generative model of music can be repurposed as learning tool, providing feedback on compositions and ideas for how to improve a composition (Huang et al., 2016). We draw an analogy to modern chess engines, which play the game of chess at a far higher level than even the best human players. Despite this, public interest in the game of chess has never been higher (CHESScom, 2023): we celebrate human performance, and it is irrelevant that the computer on our phone could beat. us every time. Chess engines offer universal access to a form of chess education: whereas chess development previously required tutoring by a strong chess player (requiring money and access) young players today can learn to play well by inspecting how the engine plays (Levinson, 2011). We anticipate similar opportunities for generative models to increase access to music education.
21

---

Published in Transactions on Machine Learning Research (04/2024)
A.5 Releasing the Anticipatory Music Transformer
We see many opportunities for generative models to support human creativity and expression. At the same time, we are concerned for the economic prospects of current participants in the music labor market, whose work may disrupted or displaced by the deployment of generative music technologies. In our work, we strive to develop methods that maximize the humanistic potential for generative models of music. Specifically, we focus on controllable generation, which places the expressive power of these models under the control of their users. We focus on generation of intermediate symbolic music representations, which could be integratod as an assistive lool analogous to writing assistant (Lee et al., 2022) into the music sequencing and synthesis workflow of modern digital audio workstation. In contrast to audio models, symbolic music generation could provide less disruptive integration of generative music technologics into existing workflows and thus support current. participants in the music labor market. Nevertheless, we find it impossible to weigh the opportunities presented by these models against the challenges this technology might pose for workers in the music economy. Criticisms of capitalist economic structures that accrue the windfalls of automation and productivvtty-cnhnniii technologics neither their inventors nor to workers whose labor they displaco- are beyond the scope of this paper. We welcome feedback on our decision to pursue this line of research; we hope to foster discussion of how steer future research in this area towards methods that serve and support composers and musicians.
B Licensing
We release the code for constructing anticipatory infilling models and weights for the models discussed in this paper under the Apache License, Version 2.0.
C Encoding Details for Anticipatory Infilling Models.
The tokenization of training examples (Section 3.3) for anticipatory infilling models using an arrival-time encoding of events (Definition 2.2) described by Definition C.1. As discussed in Section 3.5, we double the base vocabulary distinguish between anticipated events and non-anticipated events. We also include sequence separator token and global control codes z E AAR) (Section 3.3) as well as the REST token (Section 3.2).
Definition C.1. (Arrival-Time Training Example) Tet aL-(M-1)/3 be a training example, possibly re-ordered according to Definition 3.1 for infilling. An arrival-time tokenized training ezample is a sequence XI:M, defined as follows. We define two special tokens REST 27512 and SEP = 55025, and two control tokens AR = 55026 and AAR 55027. Separation between two sequences is indicated by triple of three SEP tokens. The first token X1 in every training example encodes the global control code z:
X1 E {AR, AAR).
(the anticipation control code). (18)
If a, is non-anticipated event then
X3-2 E {O...., 10000) (SEP).
(0 100s, 10ms quantized) (19)
X3:1 E (10000, 10000 + 1000) U (SEP),
(0 10s, 10ms quantized) (20)
X3: {1 (11000,.... 11000 + 16512) } U (SEP) U (REST),
(inst pitches). (21)
Otherwise (if a; is an anticipated event)
X3:-2 E (27513, 27511 + 10000) U (SEP),
27513
(0 100s, 10ms quantized) (22)
X3:- 1 E (37513, 37513 + 1000} U (SEP).
(0 10s, 10ms quantized) (23)
X3: {38513, .... 38513 16512) U (SEP) U (REST),
(instruments pitches). (24)
The total vocabulary size is 55028.
22

---

Published in Transactions on Machine Learning Research (04/2024)
Table 4: Arrival-time tokenization of "Twinkle, Twinkle, Little Star," played on a piano at tempo quar- ter=120. For clarity, we group the sequence of tokens tokens into triplets (one event per row).
Token Values
Event Description
Arrival Time
Duration
Note
t
d
P
k
55,025
55,025
55,025
Sequence Separator
Event
0
10,048
11,060
Os
480ms
C4
piano
50
10,048
11,060
0.5s
480ms
C4
piano
100
10,048
11,067
480ms
G4
piano
150
10,048
11,067
1.5s
480ms
G4
piano
200
10,048
11,069
2s
480ms
A4
piano
250
10,048
11,069
2.5s
480ms
A4
piano
300
10,095
11,067
3.5s
950ms
G4
piano
400
10,048
11,065
4s
480ms
F4
piano
450
10,048
11,065
4.58
480ms
F4
piano
500
10,048
11,064
58
480ms
E4
piano
550
10,048
11,064
5.5s
480ms
E4
piano
600
10,048
11,062
6s
480ms
D4
piano
650
10,048
11,062
6.5s
480ms
D4
piano
700
10,095
11,060
7s
950ms
C4
piano
Example C.2. The arrival-time tokenization (processed for autoregressive training) of the first for bars of the lullaby "Twinkle, Twinkle, Little Star," played piano at tempo quarter=120:
Xp:46 [55026, 55025, 55025, 55025, O, 10048, 11060, 50, 10048, 11060, 100, 10048, 11067, 150, 10048, 11067, 200, 10048, 11069, 250, 10048, 11069, 300, 10095, 11067, 400, 10048, 11065, 450, 10048, 11065, 500, 10048, 11064, 550, 10048, 11064, 600, 10048, 11062, 650, 10048, 11062, 700, 10095, 11060).
See Table 4 for structured description of this sequence.
The tokenization of training sequences for autoregressive models using an interarrival-time encoding (Defintion 2.3) is described by Definition C.3. We double the base vocabulary of notes and instruments to distinguish between onsets and offsets. We truncate interarrival times longer than 10 seconds (to 10 soconds).
Definition C.3. (Interarrival-Tee Timme Training Example) Let e1:(M/4) be a training example of events (Defini- tion 2.1). An interarrival-time tokenized training ezample is a sequence XI:M, defined as follows. We define a single special token SEP = 34024. Separation between two sequences is indicated by a single SEP token. Using the notation (/(M/2) from Definition 2.3, if x an onset,
X2 {1000,... ... 1000 + 16512],
(instruments pitches). (25)
If x is an offset,
X2; E (17512..... 17512 + 16512],
(instruments x pitchos). (26)
And regardless,
X2++1 E {0, 1000).
(0 10s, 10ms quantized). (27)
As described in Definition 2.3, omit tokens X2:41 0. The total vocabulary size 34025.
Example C.4. The interarrival-tine tokenization of the first for bars of the lullaby "Twinkle, Twinkle, Little Star," " played piano at tempo quarter=120:
23

---

Published in Transactions on Machine Learning Research (04/2024)
X0:56 [34024, 1060, 48, 17572, 2, 1060, 48, 17572, 2, 1067, 48, 17579, 2, 1067, 48, 17579, 2, 1069, 48, 17581, 2, 1069, 48, 17581, 2, 1067, 95, 17579, 5, 1065, 48, 17577, 2, 1065, 48, 17577, 2, 1064, 48, 17576, 2, 1064, 48, 17576, 2, 1062, 48, 17574, 2, 1062, 48, 17574, 2, 1060, 95, 17572].
In this case there are no interarrival times of length zero and this tokenization is slightly less compact than arrival-time tokenization. But if we were to exlend the length of each note to a full beat, the interarrival-Lime tokenization becomes:
X0:43 [34024, 1060, 50, 17572, 1060, 50, 17572, 1067, 50, 17579, 1067, 50, 17579, 1069, 50, 17581, 1069, 50, 17581, 1067, 100, 17579, 1065, 50, 17577, 1065, 50, 17577, 1064, 50, 17576, 1064, 50, 17576, 1062, 50, 17574, 1062, 50, 17574, 1060, 100, 17572]
And in this case, interarrival-time tokenization is slightly more compact.
D A Prior over Music Infilling Controls
We propose three types of anticipation, i.e., distributions over events to condition on as controls. First we propose span anticipation, whereby we anticipate all tokens a given span in order to explicitly promote the model's ability to fillin-the-middlle. Second, we propose instrumend anlicipation, whereby we anticipate all tokens except for a specified instrumental part, supporting a workflow whereby supplemental instrumental parts are generated
complement a pre-existing ensemble. And third, we propose random anticipation, whereby we randomly anticipate events at some fixed rate to accommodate broader possible range of user-specified anticipation patterns.
We apply apply these patterns of anticipation to training data according to the following distributions:
Span anticipation. We randomly anticipate consecutive subsequences of events spanning ō seconds, at an exponential rate A. We fix À = .05, and at interarrival times Exp(A) we anticipate the events Buy where j= = min : > 4 + 83.
Instrument anticipation. For an event sequence with J unique instrument parts, we uniformly sample j (I,...,Je - 1} and randomly sample j instrumental parts without replacement. We anticipate all events in the sequence associated with these j parts.
Random anticipation. We uniformly sample a rate r e (0.1..... 0.9} for each event sequence. We randomly anticipate an r fraction of events the sequence.
We balance the overall training distribution using the following mix: 10% without anticipation (standard autoregressive training), 10% with span anticipation, 40% with instrument anticipation, and 40% with random anticipation.
These anticipation patterns facilitate interaction with an anticipatory music transformer via user-specified control sequences. For example, the accompaniment task evaluated in Section 4 is modeled by instrument anticipation when j 1. Note that the prior over infilling controls facilitates much more general interaction patterns than the prompt continuation and melodie accompaniment tasks studied Section 4. We defer further study of the capabilitics of anticipatory infilling models to future work.
To train anticipatory infilling model, we augment the training dataset using the distribution of anticipation patterns specified above. We perform these augmentations during preprocessing, resulting in an augmented dataset derived from the original data. In particular, augment the Lakh MIDI datasct by factor of 30. Using the prior distribution described above, this augmented dataset contains (i) 3 copies of the original dataset, verbatim; (ii) augmented copies of the dataset with different random anticipated spans; (iii) augmented copies of the dataset with different randomly anticipated instrument subsets for each event sequence; and (iv) 12 augmented copies of the dataset with different randomly anticipated events at different rates for each event sequence.
24

---

Published in Transactions on Machine Learning Research (04/2024)
Table 5: The number of tokens in the Lakh MIDI dataset, using tokenizations described Section 2.1.
Encoding
Train
Validation
Test
Overall
arrival
1,741,830,881 1,612,129,280
123,785,048 746,06 114,519,040
125,050,497 115,409,920 1,842,058,240
1,990,665,930
interarrival
Augmenting by a factor of 30 results in an arrival-time encoded Lakh MIDI train set of approximately 52B tokens. For training examples of length 1024 and batch size 512, this results in approximately one epoch per 100,000 optimization steps. For models trained with larger step counts, we take multiple passes over the augmented dataset. Given the relatively small size of the Lakh MIDI dataset, anticipation may be of interest as a regularization technique (see Appendix K for some evidence of possible regularizing effects attributable to anticipation). In this case, it could be fruitful to (i) increase the augmentation factor further during training, and (ii) place smaller weight training without anticipation, to minimize the amount of duplication of the original (unaugmented) dataset within the augmented dataset.
E
Details of the Lakh MIDI Dataset
The Lakh MIDI dataset includes no metadata, so we have little fine- graengrained or quantitative insight into the origins and contents of this dataset. Based randomly sampling of the dataset's contents, observe that it contains many arrangements of modern pop music, transcripts of classical western compositions, and original compositions (with varying degrees of quality).
Preprocessing Out of the 178,561 sequences in the Lakh MIDI dataset, we were able to successfully parse 174.046 sequences." We discard 7032 sequences that are shorter than 100 events or 10 seconds length. We also discard 40 event sequences that longer than hour in length (inspection reveals that many of these sequences corrupt). Finally, discard 2227 sequences that contain more than 16 unique instrument parts: representing music with more than 16 parts as MIDI (for syuthesizing outputs) requires multiplexing multiple instruments onto MIDI channels; we avoid this complexity by simply excluding very large ensembles. The resulting dataset consists of 164,747 event sequences. We split this dataset in to train, validation, and test splits according to the leading hexadecimal digit of each file's MD5 hash:
Train: hashes 0-d, 144,202 event sequences, 7827 hours of music.
Validation: hash e, 10,212 event sequences, 555 hours of music.
Test: hash f, 10,333 event sequences, 561 hours of music.
We tokenize the dataset using the arrival-time and interarrival-timo encodings described in Section 2.1. As seen in Table 5, the interarrival-time lokenization is slightly more compact. To illustrate the long, variable- rate nature of symbolic music event sequences, we plot the distributions of sequence length (Figure 2) and event rate (Figure 3) for the Lakh MIDI validation split.
F Measuring Cross-Entropy in Bits per Second
Bocause the arrival-time tokenization (Definition 2.2) and the interarrival-time tokenization (Defintion 2.3) describe nearly equivalent information, the log-loss of models trained using either encoding can be meaning- fully compared via unit conversion. To make comparison agnostic to encoding, we report losses in units of bits per second: this is the total log-loss of the test set, divided by the number of seconds of music in the test set. Concretely, given per-token loss L reported in nats per token, conversions for interrarival time
We parse Midi files using the Mido library: https://githaa.ccc//iii//aiid wotps////.ommmmm/idd....oo
25

---

Published in Transactions on Machine Learning Research (04/2024)
1.25
1.00
Density
0.75
0.50
0.25
0.00
10a
104
10a
Length in Tokens (log10 scale)
Figure 2: The distribution of sequence lengths calculated for the arrival-time tokenized Lakh MIDI validation split. Mean sequence length is 12,071 tokens, with standard deviation of 9711 tokens.
1.5
Density
1.0
0.5
0.0
10
102
103
Tokens per Second (log10 scale)
Figure 3: The distribution of instantancous tokens/sccond calculated for the arrival-time tokenized Lakh MIDI validation split. Mean instantaneous tokens/second for the Lakh MIDI dataset is 68 with a standard deviation of 51 tokens/second.
and arrival time encodings of the test set defined in Section E are:
bps (L) L log(2) X teees__tokeussrraattttttttt x 3600)),
(28)
bpsinterarrival (L) = L x log(2) x belest_tokennspncarrvaaaa// (teaaiiivaavttttttatteeeeeeeeeee hours x 3600)).
(29)
Concretely, for the Lakh MIDI dataset, from Table 5 have cast_tokensarie text_ text_tokens, tokens test_
125,050,497,
15,409,920, and the length of the test set (independent of encoding) is test_hours hours 560.98 hours. The remaining factors convert units from nats to bits (the log(2) factor) and from hours to seconds (the 3600 factor). This normalized form of the log-loss is analogous to the bits per dimension loss commonly reported for image generation (Salimans et al., 2017) and the bits per beat loss for musical scores (Thickstun al., 2019).
Truncations during tokenization (see Section 2.1) result in slightly different information content between arrival-time tokenized sequence and the corresponding interarrival-time tokenized sequence. Under arrival- time tokenization, 0.05% of note durations in the Lakh MIDI dataset exceed 10 seconds, and are truncated (to 10 seconds). Under interarrival-time tokenization, a total of 3947 interarrival times (0.00%) in the Lakh MIDI dataset exceed 10 seconds, and are truncated (to 10 seconds). Finally, 0.09% training examples under arrival-time tokenization exceed 100 scconds in length, and are discarded from the tokenized dataset. These instances are sufficiently rare that they exert a negligible effect on the information content (and therefore bits per second comparisons) between the arrival-time and interarrival-time tokenized sequences.
26

---

Published in Transactions on Machine Learning Research (04/2024)
Table Model Configurations.
Small
Medium
Large
Architecture Hyperparameters
Layers
12
24
36
Attention Heads
12
20
Hidden Dimensions
768
1024
1280
Sequence Length
1024 lokens
Residual Dropout
0.1
Embedding Dropout
0.1
Attention Dropout
0.0
Weight Decay
0.1
Optimizer Hyperparameters
Max Learning Rate
0.0006
0.0003
0.0002
Optimizer
AdamW (Loshchilov & Hutter, 2018)
($1, e)
(0.9, 0.999, 1 x 10-$)
Batch Size
512 sequences (= 524,288 Tokens) 1000 steps (== 50M tokens)
Warmup
Learning Rate Schedule
Cosine decay (no restarts) (Toshchilov Hutter, 2017)
Gradient Clipping
Clipping above V
(Rae al., 2021)
Training Resources
Throughput (tokens/second)
690,000
260,000
1 140,000
Throughput (seconds/iteration))
0.76
2.02
3.74
Hardware
Google TPU v3-32 pod slice
G Hyperparameters and Resources
All models are parameterized by standard, decoder-only causally masked transformers with GeLU non- lincarities (Hendrycks & Gimpel, 2016). The models implemented in Jax (Bradbury et al., 2018) and trained Google TPU v3 hardware. We observe that large-scale pseudorandom number generation in Jax is slow, and therefore eschew the standard altention dropout regularization. The models are optimized using AdamW (Loshchilov & Hutter, 2018). The learning rate schedule consists of a 1000 step linear warmup to a maximum lcarning rate, followed by a single cycle of cosine decay (Loshchilov & Hutter, 2017) over the remaining sleps a final learning rate of zero. Following Chinchilla compute-optimality recommendations, we train each model for number of steps that approximately proportional the model's size (Hoffmann et al., 2022). Configuration details of the models and optimization are presented in Table 6.
Most of these models were trained TPU v3-32 pod slices, which in practice approximately equivalent to GPU machine with 8 NVIDIA A100's. Training throughput for each model configuration using v3-32 is reported in Table 6. Conversions of these throughput statistics to wall-clock estimates of training time for the models featured in Section 4 are shown in Table 7. Thus, the total training time on v3-32's for the models featured this paper was approximately
1121 hours = 3 x 21 hours + 2 x 169 hours + 56 hours + 112 hours + 448 hours + 104 hours.
(30)
In addition to training the models featured in this paper, substantial additional TPU hours were consumed during the development phase of this research. We crudely estimate that total TPU-hours consumed for this work were approximately 3-5 times the hours reported in Equation (30).
27

---

Published in Transactions on Machine Learning Research (04/2024)
Table 7: Estimated wall-clock training time for the Small, Medium, and Large model configurations described in Table 6, using a Google TPU v3-32 pod slice.
Config
Training Steps
Hours of Training
Small
100k
Small
800k
169
Medium
100k
56
Medium
200k
112
Medium
800k
448
Large
100k
104
H Details of the FIGARO Music Transformer Baseline
We compare our models to the implementation of Music Transformer (Huang et al., 2018) described by von Rütte et al. (2023). We use the official public implementation for training and sampling from this model. 4 While both our models and this baseline are based on the Transformer architecture and trained on Lakh MIDI, there are at several notable distinctions. The FIGARO Music Transformer architecture implements the relative attention mechanism proposed by (Huang et al., 2018). The FIGARO Music Transformer inputs are tokenized using REMI encoding that accounts for metrical structure (Huang & Yang, 2020). The FIGARO Music Transformer is also smaller-scale model than the other models considered in this paper: six-layer transformer (approximately 30M parameters) trained for 100k iterations on sequences of length 256. Some of these factors (e.g., model scale, sequence length) clearly favor the models presented in this paper; others (e.g,, relative attention) may favor the FIGARO model. With so many uncontrolled variables, we caution against drawing conclusions about individual engincering choices the design of the FIGARO Musie Transformer versus the models proposed in this paper.
While pre-trained checkpoints of the FIGARO models available, the training and evaluation splits used for the pre-trained checkpoints are incompatible with the splits defined in Section E. Therefore, re-train our own version of the model using hashes 0-d as the training split. A comparison of our re-trained model and the reference model checkpoint are presented in Table 8. For definitions and discussion of the evaluation metrics compared here, see von Rütte et al. (2023). Our version of the FIGARO model matches or slightly outperforms the reference model on these metrics.
Neither arrival-time tokenization (Definition 2.2) FIGARO's REMI encode all the nuances of music described in, e.g., MIDI file. Because FIGARO explicitly models the metrical structure of music, we prompl using a fixed number of measures (three bars) rather than fixed amount of time: this is six seconds of prompt material for music in 4/4 time when quarter=120. We select three-bar prompts for the study in the range of 4-6 seconds, thus excluding music with a very fast or slow tempo. To create fair comparisons between music composed by humans, FIGARO, and the Anticipatory Music Transformer, we apply the following procedure for constructing prompts:
Human compositions: we encode human compositions (initially expressed as MIDI) using the FIGARO tokenizer, and then re-encode these samples using our own tokenizer.
FIGARO samples: we encode prompts using the FIGARO tokenizer, and re-encode completions of these prompts generated by FIGARO using our own tokenizer.
Anticipatory Music Transformer: we encode prompts using the FIGARO tokenizer, re-encode the prompts using
tokenizer, and generate completions from own models using these re- encoded prompts.
All music thus restricted to the musical vocabulary of our arrival-time tokenization.
In Table 9, report the quantitative metrics proposed by von Rütte et al. (2023) using 800 20-sccond continuations of b-rar prompts, for both the FIGARO Music Transformer, and our Medium Anticipatory
28

---

Published in Transactions on Machine Learning Research (04/2024)
Table 8: Quantitative evaluation metrics for the FIGARO implementation (von Rütte et al., 2023) Music Transformer (Huang et al., 2018) trained on the FIGARO training data split, compared to the same model trained using the split defined in Section E.
Train Split
Tt
C
TS 1
ND
Pf
V
D f
8c f
Ag
(FIGARO split)
0.191
0.048
0.751
2.192
0.563
0.153
0.312
0.306
0.510
Hashes 0 d
0.207
0.050
0.770
1.523
0.564
0.158
0.289
0.305
0.517
Table 9: Quantitative evaluation metrics for the FIGARO implementation (von Rütte et al., 2023) of Music Transformer (Huang et al., 2018) compared to a Medium Anticipatory Model (Row in Table 1).
Model
I
C
ND
P 1
1
Sc f
Sg 1
von Rütte et al. (2023) Anticipatory
0.833
0.364
0.574
0.751
0.573
0.619
0.257
0.836
0.392
0.529
0.755
0.593
0.659
0.259
Music Transformer (Row 8 in Table 1). While the Anticipatory Music Transformer marginally outperforms the FIGARO Music Transformer in all categories, it is not clear that we can draw a strong conclusion from these results. The reported metries were designed to measure reconstruction quality: how faithfully does a generated output reproduce the original? These metrics make less sense for evaluating open-ended generation: we expect. that generated contimuations would be different from the originals. One hypothesis for the consistent out-performance of the Anticipatory Music Transformer is that music is often self-similar, and perhaps this model is better able to capture these self-similarities of music.
A Baseline Autoregressive Infilling Algorithm
Algorithm describes the baseline autoregressive infilling algorithm evaluated in Table 3. Without the ability to anticipate future controls, this algorithm proceeds by sampling from the model until the time of the sampled event time exceeds the time of the next control. At this point, insert this control (and any other controls prior to the sampled event) into the sequence prior to the sampled event. We then proceed to continue sampling from the model.
Algorithm 2: Autoregressive Sampling (Baseline)
Parameters: Autoregressive model p with context length M
Input Output
Time-localized controls un: (monotone increasing in time) A generated sequence a1:N+K
ag f SEP: + 1; k - 0: do
// special sequence separator event // Index tracks position in the generated sequence // Index k tracks position in the control sequence
Sample e ~ p(-la;-
// Sample an event from the model // Get the time t of the event // While there are controls before time
t + >-ime(ee) while Time(u:) <t; do
a; + Us i +-i+11 1: k- + 1
// Anticipate control uy at index
// Advance to index i // Consume control up
end
a; + e;
// Append the newly sampled event
i --i+1 1
// Advance to index i +
while a, # SEP:
return a
// Index 1-1 is N + K
29

---

Published in Transactions on Machine Learning Research (04/2024)
Tell us which audio excerpt is more musical.
Please headphones in quiet environment if possible.
The first few seconds of music are the same for each excerpt.
The remainder excerpts may or may not computer generated.
Indicate which excerpt you find more conventionally musical.
If both excerpts have similar musicality, indicate that they are equally musical.
You may listen the clips multiple times.
Excerpt
Excerpt
0:20 / 0:20
0:20 0:20
Musicality
Excerpt A more musical
Excerpts A and equally musical
Excerpt B Is more musical
Submit Query
Figure 4: The interface used by evaluators assess the relative musicality of paired music clips.
This algorithm is naive because the model cannot condition on upcoming controls until after the time at which they occur. In the most optimistic scenario, because the model a good predictor of events, we hope that it will assign high probability the upcoming control events and therefore that the model will generate events that are consistent with (reasonable, unsurprising) controls. But ultimately there is entropy in the music process, and the model will necessarily sometimes generate events that are inconsistent with upcoming controls. To make matters worse, these inconsistencies are then written into the history and the model makes subsequent predictions conditioned on these mistakes; this has tendency to compound the errors over time, creating highly dissonant music.
A more subtle instance of this failure of compounding errors is the algorithm's 's tendency to double the control events. It is relatively common for the model will exactly predict an upcoming control event. In this case, copies of the event are written into the history. When subsequent controls inserted into the history, the model exhibits tendeney perpetuate this doubling, generating new events that copy every control that insert into the history, like vexing child copying sibling's every word.
J Details of Human Evaluation
Workers chosen to evaluate generated music were selected from a pool of crowd workers on the Amazon Me- chanical Turk platform, according to the qualification procedure described below. Evaluators provided with the interface shown in Figure 4 and instructed to judge the relative musicality of two 20-second music clips. We presented the music as audio, synthesized using Apple's DISMusicDevice sound system. Each pair of clips begins with the same prompl: three bars for the prompted completion task and five seconds for the accompaniment task. Evaluators were instructed to judge which clip is more musical. Based on feedback from pilot study, we clarified in the detailed instructions that we are interested musicality conventional sense. We allowed evaluators indicate that the two clips are equally musical, avoiding a forced choice between the two elips.
30

---

Published in Transactions on Machine Learning Research (04/2024)
Prompt (5s)
Continuation (15s)
Prompt and Melody
Prompt Continuation
Generation
Accompaniment
+
Generation
Figure 5: Visualizations of 20-second music clips. Each rectangle indicates a musical event with an onset time, duration (width), and pitch (height). Colors indicate distinct instrumental parts. For the accompaniment task, events in the blue instrumental part are used as control events. Top: five-second prompt followed by the original continuation of only the melodic instrumental line (highest; blue). Middle: the five-second prompt followed by generated auloregressive continuation, ignoring the original melodic line. Bottom: the prompt followed by generated anticipatory accompaniment of the original melodic instrumental line.
All prompts and melodies used for evaluation are sampled randomly from the test set. A notable confounder in this ovaluation is that the Lakh MIDI contains many popular, recognizable songs. Study participants remarked that they recognized the origin of certain prompts. In these cases, we instructed the participants to 'Please try to rate the clips based on their musicality rather than recognition." However, may be difficult to aside knowledge of the canonical completion in these cases.
Details of qualification, prompt continuation, and accompaniment evaluations described below. For illustration of the prompt continuation and accompaniment tasks, see Figure 5. For complete pairwise evaluations of models in the prompt continuation task, see Table 10. For complete pairwise evaluations of algorithms in the accompaniment tasks, see Table 11. For each task, we collected evaluations from three unique workers; we defer an analysis of worker agreement to future work.
Qualification Workers who correctly identified music composed by humans as more musical than random retrieval in at least 4/5 comparisons were deemed qualified to participate in subsequent model evaluations: 15 out of 20 workers who participated in qualification advanced to the main prompt continuation and accompaniment evaluations. Workers performed total of 100 comparisons.
Prompt Continuation We compare continuations of four models: (i) the Small interarrival-time model (Row 1, Table 1) (ii) the Small anticipatory model (Row Table 1) (iii) the large anticipatory model (Row 5, Table 1) and (iv) the FIGARO Music Transformer. We also compare to (v) human compositions (skyline). For each of 50 prompts, we create 50 x (2) 500 pairwise comparisons between continuations (as well the bascline and skyline) we asked three human evaluators to indicate which clip is more musical, or that the clips are equally musical. Workers performed a total of 1500 comparisons.
Accompaniment We compare accompaniments using (i) anticipatory autoregressive sampling (Algo- rithm 1) versus (ii) baseline autoregressive sampling. We also compare to (iii) baseline completions randomly sampled from the test set and (iv) skyline human compositions. When sampling without anticipation, we
31

---

Published in Transactions on Machine Learning Research (04/2024)
insert events from the melody into the conditional history of the model once generation passes them; we describe this modified sampling procedure formally in Appendix I. For each of 20 three-bar prompts and single-part continuations, we generated accompaniments from each model. For each of the 20 x (5) 120 pairwise comparisons between continuations (as well as the baseline and skyline) we asked three human evaluators to indicate which clip is more musical, or that the clips are equally musical. Workers performed total of 360 comparisons.
Table 10: Human evaluation of paired completions of 3-bar musical prompts generated by different algo- rithms, and human compositions. P-values are reported using a Wilcoxon signed rank test. Row numbers reference Table 1.
Model A
Model
Wins (A)
Ties
Wins (B)
p-value
Human Composition
Medium (Row 8)
77
29
44
0.0027
Music Transformer
119
18
13
2.806 x 10 10-20
Small (Row 3)
84
23
43
0.0002
Small (Row 1)
103
16
31
4.976 10 10-10
Medium (Row 8)
Music Transformer
95
24
31
1.187 x 10-08
Small (Row 3)
65
27
58
0.528
Small (Row 1)
96
17
37
3.122 x 10-7
Music Transformer
Small (Row 3)
36
16
98
8.509 x 10
Small (Row 1)
46
18
86
0.0005
Small (Row 3)
Small (Row 1)
82
17
51
0.0071
Table 11:
Human evaluation of paired 15-second accompaniments generated by different models, and human- composed accompaniments. P-values are reported using a Wilcoxon signed rank test.
Algorithm A
Algorithm B
Wins (A)
Ties
Wins (B)
p-value
Human Composition
Anticipatory
11
31
18
0.194
Autoregressive
45
10
5
1.542 x 10
Retrieval
52
6
2
1.017 x 10 10-11 11
Anticipatory
Autoregressive
47
6
7
5.230 x 10-8
Retrieval
45
11
4
4.709 x 10-9
Autoregressive
Retrieval
33
12
15
0.009
Training Optimization Logs
Figure 6 and Figure 7 plot estimates of the train set and test-set losses over the course of optimization for the arrival-time models considered in Section 4. Losses are computed every 10,000 steps from logged model checkpoints; the intermediate checkpoints for of these models (as well as the small interarrival-time model) are available on request.
Training loss is significantly lower than test for all models, evidence of some amount of overfitting during many epochs of optimization on the Lakh MIDI dataset. Nevertheless, observe better test set performance for larger models, trained longer, indicating that we have not completely saturated performance MIDI dataset using this scale of models and computational resources. That said, the relative test set loss improvements vs train set improvements when we increase the model size from Small to Medium are much larger than the relative gains of increasing the model size from Medium to Large: compare train vs. test loss of the Small (Row 3) Medium (Row 6) and Large (Row 9) models 100k steps. This might suggest that we are approaching the point of diminishing returns for scaling the compute (steps) and size of models trained Lakh MIDI, and that. more data would be needed to effectively train substantially larger models.
the Lakh
32

---

Published in Transactions on Machine Learning Research (04/2024)
For the Small models, observe that autoregressive training results in slightly better test-set performance 100,000 optimization steps. But at 800,000 steps the situation reverses, and the anticipatory model performs slightly better. We suspect that this is evidence of anticipatory training having a regularizing effect on the optimization. The Small, 800k step anticipatory model also exhibits some training instability in these plots. We observed similar instabilities during training the other anticipatory and auloregressive models, but they do not appear at the 10,000-step gramularity pictured here. We undertook no measures adjust for these instabilities, simply letting the optimizations run completion without intervention.
Small Autoregressive (Row 2)
Small Anticipatory (Row 3)
Small Autoregressive (Row 4)
Small Anticipatory (Row 5)
Medium Anticipatory (Row 6)
1.00
Medium Anticipatory (Row 7)
Medium Anticipatory (Row 8)
Large Anticipatory (Row 9)
0.9
0.8
0.7
0.6
1
10
Figure 6: Mean pok-token train set log-loss Luoken (in nats) of models, estimated every 10,000 steps over the course of training. The estimates are computed using a 1 1/100 subset the train set. The per-token loss Lroken related the event loss reported in Table by the relationship Luoken = 1e3(L))/3.
33

---

Published in Transactions on Machine Learning Research (04/2024)
Small Autoregressive (Row 2)
Small Anticipatory (Row 3)
Small Autoregressive (Row 4)
Small Anticipatory (Row 5)
Medium Anticipatory (Row 6)
Medium Anticipatory (Row 7)
Medium Anticipatory (Row 8)
Large Anticipatory (Row 9)
1.00
0.9
0.8
104
10a
Figure 7: Mean per-token test set log-loss Leoken (in nats) of models, estimated every 10,000 stops over the course of training. The estimates are computed using a 1/10 subset of the test set. The per-token loss Lroken is related the event loss reported in Table by the relationship Lioken L/3
34

---

Published in Transactions on Machine Learning Research (04/2024)
L
Model Card
Table 12: Model Card (Mitchell ct al., 2019) AAticiiptoory Music Transformer.
Model Details
Organization Developing the Model
Stanford Center for Research Foundation Models
Model Date
June 2023
Model Type
Autoregressive Causal Transformer See Section
Additional Modeling Details
License
A Apache License, Version 2.0
Correspondence
Intended Use
Primary Intended Uses
Collaborative co-composition between human composer and Anticipatory Music Transformer. The role of the anticipa- tory model in this collaboration could include, e.g., infilling tedious/low-enttnee details (productivity enhancement) and suggesting possible continuations (creative ideation).
Primary Intended Users Oft-of-Scope Uses
Artists, musicians, and composers.
Long-Context Gencration. These models cannot generate full-length song structures without human control. The models have context length of 1024 tokens (331 events). At 68 to- kens/second (the average for Lakh MIDI; see Appendix E) this corresponds to approximately 15 seconds of context. Models conditioned on more than 331 events will only use the most recent 331 events (including anticipated events) to predict the next event.
Music Metadata.
These models do not explicitly model or generate metadata, including: metrical structure, key signa- ture, tempo, note-value (eighth-note, quarter-note etc.).
Extended Music Vocabulary. These models generate se- quences with a narrow vocabulary of notes, instruments, and timings. They do not model or gencrate other aspocts music, including: dynamics, articulations, or lyrics.
Factors
Western Bias
These models are trained on the Lakh MIDI datasot, a col- lection of predominantly Western music. See Section A.3 for further discussion.
Metrics
Automatic Metrics
Next-event perplexity (defined in Table 1) and bits per seconds (defined in Appendix F).
Human Evaluation
Pairwise human preferences between generated music and ref- erence compositions.
Decision Thresholds
For human evaluation, we generated samples from anticipatory models using nucleus sampling with p 0.95. See Section 4 for further discussion.
35

---

Published in Transactions on Machine Learning Research (04/2024)
Approaches to uncertainty and vari- ability
We report p-values for pairwise comparisons between music generated by different models and ground truth music using the Wilcoxon signed-rank test. Due to computational constraints, we do not account for variability in the model training process, such as datasel splits or the random seed for optimization.
Datasets
Training Data
The d splits of the Lakh MIDI dataset, augmented using anticipation (see Section 3) with the prior distribution over controls described in Appendix D.
Validation Data
The split of the Lakh MIDI dataset.
Test data -fii----Diiitttiinn Data Preprocessing
The f split of the Lakh MIDI dataset.
We do not evaluate out-of- of-distribution performance. Preprocessing and filtering of the Lakh MIDI dataset de- scribed in Appendix E.
Motivation
We chose to work with the Lakh MIDI dataset because it is the largest collection of symbolic music data currently in use by the machine learning community.
Quantitative Analyses
Aggregated Analysis
Our analysis of aggregate results based automatic metrics and human evaluation are presented in Section 4.1 and Sec- tion 4.2 respectively. Key findings include:
Anticipatory training does not interfere with autore- gressive model performance, as measured by perplex- ities of comparable anticipatory and autoregressive models.
Accompaniments generated by an Anticipatory Music Transformer have similar musicality to ground truth accompaniments according to human evaluators.
Disaggrogated Analysis
We do not perform a disaggregated analysis of the Anticipatory Music Transformer. One obstruction to conducting such an analysis is a lack of metadata associated with the Lakh MIDI dataset.
Ethical Considerations
Labor Displacement
We are broadly concerned by the transient disruptions of la- bor markets caused by the introduction of new productivity- enhancing and automative lechnologies. See Section A.1 for discussion of the possible disruptive effects of generative music models on the creative cconomy.
Copyright
The Lakh MIDI dataset contains large quantities of copyrighted music. The copyright status of models trained on this data- and music sampled from these models is an open legal ques- tion. See Section A.2 for further discussion.
36


Reviews:
[{"review_id": "Ob8AqDK8Hd", "reviewer": "Reviewer_9MMW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\nThe tokenization and shifting design is a clever idea and its experiment results are compelling. \n\nThe code and pretrained model weights will be a useful contribution as well once they are released. \n\nWeaknesses\n\nThere are 3 key weaknesses: missing references, missing comparison, and unclear presentation. \n\nMissing references. \n\nHongyuan Mei, Guanghui Qin, Jason Eisner. 2019. Imputing Missing Events in Continuous-Time Event Streams. \n\nThis paper proposes a particle smoothing method that predicts unobserved events based on past observed and unobserved events as well as future observed events. \nThis smoothing method is similar to the proposed method except that it doesn't have the \"nearby\" inductive bias; the \"nearby\" bias seems very reasonable for music specifically. \n\nMissing comparison. \n\nThe above method is a natural baseline to compare with. \n\nUnclear presentation. \n\nThe paper is very hard to follow. The term \"point process\" is a little abused and the presentation relies upon unnecessarily complicated formulation (e.g., filtration). \n\nI think the presentation could and should be significantly simplified before it is published. \n\nPlease see Requested Changes for my detailed notes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a novel model---anticipatory music Transformer---for the task of infilling control. \nThe model relies on a novel tokenization and shifting technique, such that the generation of an accompaniment note will depend on not only past notes but also nearby future melody notes. \n\nThe model achieves compelling results on Lakh MIDI dataset, under automatic and human evaluation. \n\nAll code and pretrained model weights will be released.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please correct me if I am wrong. \n\nFirst, the term \"point process\" should describe the model, not the data. \nPrecisely, a point process is a generative model of sequences of time-stamped events. So phrases like the following are wrong: \n- \"point process consists of time-ordered events\" (Def-2.1)\n- \"generative models of a temporal point process\" (Sec-1)\n\nSecond, it doesn't seem to be necessary to emphasize \"point process\" because the key technique in this paper is to tokenize the sequence such that one can use LM objective, and there is no notion of \"intensity\" in the proposed method, which is a key element in point processes. \n\nThird, the part about \"stoping time\" seems unnecessarily complicated. Some stochastic calculus terms (e.g., \"filtration\") are introduced but do not really help readers understand the essence. \nAs I understand, \"stoping time\" is important because the predicted \"events\" may go out of the boundary, passing the \"controls\" that actually should happen after them. So the method must be aware of the \"stoping time\" during generation, is it right? \nI think this point can be made clear without involving any more sophisticated concepts (e.g., \"filtration\"). \n\nGenerally, I think the paper will be much shorter and clearer if the complication above can be resolved.", "broader_impact_concerns": "No ethical concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "AIunJKbjr1", "reviewer": "Reviewer_Wdtb", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The proposed modeling approach is interesting and novel. \n2. The paper is clearly written and the notations are well defined. \n3. Results suggest the proposed method is superior to the evaluated baselines.\n\n**Weakness:**\n1. It seems the authors designed a solution that is specifically tailored to the task of MIDI generation. It is not clear whether this will be of interest to the ML community at large. \n2. The comparison to the baseline methods considers human evaluation only.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a new method, denoted as anticipation, which allows the construction of a controllable generative model. The proposed approach is based on modeling several parallel sequences conditioned asynchronously. The authors presented results on symbolic music generation (MIDI) and showed that the proposed method reached superior results than the standard auto-regressive modeling. Additionally, the authors present results on accompaniment MIDI generation and show that the proposed method is superior to both auto-regressive and retrieval-based baseline.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Questions to the authors:**\n\n1. Can the authors present results / demonstrate the applicability of the proposed approach on other tasks than MIDI generation? As stated above, it seems the solution and input representations are heavily tailored to MIDI, which might not be of high interest to the ML community at large. It is not clear whether this approach can be applied to other domains/tasks as well. \n\n2. In the FIGARO paper [1], the authors reported objective metrics alongside human study. Can the authors additionally provide objective results? It would make it easier to compare with prior work and will make the paper stronger.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "t7ZRVQNIEU", "reviewer": "Reviewer_ynfy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* The authors' use of examples throughout section 3 were very helpful in illustrating how the interleaving of events occurs. \n* The authors provided detailed informaton about their methodology with regards to human evaluation.\n* The conclusion does a good job connecting the ideas in the paper to the broader idea of using locality to improve performance of machine learning models. I also appreciate the authors discussion of the importance of controllability for facilitating the adoption of generative music models by artists and creatives. \n\n### Weaknesses\n* I wonder why the authors did not compare to Music Transformer for the accompaniment task since that model (at least the original) can also be used for accompaniment generation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present anticipation, a novel encoding for temporal point processes to be used with autoregressive sequence modeling. They apply it to the task of infilling in symbolic music (demonstrating continuation and accompaniment generation capabilities).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Small grammatical error on page 3, third paragraph. \"This generalizes the span-infilling task--which *asks* us...\"\n2. The \"Related Work on Musical Accompaniment\" section is under 4.2, and there is another entire section (section 5) for related work on controllable generative modeling. It might make more sense to group all related work discussions under one section and differentiate within with subsections.", "broader_impact_concerns": "The authors comprehensively acknowledge the ethical concerns of generative music modeling in the appendix, to a much greater extent than I have seen in related literature.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Thank you for providing manuscript updates.  I had the reviewers take a look at the latest manuscript update, particularly the one negative reviewer.  This reviewer still has some concerns, namely about some of the technical writing, and maintained their lean reject recommendation.  The other two reviewers recommended accepting the paper.  I am going to paste the comment the negative reviewer made below regarding rigor in the manuscript:

"I am confused why the authors still want---so much---to relate [their method] to temporal point process.

Their new text in Related Work says: "Sequential Monte Carlo (SMC) samplers have been considered for infilling of both continuous temporal point processes (Mei et al., 2019) and sequence models (Lin & Eisner, 2018); the later more directly relates to our setting because discretizing time (Section 2.1) reduces point process modeling to sequence modeling."

However, the former (Mei et al., 2019) is on point processes while the latter (Lin & Eisner, 2018) is on discrete-time sequences (like languages). Then it is very strange that the authors relate their work to the latter but still calls their model a "point process". For this reason, I hold my opinion that the technical arguments are lack of rigor."

The reviewer also elsewhere noted that "a lot of unnecessary math complication (e.g., filtration) should be removed."  It would be very helpful to address these concerns, but I don't see this as fundamentally problematic for publishing the paper; I think it would just clarify the contributions and clean up the presentation.  I would like to see a minor revision addressing these issues; this should not require too much additional work.
Decision: Accept (poster)