<figure><img src="image_1.png" /></figure>

HAL
open science

# Expressive Piano Generation in Moment Form

Jingwei Liu

► To cite this version:

Jingwei Liu. Expressive Piano Generation in Moment Form. 2025. hal-05287026

HAL Id: hal-05287026

https://hal.science/hal-05287026v1

Preprint submitted on 27 Sep 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Expressive Piano Generation in Moment Form

Jingwei Liu

University of California San Diego

jil121@ucsd.edu

ABSTRACT

This paper presents a situated analysis and conceptual discussion of the ideas behind our generative AI systems as an addition to the technical report [1]. We will elaborate on the development of symbolic music data processing (piano-centric) in a historical line, and highlight the improvements we made and the intuitions and implications of such treatment. By extracting the expressivity of the performance to the limit (microscopic sensitivity to timing at the level of a few milliseconds; extreme contrast of dynamics among subsequent notes; sustain pedal events informing real performance and perceptual grouping), and combining the LSTM sequential structure with exquisite interdependent conditioning, we generate contemporary moment-form-like music with undirected motion, unpolished roughness, and extremely rich polyphonic textures, micro-timing expressions, and drastic dynamical and registral contrasts from a classical music corpus¹. The model behavior can be traced back to our data processing techniques and designing mechanisms of the neural networks. Such an analytical point of view is taken throughout the text, where we aim to demystify the generations from AI and apply a musical analytical framework to AI-generated music in the same way we analyze human-composed music.

# 1. DATA PROCESSING WITH EXPRESSIVITY

As for symbolic music generation, it was an extended score-based period. Music scores function as the medium for most people to read and understand music, especially classical music. Naturally, when it comes to artificial music encoding, the earlier works tend to represent musical notes solely with pitch and duration – the best notated values on the music staff – with fixed metrical grids and quantized note values (e.g. quarter note, 8th note, 16th note).

This representation poses limitations in multiple ways, including a lack of expressive marks such as dynamics and sustain, time inhomogeneity due to tempo differences, and a potential demand for human performers to interpret the generated scores. Among all the limitations, generating polyphonic music using sequential models is the biggest challenge. Most music is polyphonic – even the simplest

folk tones are always accompanied by harmonies. To generate score-based polyphonic music, we have to deal with simultaneous notes in a temporal sequence, even worse, simultaneous notes with undetermined numbers and inhomogeneous durations. Music researchers and engineers designed several ways to mediate this situation, such as to predict the number of notes before predicting the respective pitches and durations, and to slice all notes into a quantized minimal note value (e.g. 16th or 32nd) to count for simultaneous occurrences [2] [3]. These methods are more or less expedient. They are cumbersome and musically counterintuitive. For example, if a quarter note is divided into four 16th notes, the weights of these notes should decay in the polyphonic texture as the sound is dissipating after the initial attack.

## 1.1 Related Work

It was until [4] the expressivity of symbolic music data and direct generation of music performance bypassing score came into public view. The data processing methods proposed in [4] are adopted by the Performance RNN [5] and Music Transformer [6] – the models that marked the milestones of AI music generation. The expressive piano performance MIDI data is represented as

* 128 `NOTE_ON` events: starting a note with one of the 128 MIDI pitches.

* 128 `NOTE_OFF` events: ending a note with one of the 128 MIDI pitches.

* 100 `TIME人大移` events: representing forward time shifts in 10ms increments from 10ms to 1s.

* 32 `VELOCITY` events: corresponding to MIDI velocities quantized into 32 bins that set the velocity applied to all subsequent notes (until the next velocity event).

Furthermore, the sustain pedal control event is embedded into the note's duration as a delay to its `NOTE_OFF` event. These four predictive events are concatenated into a 388-dimensional vector which operates as a one-hot vector in training and inference.

The use of flexible real time in milliseconds and the abandonment of fixed metrical grids are groundbreaking in processing symbolic music data. Especially with `TIME人大移` events, the sequential models are no longer subject to monophonic music where the notes have to be concatenated one by one based on their durations. With the time shift, the next note is flexible to start anywhere after the current note's onset, either during its sustain or after its release. This treatment allows a *homogeneous treatment of monophony*

¹ Please access the code and generated samples at the GitHub page: https://github.com/Jovie-Liu/Expressive-Piano-MIDI-Generation

Copyright: ©2025 Jingwei Liu et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License 3.0 Unported, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
and polyphony as we note that there is no simultaneity in
real time. That is to say, for any two notes that are played
by a human performer, there is always a time discrepancy
between them, no matter how unnoticeable it is. It means
that, since there are no simultaneous notes, we can always
place the musical events in sequential order by their time
onsets, thus modeling any musical temporality using se-
quential models [7].

## 1.2 Our Approach

With breakthrough innovations, there is still room left to
improve the data processing and representations in Section
1.1. Firstly, we notice that the note pitches are represented
by 128 MIDI numbers as standard MIDI codecs. How-
ever, as the dataset characterizes solo piano performances,
there is a limited range of registers that doesn't cover ex-
tremely low or high frequencies. After running statistics of
the dataset, we derived the MIDI range [21, 108]. Predict-
ing within the 88 classes instead of default 128 presents a
more compact and reasonable range for generative outputs.

Secondly, in terms of time division, although the usage
of milliseconds captures certain expressivity in the data,
it still discards lots of nuanced information. The coarse
time step of 10ms from 10ms to 1s flattens the expressive
micro-timing that takes effects in a few milliseconds, and
the equidistant time division is also a suboptimal choice
as the predictive class for [10ms, 20ms] delineates larger
difference than the class [0.99s, 1s] in human perceptions
– we can more or less tell the difference of a delay of 10ms
or 20ms at the tactus level but we are much less sensitive
to the differences between 0.99s and 1s unless it's situated
in groove-based music relative to a steady pulse [8].

The same rule also applies to the duration. In our ap-
proach, we use DURATION instead of NOTE_OFF to rep-
resent the sustain period of the note. For notes with short
durations, small changes would be noticeable. However, if
the duration is large, e.g. 5s, an increment of 10ms makes
little difference. The property that perceptible changes in-
crease with the magnitude of the stimuli is known as the
Weber's Law.

Applying Weber's Law to processing the data gives Fig-
ure 1. Notice that the divisions follow the trend of We-
ber's law at large, but the proportions are regulated by the
data statistics to balance the occurring frequencies of each
class. Data processing with cognitive Weber's law cap-
tures human perceptual truth, which corresponds to a bet-
ter compression algorithm with less perceptual loss. This
method also draws inspiration from the mel-spectrogram
where the frequencies are divided into uneven bins based
on the cognitive precision of our auditory systems.

Thirdly, the treatment of VELOCITY in Section 1.1 evens
out the dynamic contrast among notes by setting all subse-
quent notes with the same velocity value until changed. To
highlight the contrast and dynamic changes, we compute
the velocity difference between subsequent notes and cat-
egorize this VELOCITY_CHANGE term based on Weber's
law as depicted in Figure 1.

Finally, unlike previous works that embed pedal informa-
tion within the note sustain to "free up resources to focus
on better prediction of other events" [4], we propose to pre-
dict the binary pedal status as a separate event. The pedal
states inform a higher-level grouping structure of the piece

<figure><img src="image_3.png" /><figcaption>**Figure 1.** Categorical Divisions of Time Shift, Duration, and Velocity Change based on Cognitive Weber's Law and Data Statistics.</figcaption></figure>

– the notes encompassed by one sustain press are percep-
tually grouped together and naturally delimited from notes
in other pedal envelopes. Specifically as we employ the
LSTM neural networks as training models, the on/off (1/0)
pedal indicators might be particularly useful when the neu-
ral networks integrate information in determining the for-
get and update gates [9].

In summary, our data processing gives rise to five events:

* 88 `PITCH` events: MIDI pitches in range [21, 108].

* 105 `TIME SHIFT` events: 105 time shift classes as depicted in Figure 1.

* 120 `DURATION` events: 120 duration classes as depicted in Figure 1.

* 47 `VELOCITY_CHANGE` events: 47 velocity change classes as depicted in Figure 1.

* 2 `PEDAL` events: binary pedal status in {0, 1}.

## 2. MODELING FIELD INTERDEPENDENCIES

From Section 1 we can tell that for temporal music mod-
eling in the symbolic domain, it's inevitable to deal with
multiple events simultaneously. Unlike natural language
processing tasks, where there is only one token to sample
at each step and the tokens align linearly, music models are
more complicated due to their multi-argument nature.

### 2.1 Related Work

The works [4, 5, 6] mentioned in Section 1.1 and lots of
subsequent works adopted the data representation proposed
in [4]. The four constructed MIDI events are concatenated
into a single vector (in this case 128+128+100+32 = 388
dimensions) and sampled as a holistic one-hot vector. This
processing raises several systematic problems. The most
obvious one is the scheduling of events. As the four events
are sampled in a single one-hot vector, at every inference
step, it's possible to sample any of the events with a certain value. In other words, these events are not note-based. For example, it's possible to sample a NOTE_ON at C4 without a NOTE_OFF to turn it off; it's also possible to sample a NOTE_OFF before NOTE_ON, to sample a TIME SHIFT without NOTE_ON, or to sample consecutive VELOCITY events without note progressions. All these situations can be remedied by post-processing, but the lack of internal regularity and unspecified event relationships unavoidably leads to certain chaos and inaccuracy in terms of learning. The standard way to construct a one-hot vector of multiple events is to multiply all the possibilities, namely $128 \times 128 \times 100 \times 32 = 52428800$ dimensions, which is too sparse to train.

In [10], the author proposed to sample each note-based musical event separately, where the multiple outputs of a note are called fields. In this work, there are six fields related to each note. Treating these musical events as properties of notes and tying them to specific note values is more musically self-consistent. However, as the output fields are sampled independently, the model fails to capture the interdependency of note properties (e.g. pitch and duration of a note are decided separately without the knowledge of each other), thus resulting in inferior inference capacity [10].

## 2.2 Our Approach

We propose a model that predicts musical events as properties tied to each note, and sample each of the field or argument autoregressively. According to Section 1.2, we have five output arguments for each note, and each of them forms a separate one-hot vector for sampling. To model their interdependency, we apply the chain rules of conditional probabilities:

$$p(x_n, x_t, x_d, x_v, x_p) = p(x_n) \cdot p(x_t|x_n) \cdot p(x_d|x_n, x_t) \cdot p(x_v|x_n, x_t, x_d) \cdot p(x_p|x_n, x_t, x_d, x_v)$$

The model structure is shown in Figure 2. Notice that we used five LSTMs to predict each argument separately instead of integrating them into one holistic model. Such attempts were made in [7] where we designed a sequential model with local attentional devices that process multi-arguments conditioning. However, this convoluted structure disturbs the BPTT flow, thus impeding the training and convergence, and modeling multiple outputs with one sequential flow also compromises the autonomy of musical elements [1]. With the model in Figure 2, we can draw analogy from communicating channels [11] to view the parallel LSTMs as autonomous but interactive musical channels propagating in time [1].

# 3. NARRATIVE VS. MOMENT FORM: A NEW MUSIC PERSPECTIVE

The e-Piano Competition dataset [4] is a classical piano corpus, which means most of the pieces fall under the formalism or structuralism styles of composition where the music structure is regarded either hierarchical (or Schenkerian) [12] or following a well-delineated narrative plot [13]. However, the data processing and training architecture of our choice doesn't follow this convention. The JND (Just noticeable difference) based division reinvents musical time

<figure><img src="image_4.png" /><figcaption>Figure 2. Capture Interdependency with Parallel Conditioning LSTMs in Multi-argument Sequential Models.</figcaption></figure>

– an innovation that redirects our musical attention from metric/rhythm perception to microscopic expressive timing, and breaks the metrical structure of the original compositions by the abandonment of grids and uneven divisions of musical time.

On the other hand, the internal mechanism of linear sequential model also breaks the hierarchical or narrative compositional structure that emphasizes wholeness, self-regulation, and teleological progression. The conventional linear time in music is linked to process and the fulfillment of a listener's expectations in succession [14]. In contrast, non-linearity features discontinuity [15], non-processive, and a focus on in-the-moment gestures instead of the motion or succession [14].

Moment form was first articulated by Stockhausen in explicating his compositional procedure in Kontakte, an epic electronic music work. Moment form verticalizes time, renders every moment a now, avoids functional implications between moments, and avoids climaxes [15]. Moment form is a compositional approach in which a narrative overall line is deliberately avoided. It refers more to
the constituent sectional elements than a formal plan [16]. The rise of new music in the twentieth century and the dissolution of tonality, rules, and Eurocentrism all removed the priority of elaborate structure and plan in music (e.g. free improvisations). This spared room for avant-garde and experimental music to emerge – the music that exploits the extremes of instruments, registers, dynamics, and expressions (e.g. extended techniques).

If you listen to the samples generated by our model, you might be surprised to find these unorthodox elements in the generated sound, given it’s trained with pure classical music. The limited memory capacity of LSTMs and its linear structure makes it difficult to capture the narrative and hierarchical architecture of classical compositions, which results in moment-form-like progressions that refuse to participate in a globally directed narrative curve. The music is endowed with local consistencies and is self-contained in every single moment. Besides, the expressive micro-timing and drastic contrast in dynamics and register in processed data adds unpolished roughness, expands sound spectrum, and enriches polyphonic textures of the generated music, which are highlighted as outstanding features in the analysis of widely regarded extraordinary contemporary music work [17].

## 4. REFERENCES

[1] J. Liu, "Expressive Music Data Processing and Generation," *arXiv preprint arXiv:2503.11896*, 2025.

[2] F. T. Liang, M. Gotham, M. Johnson, and J. Shotton, "Automatic Stylistic Composition of Bach Chorales with Deep LSTM." in *ISMIR*, 2017, pp. 449–456.

[3] N. Boulanger-Lewandowski, Y. Bengio, and P. Vincent, "Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription," *arXiv preprint arXiv:1206.6392*, 2012.

[4] S. Oore, I. Simon, S. Dieleman, D. Eck, and K. Simonyan, "This time with feeling: Learning expressive musical performance," *Neural Computing and Applications*, vol. 32, pp. 955–967, 2020.

[5] I. Simon and S. Oore, "Performance RNN: Generating Music with Expressive Timing and Dynamics," https://magenta.tensorflow.org/performance-rnn, 2017.

[6] C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, N. Shazeer, C. Hawthorne, A. M. Dai, M. D. Hoffman, and D. Eck, "Music Transformer: Generating Music with Long-Term Structure," *arXiv preprint arXiv:1809.04281*, 2018.

[7] J. Liu, "Expressive MIDI-format Piano Performance Generation," *arXiv preprint arXiv:2408.00900*, 2024.

[8] V. Iyer, "Embodied mind, situated cognition, and expressive microtiming in African-American music," *Music perception*, vol. 19, no. 3, pp. 387–414, 2002.

[9] S. Hochreiter and J. Schmidhuber, "Long short-term memory," *Neural computation*, vol. 9, no. 8, pp. 1735–1780, 1997.

[10] H.-W. Dong, K. Chen, S. Dubnov, J. McAuley, and T. Berg-Kirkpatrick, "Multitrack music transformer," in *ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*. IEEE, 2023, pp. 1–5.

[11] S. Dubnov, V. Gokul, and G. Assayag, "Switching machine improvisation models by latent transfer entropy criteria," in *Physical Sciences Forum*, vol. 5, no. 1. MDPI, 2023, p. 49.

[12] F. Lerdahl and R. S. Jackendoff, *A Generative Theory of Tonal Music, reissue, with a new preface*. MIT press, 1996.

[13] F. E. Maus, "Music as narrative," *Indiana theory review*, vol. 12, pp. 1–34, 1991.

[14] I. Arauco, "The Time of Music," in *College Music Symposium*, vol. 30, no. 1. JSTOR, 1990, pp. 153–156.

[15] J. D. Kramer, "Moment form in twentieth century music," *The Musical Quarterly*, vol. 64, no. 2, pp. 177–194, 1978.

[16] J. Dack, "Karlheinz Stockhausen's kontakte and narrative." *Kontakte and Narrativity*, 1999.

[17] R. Adlington, "The music of Rebecca Saunders: Into the sensuous world," *The Musical Times*, vol. 140, no. 1868, pp. 48–56, 1999.