Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
Authors: Tingyu Zhu
Haoyu Liu
Ziyu Wang
Zhimin Jiang
Zeyu Zheng
Venue: ICML 2025 poster (2025)
Keywords: Symbolic Music Generation
Guided Diffusion Models
Controllable Generation
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: We introduce Fine-Grained Guidance for AI-driven diffusion-based symbolic music generation, enabling precise, real-time, and interactive control over pitch and harmony with strong theoretical and empirical results.

Abstract: Developing generative models to create or conditionally create symbolic music presents unique challenges due to the combination of limited data availability and the need for high precision in note pitch. To address these challenges, we introduce an efficient Fine-Grained Guidance (FGG) approach within diffusion models. FGG guides the diffusion models to generate music that aligns more closely with the control and intent of expert composers, which is critical to improve the accuracy, listenability, and quality of generated music. This approach empowers diffusion models to excel in advanced applications such as improvisation, and interactive music creation. We derive theoretical characterizations for both the challenges in symbolic music generation and the effects of the FGG approach. We provide numerical experiments and subjective evaluation to demonstrate the effectiveness of our approach. We have published a demo page to showcase performances, which enables real-time interactive generation.

Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
Tingyu Zhu "I
Haoyu Liu
Ziyu Wang²
Zhimin Jiang? Zeyu Zheng'
Abstract
details, leading to artifacts distortions in the generated content, such as noticeable inconsistencies distortions in videos. (2) Limited controllability: Obtaining precise con- trol over the generated content to align it with the intent of the user remains a significant challenge. For instance, correcting specific distortions in a generated video while keeping the rest of the scene unchanged is difficult with current diffusion model frameworks.
Developing generative models to create con- ditionally create symbolic music presents unique challenges due to the combination limited data availability and the need for high precision in note pitch. To address these challenges, we intro- duce efficient Fine-Grained Guidance (FGG) approach within diffusion models. FGG guides the diffusion models to generate music that aligns more closely with the control and intent of ex- pert composers, which critical to improve the accuracy, listenability, and quality of generated music. This approach empowers diffusion mod- els excel in advanced applications such as im- provisation, and interactive music creation. We derive theoretical characterizations for both the challenges in symbolic music generation and the effects of the FGG approach. We provide nu- merical experiments and subjective evaluation to demonstrate the effectiveness of our approach. We have published demo page to showcase performances, which enables real-time interac- tive generation.
These limitations are exacerbated in situations where data is scarce, which is often the case in domains like symbolic music generation, where symbolic music data is limited due to copyright constraints and the effort needed to create data. Additionally, unlike image generation, where the in- accuracy of single pixel may not significantly affect over- all quality, symbolic music generation demands high pre- cision, especially in terms of pitch. In many musical and tonal contexts, even single incorrect or inconsistent note can be glaringly obvious and disturbing.
provide more contexts, symbolic music generation is a subfield of music generation that focuses on creating mu- sie in symbolic form, typically represented as sequences of discrete events such notes, pitches, rhythms, and du- rations. These representations are analogous to traditional sheet music or MIDI files, where the structure of the mu- sic is defined by explicit musical symbols rather than audio waveforms. Many recent works in symbolic music gener- ation are based on diffusion models; see Min et (2023), Wang al. (2024) and Huang al. (2024) for example.
1. Introduction
Diffusion models (Ho et al., 2020) have consistently demonstrated effectiveness across wide range of gen- erative tasks, particularly in image and video generation (Rombach et al., 2022). Despite success, diffusion mod- els face some limitations. (1) Imprecise detail generation: Diffusion models often struggle with accurately producing
Following this branch of work, address the precision and controllability challenges in diffusion-based symbolic music generation by incorporating fine-grained guidance into the training and sampling processes. While soft con- trol schemes such as providing chord conditions may fail to ensure detailed pitch correctness, we propose to enhance chord conditioning with hard control method integrated into the sampling process, which guarantees the desired tonal correctness in every generated sample.
'Equal contribution 'University of California, Berkeley, USA NNew York University, New York, USA 'Touka Technologies. Correspondence Haoyu Liu <haoyuliu @ berkeley.edu>. Zeyu Zheng <zyzheng @ berkeley.edu>>.
Proceedings of the 42nd International Conference on Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Our results in this work are summarized as follows:
'The demo page is available at https//huqjianduzhuu- code. Saaaua/ioooG--fffsss--muiee complete source code https://gituubcco//uajjaaaaau code/FGG-music-codee
We theoretically and empirically char- acterize the challenge of precision in symbolic music generation
Motivation:
also release the

---

Efficient Finc-Grained Guidance for Diffusson Model Based Symbolic Music Generation
Methodology: We incorporate fine-grained harmonic and rhythmic guidance to symbolic music generation with diffusion models.
given by human to generate the remaining tracks based on temporal structure of that track. Wu & Yang (2021) utilized Transformer based VAE model to realize fine-grained style transfer over full songs.
The developed model is capable of generating music with high accuracy in pitch and con- sistent rhythmic patterns that align closely with the user's intent.
Functionality:
Image Inpainting, Image inpainting with diffusion mod- els has advanced rapidly, offering valuable insights for our task. our setting, harmonic conditions define con- strained or masked regions, and the model must com- plete the rest-analogous to inpainting. Recent diffusion- based methods have enabled fine-grained control during both training and sampling. For instance, Lugmayr et al. (2022) introduced a post-conditioning strategy that adapts the reverse diffusion process to reconcile known and miss- ing regions without retraining, albeit with increased in- ference time. Xie et al. (2023) combined shape and text prompts enable precise, user-guided inpainting via joint training and sampling design. Corneanu et al. (2024) im- proved sampling efficiency by conditioning directly in the latent space, supporting faster and semantically coherent completions. Inspired by these works, we adapt the idea of context-aware, guided completion to symbolic music, enabling controllable generation over structured time-pitch domains.
Effectiveness: We provide both theoretical and empir- ical evidence supporting the effectiveness of our ap- proach.
1.1. Related Work
Symbolic Music Generation. Symbolic music genera- tion literature can be classified based on the choice of data representation, among which the MIDI token-based repre- sentation adopts a sequential discrete data structure, and is often combined with sequential generative models such as Transformers and LSTMs.
To leverage well-developed generative models for sym- bolic music, Huang et al. (2018) introduced a Transformer- based model with a novel relative attention mechanism de- signed for symbolic music generation. Subsequent works have enhanced the controllability of symbolic music gen- eration by incorporating input conditions. For instance, Huang & Yang (2020) integrated metrical structures to en- hance rhythmic coherence, Ren et al. (2020) conditioned on melody and chord progressions for harmonically guided compositions, and Choi et al. (2020) encoded musical style to achieve nuanced harmonic control. These advancements have contributed to more interpretable and user-directed music generation control.
Controlled Diffusion Models. Multiple works in con- trolled diffusion models related to work in terms of methodology. Specifically, we adopt the idea of classifier- free guidance in training and generation, see Ho & Sali- mans (2021). To control the sampling process, Chung et al. (2023), Song et al. (2023) and Novack et al. (2024) guide the intermediate sampling steps using the gradients of a loss function. In contrast, Dhariwal & Nichol (2021), Sa- haria et al. (2022), Lou & Ermon (2023) and Fishman al. (2023) apply projection and reflection during the sampling process to straightforwardly incorporate data constraints. Different from these works, we design guidance for inter- mediate steps tailored to the unique characteristics of sym- bolic music data and generation. While the meaning of a specific pixel in an image is undefined until the entire im- age is generated, each position on a piano roll corresponds to a fixed time-pitch pair from the outset. This new context enables us to develop novel implementations and theoreti- cal perspectives on the guidance approach.
To better capture spatio-temppral harmonic structures in music, researchers have adopted diffusion models with var- ious control mechanisms. Min et al. (2023) incorporated control signals tailored to diffusion inputs, enabling con- trol over melody, chords, and texture. Wang et al. (2024) extended this by integrating hierarchical control for full- song generation. To further enhance control, Zhang et al. (2023) and Huang et al. (2024) leveraged the gradual de- noising process to refine sampling. Building on these ap- proaches, our work addresses the remaining challenge of precise control in real-time generation.
In parallel to diffusion-based approaches, a body of work on general symbolic music ceneration-using models such as RNNs, GANs, and VAEs-has also explored mecha- nisms for achieving precise user-controllable generation. Early work on symbolic generation already explored user- steerable conditioning. Meade et al. (2019) retrofitted an RNN method with human-intern ciman-interpretable controls such as note density and pitch range limits. Dong et al. (2017) pro- posed method that conditions a GAN model on one track
2. Background: Diffusion Models for Piano Roll Generation
In this section, we introduce the data representation of pi- ano roll. We then introduce the formulations of diffusion model, combined with an application on modeling the pi- ano roll data.
2

---

Efficient Finc-Grained Guidance for Diffusson Model Based Symbolic Music Generation
Data Representation of Piano Rolls. Let M E {0. 1)xxH be piano roll segment, where H is the pitch range and L is the number of time units in frame. For example, H can be set as 128, representing pitch range 0 127, and I. as 64, representing a 4-bar segment with time signature 4/4 (4 beats per bar) and 16th-note resolu- tion. Each element My. of M (1 E [1. L]], E [1, H]) takes value 0 or where My. 1/0 represents the pres- ence/absence of note at time index I and pitch h.? Since standard diffusion models are based on Gaussian noise, the output of the diffusion model is continuous random ma- trix X E IR LXH which is then projected to the discrete piano roll M by Ma = 1(Xm 2 1/2], where 1(-} stands for the indicator function.
3. Methodology: Fine-Grained Guidance
While generative models have achieved significant success in text, image, and audio generation, the effective model- ing and generation of symbolic music remains a relatively unexplored area. One challenge of symbolic music gener- ation involves the high-precision requirement in harmony. Unlike image generation, where a slightly misplaced pixel may not significantly affect the overall image quality, an "inaccuratel"" generated musical note can drastically dis- rupt the harmony, affecting the quality of piece.
In this section, we present control methodology that can preciscly achieve the desired harmony. Specifically, we de- sign a fine-graines conditioning and sampling control, al- together referred to as Fine-Gres Grained Guidance (FGG) that leverage the characteristic of the piano roll data.
Formulation of the Diffusion Model. To model and the
generate the distribution of M, denoted PM. the Denoising Diffusion Probabilistic Modeling (DDPM) formulation (Ho et al., 2020). The objective of DDPM training. with specific choices of parameters and reparam- eterizations, is given as
3.1. Fine-Grained Conditioning in Training
We first introduce fine-grained conditioning in training, which serves as the foundation of the more important sam- pling control in the next subsection 3.2.
A())|E= es(X.. (1)
We train a conditional diffusion model with fine-grained harmonic (C, required) and rhythmic (R, optional) condi- tions, which provided to the diffusion models in the form of piano roll M We provide illustration of Muond (C, R) and (C) via examples in Figure I and Figure 2, respectively. The mathematical descriptions are provided in Appendix B.
where is deep neural network with parameter 0. More- over, according to the connection between diffusion models and score matching (Song Ermon, 2019), the deep neural network Ey can be used to derive an estimator of the score function (X,) = Vx, log p. (X,). Specifically, 8, (X,) can be approximated by E X t)/v1- ât.
With the trained noise prediction network En, the reverse sampling process can be formulated as (Song et al., 2021a):
M (C, R)
value-1
value-))
chord Camajor
X, VI (x,
X,-1 v
Va
(2)
+V1 + ---1--0f-oll orE
where or are hyperparameters chosen corresponding to equation 1, and e, is standard Gaussian noise at each step. Going backward in time from X7 ~ (VO0,), I), the process yields the final output X,, which can be converted into piano roll M(X.).
According to Song et al. (2021b), the DDPM forward and backward processes can be regarded as discretizations of the following SDEs:
rhythm oaser pos non unact
dx, (())x,dl= + (1)X BB(JaW,,
(3)
Figure I. An illustrative example of ondd (C, R) with both con- ditions.
[1 [(1)x,+ + B(tss; dt + B(t)a,,,
dx,
"To enable the model to handle both rhythm+chord and chord- only conditions, we use negative values to indicate the absence of rhythmic input when only harmonic conditions are provided, avoiding misinterpretation of Os and 1s as active constraints. Em- pirically, removing this distinction (i.c., still using 0 and when
(4)
Thhis is slightly simplified representation model for the pur- pose of theoretical analysis, the specified version with implemen- tation details is provided in Section 5.1
3

---

Efficientt Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
M Mounlgg (C)
value
In is expected satisfy My, O, for all (1, w(t,
other words, for Xo we need
chord Camajor
A(1, h) E w(L, P P(R..n > 1/2)
(5)
Note that in the backward sampling equation 2 that derives ,, from X', have for the first term (Song et al., 2021a; Chung et al., 2023)
Xx 1 anEo (X,,))
"predicted X,"
Va
XXX -- 1,..,, 1.
Edit Intermediate-step Outputs of the Sampling Pro- cess. The primary cause of inaccurately generated notes is the estimation error of the probability density of Xo. which in turn affects the corresponding score function (X,,). The equivalence (X(, - (X,,,) therefore inspires us to project XX to the K-- constrained domain R4*H Wx x by adjusting the value of (x, t). This adjustment is interpreted as an adjustment of the estimated score. Here Wx is the set of matrices, connected to the set of positions (on the matrix) w((, K.) by
rhythm condition
Figure 2. An illustrative example of (C) with harmonic conditions only.'
ā
\Wx
3.2. Fine-Grained Control in Sampling Process
We first provide a rough idea of the harmonic sampling control. To integrate harmonic constraints into our model, employ temporary tonic key" signatures to establish the tonal center. Our sampling control mechanism guides the gradual denoising process to ensure that the final generated notes remain within specified set of pitch classes. This control mechanism removes or replaces harmonically con- flicting notes, maintaining alignment with the temporary tonic key.
Wx x R Lx 13(, h) (CI 1/2).
Specifically, at each sampling step i, we replace the guided noise prediction éØ(X,
Xx, E) with EØ (X t) such that
E (X,,)) arrgmnn
|| - Eo Ex(,,,))1
VI āre (xi
X,
E BLXH
S.l.
Preliminaries. Recall that piano roll segment M E {0, 1) where 1 E [1, [ is the time index, and h E [11 H] is the pitch index. For given chord condition se- quence C, let IC denote the corresponding key sequence. For example, when the C major chord appears as the chord condition at time index I, we would expect (C(I) to contain the pitch classes of the major scale". We note that C is essentially different from K., where C describes chord se- quences and is provided as condition for generation, and K. is a restriction of "allowed" pitch classes for sampling refinement.
The element -ise formulation of Eg (X,, t) is given as fol- lows, with calculation details provided Appendix A.
εo.un (X,,t) = (((,)) & w((;)))) Eo,th +1 +1((1,0 wx(()))
max (oo.((X,,)) Vi-
X1ihh
Vā.
2
Plugging the adjusted noise prediction Eg (X,, into equa- tion 2, we derive the adjusted X,- The sampling process is therefore summarized as the following Algorithm 1.
Let Kl; == {1, ((; VL), denote the undesired pitch positions the piano roll M. The generated piano roll M
Note that at the final step = 0, the noise correction di- rectly projects X, L H (Wk- ensuring the probabilis- tic constraint
rhythmic condition is not provided) led to 8-15% drop in chord accuracy (c.g., direct chord accuracy drops from 0.485 to 0.421, and chord similarity drops from 0.767 to 0.705), highlighting the importance of explicitly encoding missing rhythmic information.
Theoretical Property of the Sampling Control. A nat- ural concern is that enforcing precise fine-grained control over generated samples may disrupt the learned local pat- terns. The following proposition 1, proved in A.2, provides an upper bound that quantifies this potential effect and ad- dress the concern.
As clarification, instead of assigning one single key piece or big section, here we refer to each key associated with the temporary tonic.
'We note that the correspondence between C and KC is in fact flexible, and he designed by the user of the model. More discussion is provided in the next section
4

---

We provide an intuitive explanation using statistical rea- soning. Consider a piano roll segment, represented as a Efficient Finc-Grained Guidance for Diffusson Model Based Symbolic Music Generation
Algorithm DDPM sampling with fine-grained harmonic control
and the model's internal learning from the training data, e.g., melodic lines. This theoretical insight aligns with our empirical observations, which is presented in the "Mode Change" section of the demo page.
Input: Input parameters: forward process variances St, ae = -IT St, backward noise scale oc, key sig-
nature guidance K
Output: generated piano roll M {0, 13
4. Challenges for Uncontrolled Symbolic Music Generation Models
X, I);
for t - 1...,, do
Compute guided noise prediction (x,)) Perform noise correction: derive éø (X,, t) using equa- tion
In the previous section 3, we present our FGG method that guarantees the precision of generation. But why is it mean- ingful to provide such guarantee in the task of symbolic music generation? Why it hard for models to self-ensure harmonic precision without having the hard sampling con- trol? We use Section 4 to answer these questions. These discussions further motivate and justify the importance of the FGG method.
Compute X, by plugging the corrected noise
()) into equation? 2
<end
7 Convert X, into piano roll M
8 return outpul
In the rest of this section, we focus our discussion to tonic- centric genres. Although not covering every aspect of mu- sic, it still spans a wide range of genres that are deeply embedded in everyday life, including tonic-centric New Age music, light classical music, and tonic-focused movie soundtracks. Such genres rely heavily on harmony, i.e., the simultaneous sound of different notes that form a cohesive entity in the mind of the listener (Müller, 2015).
Proposition 1. Under the SDE formulation in equation 3 and equation 4, given an early-stopping time to?, if
Ex, pp Aaalllll",,,,, tX -Eo(X, 8
(9)
for all t, where e" (X,,t) is the optimal solution of the DDPM training objective (1), then we have
8 5
T
B(t)
Using the concept of temporary tonic key signatures we discussed in the previous section, we focus our discussion the presence of out-of-key notes* in generated music. In the tonic-centric genres, out-of-key notes are uncommon, and produce noticeable dissonance, if not having a "reso- lution". We often notice that out-of-key notes are usually perceived merely as mistakes when appearing in generative model outputs, as demonstrated by some examples on our demo page.
KL Pta)
20
2
dt,
lio 3(s)ds
1
8
T
B(t)
KL(Pro Pea)
dt,
30 e Sio 3(e)ds
where Pto is the distribution of Xto in the forward process, Pto is the distribution of XT, generated by the diffusion sampling process without noise adjustment, and pro is the distribution of Xto generated by the -raiind noise ad- justment.
We aim explain why the existence of out-of-key notes is issue for diffusion--base symbolic music generation models in the tonic-centric genres. Specifically, we ex- plain the following phenomenon: Suppose G is diffusion model trained to generate tonic-centric genres. In the tar- get data distribution, out-of-key notes appear at a small rate P(O) > 0. These out-of-key key notes are carefully managed (by expert composers) in the training set. However, when out-of-key notes appear in the generated samples of G, they often lack an appropriate resolution and are more likely to be perceived negatively. Why does the model often fail to learn this nuance?
Proposition provides upper bounds for the distance be- tween the controlled distribution and the uncontrolled dis- tribution, as well as between the controlled distribution and the ground truth. We remark that, our method can shape the output towards specific tonal quality. This can be for ex- ample using the Dorian scale as the key signature sequence K. to shape the generated music towards the Dorian mode (a tonal framework not present in the training data), where the generated distribution ş with fine-grainen noise adjustment fundamentally different from the ground truth distribu- tion p. Nevertheless, Proposition 1 guarantees substan- tial overlap between the two distributions and p, demon- strating a
FFor instance, a Gb note is considered as out-of-key in G> major context. Admittedly the inference of temporary tonic key is even more vague than chord recognition, due to the flexibility of harmony. However, in the following discussion, we assume that the temporary tonic key is specified.
interplay between external control
We adopt the carly-stopping time to avoid the blow-up of function, which standard in many literature (Song Er- mon, 2020; Nichol & Dhariwal, 2021)
5

---

Efficient Finc-Grained Guidance for Diffussin Model Based Symbolic Music Generation
random variable M. Suppose we are interested in whether this segment contains out-of-key note (denoted as event {O)) and whether that note is eventually resolved within the segment (denoted as event {R, O}). In our training data, almost every out-of-key note is resolved, meaning the probability of unresolved out-of-key note is close to O, i.c., P(R, 0) Rs O.
5. Experiments
In this section, we present experiments to demonstrate the effectiveness of our iie-grained guidance approach. We additionally create a demopage" 10 for demonstration, which allows for fast and stable interactive music creation with user-specified input guidance!!, and even for generating music based on tonal frameworks absent from the training set.
Now, we examine the probability in the generated music. The key question then is whether the generative model also learns to keep P( O) small. The following proposition 2 leverages analysis of statistical errors to show that P(R, O) can decrease slowly as the dataset size increases.
5.1. Numerical Experiments
We present numerical experiments on accompaniment gen- eration given both melody and chord generation, or sym- bolic music generation given only chord conditions. We focus on the former one as it provides a more effective basis for comparison. Due to page limits, we put the results and more detailed explanation of the latter one in Appendix D.3. For the accompaniment generation task, we compare with two state-of-the-ar baselines: 1) WholeSongGen (Wang et al., 2024) and 2) GETMusic (Lv et al., 2023).
Proposition 2. Consider generating piano roll M from continuous random variable X. i.e., given i.i.d. data {X')== ~ Px, let {= be given by Mi, = 1 2 1/2]. Denote the model for estimating the distribution of X as px. We have 3 C > 0 such that Vn,
inf sup ((M P(R,0)2C-n Die - P(R, O);
Px PxEPs
(10)
where P is the probability associated with the generated data px.
5.1.1. DATA REPRESENTATION AND MODEL ARCHITECTURE
The generation target X is represented by piano-roll ma- trix of shape x L x 128 under the resolution of 16th note, where L represents the total length of the music piece, and the two channels represent note onset and sustain, respec- tively. In our experiments, we set L = 64, correspond- ing to a 4-measure piece under time signature 4/4. Longer pieces can be generated autoregressively using the inpaint- ing method. The backbone of our model is 2D UNet with spatial attention.
The proof of proposition 2 is provided in appendix A.3. The term eP; is the supremum taken over the search space of the continuous generative model', and inf de- notes the best possible realization of the model. The mini- max formulation standard in works that discuss statistical convergence of generative models (Fu al., 2024).
The theoretical insights presented in proposition 2 demon- strate that the occurrence of unsolved out-of-key key note is of- ten unavoidable, and the decay rate of this error probability with respect to training set size n is slow O(n (i/(L((L))))) Thus, relying on the model itself for precision is challeng- ing for existing models, given the inherent scarcity of high- quality data and the slow decay rate of errors. There are two implications following this line: First, it would be im- mensely valuable to develop model that enjoys the abil- ity to implicitly learn contextually appropriate out-of-key notes (nevertheless, currently in our work we did not take this path). Second, with the fact that symbolic music gener- ation requires an exceptional level of precision, it is worth- while develop methods that enable the model function as well-controlled collaborative tool to aid human com- posers.
The condition matrix M is also represented by a piano roll matrix of shape 2 x L x 128, with the same resolution and length as that of the generation target X. For the ac- companiment generation experiments, we provide melody as an additional condition. Detailed construction of the condition matrices are provided in Appendix D.I.
5. 1.2. DATASET
We use the POP909 dataset (Wang et al., 2020a) for train- ing and evaluation. This dataset consists of 909 MIDI pieces of pop songs, each containing lead melodies, chord progression, and piano accompaniment tracks. We exclude 29 pieces that are triple meter. 90% of the data are used to train our model, and the remaining 10% are used for
'The exact formulation of Ps is given in appendix A.3. While real life distribution classes associated with generative models are more complicated and difficult to analyze, Ps essentially captures their characteristics, and is therefore comparable to them. This type of simplification while maintaining core characteristics ap- pears be standard in works that provide theoretical insights (Fu et al., 2024).
'Seee https:/huaqjandduzuooooom music/. We note that slow performance may result from Huggingface resource limitations and network latency.
Thhe format of user-specified input guidance limited within the constrained piano roll format, as is demonstrated the paper.
6

---

Efficient Finc-Grained Guidance for Diffussin Model Based Symbolic Music Generation
evaluation. In the training process, we split all the midi pieces into 4-measure non-overlapping segments (corre- sponding to L = 64 under the resolution of a 16th note), which in total generates 15761 segments in the entire train- ing set. Training and sampling details are provided in Ap- pendix D.2.
recognized chord of the generated output exactly matches that of the ground truth. However, since not all mismatches reflect equal harmonic deviation-for instance, C major is harmonically close to Cmaj7 but far from B major major-direct direct accuracy may fail to reflect the nuanced similarity between chords.
To address this, we further assess chord progression similarity. We divide each accompaniment into non- overlapping 2-measure segments and encode them into 256-dimensional latent space using a pre-trained disentan- gled VAE (Wang et al., 2020b). Cosine similarity is then computed between corresponding segments from the gen- erated and ground truth progressions. Table I reports the average direct accuracy and average latent similarity, along with their 95% confidence intervals. The results demon- strate that our method significantly outperforms all base- lines in chord accuracy.
5.1.3. TASK AND BASELINE MODELS
We consider accompaniment generation task based on melody and chord progression. We compare the per- formance of our model with two baseline models: 1) WholeSongGen (Wang et al., 2024) and 2) GETMusic (Lv et al., 2023). WholeSongGenn is a hierarchical music gener- ation framework that leverages cascaded diffusion models to generate full-length pop songs. introduces a four-level computational music language, with the last level being ac- companiment. The model for the last level can be directly used to generate accompaniment given music phrases, lead melody, and chord progression information. GETMusic is a versatile music generation framework that leverages a dis- crete diffusion model to generate tracks based on flexible source-target combinations. The model can also be directly applied to generate piano accompaniment conditioning on melody and chord. Since these baseline models do not sup- port rhythm control, to ensure comparability, we will use the M (C) without rhythm condition in our model.
(3) Intersection over Union (IoU) Metrics. We evaluate the similarity between the generated and ground truth accom- paniments using two loU-based metrics: loU of Chords and loU of Piano Roll. For IoU of Chords, we first ap- ply the chord recognition method from Dai al. (2020) to both the generated and ground truth accompaniments. Each chord is then represented as 12-dimensional binary vector indicating the presence of pitch classes (C through B). We compute the IoU between the generated and ground truth pitch class sets at every 16th-note time step and report the average IoU across all time steps.
5.1.4. EVALUATION
We generate accompaniments for the 88 MIDI pieces in our evaluation dataset." 12 We introduce the following objective metries to evaluate the generation quality of different meth- ods:
For loU of Piano Roll, we represent each accompaniment as binary piano roll. The IoU is computed at each 16th- note time step by comparing the sets of active pitches in the generated and ground truth piano rolls. We then re- port the average loU across all time steps. 13 The results, presented in Table 1, show that our method consistently achieves higher IoU scores than the baselines, indicating closer alignment to the ground truth both the harmonic and note level.
(1) Percentage of Out-of-Key Notes First, for each method, we present the frequency of out-of-key notes by comput- ing the percentage of steps in the generated sequences con- taining at least one out-of-key note, where each step cor- responds to a 16th note. The results, presented in Table I, indicate that frequency of out-of-key notes in the baselines is roughly 2%.4%, equating to about 1-3 occurrences in 4-measure piece. In contrast, our sampling control method effectively eliminates such dissonant notes in the generated samples.
(4) Subjective Evaluation
To compare performance of our FGG method against the baselines (ground truth, WholeSongGen, and GETMusic). we prepared sets of generated samples, with each set containing the melody paired with accompaniments gen- erated by FGG, WholeSongGen, and GETMusic, along with the ground truth accompaniment. This yields a to- tal of 6 x 4 = 24 samples. The samples are presented in a randomized order, and their sources are not disclosed
(2) Direct Chord Accuracy and Chord Progression Simi- larity We evaluate harmonic consistency by comparing the chord progressions of the generated and ground truth ac- companiments. Chords are extracted using the rule-based recognition method from Dai et al. (2020). Direct chord accuracy is computed as the percentage of beats where the
13 "While exact agreement with the ground truth is not necessar- ily optimal -sincesiieeeeee a given melody may admit multiple valid ac- companiments- the IoU still serves as useful indicator of musi- cal quality. high-quality accompaniment is expected to align reasonably well with the expert-written ground truth, and thus should not deviate substantially.
12ehe WholeSongGen model from Wang et al. (2024) is also trained on the POP909 datasct. Our evaluation set a subset of their test set so there is no in-sample evaluation issue on their model.
7

---

Efficient Finc-Grained Guidance for Diffusion Model Based Symbolic Music Generation
Methods
% Out-of-Key Notes
Direct Chord Accuracy
Chord Similarity
IoU (Chord) 1
IoU (Piano Roll)
FGG (Ours)
0.0%
0.485 + 0.006
0.767 + 0.007
0.769 + 0.003
0.281 + 0.005
WholeSongGenn
2.1%
0.314 +: 0.006
0.611 # 0.010
0.618 + 0.004
0.107 + 0.003
GETMusic
3,5%
0,153 + 0.007
0.394 + 0.012
0.412 + 0.007
0.048 + 0.003
Table 1. Evaluation of the similarity with ground truth for all methods.
to participants. Experienced listeners assess the quality of samples in 5 dimensions: creativity, harmony (whether the accompaniment is in harmony with the melody), melodi- ousness, naturalness and richness, together with an overall assessment. The results are shown in Figure 3. The bar height shows the mean rating, and the error bar shows the 95% confidence interval. FGG consistently outperforms the baselines in all dimensions. For details of our survey, please Appendix F.
notes are corrected post-sampling by rounding the nearest in-key pitch.
Training control Remove Wrong Notes After Sampling: training control is applied, and out-of-key notes are corrected post-sampling by removing them.
Training Control Only:
only training control is used; no sampling-time controls are enforced.
5.1.5. ABLATION STUDY
No Control: neither training control nor sampling control are applied.
In this section, we conduct ablation studies to better il- lustrate the effectiveness of our FGG method. We aim to demonstrate the effectiveness of both the fine-grained training condition (training control) and the sampling con- trol. We also compare with simple rule-based post-sample editing The former leverages the structured gradual de- noising process of diffusion models, ensuring a theoretical guarantee of preserving the distributional properties of the original learned distribution. In contrast, the latter employs a brute-force editing approach that disrupts the generated samples, affecting local melodic lines and rhythmic pat- terns. The numerical results further validate this analysis.
We assess overall model performance using the same quan- titative metries in the previous section. The results are shown in Table 2. In general, fine-grained condition- ing (i.e., training control) leads to substantial improve- ments across all evaluation metrics, and adding sampling control further enhances performance. While rule-based post-sampling editing (e-g.. removing or rounding out-of- key notes) yields moderate gains, it is consistently outper- formed by fine-graine fine-grained sampling control method. Our approach fully leverages the structured, gradual denoising process of diffusion models, allowing the model to itera- tively correct or replace errors while preserving the coher- ence of the original learned distribution.
Moreover, we compare with so-called "Impainting" method, which treats the pixels where there is not supposed to be a note as 0 and inpaints the remaining pixels. This information is included by adding a mask channel in the training process. We still allow for the fine-grat iin-grrined condi- tioning training.
Moreover, our method outperforms the inpainting baseline across all evaluation metrics. Unlike our approach, the in- painting method introduces additional architectural com- plexity by requiring the model to handle an auxiliary mask channel that indicates which pixels to regenerate. This not only increases implementation overhead but also adds com- putational burden during both training and inference.
Specifically, we include the following variants in our abla- tion study:
Training and Sampling Control: our full method, which applies fine-grained conditioning during both training and sampling.
5.2. Empirical Observations
Notably, harmonic control not only helps the model elimi- nate incorrect notes, but also guides to replace them with correct ones. Such representative examples are presented in Appendix G. Moreover, samples generated from ablation conditions are available in Section 3 of our demo p5ge Across all ablations, we observed occasional occurrences of excessively high-pitched notes and overly dense note clusters.
out-of-key pixels are treated as O, and the remaining positions are inpainted based on fixed con- text.
Inpainting:
Training control + Round Notes Up/Down After Sampling: training control is applied, and out-of-key
"Specifically, we compare with two rule-based post-sample editing methods: 1) Rounding wrong notes up/down and 2) Re- move wrong notes.
The demo page is available at hupss//huajianduzuu- Soddegittuh.ooFG-dififii
8

---

Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
Subjective Evaluation with 95% Confidence Intervals
Average Rating
Mothods
WholeSongGen
GETMusic
Ground Truth
Crealivity
Hermony
Melodiousness
Richness
Figure 3. Subjective evaluation results music quality.
Methods
% Out-of-Key
Direct Chord
Chord
ToU
ToU
Notes
Accuracy
Similarity
Chord
Piano Roll
Training and Sampling Control
0.0%
0.485
0.767
0.769
0.281
+0000
10.007
1..003
10.005
Inpainting
0.0%
0.458
0.710
0.743
0.271
10.006
10.008
10.003
10.005
Training Control
0.0%
0.472
0.756
0.763
0.272
Round Notes Up/Down After Sampling
10.006
+0.007
+0.003
#0.005
Training Control
0.0%
0.482
0.763
0.767
0.277
Remove Wrong Notes After Sampling
+0.006
+0.007
+0.003
+0.005
Only
3.7%
0.465
0.748
0.757
0,270
Training Control
+0.006
+0.007
+0.003
+0.005
No Control
10.1%
0.112
0.378
0.378
0.072
10.004
10.007
10.004
=0.002
Table 2. Ablation study.
6. Limitations and Future Work
align with human judgments of musicality and creativity.
While our method achieves strong performance across mul- tiple metrics, several limitations remain. First, we adopt a 16th-note quantization scheme following Wang et al. (2024), which simplifies temporal representation but re- stricts rhythmic flexibility and precludes training data without explicit beat annotations. A promising future di- rection is to integrate our pitch-class-based control mecha- nism with approaches such as Huang et al. (2024), which introduce a dynamic dimension and utilize finer 10ms time quantization better capture expressive timing variations. Second, our method supports pitch class and rhythmic con- trol in the piano roll representation, but does not accom- modate more abstract forms probabilistic control. Fi- nally, we note that evaluation remains broader challenge across the field of symbolic music generation. Since musi- cal quality evaluation is inherently detailed and partly sub- jective, objective evaluation metrics such as rule-based and structural evaluation methods used in this work have inher- limitations in reflecting perceptual or creative aspects of music. This is key reason why many recent studies supplement objective evaluation with subjective human lis- tening evaluations. A valuable future direction is to develop improved automatic evaluation metrics that more faithfully
7. Conclusion
In this work, we apply fine-grained textural guidance (FGG) on symbolic music generation models. We provide theoretical analysis and empirical evidence to highlight the need for fine-grained and precise control over the model output. We also provide theoretical analysis to quantify and upper bound the potential effect of fine-grained control on learned local patterns, and provide samples and numerical results for demonstrating the effectiveness of our approach. For the impact of our method, we note that the FGG method can be integrated with other biffusion-based symbolic mu- sic generation methods. With a moderate trade-off of flexi- bility, the FGG method prioritizes real-time generation sta- bility and enables efficient generation with precise control.
Acknowledgements
The authors gratefully acknowledge Jinghai He, Ang Lv, Yifu Tang, Gus Xia, Yaodong Yu, Yufeng Zheng, anony- mous reviewers, area chairs, and the anonymous evaluators of this work's demos.
9

---

Efficientt Finc-Grained Guidance for Diffusion Model Based Symbolic Music Generation
Impact Statement
guidance: A sharp statistical theory. arXiv preprint arXiv:2403. 1968, 2024.
This paper presents work whose goal to advance the field of Machine Learning. There are range of potential soci- etal consequences of our work, none which we feel must be specifically highlighted here.
Gao, S., Zhou, P., Cheng, M.-M., and Yan, S. Masked diffusion transformer is a strong image synthesizer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23164-23113, 2023.
References
Ho, and Salimans, T. Classifier-free diffusion guidance. In NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Chang, H., Zhang, H., Barber, J., Maschinot, A., Lezama, J., Jiang, L., Yang, M.-H., Murphy, K., Freeman, W. T.,, Rubinstein, M., et al. Muse: Text-to-image generation via masked generative transformers. In International Conference on Machine Learning, pp. 4055-4075, 2023.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion prob- abilistic models. Advances in Neural Information Pro- cessing Systems, 33:6840-688 6851, 2020.
Chen, S., Chewi, S., Li, J., Li, Y., Salim, A., and Zhang, A. Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions. In International Conference on Learning Representations, 2023.
Huang. C.-Z. A., Vaswani, A., Uszkoreit, J., Shazeer, N. M., Simon, I., Hawthorne, C., Dai, A. M., Hoffman, M. D., Dineulescu, M., and Eck, D. Music transformer: Generating music with long-term structure. In Interna- tional Conference on Learning Representations, 2018.
Choi, K., Hawthorne, C.. Simon, I., Dinculescu, M., and Engel, J. Encoding musical style with transformer au- toencoders. In International Conference on Machine Learning. pp. 1899-199999 1908. PMLR, 2020.
Huang, Y., Ghatare, A., Liu, Y., Hu, Z., Zhang, Q., Sastry, S., Gururani, S., Oore, S., and Yue, Symbolic mu- sic generation with non- oindiffeerntiablee rule guided diffu- sion. In International Conference Machine Learning, pp. 19772-19797. PMLR, 2024.
Chung, H., Kim, J., McCann, M. T., Klasky, M. L., and Ye, J. C. Diffusion posterior sampling for general noisy inverse problems. International Conference on Learn- ing Representations, 2023.
Huang. Y.S.. and Yang. Y.-H. Pop music transformer: Beat-based modeling and generation of expressive pop piano compositions. In Proceedings of the 28th ACM international conference on multimedia, pp. 1180-1188. 2020.
Corneanu, C. A., Gadde, R., and Martínez, A. M. Latent- paint: Image inpainting in latent space with diffusion models. 2024 IEEE/CVF Winter Conference on Applica- tions of Computer Vision (WACV), pp. 4322-4331, 2024.
Karatzas, I. and Shreve, S. Brownian motion and stochas- tic calculus, volume 113. Springer Science & Business Media, 1991.
Dai, S., Zhang, H., and Dannenberg, R. B. Automatic anal- ysis and influence of hierarchical structure on melody, rhythm and harmony popular music. arXiv preprint arXiv:2010.07518. 2020.
Lin, S., Liu, B., Li, J., and Yang, X. Common diffusion noise schedules and sample steps are flawed. In Proceed- ings of the IEEE/CVF winter conference on applications of computer vision, pp. 5404-5411, 2024.
Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances Neural Information Pro- cessing Systems, 34:8780-8794, 2021.
Lou, A. and Ermon, S. Reflected diffusion models. In Inter- national Conference on Machine Learning, pp. 22675- 22701. PMLR, 2023.
Dong, H.-W., Hsiao, W.-. Yang, L.-C., and Yang, Y.-H. Musegan: Multi-track sequential generative adversarial networks for symbolic music generation and accompa- niment. In AAAI Conference on Artificial Intelligence, 2017.
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timo- fte, R., and Gool, L. V. Repaint: Inpainting using de- noising diffusion probabilistic models. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11451-11461 2022.
Fishman, N., Klarner, L., De Bortoli, V., Mathieu, E., and Hutchinson, M. J. Diffusion models for constrained domains. Transactions on Machine Learning Research (TMLR), 2023.
Lv, A., Tan, X., Lu, P., Ye, W., Zhang, S., Bian, J., and Yan, R. Getmusic: Generating any music tracks with a unified representation and diffusion framework. arXiv preprint arXiv:2305. 10841, 2023.
Fu, H., Yang, Z., Wang. M. and Chen, M. Un- veil conditional diffusion models with classifier-free
10

---

Efficient Finc-Grained Guidance for Diffusion Model Based Symbolic Music Generation
Meade, N., Barreyre, N., Lowe, S. C., and Oore, S. Ex- ploring conditioning for generative music systems with hummaninterprettbbee controls. In Proceedings of the 10th International Conference on Computational Creativity (ICCC), Charlotte, North Carolina, 2019.
Song, Y. and Ermon, S. Improved techniques for training score-based generative models. Advances in Neural In- formation Processing Systems, 33:12438-12244, 2020.
Song. Y., Sohl- Dickstiin,,,, J., Kingma, D. P., Kumar, A.,
Ermon, S., and Poole, B. Score-based generative model- ing through stochastic differential equations. In Interna- tional Conference Learning Representations, 2021b.
Min, L., Jiang, J., Xia, G., and Zhao, J. Polyffusion: A dif- fusion model for polyphonic score generation with inter- nal and external controls. Proceedings of 24th Interna- tional Society for Music Information Retrieval Confer- ence, ISMIR, 2023.
von Rütte, D., Biggio, L.,, Kilcher, Y., and Hofmann, T. Figaro: Controllable music generation using learned and expert features. In International Conference on Learning Representations, 2023.
Müller, M. Fundamentals of music processing: Audio, analysis, algorithms, applications, volume 5. Springer, 2015.
Wang. Z., Chen, K., Jiang, J., Zhang, Y., Xu, M., Dai, S., Gu. X., and Xia, G. Pop909: pop-song dataset for mu- sic arrangement generation. Proceedings of 21st Inter- national Society for Music Information Retrieval Con- ference, ISMIR, 2020a.
Nichol, A. Q. and Dhariwal, Improved denoising diffu- sion probabilistic models. In International Conference on Machine Learning, pp. 8162--177. PMLR, 2021.
Wang, Z., Wang. D., Zhang, Y., and Xia, G. Learning inter- pretable representation for controllable polyphonic mu- sic generation. Proceedings of 21 International Soci- ety for Music Information Retrieval Conference, ISMIR, 2020b.
Novack, Z., McAuley, J., Berg-Kirkpattrik, T., and Bryan, N. J. Ditto: Diffusion inference- time t-optimization for music generation. In International Conference on Ma- chine Learning, pp.
PMLR, 2024.
Ren, Y. He, J., Tan, X., Qin, T., Zhao, Z., and Liu, T.- Y. Popmag: Pop music accompaniment generation. In Proceedings of the 28th ACM international conference on multimedia, pp. 1198-1206, 2020.
Wang, Z.. Min, L., and Xia, G. Whole-song hierarchical generation of symbolic music using cascaded diffusion models. In International Conference on Learning Rep- resentations, 2024.
Rombach, R., Blattmann, A.. Lorenz, D., Esser, and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF con- ference computer vision and pultern recognition, pp. 10684-10695, 2022.
Wu, S.-L. and Yang, Y.-H. Musemorphose: Full-song and fine-grained piano music style transfer with trans- former vae. IEEE/ACM Transactions Audio, Speech, and Language Processing, 31:1953-1967, 2021.
Xie, S., Zhang, Z., Lin, Z., Hinz, T., and Zhang, K. Smartbrush: Text and shape guided object inpainting with diffusion model. 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 22428-22437, 2023.
Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language un- derstanding. Advances Neural Information Processing Systems, 35:36479-36494, 2022.
Zhang, C., Ren, Y., Zhang, K., and Yan, S. Sdmuse: Stochastic differential music editing and generation via hybrid representation. IEEE Transactions on Multime- dia, 2023.
Song, J.. Meng, C., and Ermon, S. Denoising diffusion im- plicit models. In International Conference Learning Representations, 2021a.
Song, J., Zhang, Q., Yin, H., Mardani, M., Liu, M.-Y., Kautz, J., Chen, Y., and Vahdat, A. Loss-guided diffu- sion models for plug-and-play controllable generation. In International Conference on Machine Learning, pp. 32483-32498. PMLR, 2023.
Song, Y. and Ermon, S. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.

---

Efficiintt Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
A. Proof of propositions and calculation details
A.1. Calculation details 3.2
Our goal is to find the optimal solution of problem (7). Since the constraint an element-wise constraint on a linear function and the objective is separable, we can find the optimal solution by element-wiss optimization. Consider the (1, -eleemett of e.
First, if (1, h) K(); there is no constraint eth Therefore, the optimal solution of Eth is εg,i (X,, t).
If h) E wl: IC), the constraint on εt, is
V
t4Eth
1
Xe,h
S
Vāg
2
which is equivalent
1
Vāt
Eth
Xt,th
dz
2
The objective is to minimize ||E ε0,u (X,,t),, t)ll Therefore, the optimal solution of Eih is
Eth max { tIC, R),
1 Cr
Voz
VT [e,m
A.2. Proof of Proposition
Proof. Recall that According to Song et al. (2021b), the DDPM forward process X, = VorXo + v1 - aae can be regarded as discretization of the following SDE:
dx, = - d(t) + (B(t)dW.
and the corresponding denoising process takes the form of a solution to the following stochastic differential equation (SDE):
dx, 1 3(1)x, x())xx, log pogp((Xg (X.) dt + B(()aw.
where B(I/T) T3, T goes infinity, W. is the reverse time standard Wiener process, and ã. should be replaced by its continuous version e 1o ((s)ds (or Sú 3(s)ds when early-stopping time is adopted). The score function Vx, log Pt (X:) can be approximated by (X,, 1)/ ///t///vv/
fo (s)ds
Under the SDE formulation, the denoising process can take the form of solution to stochastic differential equation (SDE):
1 7 3(1)x, 3(1)xx, (X.) dt BB())WW. dX,
(11)
where B(t/T)- = T3, W, is the reverse time standard Wiener process. According to Song et al. (2021b), as + co, the solution to the SDE converges the real data distribution Po-
In the diffusion model, Vx, log Xx) is approximated by (X,, 0//11 3(s)ds Therefore, the approximated reverse-SDE sampling process without harmonic guidance is
1 B 3(1) Eo (X,,t)
dx.
dl + /B(1)aW.
(12)
Lig ((s)ds
Similarly, the sampling process with fine-gratne mme-grained harmonic guidance is
dx, = 1 -000 - B(()
Eo X,, t)
di + BB(JaW,
(13)
V 1--e Sr p(8)da
12

---

Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
where Eg is defined equation and equation 8.
For simplicity, we denote the drift terms follows:
1(X,,t)= = 2 ((t))X+(((1)))) (X.)
EN (x1,1111 1)
t(X,,t)= = B(()), B(t)
c Sig ((s)ds
EØ (x,,,tt) t)
f(X,,t)=
2 B(t)X, 3(t)
v c
lig (())ds
Since
Ex.xp (ttxn.lll=((x..00 Eo EX, 1)13] 88.
and
E" (Xx,,) =\-1 Sig Bla)ds Vx, log Pe (Xe),
we have
3(1)
Bx=p:lll((X, f(X, v
5.
Sio B(s)ds
Now we consider éo (X,,,,t) f). which is the solution of the optimization problem (7). In the continuous SDE case, the corresponding optimization problem becomes
min || - t(,,
V1-e- Sig ((s)ds X. - -
(14)
RLLH W
s.l.
6 1/2 Sig B(s)ds
According to Proposition of Chung et al. (2023), the posterior mean Xo conditioning on X,
X + (1-e - fig 8(s)do Vx. log pe (X1)
X
e Jig p(s)de
X; V1 V1-e- Sí, pls)ds e- (X.,1))
I. B(s)ds
Since the domain of X, is R.xHH (W, which is convex set, know that the posterior mean E E [X,] naturally belongs to its domain. Therefore, ex (X,,t) t) is feasible to the problem (14). Since the optimal solution of the problem is éo (X,, t), we have
(xo(X,, - E (X,, t)ll < ||E* (X.,t) - EX(Xe. Illl
t)
for all X, and t. This further leads to the result that
3(1) 8.
Ex-p, [1I((X,) () f(x, <
(15)
e St ((s)ds
Moreover, since t) is essentially the projection of es (X,,t) onto the convex set defined by the constraints in (14), and e" (X,, () also belongs to the set, we know that the inner product of (X.,,, () Eo (X.,)) () and eo(X,, () (x.X,11) () is negative, which further leads to the result that
| (X, 1) e"(X,, f)1l S (le" Eo (X,, t)|l
(16)
13

---

Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
which further implies
B(1) Sig B(s)ds
Ex [llf(X,t) f(X, t)ll -
6.
(17)
e
The following Girsanov's Theorem (Karatzas & Shreve, 1991) will be used (together with equation 15 and equation 17) to prove the upper bounds for the Kl.-divergences in Proposition 1:
Proposition Let po be any probability distribution, and Z (Zi)EEO.T)) Z' (Zi)cEJOTT) be two different processes satisfying
dz, = b(Z,, t)dt (()dBB,, Zo ~ po:
dz; = D' (Zz tjdt + o()d %o po-
We define the distributions of Ze and Zj as Pt and pe, and the path measures of Z and Z' as and P' respectively. Suppose the following Novikov's condition:
( o (t)]((
6' (z, sd(dt ] <0.
Ep
exp
(18)
Then, the Radon-Nikodym derivative of P with respect to IP' is
dP dip (Z) = exp
{
/ ott) 211(6 (Z,,)))1 S (6 (00),(006))))))))1
2 J
and therefore we have that
KL(Pr (pT) S KL(PI(P') = - T [-1 ʃ pi(z)a(t))21(C ba) ( dxdt.
2
Moreover, Chen et al. (2023) showed that if J, Pe()o have that
(2, +)l C holds for some constant C over all
kL(prlleT) - S
2 p((x)o(t)) 211(b - 6z)(z, dzdtt
1
even if the Novikov's condition equation 18 is not satisfied.
According to equation 15 and equation 17, we have
f p((=))(0 t)ldr 1
3(L)
3(1)
5 sup *E(to.T]
8,
(19)
1 St B(s)ds
v e St 8(x)ds
[ p((=))(0)= f(X, t(l,x S
3(1)
3(1)
< sup *E(to. T]
8.
(20)
S% B(s)ds
Sig (())ds
Therefore, we can apply Proposition 3 to obtain upper bounds for the KL-divergences, which leads
T
KL (L..r.10
<
(z)) (al)))(()~1((((x;))))
'l1((x, t) /(X,t)]dd
10 2
(21)
B(t)
<
8
di
t0
V
Sú Ble)do
14

---

Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
and
T
(L
-
( t) d)
10 2
(22)
S(t)
S 8 to
dl
V VI-
Sig B(o)ds
E
Remark 1. Under the SDE formulation, the forward process terminates at sufficiently large time T. Also, since the score functions blow up at t 2e O, an early-stopping time to is commonly adopted to avoid such issue (Song & Ermon, 2020; Nichol & Dhariwal, 2021). When to is sufficiently small, the distribution of Xio in the forward process is close enough to the real data distribution.
A.3. Proof of proposition 2
We first provide the following definition 1, which is adopted from Fu et al. (2024).
Definition 1. Denote the space of density functions
Po (p(x) = f(X)
L(RLXH (), 0},
where C and can be any given constants, and C(R!XH B) denotes the class Lipschitz continuous functions R R...x R with Lipschitz constant bounded by B.
Suppose that the density function of X belongs to the following space
Ps {p(X)= E Po)P((R
(23)
where the distribution of M is defined from X by
Min 1(X 2 1/2).
Proposition 4. Consider generating piano roll M from continuous random variable X. i.e., given i..dd data
[=))== px. let [= be given by Mi 1(ix, 1/2). Denote the model for estimating the distribution of X px. We have 3 C >> 0 such that Vn,
inf sup EMM P(R,O) C. -n DH P(R,O),
(24)
7x PXEP:
where P is the probability associated with the generated data fxx
Proof. We first restate special case of proposition 4.3 of Fu et al. (2024) as the following lemma.
Lemma 1. (Fu et al. (2024), proposition 4.3) Fix a constant C2 > 0. Consider estimating distribution P(x) with density function belonging to the space
P = {p(x) = [(x))xp(( f(x) L(R
Given n i.i.d. data Hi we have
inf sup (( [V P)] P) n A PEP
where the infimum is taken over all possible estimators p based on the data.
From lemma since the space Po that we define satisfies all the same conditions as the space P in lemma we know from the conclusion lemma that
inf sup Px PP
E E
TV(px. 2 Lit2
(25)
15

---

Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
where by definition of total variation,
S
TVxxx,PX)
Px
PxXX)
(26)
JREXH
For simplicity, suppose event O denotes note-out-of-key occurring at (1.h) (1, We have
/ dXn [
P(()
dY Px(X11 Y)
(t.+
RCXH-1
(27)
- (x
90
where is (LH dimeesnsiaa variable denoting the elements in matrix X excluding X11. Let C(O) denote the set of all possible realizations of piano roll M that contains (i) the note O as out-of-key note, and (ii) "resolution" to accommodate it. For each M C(O), let
8(M) (11 E [1,4 H] Mn. = 1}.
Therefore, we have
/
P(R,O)=
E MEC(O)
dY (XX((M))
(28)
Px(X)dX,
Recos
and note that Po(O)) C Ωo, we have
J
px (X)dX
(29)
To better explain and summarize equation 27, equation 28 and equation 29, the probabilities P(.) (the estimated probabil- ities of O, {R. O} or {R, 0}) are always calculated from integrating (X(X) on corresponding domain, and the key of the 3 equations are all about finding the domain on which to integrate Similarly, for the ground truth distributions and under definition which provides PM (R, 0) = 5, we have
P(R,O)= Jao
px(X)dX S 6.
Therefore,
P(R,O)= /
Px(X)dX
(xx PX(X)) (X)
(30)
[xx(X) (x -
Therefore,
P(R, 0) TV lo Roxo (fx, px) 8,
(31)
where VVlooAeco is the total variation integral restricted on the domain 2o (oc(o)-
By definition, the resolution of out-of-key refers how it integrated into the surrounding harmonic and melodic structure make sound intentional rather an error.
16

---

Efficientt Finc-Grained Guidance for Diffusion Model Based Symbolic Music Generation
By construction of packing numbers provided in the proof of proposition 4.3 of Fu et al. (2024), note that constraint PM R, O) 8 restricting the integral of total variation Ωo (o(o) does not change the order of the packing numbers, i.e. Po and Pa have the same packing numbers. Let
P/ So00o (aro)
{(( (C(()) p p(X) xEX px() E Ps).
where the constant C(oo (c(O)) ) is a scale factor to ensure that C(oo (((()) Xxxoo (oooo) is a probability density function. For simplicity we use P66, O) for short of 'P, (ooo) Therefore, from the original lemma of Fu et al. (2024) have equation 25. Only changing the Po into P(6, 0) (all the arguments above justify why this change can be made), we have
inf sup E (x.)2 TV(px, Z n IFTZ. Px PEP().))
(32)
Combining equation 32 with equation 31, and starting from our target IE PE O), we have
inf sup Exx,2 P(R, 0) + + = inf sup no Mecoo (Fx, + 8 pEPs
Px pEPs
= inf sup pEP().0)
TV Px) + -82 レĀस..
Therefore, 3C > Vn,
inl sup E [xx P(R,O) 2 C. 2H P)R,O).
Fx PEPs
which finishes the proof.
B. Details of Conditioning and Algorithms
B.1. Mathematical formulation of textural conditions in section 3.1
Denote chord progression by C, where C(I) denotes the chord at time E [[1, LlI Let Ye() C [1, pitch index h that belongs the pitch classes included in chord C(I).", and let 7R [1, L] denote the set of onset time indexes corresponding rhythmic pattern R. We define the following versions of representations for the condition:
H] denote the set of
When harmonic (C) and rhythmic (R) conditions are both provided, the corresponding conditional piano roll (C, R) is given element-wise by M un (C, R) 1(l 7R 11t Tc(()), meaning that the (1, h)eeement if pitch index h belongs to chord C(I) and there is onset notes time I, and 0 otherwise.
When only harmonic (C) condition is provided, the corresponding piano roll MC is given element- by Moond, in (C) = -1 [h E >c meaning that the (1, -leemeet is - if pitch index h belongs chord C(I), and otherwiss.
Figure and Figure 2 provides illustrative examples of Moond (c, R. and MC) The use -2 and -1 (rather than and 0) in the latter case ensures that the model can fully capture the distinctions between the two scenarios, as a unified model will be trained on both types of conditions.
B.2. Classifier Free Guidance
To enable the model to generate under varying levels of conditioning, including unconditional generation, we implement the idea of classifier-fres free guidance, and randomly apply conditions with or without rhythmic pattern in the process of training. Namely, the training loss is modified from equation 1 and given
Ex,,,x, [t)(+) Ex(X,, cond t))
(33)
+12(t)llE E (X,, oood (C, R), t)]1?]
For example, when C(T) major (consisting of pitch classes C, E. and G), nc includes all pitch values corresponding the three pitch classes across all octaves.
17

---

Efficientt Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
where A1 (1) and (1) hyper-parameters. Note that both M (C) and (C, R. derived from X via pre- designed chord recognition and rhythmic identification algorithms.
The guided noise prediction at timestep t is then computed as
Eo (x,, R) =EA (X,, M tC).t)
+ w E0 ,X, M (c, R)
(34)
(X, M (C).t)]
where is the weight parameter. Note that the general formulation Eu Ex(X,, R) includes the case where rhythmic guidance is not provided (R 0), and in equation 34 is set as O.
B.3. Additional algorithms in section 3.2
In this section, we provide the following algorithm: fine- tine-graieed sampling guidance additionally with rhythmic regulariza- tion, fine-grained sampling guidance combined with DDIM sampling.
Let B denote the rhythmic regularization. Specifically, we have the following types of regularization:
Bi: Requiring exactly N onset of note at time position I, i.e., Exep.m Mn. = N
B2: Requiring at least N onsets at time position I, i.e.
3h C [1, H], or =h [1, HI (1) if harmonic regularization jointly included
such that Mn. and hh] 2
By: Requiring no onset of notes at time position 1, i..., wh E [1, Mp.
0
Let the set of M satisfying specific regularization B be denoted Mg, and the corresponding of X be denoted Mlg note that this includes the case where multiple requirements are satisfied, resulting in
Mig
Mls Ba.... MR n MA, n....
The correction of predicted noise score is then formulated as
Eo EX,,, R)
arg min
||e
(35)
X;
vI 6,E
s.t.
Mg.
at
Further, perform predicted noise score correction with joint regularization rhythm and harmony, resulting in the corrected noise score
EØ (X,, R) = arg min
|| - Eo(Xi,t|C, R)]l
(36)
X, X.- VI aie
E (IR LxH (
s.t.
Vou
We for example provide a element-wise solution of ěø( (,, R) defined by problem (35). For given 1, suppose B(I) takes the form of B2, for simplicity take N -= 1. This gives éa in = εg.t. if max, E[X,] x 2 1 and [X X,)a 1 arg max; E X,) Xe)at. i.e.
1
Vot
Eo.th
X1.th
ai
max; E|Xo| X,JM < 3. The correction applied predicted X, (E(Xp) x,) illustrated the following figure 4.
18

---

Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
threshold-Trus True
threshold-False
Key C:major
(a) Fine Sine-grained control for X [x]] E 1L_x (We. The colored spots denote places that require E. X X 3.
original note original
removed note added note
rhythm
onset pos
nom onsset
(b) Fine-grained control for E[X, X2] E Wy. Original notes are removed By is applied. Otherwise if 13, is applied and currently no note exists, the "most likely notes" (i.e., at h = arg max X Xi)u)) are added.
Figure 4. Illustration of finenerained control on predicted Xo-
19

---

Efficientt Fine-Grained Guidance for Diffusson Model Based Symbolic Music Generation
Algorithm 2 DDPM sampling with fine-grained textural guidance
Input: Input parameters:
forward process variances St. & IT. St, backward noise scale dt, chord condition C, key signature K,, rhythmic condition R. rhythmic guidance B
Output: generated piano roll M {0, 1
X ~ (0,
to for 1 - T, 1,1-1..... 1,...,, do
11
Compute guided noise prediction ex, R)
12
Perform noise correction: derive (x,, 1|c, R) optimization equation 36
13
Compute x, by plugging the corrected noise (x,,,, R) into equation 2
end
Convert Xo into piano roll M return outpul
15
16
We additionally remark that the fine- the-grained sampling guidance empirically effective with the DDIM sampling scheme, which drastically improves the generation speed. Specifically, select subset {T) C [[1,7]. and denote
X1
V ā,, Eo XT. Ti) ) + V1- 1-āā. ār - oo E (X.,,
X Van,
(X,. Ti) + o+,
V@n,
we similarly perform the DDIM noise correction
Xx: ICC, = arg min
||e Ex(X r2r T/C,, R)ll
X₁ v1 â=e ) ( nMg
s.t.
Vān,
on each step i.
C. Comparison with Related Works
We provide detailed comparison between our method and two related works in controlled diffusion models with con- strained or guided intermediate sampling steps:
Comparison with reflected diffusion models In Lou & Ermon (2023), a bounded setting is used for both the forward and backward processes, ensuring that the bound applies to the training objective as well as the entire sampling process. In contrast, we do not adopt the framework of bounded Brownian motion, because we do not require the entire sampling process to bounded within a given domain; instead, we only enforce that the final sample outcome aligns with the constraint. While Lou Ermon (2023) enforces thresholding on X, in both forward and backward processes, our approach perform thresholding- projection method on the predicted noise (X(,,, t). interpreted as noise correction.
Comparison with non-differentiable rule guided diffusion Huang et al. (2024) guides the output with musical rules by sampling multiple times at intermediate steps, and continuing with the sample that best fits the musical rule, producing high-quality, rule-guided music. Our work centers on a different aspect, prioritizing precise control to tackle the challenges of accuracy and regularization in symbolic music generation. Also, we place additional emphasis on sampling speed, ensuring stable generation samples within seconds to facilitate interactive music creation and improvisation.
D. Numerical Experiment Details
D.1. Detailed Data Representation
The two-channel version of piano roll with with both harmonic and rhythm conditions (Moon (C, R)) and with harmonic condition (Monnd()) with onset and sustain are represented as:
(C, R): : In the first channel, the (I, h)-element 1 if there are onset notes at time and pitch index belongs to the chord (I), and otherwise. In the second channel, the (I, h) -lleemenntt is 1 if pitch index h belongs to the chord C(I) and there is no onset note at time I.
20

---

Efficient Fine-Grained Guidance for Difflusin Model Based Symbolic Music Generation
(cond(C): In both channels, the hileemet is if pitch index h belongs to the chord C()), and 0 otherwise.
In each diffusion step i, the model input is a concatenated 4-channel piano roll with shape L x 128, where the first two channels correspond to the noisy target X, and the last two channels correspond the condition M (either Meand(C (C, R) MC (C)). The output is the noise prediction Eo, which is 2-channel piano roll with the same shape as X, For the accompaniment generation experiments,
provide melody as an additional condition, which is also represented by 2-channel piano roll with shape 2 x L x 128, with the same resolution and length as X. The melody condition is also concatenated with X, and Meond as model input, which results in a full 6-channel matrix with shape
6 x L X 128.
D.2. Training and Sampling Details
We set diffiusion timesteps T = 1000 with 3p - 4.5e-4 and BT = 1.2e-2. We use Adam optimizer with a learning rate of 5,-5, 31 = 0.9, and Bg = 0.999. We applicd data augmentation by transposing each 4-measure piece into all 12 keys. This involves uniformly shifting the pitch of all notes and adjusting the corresponding chords accordingly. This augmentation expands the dataset to 189,132 samples. Training is conducted with a batch size of 16, utilizing random sampling without replacement. Specifically, in each iteration, 16 samples are randomly selected without replacement until all samples are utilized, constituting one epoch. This procedure is repeated to ensure each sample was processed twice during training, resulting in total of 23,642 iterations.
To speed up the sampling process, we select a sub-sequence of length 10 from (I....,7) and apply the accelerated sampling process in Song et al. (2021a). It takes 0.4 seconds to generate the 4- 4-measure accompaniment on a NVIDIA RTX 6000 Ada Generation GPU.
D.3. Experiments on Symbolic Music Generation Given only Chord Conditions
As mentioned in Section 5.1, we also run numerical experiments on symbolic music generation tasks given only chord condition. However, compared with the accompaniment generation task, we remark that this experiment does not have enough effective basis for comparison.
For the accompaniment generation task, we evaluate the cosine similarity of chord progression between the generated samples and the ground truth, as well as the IoU of chord and piano roll. The comparison with ground truth on those features make sense in the accompaniment generation task, because the leading melody inherently contains many constraints on the rhythm and pitch range of the accompaniment, ensuring coherence with the melody. Thus, similarity with ground truth on those metrics serves as an indicator of how well the generated samples adhere to the melody.
However, in symbolic music generation conditioned only on chord sequence, while chord progression similarity remains comparable (as the chord sequence is provided), evaluating IoU of piano roll against ground truth is less informative. This is because multiple different pitch range and rhythm could appropriately align with given chord progression, making deviations from the ground truth in these features less indicative of sample quality. Therefore, chord similarity emerges as the sole applicable metric in this context.
Additionally, WholeSongGen's architecture does not support music generation conditioned solely on chord progressions, as it utilizes a shared piano-roll for both chord and melody, rendering unsuitable for comparison. Conversely, GETMusic facilitates the generation of both melody and piano accompaniment based on chord conditions, allowing for viable comparison.
Consequently, we present results focusing on chord similarity between our model and GETMusic. For our model, we evaluate performance under two conditions: with both conditioning and control during training and sampling, and with conditioning during training but without control during sampling. The outcomes, summarized in Table 3, indicate that our fully controlled FGG method surpasses both the one without sampling control and GETMusic.
Methods
FGG (Ours)
FGG, only Training control
GETMusic
Chord Similarity
0.676 + 0.007
0.645 + 0.008
0.499 + 0.013
Table 3. Evaluation of the similarity with ground truth, chord-conditioned music generation.
21

---

Efficient Fine-Grained Guidance for Diffusson Model Based Symbolic Music Generation
E. Demo Page Details
In this section, we briefly introduce how the Dorian mode and Chinese style clips are generated. We note that both styles are shaped not only by key-constraint K, but also with designed chord progressions C.
The key constraint for Dorian mode, example 1, is K2 {A, C, D, E, # G) throughout the 4 bars, which means all generated notes have to be in the pitch classes in K1. The the chord progression for Dorian mode, example is
C Am(4) Em(2)
Am(2) C(2) D(2) Am(2) D(2). ((2).
For example 2, (D, F. F. F. G, A, B, C). and
C2 Dm(() G C(4) F(4).
The number in parentheses corresponds the number of beats the chord lasts. For example, at the beginning of Ct, the chord Am lasts beats. Therefore, for the condition matrix under the 16th resolution, the positions corresponding to pitch classes A, C and E have value 1, where the rest have value O, for t O, 2... 15. The condition is passed to the diffusion model as generation condition. Then 11 is applied as sampling control to shape and refine the tonal quality.
Similarly, for Chinese mode, we have Ki = {C, D, E,
E, G, A} and
C G22) - Am(2) - C(2) - G(2) Em(2) - G(2) - D(4).
For the second example, KC2 = F. and
C2 A44 - (m(()) - Bm(2) A(2) - Fm(2) A(2).
F. Subjective Evaluation
To compare performance of our FGG method against the baselines (WholeSongGen and GETMusic). we prepared 6 sets of generated samples, with each set containing the melody paired with accompaniments generated by FGG, WholeSongGien, and GETMusic, along with the ground truth accompaniment. This yields a total of 6 x 4 = 24 samples. The samples are presented in a randomized order, and their sources are not disclosed participants. Experienced listeners assess the quality of samples in 5 dimensions: creativity, harmony (whether the accompaniment is in harmony with the melody), melodiousness, naturalness and richness, together with overall assessment.
F.1. Background of Participants
To evaluate the musical background of the participants, we first present the following questions:
How many instruments (including vocal) are you playing or have you played?
Please list all instruments (including vocal) that you are playing or have played.
What is the instrument (including vocal) you have played the longest, and how many years have you been playing it? (e.g., piano, 3 years)
We recruited participants with substantial musical experience for our survey, The number of instruments these partic- ipants play range from 0 to with an average value of 2.03, and a standard deviation of 1.31. Examples of instrument played include piano, violin, vocal, guitar, saxphone. Dizi, Yangqin and Guzheng. The average years of playing has an average of 8.61 and standard deviation of 8.08. Specifically, the percentage of participants with 2 3 years of playing music is 67.74%, and the percentage of participants with > 10 years of playing music is 16%. The distributions are given in the following figure 5.
F22 Evaluation Questions
Thank you for taking the time to participate in this experiment. You will be presented with 6 sets of clips, each containing clips. The first clip in each set features the melody alone, while the remaining three include the melody accompanied by different accompaniments After listening to each clip, please evaluate the accompaniments in the following dimensions based on your own experience.
22

---

Efficient Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
3%
4, 10%
3, 23%
0, 16%
1, 16%
2, 32%
Number instruments played by the participants.
Distribution of the Participants Years Playing Music
frequency
Years of Playing Music
(b) Distribution of the participants' years of playing instruments.
Figure Information the musical background of the participants the subjective evaluation.
Does the accompaniment sound pleasant to you?
How would you rate the richness (i.e., the complexity, fullness, and expressive depth) of the accompaniment?
Does the accompaniment sound natural to you?
Do you think the accompaniment aligns well with the melody?
Does the accompaniment sound creative to you?
Please give an overall score for the clip.
For each question, participants are provided with a Likert scale ranging from 1 to 5, where represents "very poor" and represents "very good."
G. Representative Examples of Sampling Control
In this section, we provide empirical examples of how model output reshaped by fine-grained correction in Figure 6. Notably, harmonic control not only helps the model eliminate incorrect notes, but also guides it to replace them with correct ones.
23

---

Efficientt Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
Plano
Piano
9
Piano
Chords
Chords
(a) An example of replacing an out-of-key note Bb> with the in in-key note B
(b) An example of replacing an out-of-key note D2 with the in-key note D)
Figure 6. Examples resulting from symbolic music generation with FGG. The first track is generated without key-signature control in sampling, the second track generated with key-signature sampling control. The third track presents the chord condition. In each subfigure, the tracks are generated with the same conditions and the same set of noise.
H. The Effect of Guidance Weight for Classifier-free Guidance
In Section 3.1, we discussed the implementation of classifier-free guidance for rhythmic patterns, designed to enable the model generate outputs under varying levels of conditioning. Specifically, randomly apply conditions with without rhythmic patter in the process of training. This approach ensures that the model can function effectively with both chord and rhythmic conditions with chord conditions alone. Following Ho & Salimans (2021), when generating with both chord and rhythmic conditions, the guided noise prediction at timestep t is computed as:
Eo t|C, =Eg [X,, M c MC tC),t)
+ Eg (X,, Mood (C, R), EX,, Mond M
where Eo (X,, Mooodd (C, t) is the model's predicted noise without rhythmic condition, and EØ ex(X,, Moond R), is the model's predicted noise with rhythmic condition, and w is the guidance weight.
The literature has consistently demonstrated that the guidance weight w plays pivotal role balancing diversity and stability in generation tasks (Ho & Salimans, 2021; Chang et al., 2023: Gao al., 2023; Lin et al., 2024). In general, lower weight enhances sample diversity and quality, but this may conditions. Conversely, higher values of w promote closer adherence to the conditioning input, but excessively high can degrade output quality by over-constrainin the model, resulting in less natural or lower-qualic samples.
the cost of deviation from the provided
In this section, we hope to investigate the effect the guidance weight w on our music generation task. We focus on the same accompaniment generation task as mentioned Section 5. To measure the samples' adherence to rhythmic controls, we use the rhythm of the ground truth as the rhythmic condition and assess the overlapping area (OA) of note duration and split both the generated accompaniments
note density between the generated and ground-truth samples. Specifically, and the ground truth into non-overlapping 2-measure segments. Following (von Rütte et al., 2023), for each feature (f E {note duration, note density) )), we calculate the macro overlapping area (MOA) in segment-level feature distributions so that the metric also considers the temporal order of the features. MOA is defined as
MOA(F) Σ overlap(m; (f) mf)),
where (f) is the distribution of feature f in the i-th generated segment, and (f)) is the distribution of feature ff in the i-th ground truth segment. Additionally, we measured the percentage of out-of-key notes as a proxy for sample quality.
In these experiments, we only use the fine- grainnd control training, but not insert any sampling control so that we can
24

---

Efficientt Fine-Grained Guidance for Diffusion Model Based Symbolic Music Generation
evaluate the inherent performance of the models themselves. The experiments were conducted across range of guidance weights (w from 0.5 to 10), and he results are summarized in Table 4.
Values w
% Out-of-Key Notes
OA
OA
(duration)
(note density)
1.3%
0.592
0.803
+0.005
#0.004
1.0
1.4%
0.617
0.830
+0.005
++0.03
3.0
1.7%
0.644
0.848
+0.003
+0.003
5.0
2.6%
0.638
0.846
+0.005
+:0.003
7.5
6.0%
0.643
0.829
+0.005
+00004
10.0
14.3%
0.630
0.779
1.0.055
1.0.005
Table 4. Comparison of the results with and without control in the sampling process.
The findings indicate that the guidance weight w increases, the percentage of out-of-key notes rises, suggesting that lower w values yield higher-quality samples. Meanwhile, the OA of duration and note density improves as w increases from 0.5 to 3.0, indicating better alignment with rhythmic conditions. However, when w exceeds 5.0, a notable decline observed in both the OA metrics and the percentage of out-of-key notes. This degradation is likely due to a significant drop in sample quality at excessively high w values, where unnatural outputs undermine adherence to the rhythmic conditions. These observations coherent with the existing results about the trade-off between sample quality and adherence conditions in literature.
I. Discussion
The role generative Al in music and art remains an intriguing question. While has demonstrated remarkable perfor- mance in fields such as image generation and language processing, these domains possess two characteristics that symbolic music lacks: an abundance of training data and well-designed objective metries for evaluating quality. In contrast, for mu- sic, it is even unclear whether it is necessary to set the goal generating compositions that closely resemble some "ground truth".
In this work, we apply fine-grained sampling control to climinate out-of-key notes, ensuring that generated music adheres to the most common harmonies and chromatic progressions. This approach allows the model to consistently and efficiently produce music that is (in some ways) "pleasing to the ear". While suitable for the task quickly creating large amounts of mediocre pieces, such models have a limited capability of replicating the artistry of real composer, creating sparkles with unexpected "wrong" keys by themselves.
or, what sense?
25


Reviews:
[{"review_id": "VcVPqkBcHd", "reviewer": "Reviewer_uD9t", "summary": "The two main motivators of this work are (1) the importance of providing user control for symbolic music generation, and (2) specifically the importance of *exact* pitch control, and the particular challenge of achieving this when using an image-based representation (i.e. piano roll). The authors solve this by proposing a conditioning-based approach that allows some harmonic and some rhythmic control. They introduce a piano roll representation of the conditioning signal, and they use these signals both during train time of a conditional diffusion model, and also as a mask-like representation of the constraints to provide further guidance during the sampling process. Essentially—if I have understood correctly—if the constraint says “don’t use note X at time T”, then at each step of denoising, the sample value of note X at time T is pushed toward zero. They provide a theoretical bound on how much this adjustment can affect the overall resulting distribution. The authors also provide a theoretical argument to try to explain why harmonic precision is hard for uncontrolled (i.e. unconditional) models.\n\nThey provide experimental results using the POP909 dataset, with piano rolls of size 2x64x128, corresponding to 4-bars of 4/4 time quantized into 16th notes, with one channel for onset and one channel for sustain. They measure both objective (e.g. percentage of out-of-key notes) and subjective quantities. They present an ablation study where (1) the control signal is provided as a conditioning input,  but “wrong” notes are removed only after the completing the reverse-diffusion process; (2) one study where the control signal is again provided as input but no notes are removed; and (3) one study where an unconditional model is used and no control occurs during the sampling process.\n\nThey find that providing the conditioning signal (i.e. training a conditional model) helps, and providing both conditioning signal together with interventions during the sampling process helps even more.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's suggestions on revising! Due to this year's policy, it is not allowed to upload a revised paper, external links can only contain figures/tables, and rebuttal has a 5000 length requirement. Please allow us to describe a revision plan in the follows and promise to follow through in next phase.\n\n**Responses**\n\n1. To enforce Dorian, we restrict pitch classes to the Dorian scale, and additionally use the Am-Em-C-D chord progression for shaping. The combination of them makes it different with G major.\n\n2. We added a comparison with the inpainting baseline (see **4. Additional Experiments and demo page** below). We did not initially frame our methodology from an inpainting perspective. Generating with minor correction looks more efficient to us, since a well-trained model aligns with chords, yielding only 2% incorrect notes. In contrast, inpainting adds complexity to the model, while much of the Roll[time,pitch]=0 in the input might be redundant.\n\n3. We actually use [0,1] data for training.\n\n4. \"simple rule-based post-sample editing\" is “training control edit after sampling”\n\n**Revisions**\n\n**1. Additional related work.** We will add two sections, one on precise control over symbolic music generation with other generative models, the other on image inpainting.\n\n**2. Modified claims.**\n- We will explain that the user-specified control refers specifically to user-designed control within the constrained piano roll format.\n- We will remove the misleading word “out-of-sample”, and replace with the statement “our method can shape the output towards a specific tonal quality.”\n- We will describe the demo page as “a demo page to showcase performances, which enables real-time generation”, removing \"one of the first\".\n\n**3. Theory.** We decide to remove the argument regarding conditional probability  $\\widehat{P}(\\bar{R}|O)$, which seems too intricate. Further, the introduction of proposition 2 (after the discussion of out-of-key notes and resolutions and before proposition 2) will be:\n\n>We provide an explanation using statistical reasoning. Consider a piano roll segment, represented as a random variable $M$.  Suppose we are interested in whether this segment contains an out-of-key note (denoted as event $\\{O\\}$) and whether that note is eventually resolved within the segment (denoted as event \n$\\{R,O\\}$). In our training data, almost every out-of-key note is resolved, meaning the probability of unresolved out-of-key note is close to 0, i.e., $P(\\bar{R},O)\\approx 0$.\n\n>Now, we examine the probability in the generated music. The key question then is whether the generative model also learns to keep $\\widehat{P}(\\bar{R},O)$ small. The following proposition 2... (same as in manuscript).\n\n**4. Additional Experiments and Samples.**\n\nIn numerical experiments, we have added a baseline named **inpainting**, in which we treat the pixels where there should not be a note as known (value should be 0), and let the model inpaint the remaining parts. To do this, we add a mask to model inputs, and trained an inpainting model. We also added **rounding the out-of-key notes to the nearest allowable** in our ablation study. We also added more interpretable metrics. Results are shown in [https://drive.google.com/file/d/1IAcAqK4qK4AiQVKWriFSJ91QNhrg5at-/view](https://drive.google.com/file/d/1IAcAqK4qK4AiQVKWriFSJ91QNhrg5at-/view).\n\nSamples generated from ablation conditions are **added to Section 3 of demo page**. Across all ablations, we observed occasional occurrences of excessively high-pitched notes and overly dense note clusters.\n\n**5. Discussion of limitation:**\n\nThe 16th-note quantization follows prior work (Wang et al., 2024), which admittedly reduces rhythmic flexibility, and cannot ingest data without beat information. A potential improvement is integrating our pitch class control method with (Huang et al., 2024), which adds a dynamic dimension, and uses 10ms time quantization for greater rhythmic flexibility. Another key limitation is the control format. Our method supports pitch class and rhythmic control in the piano roll representation, but does not accommodate more abstract forms or probabilistic control. Additionally, the evaluation methods and datasets present challenges in accurately assessing generated music quality. Since music evaluation is inherently detailed and partly subjective, the metrics used in this work has fundamental limitation in metrizing quality improvement.\n\n**6. More explanations of the demo page**\n\nIn section 2 of demo page, the chord conditions are converted to condition matrices exactly following Figure 2 of the paper, the melody conditions are provided in an additional channel, also in the form of a piano roll, and we did not use rhythmic constraint in the generation. We will add a section in appendix to provide description of the matrix. Sample melodies are either randomly picked from the test set of POP909, or extracted by hand from some of our favorite existing pieces.", "claims_and_evidence": "**NOTE**: Since this question requires me to “convert” the standard contributions-based representation into a claim-based representation (some of which are implicit in my reading of the paper), I invite the authors to restate their claims explicitly if I have misrepresented anything here. \n\n**Claim**: The authors provide statistical theory evidence to characterize “the precision challenge” in symbolic music generation. (e.g. see Section 4).  \n**Assessment**: I think I like the provided reasoning/argument, in as much as I followed it. However, I found parts of the argument hard to follow (i.e. unclear), perhaps unnecessarily so (see “strengths/weaknesses” below)).\n\n\n**Claim**: The proposed model allows future users precise control over harmonic and rhythmic components of symbolic music generation.  (e.g. see Section 3, first few paragraphs).  \n**Assessment**: This claim fails to highlight that of course it can only do this in as much as the conditioning representation can support the desired guidance. I think the conditioning representation is reasonable and good (it makes sense to present represent constraints in a piano roll format!) but also it is quite limited. For example, it does not allow probabilistic constraints (as in the demo link earlier). As other examples, one could certainly imagine other kinds of rhythmic constraints as well, and there are also different ways of specifying harmonic constraints too. But one paper cannot provide all possible forms of control!    \n**Easy Fix**: This could be fixed with a clear and explicit discussion on limitations, which I believe are significant.\n\n**Claim**: The proposed controlled diffusion model is capable of generating music with high accuracy and consistency with respect to harmonic and rhythmic guidance, despite limited training data. (e.g. see Section 1, ‘methodology’ and ‘effectiveness’ paragraphs).   \n**Assessment**: I believe this claim is basically supported by clear evidence.    \n\n\n**Claim**: The model supports this control even when the controls push the music towards an out-of-sample style. (e.g. see Section 1, ‘methodology’ paragraph).     \n**Assessment**: This depends on what one means by “out-of-sample”. If one thinks of the training+sampling procedures as teaching+enforcing where to place or not place notes, based on the conditioning signal, then I think that providing a slightly different subset of notes (i.e. a different scale or chord from what is used in the training signal) is not necessarily an out-of-sample task. In particular, the adjusted sampling procedure will absolutely guarantee that only the “allowed” notes will be used (i.e. the others will be removed) so if the set of “allowed” notes is very far from anything seen in the training data, then the generated samples will still satisfy the constraints; the only remaining question is whether the generated samples sound good (and perhaps subquestions such as: how much did the conditional training alone help in those particular cases, versus how much was the sample-editing procedure required?).      \n**Fix**: Either justify exactly what is meant by out-of-distribution, and answer my questions above, or remove/carefully-qualify this claim (at no great cost to the paper, in my opinion).\n\n\n\n**Claim**: “We have published a demo page to showcase performances, as one of the first in the symbolic music literature’s demo pages that enables real-time interactive generation.” [L 32-34].   \n**Assessment**:    \n(1) “interactive”: The demo allows the user to switch between 4 presets (and regenerate multiple times for each one). This is good (far, far better than no demo!), but when I read about an interactive demo, I was excited to try inputting more complex chords and melodies myself, and melodies that include, e.g. less harmonically obvious notes, and see how the system sounds. I do think it’s OK to call it an interactive demo, though.   \n(2) “one of the first in the symbolic music literature’s demo pages”: That reads to me like a significant exaggeration. For example, there are over 15 interactive musical demos on this page alone (https://magenta.tensorflow.org/demos/web/ , which includes [1]) many of which allow the user to input, e.g.  melodies and constraints (i.e. more than just from a dropdown of 4 choices). An effective demo where the user provided melodic contour constraints in real-time was presented in [2]. There are many other online MIDI generation demos as well, e.g. a transformer model here: https://huggingface.co/spaces/skytnt/midi-composer, and many others are available as well. Incidentally, OpenAI’s MuseNet was available interactively as well a few years ago, although it’s not available anymore, so it’s understandable if the authors had not come across it.    \n\n\n\nThat said, I think that providing any kind of user interaction is important and commendable and in fact should generally be expected for generative models. The issue here is simply that the claim is incorrect and can be easily fixed.    \n\n\n**[1]** Roberts, A. and Engel, J. and Hawthorne, C. and Simon, I. and Waite, E. and Oore, S. and Jaques, N. and Resnick, C. and Eck, D., “Interactive Musical Improvisation with Magenta”, NeurIPS 2016 (Demo).    \n**[2]** Donahue, C, and Simon, I., and Dieleman, S., “Piano Genie”, IUI 2019 (see https://www.i-am.ai/piano-genie.html for an online demo)", "methods_and_evaluation_criteria": "Yes, the proposed methods and evaluations do make sense, generally speaking, especially in relation to the conventions in this field.     \n\nAlso: No, the conventional evaluations do not necessarily make sense, but the current authors are not responsible for that. However, it would be helpful to mention potential limitations associated with the evaluation criteria and dataset.     \n\n\nI found the demo page to be very helpful. I would like to see more examples and information on that page (and/or in the appendix, whichever is easier):    \n*  to help make sure I understand how the accompaniment was generated for the sample melodies, I would like to see the constraint matrices (i.e. in “piano roll format”) that were used to generate those accompaniments.    \n* it would be interesting and helpful to see some samples generated for each of the ablative conditions    \n* for the sample melodies in the demo page, how were these melodies obtained? In particular, one of them sounds almost as though it was taken from an existing piece (from the dataset? ), and the melody was “extracted” by hand– is that the case, or is that just a coincidence?\n\n\n**Baselines**. The authors have essentially posed/re-framed the symbolic-control problem as an image in-painting problem, where certain pixels have constrained values. The in-filling literature for images is extensive, so I would expect there to be at least one baseline from that literature that would be reasonable to apply here. Could the authors please respond to this?", "theoretical_claims": "No; I looked at parts of the proofs, but I did not check them line-by-line.", "experimental_designs_or_analyses": "Yes, the experimental designs seemed fairly reasonable. \nFor the ablations, the authors mentioned a comparison to a “simple rule-based post-sample editing”, but I don’t think I saw this in Table 2. Am I missing something? (unless this refers to the case where the *conditional*-trained model is still used, but the editing only happens once at the end, i.e. “training control edit after sampling”). I assumed  “simple rule-based post-sample editing” meant an *unconditional* model, but with an edit applied after sampling. Now I’m thinking maybe it means a conditional model after all– clarification on this would be welcome.\n\nIn 5.1.5, regarding the ablations, the authors write “In contrast, the latter employs a brute-force editing approach that disrupts the generated samples, affecting local melodic lines and rhythmic patterns. The numerical results further validate this analysis.” However, as far as I can tell, the numbers in Row 1 and Row 2 of Table 2 look nearly identical. E.g. OA(pitch) is 0.628+/- 0.005 versus 0.624 +/- 0.005. I do believe that the brute-force editing might disrupt the generated samples, affecting melodic lines, but to me, this absolutely demands qualitative listening-samples because (if I’ve understood correctly) the numerical results do *not* validate this analysis. This also points to shortcomings in the standard approaches for evaluating this kind of work (see my earlier comment in the Methods & Evaluations section about limitations of evaluations).\n\nAn ablation or comparison that would have been interesting is to simply zero out—at every step of the sampling process—those pixels/cells of the piano roll where there is not “supposed” to be a note. I wonder if this would capture the main benefits of the current proposed method, in that it would reduce the possibility of “wrong notes” directly at every iteration of the sampling process. I am not requesting that the authors do this, but if it’s feasible, then I would certainly be very interested to see (and hear) the results of such a modified sampling process. I realize it might be theoretically less sound, but would still be an interesting comparison point. (Or, again, am I misunderstanding something?) Related, see Q2 below.\n\nIncidentally, another related comparison could be some equivalent of the 'MIDI scale' function that Ableton provides, i.e. just \"round notes up/down\" to the nearest allowable note that satisfies the constraints.", "relation_to_broader_scientific_literature": "Some parts of the paper's motivation relate to challenges that are specific to diffusion on piano rolls, not to symbolic music generation in general. For example, try the demo at https://magenta.tensorflow.org/demos/performance_rnn/ to see that it is possible to use a simple language model to enforce certain precise harmonic controls easily and effectively (i.e. probability distribution over pitch classes). Of course language models have their own challenges that diffusion models don’t face. For a more detailed discussion and evaluation of conditioning MIDI-based language models with a variety of controls, see for example [3].     \n\n\n**[3]** Nicholas Meade, Nicholas Barreyre, Scott C Lowe, Sageev Oore, “Exploring Conditioning for Generative Music Systems with Human-Interpretable Controls”, Int’l Conf on Computational Creativity (ICCC) 2019", "essential_references_not_discussed": "Could optionally consider including any of the references mentioned above.\n\n\n**Image inpainting**. A significant area of related work is image inpainting/infilling, since a premise of this paper is to convert symbolic control into an infilling task. (See also my comment on Baselines in the section above on Experimental Designs.) Some of these papers could be discussed explicitly.", "other_strengths_and_weaknesses": "I appreciated occasional well-articulated observations throughout the paper (e.g. in the introduction regarding common limitations, precision demands for music generation; also at the end of the appendix, etc). As one example: (Sec3): \"One challenge of symbolic music generation involves the high-precision requirement in harmony. Unlike image generation, where a slightly misplaced pixel may not significantly affect the overall image quality, an `inaccurately' generated musical note can drastically disrupt the harmony, affecting the quality of a piece.\" Absolutely!\n\nIn Section 4, the authors write: “We provide an intuitive explanation under the statistical convergence framework.” Personally, I found the explanation, including Proposition 2, highly unintuitive (or at least unclear). Once I spent time parsing it, then I did appreciate the argument (assuming I understood it correctly). Some intuition and clarity would help. \n\n\nQuantizing to 16th notes and {1,0} note indicators is both ignoring velocity (i.e. dynamics, rhythmic accents), and also ignoring a large class of rhythms (e.g. triplets, swing, other groupings). Also, if I understood correctly, it requires the data to be “beat-aligned”, e.g. would this representation allow ingesting data such as any sophisticated but unquantized performance data which does not have barline/beat information (e.g. MAESTRO dataset)? Quantization is OK in the sense that simplifications need to be made to get ML systems to work, but it can also be a significant limitation that needs to be addressed as such. How complicated would it be to “scale” up to incorporate some of these aspects? How much is lost by not incorporating these? Again, I do understand the need to simplify; it is simply important to be clear and thoughtful about the extent and impact of the simplification.\n\n**Lack of Limitations.** This paper is missing almost any discussion of limitations whatsoever. I would like to see this such discussion added, either in one place, and/or throughout the paper in order as appropriate.\n\n**Overall**: This is an interesting paper and I look forward to seeing the authors' response and any discussion.", "other_comments_or_suggestions": "My current score is a placeholder.\nIf my concerns are addressed then I will consider raising my mark.", "questions_for_authors": "(See any questions above). Also:\n\nQ1. I am a bit confused about the Dorian mode generation. If harmonic constraints are implemented as “sets” of eligible notes, then how does this allow generation in specific modes that is any different from a major key? For example, G major consists of exactly the same set of notes as A dorian, B phrygian, etc. So in the demo page, which shows A dorian, how is this different from generating in G major? I do agree that this example has a bit of a dorian sound to it, but what is causing that? Presumably sometimes you would have just got something that clearly sounds in G major, right?\n\nQ2. Essentially, this problem appears to be an instance of inpainting where parts of the image are known and other parts need to be filled-in. This has been studied extensively. Why ensure Roll[time,pitch] <=0.5 rather than ensure Roll[time,pitch] == 0?  I.e. why not explore an inpainting method to fill in the remainder of the piano-roll by grounding the Roll[time,pitch]==0 wherever required?\n\nQ3. Out of curiosity: In general, the data-distribution is normalized to [-1,1] before training a diffusion model. So, I was just wondering if the equations use [0,1] data for notational convenience or did the authors actually use [0,1] data for training?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qd3BRlqhwi", "reviewer": "Reviewer_2Mem", "summary": "This paper introduces a Fine-Grained Guidance (FGG) approach for diffusion-based symbolic music generation, addressing precision and controllability challenges. \n\nFGG incorporates harmonic and rhythmic constraints during both training and sampling, ensuring generated music aligns with user intent. \n\nTheoretical analysis bounds the impact of guidance on learned distributions, while experiments demonstrate reduced out-of-key notes, improved chord similarity, and subjective quality. \n\nA demo showcases real-time interactive generation, highlighting practical applicability.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer’s recognition of our work's theory, experiments, and the practicality shown by the interactive demo. \n\nRegarding OOD datasets, we unfortunately have not yet found a high-quality dataset other than the one we used. Alternatively, in the numerical experiments, the songs of POP909 dataset were split upfront into training and testing sets, to prevent data-leakage in the training process.\n\nMeanwhile, although there is not an appropriate OOD dataset for us to run a thorough experiment, we tried to use our model to generate stylized music pieces in section 2 of our demo page. Those styles did not appear in the POP909 dataset but the generated results seem satisfying, which hopefully serves as an indication of our model’s generalization ability.\n\nWe would love to thank Reviewer 2Mem again for the comments and suggestions, which are very helpful to us.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Basically yes. The hypothesis seems to be a little bit strong, but empirical results and ablations can support them.", "experimental_designs_or_analyses": "The experiments are enough for this paper. I am satisfied at chosen baselines (GETMusic, WholesongGen), where WholeSongGen is a strong baseline.\n\nThe Analysis is firmly around of its hypothesis. It highlights the chord, pitch, rhythm attributes, showing a significant result.\n\nI would be happy if more experiments can be conducted on OOD datasets, which showing the model generalisation ability in the wild.\n\nNote that a subjective study is provided in the appendix.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "I believe current references are enough under the topic of symbolic diffusion-based piano music generation.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "QEdbjC5PET", "reviewer": "Reviewer_LX3j", "summary": "This paper presents a fine-grained guidance (FGG) mechanism for improving symbolic music generation using diffusion models.\nThe proposed approach incorporates strict harmonic control by integrating domain knowledge into the generative process, ensuring that generated musical sequences adhere to predefined chord progressions and key signatures.\nThe paper introduces a conditional generation setup, where fine-grained harmonic constraints guide the sampling process, leading to a controlled and structured output.\nTheoretical justifications are provided, demonstrating the necessity of such structured control in diffusion-based symbolic music generation.\nTo validate the approach, the authors provide both theoretical and empirical evidence, showing that FGG prevents out-of-key generation and enhances structural coherence in generated compositions.\nThe paper also presents an interactive demonstration, showcasing  generation capabilities.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's suggestions on revising! Due to this year's policy, it is not allowed to upload a revised paper, external links can only contain figures/tables, and rebuttal has a 5000 length requirement. Please allow us to describe a revision plan in the follows and promise to follow through in next phase.\n\n**Responses**\n1. To train a model that can handle both rhythm+chord and chord-only conditions, we must distinguish when rhythmic conditions are provided. We achieve this by using negative values to indicate the absence of rhythmic conditions, preventing the model from misinterpreting 0s and 1s as rhythmic constraints applied everywhere. This design choice was determined through empirical experimentation.\n\n2. **Chord progressions** are harmonic structure of music, and the latent representation better accounts for the degrees of similarity between chords. For features like **pitch, duration, and note density**, we evaluate the **overlapping area** between their distributions in generated and ground truth segments, to evaluate how well the generative model captures the statistical properties inherent in the original compositions.​ We also **added three metrics** to directly measure generation quality: Direct Chord Accuracy, IoU of Chord, and IoU of Piano Roll. New results are shown in [https://drive.google.com/file/d/1IAcAqK4qK4AiQVKWriFSJ91QNhrg5at-/view]( https://drive.google.com/file/d/1IAcAqK4qK4AiQVKWriFSJ91QNhrg5at-/view).\n\n3. See \"related literature\" in the following.\n\n**Revisions in writing**\n\n**1. Contribution**:\n\nMotivation: We theoretically and empirically characterize the challenge of *precision* in symbolic music generation\n\nMethodology: We incorporate fine-grained harmonic and rhythmic guidance to symbolic music generation with diffusion models.\n\nFunctionality: The developed model is capable of generating music with high accuracy in pitch and consistent rhythmic patterns that align closely with the user’s intent.\n\nEffectiveness: We provide both theoretical and empirical evidence supporting the effectiveness of our approach.\n\n**2. Related Work**:\n\nThank you for your suggestions, we really appreciate the structure line of organizing this paragraph that you have provided us. We will integrate it as:\n\nTo leverage well-developed generative models for symbolic music, Huang et al. (2018) introduced a Transformer-based model with a novel relative attention mechanism designed for symbolic music generation. Subsequent works have enhanced the controllability of symbolic music generation by incorporating input conditions. For instance, Huang and Yang (2020) integrated metrical structures to enhance rhythmic coherence, Ren et al. (2020) conditioned on melody and chord progressions for harmonically guided compositions, and Choi et al. (2020) encoded musical style to achieve nuanced harmonic control. These advancements have contributed to more interpretable and user-directed music generation control.\n\nTo better capture spatio-temporal harmonic structures in music, researchers have adopted diffusion models with various control mechanisms. Min et al. (2023) incorporated control signals tailored to diffusion inputs, enabling control over melody, chords, and texture. Wang et al. (2024) extended this by integrating hierarchical control for full-song generation. To further enhance control, Zhang et al. (2023) and Huang et al. (2024) leveraged the gradual denoising process to refine sampling. Building on these approaches, our work addresses the remaining challenge of precise control in real-time generation.\n\n**3. Theory**\n\nWe will make the theoretical proofs in appendix more detailed by (1) expanding more steps and (2) adding more recalls of formulations from the text directly to the appendix. We will also simplify proposition 2 by removing the conditional probability argument, the details of which are in our response to reviewer uD9t.\n\n**4. Sections 2 and 3.**\n\nWe will add subtitles to section 2, namely “Data representation of the piano roll” and \"Formulation of the diffusion model\", at corresponding places.\n\nWe will delete the third paragraph “The FGG method improves…” of section 3.\n\nWe will revise the first paragraph of section 3.2 to:\n\n> We first provide a rough idea of the harmonic sampling control. To integrate harmonic constraints into our model, we employ temporary tonic key signatures to establish the tonal center. Our sampling control mechanism guides the gradual denoising process to ensure that the final generated notes remain within a specified set of pitch classes. This control mechanism removes or replaces harmonically conflicting notes, maintaining alignment with the temporary tonic key.\n\nWe will add subtitles or topic sentences “mathematical formulation of the harmonic sampling control”, \"preliminaries\", \"Edit intermediate-step outputs of the sampling process\" and “theoretical property of the sampling control” to corresponding places of section 3.2, and do further refinement.", "claims_and_evidence": "### The paper makes three primary claims:\n1. The proposed fine-grained guidance mechanism ensures strict in-key generation.\n2. Theoretical justifications demonstrate the necessity and effectiveness of FGG.\n3. Empirical evaluation supports the effectiveness of FGG in improving symbolic music generation.\n\n\n### Assessment of Claims and Supporting Evidence\n1. Fine-Grained Guidance (FGG) for Strict Harmonic Control\n- Supported Evidence: The methodology section details how fine-grained constraints are integrated into the generative process, ensuring adherence to predefined harmonic structures. The 0.0% off-key note generationresult confirms this strict control.\n- Evaluation: Well-supported. The claim is effectively demonstrated, as the model enforces harmonic rules explicitly, leading to an expected perfect in-key generation.\n2. Theoretical Justification of FGG’s Necessity and Effectiveness\n- Supported Evidence: Theoretical formulations, including Proposition 1 and Proposition 2, argue the necessity of structured control in diffusion-based symbolic music generation.\n- Evaluation: Partially supported. The paper is not self-contained, requiring extensive prior knowledge in statistical methods and diffusion models. Several key derivations rely on unstated assumptions, making it difficult for non-expert readers to follow.\n- Recommended Improvement: The presentation should be more self-contained, with additional intermediate steps, particularly in the appendix, to improve accessibility. Re-formalizing the problem setup and explicitly deriving key constraints, such as the first constraint over $\\epsilon_{l,h}$ in Appendix A.1, would make the proofs much more understandable.\n3. Empirical Evaluation Demonstrating Effectiveness\n- Supported Evidence: The authors conduct experiments using cosine similarity via a VAE latent representation and overlapping area (OA) measures to assess generation quality.\n- Evaluation: Not convincingly supported. Aside from strict off-key detection, the evaluation metrics lack clear intuition regarding what they measure. The rationale behind using VAE-based cosine similarity and OA instead of direct accuracy or intersection-over-union (IoU) is unclear.\n- Recommended Improvement: The authors should either justify their choice of evaluation metrics or adopt more interpretable alternatives that directly measure harmonic consistency and generation quality.", "methods_and_evaluation_criteria": "The methodological approach of integrating harmonic constraints in diffusion-based symbolic music generation is novel and well-motivated. However, the evaluation criteria lack clarity and justification. The authors use VAE-based cosine similarity and overlapping area (OA) measures to assess performance, but these metrics are not well explained, and their relevance to assessing generation quality is uncertain.\nAlternative evaluation methods, such as direct accuracy, intersection-over-union (IoU), or out-of-key note ratios beyond 0.0%, could provide more interpretable and robust validation.\nMoreover, justification for some design choices are lacking, e.g. why did you use negative values for the condition piano roll when no rhythm control is given?", "theoretical_claims": "See claims and evidence.", "experimental_designs_or_analyses": "The experimental setup demonstrates the technical feasibility of the approach but fails to convincingly validate the claimed improvements. The 0.0% off-key rate is expected due to strict constraints, and the other evaluation metrics do not provide clear insight into how well the method improves generative quality.\n\nThe paper would benefit from:\n- Justifying the choice of VAE-based cosine similarity and OA as evaluation metrics.\n- Introducing alternative, more interpretable accuracy metrics.\n- Expanding empirical analysis beyond off-key note prevention.", "relation_to_broader_scientific_literature": "The progression made prior to this work is insufficiently described at the opening sections of this work. \nThe related work section does name dropping rather than giving context as to what recent works did to tackle the challenges specified by you in the introduction part, what is the progression made from one to the other? What key contribution did they introduce? How do they aim to address the gaps you pointed out in the introduction?\nYou should draw the progression for the reader to better understand the context before diving into the specifics in the next sections.\nThe current related work serves to specify some differences of the suggested approach from some other prior works, not addressing any of the above.", "essential_references_not_discussed": "- I believe that the related work section needs to be revised as a whole and to draw the progression made in recent years, point out the gaps and how other people tried to address them. \n- Theoretical derivations could benefit from expanding derivation steps, see comment in Claims And Evidence.\n- Justification for Empirical evaluation metrics are required.", "other_strengths_and_weaknesses": "### Strengths:\n1. Novel structured control mechanism for enforcing harmonic accuracy.\n2. Theoretical rigor in proving the necessity and effectiveness of the approach.\n3. Well-illustrated methodology and results.\n4. Practical value through an interactive demo.\n\n\n### Weaknesses:\n1. Writing clarity issues: The writing requires significant effort to follow due to structural issues and ambiguous wording. Examples:\n- The contribution summary should be clearer and more structured. Instead of long paragraphs, each contribution should be presented in a concise sentence.\n- Related work section lacks coherence, with frequent references to the authors' own work disrupting the logical flow. Example: \"We adopt classifier-free guidance...\" appears prematurely in Related Work rather than in Methods.\n2. Justification for evaluation metrics and design choices:\n- The authors do not justify why negative values are used for the condition piano roll during training when no rhythm control is given.\n- The choice of evaluation metrics (VAE-based cosine similarity, OA) lacks clear reasoning.\n3. Lack of self-containment:\n- Key derivations require extensive prior knowledge and are not well explained.\n- Theoretical justifications, especially in Appendix A.1, should explicitly present intermediate steps and provide clearer intuition behind their design choices.", "other_comments_or_suggestions": "Overall I think that this work is interesting, and shows great performance by imposing domain knowledge constraints on the generative process, and demo also is very fun to play with and shows very nice generations.\nThat being said I strongly suggest a careful revision to improve clarity, as I found it difficult to follow key claims throughout the paper..\nYou should be more structured, well defined and avoid vague wording. \nSpecifically related work, background and method should be revised. make sure your claims are plainly stated, your design choices justified and that the problem setup is based before moving on.\nAlthough I overall like this work yet due to lacking clarity along the paper I find it difficult to confidently give a high ranking at this stage.\nPlease address my concerns, revise your writing and I shall increase my graded score.", "questions_for_authors": "1. Why do you use negative values for the condition piano roll when no rhythm control is given?\n2. How do you justify your choice of evaluation metrics? could you give motivation to what they intend to capture and how does the score demonstrates that? Could you supply additional, more interpretable metrics of evaluation?\n3. Could you elaborate on how prior work in this field addressed the gaps you highlight in the introduction? what gaps remain?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mpxGAj2VGM", "reviewer": "Reviewer_XA1t", "summary": "This paper proposes a method called Fine-Grained Guidance, FGG, to improve precision and controllability in diffusion-based symbolic music generation, especially in the domain of tonal music where out-of-key notes can be perceived as mistakes.\n\n## update after rebuttal\n\n**I raised the score to 4 as the rebuttal addressed my concerns.**", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's valuable comments. Please allow us to provide responses as follows:\n\n**1. Using this FGG is of course beneficial for most of the cases, but it \"eliminates\" the generation of such tension-imposed music. How can the authors justify this?**\n\nWe agree that some composers intentionally put out-of-key notes to add “creativity”. However, it is observed that generative models often fail to create an accommodating context for such “creativity”. In other words, most of the out-of-key notes generated by models cannot add creativity, but instead disrupts harmony. We think the main reason is that out-of-key notes are relatively rare in most datasets, which makes it difficult for the model to learn how to “correctly” use out-of-key notes. Given such a limitation in data, the benefit of avoiding out-of-key notes could outweigh the loss of potential creativity.\n\n**2. I'm not sure if Proposition 2 is tight enough. If it's not tight, then the ground of Section 4 is largely weaken.**\n\nWe also cannot theoretically verify whether it is tight. We would like to clarify the following: If Proposition 2 is not tight—meaning the error probability decays *even slower* than $O(n^{−1/(LH)})$—then the probability of generating \"unresolved wrong notes\" remains even higher. Therefore, the lack of tightness in Proposition 2 does not undermine the main argument in Section 4.\n\n**3. Could the authors provide more intuition of Eq. (8)?**\n\nEq. (8) ensures that: if (l,h) is not an out-of-key position, predicted $X_0$ will be unchanged; if (l,h) is out-of-key and predicted $X_0$ exceeds 0.5, then predicted $X_0$ will be reset to 0.5. This acts as a projection method, enforcing that out-of-key notes cannot be present. The threshold of 0.5 is chosen because, in piano roll quantization, values <= 0.5 mean no note, while values >0.5 indicate a note. Instead of correcting only at the final step, we apply this constraint throughout the sampling process to maintain the flexibility of diffusion models while ensuring harmonic coherence.\n\n**4. Why is music treated as fundamentally different compared to image or video generation?**\n\nMusic generation follows two main approaches: treating music as an image (e.g., piano roll representation) or as language (e.g., sequences of pitch, duration, and timing tokens). However, it has the following challenges:\n\n- Precision Sensitivity – In image generation, slightly changing a single pixel usually has minimal impact, whereas one misplaced note in music can significantly disrupt harmony.\n\n- Temporal Overlaps – Unlike text, where words follow a sequential order, musical notes often overlap, which leads to challenges when conducting music generation in the “next token predictor” manner.\n\n- Genre-Specific Rules – Music follows complex, style-dependent structures, such as tonal harmony in classical music, or intricate rhythms in jazz.\n\nThese factors make it difficult to develop a unified, generalized model for music generation. For now, the fragmentation in music models is partially due to the complexity of musical structure and the precision required to generate high-quality music. But as generative AI matures, we should aim for models that can adapt dynamically to different musical styles rather than being strictly bound to a single one. In fact, our control method can also be applied in stylized music generation (see the next following paragraphs).\n\n**5. Does our method has a broader impact or scalability?**\n\nWe think that our proposed method (especially the sampling control part) is generalizable. In our paper, we enforce a \"no out-of-key notes\" constraint within the sampling process. As shown in the demo page, this method can be applied to generate stylized music with special scales (e.g., Chinese pentatonic scale). \n\nMore broadly, our method can be adapted to various musical constraints. Many genres and styles can be defined through rules (e.g., harmonic, rhythmic, or structural constraints), which our sampling method can also incorporate. For example, rhythmic control could be implemented by ensuring that certain time positions must (or must not) contain notes. Then we can incorporate this requirement in the sampling process as follows: in each step, we edit $\\epsilon$ to project predicted $X_0$ to the correct domain where the rhythmic constraint is satisfied. Such rhythmic control method could be used for generating jazz music which has special rhythmic requirements.", "claims_and_evidence": "- Tables give evidence that the out-of-key notes are never generated. However, the composers intentionally put these out-of-key notes to give some tension to the model. Using this FGG is of course beneficial for most of the cases, but it \"eliminates\" the generation of such tension-imposed music. How can the authors justify this?\n- I'm not sure if Proposition 2 is tight enough. If it's not tight, then the ground of Section 4 is largely weaken.\n- Could the authors provide more intuition of Eq. (8)? If (l,h) is in w_K(l), then epsilon will be the maximum value, but I could not imagine easily of the intuition of $\\frac{1}{\\sqrt{1-\\alpha_t}}$ $(X_{t,lh}-\\frac{\\sqrt{\\alpha_{t}}}{2})$. If you plug in the value to Eq. (6), then the predicted X_0 will be 0.5. Could you explain more about this and the general intuition behind it?\n- Can you apply this method to non-tonal, avant-garde, or jazz style music? While I appreciate the strong technical contributions of this work, I would like to raise a broader discussion point regarding modality-specific research directions, particularly in the audio domain. Unlike other generative domains such as vision or language, where models are typically expected to generalize across all possible natural images or text, audio — and especially symbolic music — remains heavily siloed. We often see separate models for speech, music, and even for specific genres like pop, classical, or jazz. This raises the question: why is music treated as fundamentally different in its modeling approach compared to image or video generation? Should we continue to develop highly specialized generative models for narrow musical domains, or should the field aim toward more unified, generalized audio models? The current direction in symbolic music seems to serve a small subset of expert users, and it’s unclear whether this granularity is justified in terms of broader impact or scalability. I fully acknowledge that each modality has unique properties — but from a reviewer’s perspective, this fragmentation affects how we measure and compare contributions across fields. I do not see this as a flaw of the paper per se, but I encourage the authors and the community to consider this broader question of generality, user impact, and the future direction of audio generation research.", "methods_and_evaluation_criteria": "I did not carefully read the experiments.", "theoretical_claims": "I didn't read proofs.", "experimental_designs_or_analyses": "The experimental section looks solid enough.", "relation_to_broader_scientific_literature": "Related to the general public in this community who are particularly interested in music domain (and AI for science given that they also want to generate a sample that a certain constraint is met)", "essential_references_not_discussed": "References are enough", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "-", "questions_for_authors": "-", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes fine-grained guidance to control harmonic characteristics in diffusion-based symbolic music generation on piano-roll format.

- Strength
    
    - Reviewers all acknowledged the necessity of precise control of harmonic quality in symbolic music generation.
    - The proposed method has a theoretical justification and empirical evidence to support its effectiveness.
    - The generated results are of high quality, both in subjective listening tests and interactive demos.
- Weakness
    
    - The writing and structure of the paper has clear margin to be improved. The reviewers suggested several ideas for the revision, and the authors provided revision plan to address this issue.

The proposed method clearly achieved the goal. It would benefit symbolic music research by providing powerful controllability and the broader ML community through potential application of guiding diffusion with specific fine-grained conditions.
Decision: Accept (poster)