NEURAL BAND-TO-PIANO SCORE ARRANGEMENTWITH STEPLESS DIFFICULTY CONTROL
Moyu Terao1

Eita Nakamura1,2

Kazuyoshi Yoshii1,3

12Graduate School of Informatics, Kyoto University, JapanThe Hakubi Center for Advanced Research, Kyoto University, Japan
3
PRESTO, Japan Science and Technology Agency (JST), Japan

ABSTRACTThis paper describes a music arrangement method of popular musicthat can convert a band score into a piano score with a steplesslyspecified level of performance difficulty. The basic strategy of bandto-piano score arrangement is to select notes from an augmented
band score obtained by up- and down-shifting the notes of an original
band score by one octave. Given band scores and the corresponding
piano scores with elementary- and advanced-levels, one can train a
deep neural network (DNN) that estimates note masks conditionedby the difficulty levels. Conditioned by an intermediate level at runtime, however, the DNN tends to generate an advanced-level score.
To solve this problem, assuming that an easier piano score is a subsetof harder one, we estimate the basic importance of each note with adifficulty-agnostic DNN and then warp it with a power function depending on a specified difficulty level. To achieve the fine controllability of the difficulty level, we propose a training method that subjects the DNN to generating piano scores with various intermediatelevels, where the note-level loss for those scores is evaluated using
only the ground-truth elementary- and advanced-level scores. Considering the non-uniqueness of piano arrangement, the statistic-levelloss with respect to the note density and polyphony level is also computed according to the given levels. The experimental results showedthat the proposed method attained both the performance gain and thestepless difficulty control.
Index Terms— Automatic piano arrangement, score reduction,
symbolic music processing, deep learning
1. INTRODUCTION
One typical way of music arrangement is to change the instrumentation of a musical piece while preserving its essential musical content(rhythm, melody, and harmony). Much effort has been devoted toautomatic piano arrangement [1–5], guitar arrangement [6–8], andorchestration [9–11]. In this paper we tackle piano arrangement ofband music with stepless difficulty control.
One of the most important requirements in band-to-piano scorearrangement is to allow a user to steplessly control the difficulty levelof the generated piano score according to his or her skill and preference (Fig. 1). If a user gradually moves the slider of the difficultylevel from the elementary level to the advanced level, it would be better in terms of usability that the generated piano score also changes
gradually, i.e., notes are added gradually. The underlying assumption is that an easier-level piano score can be obtained as a subset of aharder-level score if both scores originate from the same band score.
Since the music arrangement is a challenging one-to-many mappingtask, this assumption can effectively reduce the search space.
This work is supported in part by JST PRESTO No. JPMJPR20CB and
JSPS KAKENHI Nos. 19H04137, 20K21813, and 22H03661.

Melody{Band score1

Accompaniment

0.3
0Elementary

Estimated piano score
0.3
1
Advanced

0

Fig. 1. The proposed band-to-piano score arrangement system withstepless difficulty control.
The score reduction approach has typically been taken for bandto-piano score arrangement. One can obtain a piano score by selecting notes from an augmented band score obtained by up- anddown-shifting the notes of an original band score by one octave.
Given paired data of band scores and the corresponding piano arrangements, which are typically categorized into only elementary
and advanced levels, one can train a DNN that estimates masks used
for note selection, conditioned by the difficulty levels [12]. Conditioned by an unseen intermediate level at run-time, however, theDNN trained with binary conditions tends to generate an advancedlevel piano score.
Under these circumstances, we propose a difficulty-controllable
band-to-piano score arrangement method based on the score reduction approach (Fig 2). This method estimates the basic importanceφ of each note in an augmented band score with a difficulty-agnostic
DNN and then warps it with a difficulty-dependent power function
φα , where a harder level takes a smaller α. If the resulting importance φα exceeds a threshold, the notes is selected to be included in
the piano arrangement. The estimated importance φα is thus guaranteed to monotonically increase according to the difficulty level.
Thanks to the differentiable warping function, the DNN and the values of α for the elementary and advanced levels can be optimized
jointly using paired data consisting of only these two levels. The
value of α between these two values can then be used for an intermediate level.
To improve the controllability of the difficulty level at run-time,i.e., to let the generated piano score steadily change from the elementary level to the advanced level, we subject the DNN to generating
piano scores with various intermediate levels not limited to the elementary and advanced levels in the training phase. The essential
problem is that such generated piano scores cannot be evaluated because only the elementary and advanced piano scores are given as
the ground-truth data.
Considering the non-uniqueness of band-to-piano score arrangement as suggested in [12], we propose a regularized training method

3. PROPOSED METHOD
Difficulty
Mask(Elementary)Band score

U-Net

This section describes the proposed method that converts a bandscore into a piano score with an arbitrary difficulty level.

Mask(Advanced)

3.1. Problem Specification
Onset importance(Elementary)Band score

U-Net

Basic importance
Onset importance(Advanced)

Fig. 2. Comparison of the conventional and proposed methods. The
conventional method directly estimates the selection probability foreach note of a band score with a DNN conditioned by a difficultylevel. The proposed method first estimates the basic selection probability for each note with a difficulty-agnostic DNN and then warps
it according to the difficulty level.
based on note- and statistic-level (instance- and distribution-level)
criteria that can evaluate piano scores with arbitrary difficulty levels.
When a piano score with a randomly-sampled level is generated, its
note-level loss is a weighted sum of the loss using available groundtruth scores. To regularize the training, the statistic-level loss of thegenerated score is also introduced such that the distributions of thenote densities and the polyphony levels are made close to those corresponding to the randomly-sampled level.
The main contribution of this study is to propose neural piano
arrangement based on note importance estimation for run-time stepless difficulty control. We experimentally investigate the potential ofnote importance estimation in difficulty control and the effectivenessof considering the both the note- and static-level losses.
2. RELATED WORK
A standard approach to piano arrangement of multi-instrument music (e.g., orchestra and popular music) is score reduction based onnote selection. Huang et al. [11] proposed an orchestration methodthat generates a score of a specified musical instrument from a scoreof ensemble music by segmenting the instrument tracks in termsof musical content, determining their musical roles (e.g., lead andpad), and selecting phrases. Takamori et al. [3] proposed a band-topiano arrangement method that focuses on musical content such as
melodies, chords, rhythms, and the number of notes extracted froman original score. This method selects accompaniment patterns froma database of accompaniment parts collected from piano scores of
popular music. They further attempted piano arrangement from audio data [13]. Wang et al. [14] pointed out that the use of predefined
accompaniment parts leads to a limited variety of arrangements andthus used a DNN for flexible piano arrangement. These arrangement
methods focus on the constraints of piano scores in terms of the notedensity and the number of simultaneous notes.
The player’s performance skill has recently been considered as
an important factor for music arrangement. Nakamura et al. [4]pointed out that the weights of the constraints on piano score should
be varied according to the player’s skill and that more strict constraints are needed for elementary-level players. They proposed amethod that represents the performance difficulty as a continuous
variable and also compared the difficulty levels in terms of the frequency of performance errors. Our previous band-to-piano arrangement method can generate only elementary- and advanced-level piano scores, where the note statistics are considered to deal with the
non-uniqueness of piano arrangement [12].

Our goal is to convert a band score X , {XA , XM } into a pianoscore Y , {YL , YR } with an arbitrary difficulty level γ ∈ [0, 1],
where XA , {OA , PA } and XM , {OM , PM } represents the condensed accompaniment part and the melody (vocal) parts, respectively, and YL , {OL , PL } and YR , {OR , PR } represent the leftand right-hand parts, respectively. Each part is represented as an onset matrix O∗ ∈ {0, 1}P ×N and a pitch matrix P∗ ∈ {0, 1}P ×N(∗ ∈ {A, M, L, R}), where P represents the number of pitches(P = 128) and N represents the number of tatums (sixteenth notes)of the piece. In the melody part of the band score, for example,
OM (p, n) = 1 represents the presence of an onset at pitch p andtatum n and PM (p, n) = 1 represents the presence of pitch p attatum n. Let h ∈ {L, R} denote the left or right hand part.
To train a DNN-based conversion model, we use pairs of band
and piano scores as training data. Note that each X in the training
data is associated with a piano score Y with either of the elementary
and advanced levels, i.e., intermediate-level piano scores are not included in the training data.
3.2. Score Reduction Approach
We take the score reduction approach to band-to-piano arrangement.
The statistical analysis of existing piano arrangements showed thaton average, 92% of the notes on the right-hand part and 81% of theleft-hand part are derived from the notes from an augmented bandscore obtained by up- and down-shifting the notes of an original bandscore by one octave [12]. This fact justifies the assumption underlying our score reduction approach that a reasonable piano score can beobtained by selecting important notes, with octave shifts if necessary,from an augmented band score, i.e., a piano score can be obtained asa subset of an augmented band score. Let OB ∈ {0, 1}P ×N be anaugmented onset matrix given by
OB (p, n) =max
j∈{−12,0,12}(OA (p + j, n), OM (p + j, n)) ,(1)

where j ∈ {−12, 0, 12} represents the octave shift.
Given a difficulty level γ, we estimate selection probability (noteimportance) matrices π , {πL , πR } for the notes of an augmented
band score, where πL , πR ∈ [0, 1]P ×N represent the importancematrices used for estimating the left- and right-hand parts of the piano score. The note importance is defined at only the note onsetsof the augmented band score, i.e., if OB (p, n) = 0, πL (p, n) =πR (p, n) = 0. The piano onset matrices OL and OR are obtainedby binarizing πL and πR with a threshold, respectively. The piano
pitch matrices PL and PR are determined by referring to the onsets
and durations of the selected notes of the augmented band score.
3.3. Supervised Training
In the training phase, a pair of a band score X and the corresponding
elementary- or advanced-level piano score Y is given. A lot of suchpairs are given as training data in practice, but no intermediate-level
piano scores are available, i.e., only elementary- and advanced-level
piano scores can be used as target data. Using such training data, we
jointly optimize the U-Net parameters and the power exponents α0
and α1 corresponding to the elementary and advanced levels. A key

Basic importance

Elementary1

Conv (4→4)
Conv (4→8)
Conv (8→16)

skip connection
skip connection
skip connection

Conv (16→32)

Deconv (8→4)

Threshold

Deconv (16→4)0

Deconv (32→8)

OB ,

α

πR = φRγ

OB ,

1

0

1

αγ = (1 − γ)α0 + γα1 .

(3)Using a difficulty-independent threshold for πL and πR , a larger
number of notes are selected for a higher-level piano score.
Considering the non-uniqueness of piano arrangement, the importance matrices π given by Eq. (2) are evaluated at the note andstatistic levels (instance and distribution-level matching). When atarget level γ is equal to the difficulty level (0 or 1) of the groundtruth piano score, the note- and statistic-level losses can be computedin the same way as [12]. We here propose a novel regularized training method that minimizes the sum of the note- and statistic-levellosses computable for an arbitrary level γ.
3.3.1. Note-Level Loss
We define the note-level loss for evaluating the importance matrices
π estimated for a difficulty level γ ∈ [0, 1]. If the ground-truth onset
matrices O , {OL , OR } of the left- and right-hand parts with thesame difficulty level γ are available, one can use the modified binary
cross entropy defined as follows:
P X
N
X

γ∈
/ {0, 1}. We thus propose to use the following loss:((1 − γ)BCE(π, O) if level(O) = 0,nt
L =γBCE(π, O) if level(O) = 1,

w · Oh (p, n) log πh (p, n)h∈{L,R} p=1 n=1+ 1 − Oh (p, n) log 1 − πh (p, n) , (4)where w ≥ 0 is a weighting factor used for compensating for theimbalance of the numbers of onset and non-onset frames (w = 4 inthis paper). This strategy, however, cannot be applied to the case of(5)

where level(O) represents the difficulty level of O. For example,
when γ = 0.3 (closer to the elementary level), the binary crossentropy of π for the elementary- or advanced-level piano score isconsidered with a wight of 0.7 or 0.3, respectively.
3.3.2. Statistic-Level Losses
We define the statistic-level losses used for regularizing the supervised training of the U-Net with the note-level loss. Let Clv
h (n) denote the polyphony level (the number of concurrent pitches) at tatumn, and let Cds
h (m) denote the note density (the number of onsets ina measure) at measure m, in the estimated piano onset matrices O(h ∈ {L, R}), which are given by(2)

where denotes the element-wise product and αγ is a power exponent depending on the difficulty level γ ∈ [0, 1]. Note that alower αγ is used for a higher level γ, i.e., α0 ≥ αγ ≥ α1 and
αγ
0
1
φα
≤ φα
h ≤ φhh (Fig. 4). The power exponent αγ corresponding to an intermediate level γ is computed with linear interpolation
between α0 and α1 corresponding to the elementary and advanced
levels (optimized as explained later) as follows:

BCE(π, O) = −Threshold1
Basic importanceFig. 4. Note importance warping with power functions.

feature of the proposed method is that piano scores with randomlysampled difficulty levels not limited to the elementary and advanced
levels are generated in the training phase.
We first estimate difficulty-independent importance matrices
φ , {φL , φR } with a convolutional neural network (CNN) called
the U-Net [15] that takes the band score X as input (Fig. 3), where
φL , φR ∈ [0, 1]P ×N represent the basic importance values over the
pitch-tatum space used for estimating the left- and right-hand parts,
respectively. To estimate the difficulty-dependent matrices π, wethen warp φ with an element-wise power function as follows:X

1

Deconv (32→16)

Fig. 3. The detailed architecture of the U-Net. The numbers in theparentheses represent the change in the number of channels. Skip
connection denotes concatenation of channels.

αAdvanced
1

0

U-Net

πL = φL γ

Onset importanceBand scoreClv
h (n) =PX

Oh (p, n), Cds
h (m) =

p=1

P
X X

Oh (p, n). (6)n∈G(m) p=1

where G(m) represents the set of tatum indices in measure m. Notethat the piano onset matrices OL and OR are stochastically determined according to πL and πR , respectively, in a differentiable manner with the gumbel sigmoid function [16], instead of performing
simple thresholding used at run-time.
We then compute the empirical distribution (histogram) Qdh ∈d[0, 1]I +1 by normalizing the tatum- or measure-level frequenciesd
Ch over all possible values, where I d represents the maximum
polyphony level or the maximum note density and d ∈ {lv, ds}
represents the polyphony level or the note density.
We aim to make the histogram Qdh close to the ground-truth
histogram Q̄dh,γ corresponding to the difficulty level γ in a distributional sense. What we can do with the training data, however,is to estimate the ground-truth histograms Q̄dh,0 and Q̄dh,1 from theexisting elementary- and advanced-level piano scores, respectively.
Another key feature of the proposed method is to estimate the intermediate histogram Q̄dh,γ with Wasserstein interpolation between
Q̄dh,0 and Q̄dh,1 [17]. The static loss with respect to d ∈ {lv, ds} isgiven by the Jensen-Shannon (JS) divergence as follows:


X
Ld =
DJS Q̄dh,γ k Qdh .
(7)h∈{L,R}

4. EVALUATIONThis section reports an experiment conducted for evaluating the effectiveness of the proposed method.
4.1. Experimental Conditions
We collected 184 pairs of band and piano scores (85 elementarylevel arrangements and 99 advanced-level arrangements). We used
randomly-chosen 138 pairs as a training set, and the remaining 46
pairs as a test set. We set the analysis window size to 12 measures

Table 1. The F-measures were evaluated for the ground-truth scores (left side of slash) and for their modified version obtained by removing
notes not included in the augmented scores (right side of slash). The statistic-level losses were measured for three difficulty levels γ = 0, 0.5, 1

Conventional
Proposed

Lnt
X
X
X

Llv

Lds

X

X

Llv
med

Lds
med

X
X

X

X

F [%]
Left
Right
20.9 / 22.8 55.3 / 58.0
25.3 / 28.0 56.2 / 58.6
25.2 / 27.5 57.9 / 60.7
24.8 / 26.7 57.8 / 60.6
24.8 / 26.8 55.9 / 58.4

Ground-truth
0.2
Elm

0.8

Med
Adv

0.4

Conventional(
0.2

0.8
0.4

Llv (×102 )Elm Med Adv
15.3 12.6 11.4
15.9 8.82 8.43
11.0 7.41 9.66
11.4 9.29 13.4
11.4 7.96 13.7

Lds (×102 )Elm Med Adv
5.54 17.2 10.1
5.43 3.55 5.35
4.81 4.11 9.70
9.65 7.63 13.9
6.87 5.43 14.6

Ground-truth

0.1
0.0

0.0

)

0.1

0.0

Proposed (

0.8
0.4
0.0

Losses
Lnt
med

0.0

)

0.2
0.1

0

1

2 3
Left

4 0

1

2 3
Right

4

0.0

0

10
20
Left

0

10
20
Right

Fig. 5. The distributions of polyphony levels for the left- and righthand parts with difficulty levels γ = 0, 0.5, 1. The top row shows
the ground-truth distributions for γ = 0, 1 and their barycenter for
γ = 0.5. The middle and bottom rows show the distributions computed from piano scores estimated by the conventional and proposed
methods.
and assumed the time signature to be 4/4. When the length of a measure was shorter than 16 tatums, it was fill with silent. Conversely,
when a measure was longer than 16 tatums, the first 16 tatums were
preserved. Data augmentation was performed on the training set bytransposing the keys by from 1 to 11 semitones.
For the U-Net, we set the kernel size to 4, stride to 2, andpadding to 1. Dropout (p = 0.5) was applied to all the deconvolutional layers. The Adam optimizer with a learning rate of 10−4 wasused for training [18]. We set the threshold to 0.5. The note-leveland statistic-level losses were treated equally.

Fig. 6. The output piano score Y estimated by the model optimized
with Lnt + Llv + Lds for a difficulty level γ. The larger γ, the moredifficult the piano score. The red notes represent the notes added tothe one-step easier score.
termediate losses, where γ represents the difficulty level. It was observed that the more difficult the piano score, the greater the numberof notes (see other examples on our webpage1 ).

4.2. Experimental Results
We examined ground-truth piano scores and investigated how muchnotes of the elementary-level scores were included in the advancedlevel scores. In 55 pairs of elementary- and advanced-level pianoscores, 75% of the right-hand notes and 47% of the left-hand notesof the elementary-level scores were included in the advanced-level
scores. This result shows that for the right-hand part the assumptionthat an elementary-level score is a subset of an advanced-level scoreis appropriate, and for the left-hand part the assumption might not
hold. Considering the non-uniqueness of piano arrangement, thisassumption still seemed reasonable for our task to effectively reduce
the solution space.
Table 1 shows the evaluation results. The proposed method outperformed the conventional method in terms of not only F but also
Llv and Lds . We found that Llv and Lds were reduced for intermediate levels. Using intermediate scores (Lmed ) with random difficultylevels was not found to be effective. Fig. 5 shows that the proposed
method is capable of fine control at intermediate levels.
Fig. 6 shows examples of piano arrangement estimated with theproposed model trained with the statistic-level losses without the in-

5. CONCLUSION
We proposed a neural piano arrangement method that can control
the difficulty levels of output piano scores in a stepless manner. Thismethod uses a difficulty-agnostic U-Net for estimating the basic importance of each note in an augmented band score and then warps
it with a power function depending on a specified difficulty level.
Given a band score, an easier-level piano score can thus be obtained
as a subset of harder-level one.
Experimental results showed that the regularized training with
the note- and statistic-level losses improved the onset match rate andstatistical characteristics. We quantitatively confirmed that the proposed method can control the difficulty level steplessly. On the other
hand, we have observed that the intermediate loss does not effectively improve the onset match rate and statistic distance. We plan to
conduct a subjective evaluation experiment to examine the validityof the output piano scores for human performance.
1 https://teraomoyu.github.io/difficulty-controllable-pianoarrangement.github.io/6. REFERENCES[1] S. Chiu, M. Shan, and J. Huang, “Automatic system for thearrangement of piano reduction,” in Proc. International Symposium on Multimedia, 2009, pp. 459–464.
[2] S. Onuma and M. Hamanaka, “Piano arrangement system
based on composers’ arrangement processes,” in Proc. International Computer Music Conference, 2010, pp. 191–194.
[3] H. Takamori, H. Sato, T. Nakatsuka, and S. Morishima, “Automatic arranging musical score for piano using important musical elements,” in Proc. Sound and Music Computing Conference, 2017, pp. 35–41.
[4] E. Nakamura and K. Yoshii, “Statistical piano reduction controlling performance difficulty,” APSIPA Transactions on Signal and Information Processing, vol. 7, no. e13, pp. 1–12,
2018.
[5] Z. Wang, K. Chen, J. Jiang, Y. Zhang, M. Xu, S. Dai, X. Gu,
and G. Xia, “Pop909: A pop-song dataset for music arrangement generation,” in Proc. International Society for Music Information Retrieval, 2020, pp. 38–45.
[6] D. Tuohy and WD. Potter, “A genetic algorithm for the automatic generation of playable guitar tablature,” in Proc. International Computer Music Conference, 2005, pp. 499–502.
[7] Y. Yoshinaga, S. Fukayama, H. Kameoka, and S. Sagayama,“Automatic arrangement for guitars using hidden Markov
model,” in Proc. Sound and Music Computing Conference,
2012, pp. 450–456.
[8] G. Hori, H. Kameoka, and S. Sagayama, “Input-output HMM
applied to automatic arrangement for guitars,” Journal of Information Processing, vol. 21, no. 2, pp. 264–271, 2013.
[9] H. Maekawa, N. Emura, M. Miura, and M. Yanagida, “On machine arrangement for smaller wind-orchestras based on scoresfor standard wind-orchestras,” in Proc. International Conference on Music Perception and Cognition, 2006, pp. 268–273.
[10] L. Crestel and P. Esling, “Live orchestral piano, a system forreal-time orchestral music generation,” in Proc. Sound andMusic Computing Conference, 2017, pp. 434–442.
[11] J. Huang, S. Chiu, and M. Shan, “Towards an automatic musicarrangement framework using score reduction,” ACM Transactions on Multimedia Computing, Communications, and Applications, vol. 8, no. 1, pp. 8:1–8:23, 2012.
[12] M. Terao, Y. Hiramatsu, R. Ishizuka, Y. Wu, and K. Yoshii,“Difficulty-aware neural band-to-piano score arrangement
based on note- and statistic-level criteria,” in Proc. IEEE International Conference on Acoustics, Speech and Signal Processing, 2022, pp. 196–200.
[13] H. Takamori, T. Nakatsuka, S. Fukayama, M. Goto, and
S. Morishima, “Audio-based automatic generation of a piano
reduction score by considering the musical structure,” in Proc.
International Conference on Multimedia Modeling, 2019, pp.
169–181.
[14] Z. Wang, D. Xu, G. Xia, and Y. Shan, “Audio-to-symbolicarrangement via cross-modal music representation learning,”
in Proc. IEEE International Conference on Acoustics, Speechand Signal Processing, 2022, pp. 181–185.
[15] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in Proc.
International Conference on Medical image computing andcomputer-assisted intervention, 2015, pp. 234–241.

[16] E. Jang, S. Gu, and B. Poole, “Categorical reparameterization
with Gumbel-Softmax,” in Proc. International Conference on
Learning Representations, 2017.
[17] M. Agueh and G. Carlier, “Barycenters in the Wassersteinspace,” in SIAM Journal on Mathematical Analysis, 2011, pp.
904–924.
[18] D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in The International Conference on Learning Representations (ICLR), 2015.