PIANO ARRANGEMENT SYSTEM BASED ON COMPOSERS’ARRANGEMENT PROCESSES
Sho Onuma

Masatoshi Hamanaka

University of Tsukuba
sho@music.iit.tsukuba.ac.jp

University of Tsukuba
hamanaka@music.iit.tsukuba.ac.jp

ABSTRACTOur goal is to construct an automatic musical arrangement
system by applying analysis results of composers’arrangement processes. We arranged an orchestral score
for a solo pianist to evaluate our system. Usually, anorchestral score has several instrumental parts, which isdifficult for a soloist to play. Therefore, an orchestral score
needs to be arranged with several instrumental parts into
both hands parts, which enables a solo pianist to play withboth hands. However, there is a problem with anarrangement of more than five notes for each hand, since
one hand has only five fingers. To solve this problem, we
introduce a protocol analysis for analyzing the process ofarranging music. We applied this protocol analysis to amusical arrangement, and analyzed the arrangement
process. By using the analysis results, we constructed a
piano arrangement system for warning the composer ofproblems in the score that need to be arranged.

this analysis, we can determine what the problems are in
an arrangement and how to solve them.
In this protocol analysis, we recorded the five composers’
thought process for arranging an orchestral score for pianoby using an eye mark camera. After the music was arranged,
we queried the composers on how or why they arranged
their music they way they did. The data from eye markcamera enabled us to perform detailed protocol analysis
using not only their thinking-aloud processes but also visual
points. Using results of the think-aloud protocol analysis, weconstructed a piano arrangement system. There were exactly
the same points that a musician arranged and the systemwarned, and musical novices were able to choose fromseveral arrangement processes.
We described our protocol analysis in Section 2, asystem implementation in Section 3, an example ofanalysis using the system in Section 4, and we making
concluding remarks in Section 5.

1. INTRODUCTION

2. ANALYZING MUSICAL ARRANGEMENTPROCESSES

We propose an automated method for arranging an
orchestral for piano, which enables a solo pianist to
comfortably play an orchestral score.
Musical arrangements have several variations. For example,
a folk song arranged in a jazz style or rock song arranged witha heavy use of strings. There are also arrangements forsoloists. For example, there are musical arrangements
originally for multiple-instruments for solo pianists.
Musical arrangements are divided into two types, thosewith many notes and those with fewer notes. Althoughprevious research has mostly focused on the former [1] [2],we investigated musical arrangements with fewer notes.
We used an original orchestral score, and arranged it for asolo pianist. This type of arrangement is called a “pianoarrange score”, which is widely available. However, a
piano-arrange score is usually written for professional orsemi-professional soloists. Therefore, a musical novice
may have difficulty in playing it.
We used a think-aloud protocol analysis to examine fivecomposers’ arrangement processes [3]. Originally, a thinkaloud protocol analysis a usability testing technique used
to analyze composers’ thinking-aloud process. Based on

In arranging an orchestral score for piano, we have to pay
attention to several problems concerning the use of amusician’s fingers. For example, it is difficult to play twonotes that are more than one octave distant with one handbecause the fingers cannot stretch that far. Also, thenumber of notes that can be played at the same time islimited due to the number of fingers. In addition, it isdifficult to cross the hands while playing. Furthermore, it
is important to maintain an original piece of music andconsidered melody repetition because there are repetitionsof similar melodies in an orchestral score. For example,
there are two similar orchestral melodies, “p” and “q”, andonly “q” has a flute part. When “p” and “q” are arranged
without the flute part, the arranged melodies “p” and “q”
are exactly the same. The arrangement cannot be
reproduced from an original piece because there isessentially no difference between “p” and “q” in thearrangement. Repetition and maintenance are complicatedproblems because of recognized the entire score and in amusical arrangement. However, composers are able toarrange music that has these problems. Therefore, we
analyzed a composer’s arrangement process.

2.1. RECORDING COMPOSERS’ ARRANGEMENTS
We recorded five composers’ arrangement process. In
Figure 1, a piano arrangement of an orchestral score. To
record a video of a composer’s visual points, we used aneye mark camera. We also recorded each composer’s voice
when he/she is thinking-aloud by using microphones.
After the music arranging process, we queried thecomposers on the details of their arrangement processes
using the video from the eye mark camera. For example,
we asked composers questions such as. Why did you focus
on this section for a long time? What is troubling you now
about your arrangement? Please tell me if you have any
good ideas for the arrangement.
Orchestra

Piano

Arrange

Eye mark camera

In another example listed in Table 1, the Time was 4:48,
the “Events” were that the composer’s thought-aloud. He
said, “It is difficult to play cornet because there are many
notes”. The composer’s thinking-aloud indicates that he
was having difficulty because of the “many notes”, and hesolved this problem by not including the cornet part.
Thus, using the results of the think-aloud protocol
analysis, we labeled the problems and problem-solving
operations which appear plural times and examined therelations between them. We found that certain problems andproblem-solving operations occurred when composers are introuble in arrangement process. These problems indicate thatproblems occurred during arrangement, and solving the
problems indicated that the problems were also solved.
In the third example in Table 1, the time 7:43, and thecomposer is in trouble because of over-lapping notes between
several parts. In Table 1, the problems are described as P5(Problem 5) under “Problems”. Then, the composer solved the
problem by “Removing a note”. In Table 1, the solution is
described as j4 (problem-solving operation 4) under “Operation”.
Consequently, we found nine problems and six operations in
the arrangement process.
2.3. MODELING PROBLEMS AND PLOBLEMSOLVING OPERATIONS

Figure 1: Recording of Musical Arrangement Process
2.2. PROTOCOL ANALYSIS
We analyzed the arrangement process by using a thinkaloud protocol analysis. Table 1 list the details of protocol
analysis of a composer arrangement processes. “Time”
denotes the time it took to arrange the music. “Events”
denotes important actions during the process, for example,
playing a midi keyboard or thinking-aloud. “Arrangementwork” denotes details of arrangement.
In the first example in Table 1, the Time was 1:48, the“Events” were that the composer played the viola and celloparts in midi keyboard. When the composer looked for thebase line, he said, “Difficult to play”. It would be difficult toplay if both viola and cello parts are included. The composer
solved this problem by including only the cello part and left
out the composer viola part.
Time
Eventsmin:sec
1:48 play：[viola + cello][3-4 bars](consider left hand)

Arrangement

Problem Operation

Playing viola and cello parts
Looking for base line (Difficult)If playing both viola and cello parts
Difficult to play
Use cello, remove viola

P0

In Figure 2, “P” indicates a problem, and “j” indicates aproblem-solving operation. We used them, to construct astate-transition model. The model showed the arrangement
process started in P0 and ended in S9 (Solution 9). P0 involves
inputting composed orchestral melodies, and S9 involves
outputting piano melodies. P0 is actually P1-8 via j1-6 ortransition to another problem except for S9. Several problems
are formed simultaneously, but a musical novice solved eachproblem one by one, enabling him/her to transition to S9.
For example, when a pianist’s fingers cannot reach (P7)
when melodies are composed, Figure 2 produces threeproblem-solving operations, j4-6. When no problems occur,
the state-transitions to S9 and the model output the melody.
Based on this model, we constructed a piano arrangement
system. In the next section, we describe these various
problems, and how they are transitioned.
Composedorchestra melodies

Table 1: Example of Think-Aloud Protocol Analysis

OPERATION

P1
P2

P7

4:48 Think-aloud:“It is difficult to playcornet, because of
P6
many notes.”“Not using cornet.”
7:43 Play[1st violin + 2nd violin] Playing 1st violin and 2nd violin[2 bar]Last 2 bars, there are over-lapped notes P5
Remove note

P0

PROBLEMS
P1: Dynamics
P2: Same melodies
P3: Repeated melodies

P3

j1: Rating melodies
j2: Unison
j3: Unrepeated

j4

j4

j4

PROBLEMS
P4: Close notes
P5: Over-lapped notes
P6: Many notes
P7: Unreachable notes
P8: Difficult melodies

P4

OPERATION

P5

P8

P6

P7

S9

Piano melodies

Figure 2: State-Transition Model

j4: Remove notes
j5: Move notes
up/down octave
j6: Change hand

2.4. PROBLEMS
P1-3 are transitioned by j1-3. P1 is where there are noticeable
dynamics. Dynamics are important in deciding a piece’s
ambience. The problem determines the melody’s rating fromforte (f) or piano (p). P2 is where the same melodies exist
several parts. When different parts with the same melodies
are played together, we call this Unison. The Unison melody
is important, and the melody’s rate is added. P3 is where
there is a repeated melody. Our system records the melody
and warns if they are the same. To implement these threeproblems, we need further protocol analysis because there
are many solutions. We describe P4-8 in Section 3.
3. IMPLEMENTATION
We implemented a part of the state-transition model(Figure 2) as a system, P4-8 (Figure 3). Before starting thearrangement, we map the each part for the left or righthand. According to iterate problems and problem-solving
operations, a musical novice is able to arrange byinteractively reducing the system’s warnings.
For these problems, the model shows three problemsolving operations; j4-6. In this system, a musical novice isable to solve problems using these operations. For input
files, we used MIDI [4] files consisting of several parts.
We transcribed variables as follows; PitchX is the pitchof noteX (MIDI note number), OX is onset of noteX, LX is
the length of noteX, and HX is playing hand of noteX.
Display
pianoroll• Read files• Start analysis

notes because the pianist’s fingers cannot reach them. P 7 isdescribed by the following formulas. The system displays
the notes in red (Figure 4 P7).
𝑂𝐴 ≤ 𝑂𝐵 < 𝑂𝐴 + 𝐿𝐴 and 𝑃𝑖𝑡𝑐ℎ𝐴 + 13 ≤ 𝑃𝑖𝑡𝑐ℎ𝐵(2)
𝐻𝐴 = 𝐻𝐵
P6 (Many notes) is formed when there are many notes
played together in one hand. It is difficult to play morethan five notes with one hand. P6 is described by thefollowing formulas. Then, 𝑁𝑢𝑚x is number of noteX and“𝑅𝑣” is number of notes which is limitation of playing
together, initial value is four. The system displays thenotes as green circles (Figure 4 P6).
𝑂𝐴 ≤ 𝑂𝑋 < 𝑂𝐴 + 𝐿𝑋 and 𝑃𝑖𝑡𝑐ℎ𝐴 ≠ 𝑃𝑖𝑡𝑐ℎ𝑋(3)
𝐻𝐴 = 𝐻𝑋 and 𝑁𝑢𝑚x ≥ 𝑅𝑣
P4 (Close notes) is formed when it is difficult to play twonotes that are close together with each hand. When we play
the piano with both hands, it is difficult to play when our hand
are too close. The system warns the musical novice if notes
are too close. P4 is described by the following formulas. The
system displays the notes as orange squares (Figure 4 P4).
𝑂𝐴 ≤ 𝑂𝐵 < 𝑂𝐴 + 𝐿𝐴 and 𝐻𝐴 ≠ 𝐻𝐵
(4)
𝑃𝑖𝑡𝑐ℎ𝐴 + 4 ≥ 𝑃𝑖𝑡𝑐ℎ𝐵 and 𝑃𝑖𝑡𝑐ℎ𝐴 < 𝑃𝑖𝑡𝑐ℎ𝐵
P8 (Difficult melodies; Crossing) is formed when a melody
is difficult to play. The system warns the musical novice
when the arranged melodies became too difficult. For
example, fingers need to move fast or too far apart. However,a melody’s difficulty is different from each player.
Therefore, detecting this problem is difficult.
Therefore, we implemented a version of P 8, called
Crossing. This means that the system warns the musicalnovice when the arrangement requires the crossing ofhands. P8 is described by the following formulas. The
system displays the notes in pink (Figure 4 P8).

P4

P5

P6

Close
notes

Over-lapped
notes

Many
notes

• Show track• Arrangement set up

Figure 3: Overview of Piano Arrangement System
3.1. IMPLEMENTATION OF PROBLEMS
P5 (Over-lapped notes) is formed when there are twonotes of the same pitch and over-lapped pronunciation time.
When notes of different parts over-lap, the system warns
the musical novice because they are difficult to play. P5 isdescribed by the following formula. The system displaysthese notes in black (Figure 4 P5).
𝑂𝐴 ≤ 𝑂𝐵 < 𝑂𝐴 + 𝐿𝐴 and 𝑃𝑖𝑡𝑐ℎ𝐴 = 𝑃𝑖𝑡𝑐ℎ𝐵(1)
P7 (Notes over one octave) is formed when it is difficult toplay two notes that are over one octave apart together in
one hand. The system warns the musical novice of these

P7

P8Play with
LEFT handPlay withRIGHT hand

Notes over
one octave

Difficult(crossing hands)

Figure 4: P4-8

𝑂𝐴 ≤ 𝑂𝐵 < 𝑂𝐴 + 𝐿𝐴 and 𝑃𝑖𝑡𝑐ℎ𝐴 < 𝑃𝑖𝑡𝑐ℎ𝐵(5)
𝐻𝐴 ≠ 𝐻𝐵 and 𝐻𝐴 = 𝑅𝑖𝑔ℎ𝑡
Thus, the system can be used to analyze music and display
warnings in various colors.
3.2. PROBLEM-SOLVING OPERATION
IMPLEMATION

Original

Warning(P5)

When these warning appear, the musical novice needs tosolve the problem. In this system, musical novice canperform the following problem-solving operations; removenotes (j4), move notes up or down in octave (j5), and change
the playing hand (j6). These operations produce warnings in
the system so the musical novice can solve the problem.
4. EXAMPLE OF ANALYSIS
We constructed our piano arrangement system using themodel. Then, we compared the arrangement produced using
the system with that arranged by composer without the systemusing two problems; over-lapped notes (P5) and notes over
one octave (P7). For comparison, we used SYMPHONIE No.
14 by W. A.MOZART K.114 Trio, bars one to sixteen.
Figure 5 shows the results of analyzing this originalpiece of music. From the composers’ arrangement processes,
we determined that tracks 1 and 2 are played with the righthand and tracks 3 and 4 are played with the left. Figure 5
shows that there were many warnings, signified as arrows.
Therefore, this original piece needed to be arranged.

Many warnings
appearFigure 5: Results from Original Piece
4.1. ORIGINAL PIECE AND ARRANGEMENT OFCOMPOSER
We then compared the original piece with one of the fivecomposer’s arrangement. Figure 6 is a part of result thatthe system warned P5 or P7.
First, we described P5. In Figure 6 “Warning (P5)”, the system
detects that there is an P5 in the original piece. The composer
solved the problem by moving a note one octave down.
Applying j5 to the original piece produced the same result asthat during a composer’s traditional arrangement process.
Then, we described P7. In Figure 6 “Warning (P7)”, thenotes enclosed in circles are played with the right hand in
the original piece. The composer solved the problem bychanging the playing hand to the left one. Applying j6 tothe original piece produced the same result as that during acomposer’s traditional arrangement process.
Therefore, this warning system and the composerarrangement process are the same. Accordingly, theproblems detected by the system were correct.

ComposerMove down
an Octave(B)

Warning(P7)

Changehand(C)Figure 6: Comparison of Original Piece and Composer’s
Arrangement (Move down an Octave B->j5 Change hand C->j6)
5. CONCLUSIONS
We constructed a piano arrangement system by applying
protocol analysis of composers’ arrangement process. In
the think-aloud protocol analysis, we recorded thecomposers thinking-aloud and visual points, and wedetermined nine problems and six operations in thearrangement process. Using the result of the analysis, weconstructed a piano arrangement system. There are exactly
same points that a musician arranged and system warned,
and musical novice enable to choose from several
arrangement processes.
We plan to implement P1-3 and operations j1-3 in oursystem by analyzing other composers’ arrangement process.
This will also be automated.
6. REFERENCES[1] Dan Gang, Daniel Lehmann, Naftali Wagner:“Tuning a Neural Network for Harmonizing
Melodies in Real-Time,” International Computer
Music Conference, 1998.
[2] Takashi K., Mitsuru N., Hiroshi S., Hideki S.: Hidden
Markov Model Applied to Automatic Harmonizationof Given Melodies, IEICE technical report.
Speech 99(627) pp. 25-32 2000. (in Japanese)[3] Sho O., Masatoshi H.: Time Series Analysis of Music
Arrangement Work -Reduction from orchestra’sscore to piano’s score-, 09-IPSJ-71-2, pp. 2.227-

2.228,2009 (in Japanese)[4] MIDI Manufacturers Association (Reference 12/22

2009) http://www.midi.org/