Conversion from Standard MIDI Files to Vertical Line Notation Scoresand Automatic Decision of Piano Fingering for Beginners
Yasuyuki SAITO
Nat. Inst. of Tech., Kisarazu College
Chiba 292-0041, Japansaito@j.kisarazu.ac.jp
Suguru AGATA
Showa Univ. of Music
Kanagawa 215-0021, Japanagata-s@snow.ocn.ne.jp

Eita NAKAMURA
Kyoto University
Kyoto 606-8501, Japanenakamura@sap.ist.i.kyoto-u.ac.jp

Yuu IGARASHI
Showa Univ. of Music
Kanagawa 215-0021, Japany uu @yuxmu.com

ABSTRACTThis paper introduces a method for converting standard
MIDI files to the “vertical line notation” (VLN) and an algorithm for automatic decision of piano fingering for beginners.
Currently, staff notation is widely used for various instruments including piano. However, this notation often appears hard to beginners. On the other hand, VLN is intuitive and easy to understand for piano beginners since itgraphically indicates the time order of notes as well as fingering. With the VLN score, piano beginners can makesmooth progress with correct fingering, and it is expected
to be useful for early education in music.
A problem with VLN is that it is currently created byhand with a spreadsheet software. It would be desirable
to automatically produce VLN scores from existing digitalscores.
Our proposed method can solve above problem. In addition, this paper also presents some examples of practicaland successful use of VLN scores.
1. INTRODUCTION
Among many existing types of music notations including
staff notation, tablature notation for guitar and luto, oneline notation for percussion, graphic notation, numeral notation, and Japanese traditional notation [1], staff notation
is most commonly used for a wide variety of musical instruments.
Notations have evolved to express composer’s intentionsas accurately as possible, and therefore, they became advanced and complex with the times. For instance, staff
notation provides musician with a variety of information,
including pitch, accidentals, note length, dynamics, trills,
turns, articulations (including staccato, tenuto, accent andattack), etc.
Copyright: c 2016 Yasuyuki SAITO et al. This is an open-access article distributed
under the terms of the Creative Commons Attribution 3.0 Unported License, which
permits unrestricted use, distribution, and reproduction in any medium, provided
the original author and source are credited.

200

Riku SATO
Nat. Inst. of Tech., Kisarazu College
Chiba 292-0041, Japanj11424@j.kisarazu.ac.jp

Shigeki SAGAYAMA
Meiji University
Tokyo 164-8525, Japansagayama@meiji.ac.jp

However, to music beginners, it is difficult to read evenpitch and the length of a note in staff notation. For thisreason, in many cases, people who are not accustomed toplay the piano often give up in the process of reading a staff
notation scores, even for a short musical piece and simple
melody (see section 3.4).
With this background in mind, in 1985, Suguru Agata
proposed vertical line notation (VLN) which is designed
by piano roll style for the piano beginners [2]. It is very
easy to understand the pitch of notes on a VLN score, andit can also show piano finger numbers.
In this paper, we first review how to read VLN scores.
Next, we explain a conversion method from standard MIDI
files to VLN scores, and propose an automatic decisionof piano fingering for the piano beginners. Moreover, weshow applied examples in which piano beginners could
play the piano easily using VLN scores.
2. VERTICAL LINE NOTATION
Figure 1 illustrates how to read a VLN score. In a VLN
score, the horizontal direction expresses note pitch; the left
side is for lower pitches and right side is for higher pitches.
The vertical direction shows time flow; the direction toward the bottom represents the progression of the music.
The horizontal positions of circles represent the pitch ofnotes while blue bars designate the temporal order of notes.
We call the circle “node” and the bar “link,” after the research field on graphical model [3]. Each node contains afinger number. Therefore even piano beginners can play amusical piece with correct fingering very easily. They only
need to strike the keys by following the finger numbers in
the nodes. For piano players, it is very important to get into
the habit of playing with correct fingering. The link showswhether the pitch of the next node is higher or lower thanthe current node. This is useful in searching for the next
node.
Figure 2 shows an example of staff notation score of achildren’s song “tulip.” Figure 3 shows a VLN score of
the song “tulip.” This VLN score is made for Japanesechildren or for older persons, so the title is written on the
top of the VLN score in hiragana letters. The lyrics arealso written on the left side using hiragana letters.

International Conference on Technologies for Music Notation and Representation - TENOR 2016

3

time ﬂow

2

node
link

3
4

higher pitch

lower pitch

Figure 1. Format of VLN score.

4
&4 œ
3

& œ

œ

œ

œ

œ

5

& œ

Œ

œ

œ

œ
Œ

œ

œ

œ

œ

œ

œ

œ

œ

œ

Œ
Œ
Œ

〜

Figure 2. Staff notation score of a children’s song “tulip.”

If the appropriate adjustment of magnification ratio of a
VLN score is achieved, the graph of keys in the VLN scorecan be made to correspond to the keys of a real piano keyboard. As a result, the piano player can easily find the key
to be pressed on the keyboard just below the node. Moreover, because a VLN score always shows real pitches, it
is not influenced by key and clef sign in staff notation. In
contrast, in the staff notation, pitch is shown as vertical position, and therefore the correspondence of pitch in staff
notation to piano keyboard is not intuitive.
At the same time, it would be necessary for the beginners to gradually be capable of reading staff notation. We
will discuss the transition from using VLN scores to staff
notation scores in Section 8.3.
3. PERFORMANCE BY USING VLN SCORES
3.1 Beginners’ piano class for senior persons in Showa
Univ. of MusicWe began a beginners’ piano class for elder people in
Showa University of Music in 2010. The age of participants is from 60s to 80s. In this piano class, we have usedelectronic keyboards. Figure 4 shows the configuration of
the piano class.
In the lesson, we teach through three stages instead ofplaying music from the beginning. First, the participants
hit table by finger in accordance with VLN score. In thisstage, the participants concentrate on only correct order ofthe finger without much attention to the position of key on
the keyboard.
Second, while turning off the power of the electronic keyboard, they practice key strokes. Even if they make mis-

Figure 3. VLN score (handmade) of “tulip” with Japaneselyrics.

takes in the fingering, they need not to worry about it. The
important things are to continue until the end, and not tobe afraid to fail. This “silent practice” characteristics is
unique to the electronic keyboard, and cannot be done on
the acoustic piano.*1
Third, participants play the music with sound after turning on the electronic keyboard.
We have found the all the participants can play the pianoand have enjoyed the practice.
3.2 Piano class in Rest Villa Ebina
We have held piano classes with VLN scores at Rest Villa
Ebina which is a private facility providing long-term careto the elderly. Figure 5 shows the situation. In this photo,
two persons use wheelchairs. One of the residents of the
facility suffers from Parkinson’s disease. The upper halfof her body usually leaned to the right side. However, herposture has improved, since she started the keyboard performance using VLN scores. In addition, her speech ability has also changed for the better, and she has been able
to evaluate the performance of others. It is considered thatmuscle and brain are activated by playing the keyboard,
and we suggest that VLN score can be effectively utilized
in the welfare field and it gives purpose in life for seniorpersons.
If we had used a complex staff notation score, people
might have lost interest. However, by using the VLN score,*1 The same effect is obtained by using acoustic pianos with “silent
mode”, for example, YAMAHA Silent Piano series.

International Conference on Technologies for Music Notation and Representation - TENOR 2016

201

Figure 4. Piano class for senior people at Showa Univ. ofMusic.

Figure 5. Piano class for senior people at an elderly persons’ home.

they have been interested in the piano performance, andthey have practiced enthusiastically.
3.3 Beginners’ piano class for students at NITKC
We tried to apply VLN scores to piano class for young
people. We held a beginners’ piano class for students
at the National Institute of Technology, Kisarazu College(NITKC) in May 2015. Actually, 13 students who wereinterested in playing the piano volunteered for this class.
They experienced the same three stage practice method as
the senior persons. Amazingly, after 90 minutes of practice, they could play the piano using both hands.*2
3.4 Open campus at NITKC
In open campus at NITKC, various studies are introduced
to junior high school students and their parents. We explained the automatic accompaniment system (AAS). The
name of the AAS is “Eurydice”. This AAS is developed by*2 Some video clips are shown in the following Website:

http://beam.kisarazu.ac.jp/˜saito/research/VLN/

202

Figure 6. Piano class for students at Nat. Inst. of Tech.,
Kisarazu College.

Figure 7. Piano performance by a junior high school student in open campus of NITKC. She could play the VLN
score at sight.

using C++ language on “Qt” which is a cross-platform development framework for graphical user interface (GUI).
Eurydice deals with MIDI signals; the input MIDI signal is given from MIDI piano as performance data of human player, and the output MIDI signal is accompaniment
data [5]. A standard MIDI file (SMF) as musical score information is given to Eurydice in advance, and the AAS
estimates a musical score position of the performance ofthe human player. Eurydice allows errors in performancessuch as insertion of unnecessary notes, deletion of notes,
and lack of accidentals. Notably, Eurydice is the world’s
first system that allows arbitrary jumps; the human player
may practice playing the same bars again and again, or go
to a later section of the score by skipping over some bars;
in either case Eurydice follows the player’s performance.
Until 2014, we have used staff notation at the presentation. At first, as a performance example, we played a
melody of a short piece of music, and Eurydice played anaccompaniment part at the same time. After that, we asked

International Conference on Technologies for Music Notation and Representation - TENOR 2016

Titlejunior high school students to play the music. However,most of the students did not try to perform on the piano.
In 2015, we explained VLN and demonstrated performance by using a VLN score with Eurydice. Figure 7
shows the piano performance. Consequently, all the juniorhigh school students who visited our demonstration (about
20 persons) tried playing the piano, and performed to theend at sight without regular practice. We consider that thepsychological distance for piano of junior high school students is shortened by using VLN score.

44
Figure 8. An example score in staff notation which includes eighth notes.

4. AUTOMATIC CONVERSION FROM SMFS TOVLN SCORES
Currently, VLN scores are made by using Microsoft Excel
or fully hand written. The work takes a long time and is laborious. Therefore, it would be desirable to automatically
produce VLN scores from existing digital scores.
There are several formats of digital score such as standard
MIDI file (SMF), MusicXML, and custom formats of music score editors. Among these formats, SMF is the mostwidely used. It is easy to make SMFs by using commercial
or free music software. Fortunately SMFs are distributedon the internet, in that case, people do not need to make
SMFs themselves. This is a great advantage. We implemented a conversion system from SMFs to VLN scores [4].

h1h2

4.1 MIDI standard
Musical instrument digital interface (MIDI) is a universal
standard for the digital transmitting of performance data
between electronic musical instruments [5]. It consistsof certain standards, such as a physical transceiver circuit
and interface, a communication protocol, a file format, etc.
Transmission and reception of data on the MIDI standard
are all carried out in MIDI messages. A MIDI message
is constructed of plural bytes (8 bits per one byte). To
transmit MIDI messages efficiently, the bytes which express MIDI messages are divided into two types: a “statusbyte” and a “data byte.” The status byte has “1” as themost significant bit (MSB); namely, it contains 128 datatypes from 80H to FFH in hexadecimal format. In contrast, the data byte includes “0” as the least significant bit(LSB). Thus the range of the data is from 00H to 7FH.
4.2 Formats of SMF
There are three types of formats of SMFs. The format 0
has only one track which includes all note information. It
is equivalent to mixed down music data. Therefore, thisformat is for the purpose of playback only. The format 1
can contain many independent tracks. This format easily
deals with decomposed data on a MIDI sequencer. The
format 2 has multitracks and multisequence patterns. In
the case of playing the SMF format 2 data, one of the sequence patterns is chosen in each track. This format seems
to have been originally developed and intended for use in,for example, karaoke. However, it is little used in otherpractice.
The SMF is composed of several parts. One is the “header
chunk,” which includes information of the whole SMF. The

Figure 9. Previous style of VLN score (handmade) for
Figure 8.

other is the “track chunk,” which contains real performancedata.
4.3 New style of VLN
Figure 8 shows an example score in staff notation which
includes eighth notes. Figure 9 represents the conventional
style of VLN score (handmade) which is equivalent to thescore in Figure 8. In this case, in Figure 9, the length ofthe last eighth note is expressed shorter than the length ofthe previous one: h2 < h1 .
Therefore we propose a new style of VLN as shown in
Figure 10: the first note is put on the bar line. The length ofthe last eighth note is equal to the previous one: h2 = h1 .
In the next section, we will explain the conversion
method for making this style of VLN scores from SMFs.
4.4 Conversion from SMFs to VLN scores
In SMFs, the header chunk and the track chunk begin witha magic number as “MThd” and “MTrk.” First, it is necessary to find these bytes in a MIDI file. In the header
chunk, a time resolution, which is called “time base,” isdefined. This shows a capacity to decompose a quarter

International Conference on Technologies for Music Notation and Representation - TENOR 2016

203

P

M

l
Figure 11. Measure number M , note position P , and note
length (duration) l.

hm

hp

h1
h2

hb

Figure 10. New style of VLN score (automatic converted
from SMF) for Figure 8.

wm
note. For example, if a time base is equal to 480, it is possible to express the quarter note as 480 divided lengths.
“Delta time” is also described together in all MIDI eventsin which the MIDI file is represented. It represents the time
until the next event is performed. Both “note-on” (a command for turning the sound on) and “note-off” (turning off
the sound) are used to represent a note length. For example, in the case that a time length from note-on to note-off
is equal to time base, it is equivalent to a quarter note.*3
If we let S represent a time for note-on, and E be a time
for note-off, then D means a time base. The meter will
be four-four time, and a piece of music has no anacrusis.
In these conditions, a note length l is calculated by Equation (1):l = (ES) / (D ⇥ 4),(1)

where D ⇥ 4 is equal to the whole time length in a musicalbar.
A note position P from the beginning in a measure is
obtained by Equation (2), and a measure number M of anote is given as Equation (3):P = S mod (D ⇥ 4),(2)

M = [S / (D ⇥ 4)] + 1,

(3)

where “mod” means the remaining operations, “[ ]” indicates the floor function, and the multiplier “4” is equal tothe denominator of the musical time. By using these threeequations, performance data of a MIDI file are converted
to data sequences which have only note length, note position, and measure number. Figure 11 shows l, P , and M .
*3 Ref. [6] describes formalisms of duration-onset of notes which reminds to performance theory.

204

hn

hl

wkey
1

Figure 12. Parameters which are used for calculating yand x-coordinates of node.

These parameters are used to indicate the vertical position
ynode of the center of a node in a VLN score;ynode = hm + hp + hb M + hn ,(4)where hm , hp , and hb shows the height of the top margin,of the piano area, and of musical bar in the VLN score, andhn corresponds to the position of the node is the musicalbar (see Figure 12):hn = hb P.

(5)Note that P is a common fraction. hl shows the note length(in a VLN score) which is given as Equation (6):hl = l hb .

(6)

The horizontal position of the node is calculated easily because SMF includes pitch information clearly. Let
nlowest mean the note number of the lowest pitch in the
VLN score. This note number corresponds to a pitch
vlowest in Table 1 which is the lowest pitch in the VLN
score. Let nnode represent the note number of the node,
xnode which is x-coordinates of the center of the node isInternational Conference on Technologies for Music Notation and Representation - TENOR 2016

plowest
C, D, EF, G, A, H

vlowestC
F

2

Table 1. Lowest pitch in VLN score. plowest means lowest
pitch in music piece, and vlowest shows the lowest pitch in
VLN score.
prangewithin 1 octave
over 1 octave

vhighest
1 octave higher of lowest pitchhighest pitch

Table 2. Highest pitch in VLN score. prange means pitch
range in music piece, and vhighest shows highest pitch in
VLN score.
given as Equation (7):xnode = wm +wkey
+ (nnode2

nlowest )

wkey
,
2

(7)

where wm shows the width of the left margin in the VLN
score, and wkey means the key width on the VLN score.
The reason why wkey is divided by 2 is that the unit ofxnode is semitone.
The highest pitch in VLN score is obtained from Table 2.
For example, in Figure 12, the lowest pitch C4 is assigned
to vlowest . Therefore nlowest is equal to 64 which is MIDI
note number of C4. The highest pitch is within the 1 octave
from nlowest (C4). Thus C5 is substituted into vhighest .
5. AUTOMATIC DECISION OF
PIANO FINGERING FOR BEGINNERS
As shown in the previous section, VLN scores can be madefrom SMFs automatically, however, it is necessary to put
a finger number into each node. VLN scores have finger
numbers which are very important to pianist. On the other
hand, it is not easy to decide the piano fingering. Hence,
we have studied automatic decision of it.
Models and algorithms for decisions deciding appropriate
piano fingerings have been studied previously [7, 8, 9, 10,11]. To enable the adaptation of fingerings for beginners,
we present a method based on the hidden Markov model(HMM) [10].
Piano fingering for one hand, say, the right hand, is indicated by associating a finger number fn = 1, . . . , 5(1 = thumb, 2 = the index finger, . . . , 5 = the little finger) to each note pn in a score,*4 where n =1, . . . , N indexes notes in the score and N is the numberof notes. We consider the probability of a fingering sequence f1:N = (fn )N
n=1 given a score, or a pitch sequence,
p1:N = (pn )N
n=1 , which is written as P (f1:N |p1:N ). As
explained below, an algorithm for fingering decision canbe obtained by estimating the most probable candidatefˆ1:N = argmax P (f1:N |p1:N ). The fingering of a particuf1:N

lar note is more influenced by neighboring notes than notesthat are far away in score position. Dependence between*4 We do not consider the so-called finger substitution in this paper.

3 4
5

1

Figure 13. Schematic illustration of the piano fingeringmodel based on HMM.

adjacent notes is most important, and it can be incorporated within a Markov model. It also has advantages in efficiency in maximizing probability and setting model parameters. Although the probability of fingering may depend on
inter-onset intervals between notes, the dependence is notconsidered here for simplicity.
Supposing that notes in a score are generated by finger movements and the resulting performed pitches, theirprobability is represented by the probability that one finger would be used after another finger P (fn |fn 1 ), andthe probability that a pitch would result from two consecutively used fingers (Figure 13). The former is called
the transition probability, and the latter output probability. The output probability of pitch depends on the previous pitch in addition to the corresponding used fingers, and it is described with a conditional probability
P (pn |pn 1 , fn 1 , fn ). In terms of these probabilities, theprobability of notes and fingerings is given as
P (p1:N , f1:N ) =

N
Y

n=1

P (pn |pn 1 , fn 1 , fn )P (fn |fn 1 ),(8)

where the initial probabilities are written as P (f1 |f0 ) ⌘P (f1 ) and P (p1 |p0 , f0 , f1 ) ⌘ P (p1 |f1 ). The probability
P (f1:N |p1:N ) can also be given accordingly.
To train the model efficiently, we assume some reasonable constraints on the parameters. First we assume
that the probability depends on pitches only through theirgeometrical positions on the keyboard which is represented as a two-dimensional lattice (Figure 13). We
also assume the translational symmetry in the x-direction
and the time inversion symmetry for the output probability. If the coordinate on the keyboard is written as`(p) = (`x (p), `y (p)), the assumptions mean that the output probability has a form P (p0 |p, f, f 0 ) = F (`x (p0 )`x (p), `y (p0 ) `y (p); f, f 0 ), and it satisfies F (`x (p0 )`x (p), `y (p0 ) `y (p); f, f 0 ) = F (`x (p) `x (p0 ), `y (p)`y (p0 ); f 0 , f ). A model for each hand can be obtained inthis way, and is written as F⌘ (`x (p0 ) `x (p), `y (p0 )`y (p); f, f 0 ) where ⌘ shows left and right hand respectively, and each model exists independent. It is further assumed that these probabilities are related by reflection in
the x-direction, which yields FL (`x (p0 ) `x (p), `y (p0 )`y (p); f, f 0 ) = FR (`x (p0 ) `x (p), `y (p0 ) `y (p); f, f 0 ).
The present model can be extended to be applied to pas-

International Conference on Technologies for Music Notation and Representation - TENOR 2016

205

Figure 14. Drop-down menu of “File” on VLNMaker.
sages with chords, by converting a polyphonic passage
to a monophonic passage by virtually arpeggiating thechords [9]. Here, notes in a chord are ordered from low
pitch to high pitch.
To find the optimal fingering for a given piano scoreusing the model, we need to maximize the probability
P (f1:N |p1:N ). This can be computationally efficiently
solved with a dynamic programming called Viterbi algorithm [12]. An extension of the above model for both hands
is discussed in Ref. [13]. With this extension, it is possible
to simultaneously estimate the optimal fingering for both
hands from a MIDI file in which notes for the left- andright-hand parts are not separated.
The parameter values for the transition and output probabilities have been obtained in a previous study [13]. Because the used dataset consisted of fingerings for experienced piano players, the obtained fingerings were not fully
appropriate for the beginners. For example, using different fingers for pressing successive identical keys is common for the experienced players, but it is not easy to perform for the beginners. To solve this problem, we adapted
the trained model parameters to increase the probabilities
of self transition for using identical fingers for successivenotes.
6. IMPLEMENTATION
We developed a conversion software from SMFs to VLN
scores, and we named it “VLNMaker”. We use “Qt” which
is a cross-platform programming environment for graphical user interface (GUI). Qt works on Windows, Mac OS
X, and Linux. Moreover, it can make an executable file foreach OS by using C++ compiler respectively from same
source code.
Figure 14 shows drop-down menu of “File” on VLNMaker: “Open score file” to load SMF, and “Open fingering file” to read piano fingering information, and “export” to output PDF file.*5 After loading SMF and pi*5 “Save” and “Save as” have not implemented yet. These menus will
be used to save the modified piano fingering by hand. The interactive
hand modification function of piano fingering also has not implemented
yet.

206

Figure 15. Window state after loaded SMF and piano fingering file.
Subject
No.
1
2
3
4
5
6
7
8
9
10
11
12
13

EPI
none
none
none
none
Tb
Drums
Gt
E.Gt
Cla
Gt
Euph
Piano
A.Sax

reading
staff N.
C
C
C
C
C
C
B
B
B
B
B
A
A

reading
VLN
A
A
A
A
B
B
A
A
A
B
B
A
B

play
of the
piano
C
C
C
C
C
C
C
C
B
B
B
B
C

play
by using
VLN
A
A
B
B
B
B
B
A
A
B
A
B
B

Table 3. An inquiry result of beginners’ piano class at Nat.
Inst. Tech., Kisarazu College (A: excellent, B: good, C:poor). EPI means experience of playing instrument.

ano fingering information, user can choose “zoom in” and“zoom out” for changing scale of view, and “PrevPage”
and “NextPage” for moving previous and next page.
7. EXPERIMENTAL RESULTS
7.1 Inquiry of beginners’ piano class at NITKC
Table 3 shows a result of inquiry of beginners’ piano class
at NITKC (see Section 3.3). 13 students who were interested in playing the piano volunteered for this class which
took 90 minutes.
All the subjects could read VLN scores easier than orequal to staff notation scores except one of them. The subject 13 is used to reading staff notation scores and usuallyplays alto saxophone, and this is the first time seeing VLN
scores. Therefore, it seems that prior knowledge of traditional staff notation caused interference in reading VLN
scores for this subject. Although most of the subjects haveInternational Conference on Technologies for Music Notation and Representation - TENOR 2016

Test 1

Test 1

Test 4

Test 444

44

Test 2

Test 5

Test 2

Test 544

44

Test 3

Test 6

Test 7

Figure 16. Scores used reading test.

Test 744

not took any regular piano lesson, all the subjects could
play the piano well in the class.
These results suggest that VLN scores provide a greater
chance of success when introducing the piano to beginners.
7.2 Comparison of performance with VLN scores andstaff notation scores
In order to examine the readability of VLN scores, we
compared with piano performances between in case of using VLN scores and using staff notation scores. We usedexample scores as shown in Figure 16 and Figure 17. The
score of Test 3 and Test 6 does not contain finger number because we expected that subject just concentrates onpressing correct key. Therefore all the subjects pushed the
key only with index finger. In the score of Test 4 and Test
5, each note has an accidental respectively. Note that thescore of Test 3 and Test 7 are the same music piece. Subjects were NITKC students. Among them, each memberof group A was not used to read staff notation score, andgroup B was used to. We taught them the pitch of first noteof each score and instructed them to continue playing the
piano even if the subject noticed mistake by oneself. The
playing tempo was free.
Table 4 shows the result of the number of miss-key hitting. Most subjects Ai made mistakes in the case of using

Figure 17. Scores used reading test (continued).
staff notation score. On the other hand, subjects Bi playedcompletely without errors. All subjects performed withoutor with a few mistakes in the case of using VLN scores.
Especially, although the score of Test 3 and Test 7 are thesame music piece, each number of mistakes for Test 3 isless than that for Test 7 for subjects Ai .
Table 5 shows the results of playing duration. The durations in the case of using VLN scores are less or equal
to the duration in the case of using staff notation scoresfor most subjects Ai . Although subject A10 finished Test 7
faster than Test 3, the number of mistakes in Test 7 is larger
than in Test 3. For subject Bi , in contrast, the duration in
the case of using VLN score is greater or equal than using
staff notation score.
From the above results, we consider that the readability of
VLN scores is high for music beginner players. However,
it is not necessarily the case for people who are accustomed
to read staff notation score. The most important thing is
to be able to play the piano for music beginner, namely
first consideration is that subject experiences success performances. Therefore, at first, piano beginners may notrecognize pitch name clearly, and they only have to under-

International Conference on Technologies for Music Notation and Representation - TENOR 2016

207

Subject
No.
A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
B1
B2

Scores / Number of miss-key hitting
1 2
3
4 5
6
7
G F VLN G F VLN G
8 2
0
6 5
3
1
6 6
0
11 6
0
9
9 8
0
7 3
1
8
10 9
0
13 8
0
8
8 8
0
9 3
0
8
6 6
0
1 0
0
6
8 8
0
3 3
0
8
6 8
0
9 7
0
6
6 0
0
1 1
0
6
1 0
0
1 0
0
7
0 0
0
0 0
0
0
0 0
0
0 0
0
0

Table 4. The number of miss-key hitting. Subjects Ai are
not used to read staff notation score, subjects Bi are used
to. G and F mean staff notation score in G clef and F clef,
and VLN signifies VLN score. Note that score 3 and score
7 are same music piece.

stand the position of the key which will be pressed. Of
course, understanding pitch name is also important, thus
we will discuss in Section 8.2 and Section 8.3.

7.3 Production of VLN scores for beginner players
and advanced players, and their influence on
performances
Figure 18 shows an example of VLN score: the beginningof “March” from “The Nutcracker Suite op.71a” by Peter
Ilyich Tchaikovsky. Finger numbers in Figure 19 are forbeginners, and the VLN score is produced by VLNMaker.
Finger numbers in Figure 20 are for advanced player, andthe numbers are given by a professional piano player.*6
We examined differences of ease of playing in the case ofusing VLN scores for beginners versus advanced players.
The subjects were students who belonged to NITKC, andtheir age was from 19 to 20 years old. First, we instructed
how to read VLN score to subjects. Second, we asked themto play the song “tulip” to be used to read a VLN score. In
this case, if the subjects made mistakes in their piano playing such as missed-key strokes and/or wrong fingering, we
ignored those errors. Third, they played the only 2 bars of
the beginning of “March” twice by changing the fingering.
In this case, we asked them to repeat their practicing of thepiece until they could play completely without errors.
Table 6 shows results of inquiry about piano playing using VLN scores. None of the subjects had previously practiced playing the piano in regular lesson. The results showthat they could play easily when they used a VLN score forbeginners versus one designed for an advanced player.
*6 VLNMaker can also load a text file which contains piano fingering
information which is made by hand. In this case, we prioritized ease ofunderstanding, hence we omitted rests in the VLN scores.

208

Subject
No.
A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
B1
B2

1
G
140
88
72
140
176
64
64
64
40
84
40
20

2
F
96
76
68
88
76
80
52
64
48
84
52
16

Scores / Duration [sec]
3
4
5
6
VLN
G
F VLN
40
144 68
52
68
80 76
44
48
72 68
32
88
116 6476
44
108 80
36
48
52 60
52
36
80 76
40
48
60 52
36
32
64 52
28
72
82 60
52
44
44 40
44
28
20 20
28

7
G
64
84
52
88
72
60
88
48
52
40
32
20

Table 5. Playing duration. Subjects Ai are not used to readstaff notation score, subjects Bi are used to. G and F mean
staff notation score in G clef and F clef, and VLN signifies
VLN score. Note that score 3 and score 7 are same musicpiece.

8. DISCUSSION
8.1 Score arrangement
We consider that it is necessary to establish an arrangement method. Although the vertical line notation can show
music faithfully, it is difficult to play complex chords andphrases, a sequence of very short notes, or a large jumpof pitch for beginners. We will solve these problems by
reducing the number of notes, simplifying rhythms, andadjusting pitches to match player’s skill while keeping theatmosphere of the original music as much as possible.
8.2 Expansion of VLN
We plan to expand the vertical line notation. A characteristic of the vertical line notation is simplicity comparedwith staff notation. On the other hand, we are considering
giving it a more functional representation. Figure 21 andFigure 22 show examples of musical scores (“For Elise” byLudwig van Beethoven) in expanded vertical line notation.
Both figures contain the left hand part, which is arranged,
and repeat marks shown as a green line and dots on theupper portion in Figure 21 and on the lower portion in Figure 22 respectively. Moreover, these VLN scores containlarge jump of pitch relatively, in these part, same finger
does not always use. Another examples are shown in Figure 23 and Figure 24. These figures include chords whichis shown as connected two nodes with a horizontal broken
line respectively, and left hand part as orange nodes. Ourdeveloped software will be able to deal with informationof these kinds.
Furthermore, there is room for improvement for the useof color. For example, the background of the keys of C andF are expressed using pale red and green.
In addition, it is important to express a length of eachnote. Currently, the vertical line notation shows the lengthof the note by the vertical distance between nodes. Al-

International Conference on Technologies for Music Notation and Representation - TENOR 2016

TitleComposer1

4
4
3

Figure 18. Staff notation score at the beginning of“March” by Tchaikovsky.

3
2
1

2

1
1
1
1
1

3
2
1
2
2
3

1
2

1

Figure 20. A converted result of VLN score of “March”
for advanced players.

1

Figure 19. A converted VLN score from SMF for
beginners of “March” from “The Nutcracker Suite” by
Tchaikovsky.

though it is simple, the player may not understand thelength of the note immediately. Therefore, for example, weare considering showing each node by a rectangle insteadof a circle, and the height of the rectangle would indicate
the length of the note. Furthermore, it may be useful to use
different colors for various note lengths.
However we should pay attention to complexity of notation. Although complex notation scores have various anddetailed information of music, such scores cause confusion
for beginner player. Namely, readability and complexity
are a trade-off relationship.
8.3 Transition from VLN to staff notation
At senior keyboard class in Showa University of Music,
some of the early joined members tried to play the piano
while putting VLN scores in a landscape orientation. Itshows as same as staff notation: vertical direction represents the pitch of notes and horizontal direction expresses
time flow.
Currently, most of the members proceed to using staff
notation scores which are chosen as liking music pieces

subject
No.
1
2
3
4
5
6
7
8
9
10

reading
VLN
B
B
B
B
B
B
B
A
A
B

playing by using VLN
TMbgn Madv
B
B
C
A
A
B
A
B
B
B
A
B
B
B
C
B
C
C
B
B
C
A
A
C
A
A
C
B
A
C

Table 6. Results of inquiry about piano playing using VLN
scores (A: excellent, B: good, C: poor) and situation ofplay. Each T, Mbgn and Madv shows use of a VLN scoreof the song “Tulip,” “March” for beginners and “March”
for advanced players. All the subjects had not formerly
practiced playing the piano in regular lesson.

themselves. In fact, many piano score books in staff notation are sold, and this is one of advantages of staff notation.
It seems that the members are used to play and have confidence in playing the piano gradually, thus they are going
to be more aggressive. This case is a success instance ofintroduction of playing the piano, and is also “graduation”
of VLN scores.
We will try to construct a more systematic learning
method for transition from VLN score to staff notation

International Conference on Technologies for Music Notation and Representation - TENOR 2016

209

Figure 21. VLN score of the first page of “For Elise” byBeethoven (handmade).

Figure 22. VLN score of the second page of “For Elise”(handmade).

score: it is necessary to learn the various information in
staff notation such as pitch names, rule of accidentals, note
lengths, expression marks, etc.

problem we believe that our study will solve it. Anotherone is that practice method is developing. Because thenumber of piano teacher with VLN score is not enough, it
is necessary to teach the learning method using VLN score
to teachers rather than students in advance.
In the near future, we will apply VLN scores not only formusic education but also rehabilitation, music therapy, etc.

8.4 Fingering check system
We plan to construct a fingering check system which obtains images of the moving state of a piano player’s handand fingers by using a movie camera, specifically, an RGBD camera such as Microsoft Kinect sensor [14], ASUS
Xtion Pro Live [15], Leap Motion [16, 17], etc. Using
these device, we can implement non-contact type system.
It is important to use “natural input” for constructing user
friendly system which does not ideally bring any stressfor users. The fingering check system can provide a selflearning material for beginner player, and also automatically estimate the playing skill level of a piano player by
analyzing the images. Hence, an automatic conversion system for VLN scores from SMFs will be improved to provide VLN scores with appropriate difficulty of piano fingerings which would correspond to the player’s piano skill.
8.5 The spread of VLN
We consider spreading VLN score widely. Now, regularpiano lessons with VLN scores have held in limited places
yet such as Showa Music University, Heisei Music University, Ooizumi Elementary School, etc. One of the reason
is that VLN scores are made by hand. With regard to this210

9. CONCLUSIONSThis paper described a conversion method from standard
MIDI files to the “vertical line notation” (VLN), and analgorithm of automatic decision of piano fingering for piano beginners. Because VLN scores have been made byhand on Microsoft Excel currently, we developed an automatic conversion system from standard MIDI files to VLN
scores. Moreover, we proposed an automatic decision of
piano fingering for beginners. We confirmed that fingeringfor beginners is easier to play than fingering designed forthe advanced players.
As future work, we plan to solve several problems described in section 8, and improve both VLN itself, and theautomatic conversion system of VLN scores from SMFs.
Acknowledgments
This study is supported in part by a Grant-in-Aid for Scientific Research from the Japan Society for the Promotionof Science, No. 26240025, 15K16054, 16K00501.

International Conference on Technologies for Music Notation and Representation - TENOR 2016

Figure 23. VLN score of first page of “Flea Waltz” (handmade).

Figure 24. VLN score of second page of “Flea Walz”(handmade).

10. REFERENCES
http://en.wikipedia.org/wiki/Musical_notation[10] Y. Yonebayashi et al., “Automatic decision of piano
fingering based on hidden Markov models,” Proc. IJCAI, pp. 2915–2921, 2007.

[2] S. Agata, “YAMAHA portable keyboard self tutor for
people who have never played a keyboard before,”
Yamaha Europa G.m.b.H, 1985.

[11] M. Balliauw et al., “Generating fingerings for polyphonic piano music with a tabu search algorithm,”
Proc. MCM, pp. 149–160, 2015

[3] C.M.Bishop, “Pattern Recognition and Machine
Learning,” Springer, pp. 359–418, 2010.

[12] L. Rabiner, “A tutorial on hidden Markov models
and selected applications in speech recognition,” Proc.
IEEE, 77(2), pp. 257–286, 1989.

[1] Musical notation,[4] R. Sato et al., “Automatic generation of musical scorein vertical line notation from MIDI file,” Proc. ICAI,
T105-04, pp. 559–562, 2015.
[5] MIDI,
http://en.wikipedia.org/wiki/MIDI[6] G. Mazzola, “The Topos of Music,” Chapter VIII,
Birkhuser, 2002.

[13] E. Nakamura et al., “Merged-output HMM for piano
fingering of both hands,” Proc. ISMIR, pp. 531–536,
2014.
[14] Kinect sensor,
http://www.xbox.com/en-US/en/kinect/[15] Xtion Pro Live,https://www.asus.com/3D-Sensor/Xtion_PRO_LIVE/[7] R. Parncutt et al., “An ergonomic model of keyboard
fingering for melodic fragments,” Music Perception,
14(4), pp. 341–382, 1997.

[16] Leap Motion,[8] M. Hart and E. Tsai, “Finding optimal piano fingerings,” The UMAP Journal, 21(1), pp. 167–177, 2000.

[17] “Playing the Grand Piano with Leap Motion Gestures
and BigBang”,[9] A. Al Kasimi et al., “A simple algorithm for automatic
generation of polyphonic piano fingerings,” Proc. ISMIR, pp. 355–356, 2007.

https://www.youtube.com/watch?v=ytGcKfhzF2Q

https://www.leapmotion.com/?lang=en

International Conference on Technologies for Music Notation and Representation - TENOR 2016211