T105-04

The 1st International Conference on Advanced Imaging

Automatic generation of musical score in vertical line notationfrom MIDI file
Riku SATO*1, Yasuyuki SAITO*1, Suguru AGATA*2,
Eita NAKAMURA*3, Shigeki SAGAYAMA*3*1 Dept. of Information Engineering, National Institute of Technology, Kisarazu College*2 Showa University of Music*3 School of Interdisciplinary Mathematical Sciences, Dept. of Frontier Media Science, Meiji
University
Current mainstream music notation uses a 5-line staff. Staff notation is used for many types of
instruments, and contains a great deal of information. However, to understand staff notation is difficult for
beginner players who are not used to reading it because of its complexity. Vertical line notation is a new
type of notation that is easy to understand because of its simple and intuitive design. It was created by
Suguru Agata who is one of the authors of this research. In this study, we examine a method for creating a
musical score in vertical line notation from a MIDI file.
1. Introduction
Generally, instrumental players use staff notation.
There are some other types of notation such as tablature
notation for guitar, one line notation for percussion,graphic notation, numeral notation, Japanese traditional
notation, and so on1). 1
In ancient Greece, character notation was used. The
characters show the pitch of notes, and were written
above the lyrics. The oldest Christian chant in existence,
which is said to have been created in A.D. 280, is written
using this notation. In the ninth century, neumatic
notation was introduced for use in Gregorian chant. At
first, the neumatic notation included just a curve andstraight lines which indicated a pitch and length of note.
Next, the notation introduced a horizontal line as areference pitch, and the line went increased graduallyfour and then five, finally it is same as a contemporary
staff notation. Around 1025, Guido d’Arezzo, who wasan Italian music theorist and monk devised a notation
which is considered a prototype of current staff notation.
The notation used square notes on four lines. The
number of horizontal lines of music notation was settled
at five in the early 17th century.
Current mainstream music uses staff notation, and is
not limit to the piano. In fact, staff notation is used formany types of instruments. It contains a large amount of
information such as pitch, length, dynamics, tempo
marks, expression marks, articulation, repeat signs,
ornamentation, abbreviations, accidentals, key signatures,and so on. Therefore, to understand staff notation isdifficult for beginner players, especially older persons,

who are not used to reading it because of its complexity,
even though they grasp only the pitch information.
With this background in mind, vertical line notation
was devised in 1985 for keyboard education purposes by
Suguru Agata, who is one of the authors of this study.
Agata made a textbook in vertical line notation in
English2) and German while he was in Germany between
1982 to 1985. At that time, he worked for an instrument
manufacturer. Unfortunately, his superiors did not see thevalue in vertical line notation. However, he has
frequently demonstrated the effectiveness of thisapproach in open lectures of keyboard class for older
persons at Showa University of Music since August
2010.
Vertical line notation is a new type of music notation,and even the beginner can understand it very easily.
Currently, musical score in vertical line notation is madeby using Microsoft Excel manually.
In this study, we present an automatic conversion
system from a MIDI file to a musical score in verticalline notation.
2. Vertical line notation
Fig.1 shows an example of musical score in staff
notation of Japanese children’s song “Tulip”. And Fig.2
is in the vertical line notation which corresponds to Fig.1.
This musical score is made for a little child or an older
person, so title is written on the top of the musical scorein hiragana-character. The lyric is also described on theleft side in hiragana-character.
In the vertical line notation, horizontal direction means
the pitch of note: the left side is lower pitch and right*1 2-11-1 Kiyomidai-higashi, Kisarazu, Chiba, Japane-mail: saito@j.kisarazu.ac.jp− 559 −

The 1st International Conference on Advanced Imaging

Fig.1 Example of staff
children’s song “Tulip”)

notation(Japanese

Fig.2 Example of vertical line notation (“Tulip”)
side is higher pitch. The vertical direction signifies time;going in the downward direction moves forward.
Each circle is a note located on a corresponded pitch.
The circle includes numeric figure which shows which
finger to use. In this way, the piano player can perform
correct fingering very easily. The correct fingering isimportant to play smoothly and without wasted motion.
The blue beam does not have a special meaning;
however, it is useful for seeing whether the pitch of thenext note is up or down compared with the current pitch.
Moreover, musical score in vertical line notation cancontain chord names. In Fig.2, they are expressed insquares with orange background color. Therefore, theaccompanist or a senior player can perform the chordwith the left hand.
Fig.3 shows another example of staff notation. This iscalled “I Stepped on a Cat” in Japan, and the name ofthis song exists about 28 versions in the world. For
example, this song is called “Circus Song” in USA, UK,
and Canada. This music is G flat major, which has six
flats. Therefore, this song looks very difficult. On theother hand, a musical score in vertical line notation
shown in Fig.4 can be read easily. In this case, notes
which are colored with orange background on the left
side indicate to play with left hand, and each note whichis connected a dotted line is performed at the same time
as a chord.
3. Musical Instrument Digital Interface
In digital music area, Musical Instrument Digital
Interface (MIDI) has been most often used. It wasformulated by the Japan MIDI Standard Committee(JMSC; in 1996, this organization was incorporated into

Fig.3 Example of staff notation (“I Stepped on aCat”)

Fig.4 Example of vertical line notation (“I Steppedon a Cat”)
the Association of Musical Electronics Industry, AMEI)
and the MIDI Manufacturers Association which is aninternational organization3). MIDI is a universal standard
for the digital transmitting of performance data between
electronic musical instruments. It consists of someregulations, such as physical transceiver circuit andinterface, communication protocol, file format, et cetera.
Transmission and reception of data on the MIDI
standard are all carried out on MIDI messages. Plural
bytes (8 bits per one byte) construct a MIDI message. Totransmit MIDI messages efficiently, the bytes which
express MIDI messages are divided into two types:“status byte” and “data byte”. The status byte has “1” at
the most significant bit (MSB); namely, it contains 128
data types from 80H to FFH in hexadecimal. In contrast,
the data byte includes “0” at the least significant bit(LSB). Thus the range of the data is from 00H to 7FH.
There are three types of formats of standard MIDI files.
The format 0 has only one track which includes all note
information. It is, so to speak, mixed down music data.
Therefore, this format is for the purpose of playback only.
The format 1 can contain many independent tracks. This
format easily deals with decomposed data on a MIDI
sequencer. The Format 2 seems to have been originallydeveloped and intended for use in, for example karaoke.
However, it is little used in practice.
The standard MIDI file is composed of several parts.
One is the “header chunk,” which includes informationof the whole standard MIDI file. The other is “trackchunk,” which contains real performance data.

− 560 −

The 1st International Conference on Advanced Imaging

4. Transformation method from MIDI file tovertical line notation
The header chunk and the track chunk begin with amagic number as “MThd” and “MTrk”. First, it isnecessary to find these bytes in a MIDI file. In the header
chunk, a time resolution, which is called “delta time”, isdefined. This shows a capacity to decompose a quarter
note. For example, in case that the delta time is 480, it ispossible to express the quarter note to 480 divided
lengths.
The delta time is also described together in all events
the MIDI file is represented. In this case, the delta time
represents the time until the next event is performed. To
represent the note by MIDI events, use two types of deltatime of the above, and both of “note-on” (a command ofturning on the sound) and “note-off” (turning off thesound). In case that the time length from note-on to noteoff is equal to delta time, it means a quarter note. Let S
show a time of note-on, E be a time of note-off, D meana delta time for whole MIDI file, the meter be four-fourtime, and the music has no Auftakt. In these conditions,the note length l is calculated by Equation (1).
l = (E – S) / (D * 4).
(1)
A note position P from the beginning in a measure is
obtained by Equation (2), and a measure number N of anote is calculated by Equation (3).
P = S mod (D * 4),(2)
N = [ S / (D * 4) ] + 1,
(3)
where “mod” means the remainder operation, “[ ]”indicates the floor function, and 4 is the denominator of
the musical time. By using these three equations,performance data of a MIDI file are converted to data
sequences which have only note length, note position,and measure number.
5. Implementation
We have developed software which inputs a MIDI fileand converts it into a musical score in vertical line
notation. The software creates the musical score as agraphics data by using OpenCV4).
OpenCV is a very useful computer vision library. Thisis developed and released by Intel, and is an open source.
Currently, development of OpenCV has been taken overby Willow Garage.
We use C++ and introduce Qt for GUI programming5).
Qt provides a cross platform environment such as
Windows, Mac OS, Android, and so on. It contains
integrated development environment (IDE) which iscalled “Qt Creator”. Using the Qt Creator, we canperform the debug process easily, and design a window
interface which includes a push button, radio button,slider bar, table, menu bar, and so on.

(a)

(b)

Fig.5 Experimental results. (a) initial result oftransformation from MIDI file into vertical line
notation, (b) final result with suitable finger number

Fig.6 Control panel to input finger numbers
6. Experimental results
Fig.5 (a) shows the result of the conversion from a
MIDI file into in vertical line notation immediately after
reading the MIDI file. At this time, the finger numbers
are all zeros. Fig.5 (b) shows the final result which
includes suitable finger numbers. Fig.6 presents a control
panel to input finger numbers. To push each button from“1” to “5”, the finger number is placed into the circle inthe musical score in vertical line notation. A notice circle
is expressed in red. The “prev” and “next” buttonchanges the notice node. The “scale up” button magnifies
the score and “scale down” button reduces respectively.
7. Discussion
7.1 Improvement of software and vertical line
notation itself
This developed software can make the fundamental
musical score in vertical line notation. However, several
issues are left and are included to improve the verticalline notation itself.
First, it is necessary to introduce an automatic
fingering determination method. To make the musicalscore in vertical line notation, the finger numbers is set
manually both in conventional way and even in this study.

− 561 −

The 1st International Conference on Advanced Imaging

7.2 Potential of vertical line notation
The vertical line notation is designed for education of
keyboard performance. In addition to this purpose,
vertical line notation has a very big potential.
Agata has used musical scores in vertical line notation
at a facility covered by public aid providing long-termcare to the elderly. One of the residents of the facility
suffers from Parkinson's disease. This person’s bodyusually leaned to one side. However, since the start of the
keyboard performance using musical scores in verticalline notation, his posture has improved. In addition, his
speech ability has also changed for the better, and he has
been able to evaluate the performance of others. It isconsidered that muscle and brain are activated by playing
the keyboard. If Agata used a complex musical score, theperson might lost interest. The vertical line notation issuggested to be fully utilized in the welfare field.

Fig.7 Example of staff notation (“For Elise”)

Fig.8 Example of expanded vertical line notation(“For Elise”)
However, it takes a great deal of time and effort. We willrefer to some prior studies6-8).
Second, it is necessary to establish an arrangement
method. Although the vertical line notation can show
music faithfully, for beginner and older player, it isdifficult to play a complex chord and phrase, a range ofvery short notes, or a large jump of pitch. We will try tosolve these problems by implementing reducing note ormeasure, simplifying rhythm, and adjusting pitch.
Third, it is important to express a length of each note.
Currently, the vertical line notation shows the length of
the note by the vertical distance between nodes.
Although it is simple, the player may not understand thelength of the note immediately. Therefore, for example,
we consider showing each node by a rectangle instead ofa circle, and the height of the rectangle means the lengthof the note.
Finally, we will try to expand the vertical line notation.
A characteristic of the vertical line notation is simplicity
compared with the staff notation. On the other hand, we
have considered giving it a more functional
representation. Fig.7 and Fig.8 show examples ofmusical scores (“For Elise” by Beethoven) in staff andexpanded vertical line notation. Fig.8 contains left handpart, which is arranged, and repeat marks shown as green
line and dots on upper in the left score and on lower inthe right score respectively. The software will be able todeal with these kinds of information.

8. Conclusions
In this study, we developed software which converts
from a MIDI file into in vertical line notation. The
vertical line notation is a new notation for piano
beginners and older persons. Its horizontal direction
indicates the pitch of note and its vertical direction
presents time sequence. The software, which isconstructed by using OpenCV based on C++ on Qt
Creator, works properly to read MIDI file and processes
finger numbers input from control panel.
We have examined the effectiveness of vertical line
notation on the field of keyboard education and welfare.
For our future work, we will solve several issues
described in section 7.
Acknowledgement: This study is supported in part by
Grant-in-Aid for Scientific Research from Japan Society
for the Promotion of Science, No. 26240025.
References
1)
2)

3)
4)
5)
6)

7)

8)

− 562 −Musical notation,
http://en.wikipedia.org/wiki/Musical_notation
S. Agata: YAMAHA portable keyboard self tutor for
people who have never played a keyboard before,(Yamaha Europa G.m.b.H, 1985)
MIDI, http://en.wikipedia.org/wiki/MIDI
OpenCV, http://opencv.org/
Qt, http://qt-project.org/
A. Al Kasimi, E. Nichols, C. Raphael: A Simple Algorithm
for Automatic Generation of Polyphonic Piano Fingerings,
ISMIR2007, 355-356 (2007)
Y. Yonebayashi, H. Kameoka, S. Sagayama: Automatic
decision of piano fingering based on hidden Markov
models, IJCAI'07, 2915-2921 (2007)
E. Nakamura, N. Ono, S. Sagayama: Merged-Output
HMM for Piano Fingering of Both Hands, Proc. of
ISMIR2014, 531-536 (2014)