Journal of Robotics, Networking and Artificial Life
Vol. 7(1); June (2020), pp. 73–76
DOI: https://doi.org/10.2991/jrnal.k.200512.015; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

The Research about Editing System of Performance
Information for Player Piano
Mami Ezaki*, Eiji Hayashi
Department of Intelligent and Control Systems, Kyushu Institute of Technology, 680-4, Kawazu, Iizuka-City, Fukuoka, 820-8502, Japan

ARTICLE INFOABSTRACT

Article History

We have developed a system that allows a piano to perform automatically. In order to play music in the manner of a live pianist,
we must add expression to the piano’s performance. In the case of music, there are often 1000 or more notes in the score,requiring that an editor spend a huge amount of time to edit. Therefore, we have developed an interactive musical editing systemthat utilizes a database to edit music more efficiently.

Received 06 November 2019
Accepted 26 March 2020
Keywords
Automatic pianoknowledge database
computer music
music interface© 2020 The Authors. Published by Atlantis Press SARL.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
We have developed a system that allows a piano to perform automatically [1]. In this system, 90 actuators have been installed on
the keys and pedals of a grand piano. These actuators execute key
strokes and pedal movements to govern the piano’s performance,e.g., “Figure 1 (Continued)” [2,3].
Our research focuses on giving player piano performance withhuman-like skill and expression. Playing the piano expressiveby player piano, it is necessary to adjust the volume, length, andtiming of music. We aimed to develop a system that, like a skilled
pianist, can perform even the first musical score based on information related to previous skills and experience. In this paper,
we developed a system that automatically estimates the performance expression of unedited music using edited performancedata and score data. We described a phrase search using Dynamic
Programming (DP) matching and a method for selecting an optimal phrase for inference, and evaluated the entire song [4].

exhibits tempo. “Bar” is the vertical line placed on the staff to divide
the music into measures.
The pedal information consists of four parameters. Specifically,
there are four parameters: “key” (“Damper” or “Shifting”), whichindicates the type of pedal, “Velo”, which indicates the amount ofpedaling, “Time”, which indicates the time to pedal, and “Bar”,
which indicates the bar number.

2.2. How to Make the Data for Player PianoWe show the structure of the edit system how to make the data forplayer piano in “Figure 2 (Continued)”. When we have the music
data A that the pianist has played, and do not have the music data
B that the pianist has played, we extract a characteristic of the pianist from music data A. Then we make the music data B’s musicalperformance by doing reasoning from characteristic of musicalperformance and information of musical grammar and score ofmusic B.

2. EDITING SUPPORT SYSTEM
2.1. Performance Information
The automatic piano that we have developed uses a music data structure that is similar to Musical Instrument Digital Interface (MIDI).
We defined performance information, dividing it into two categories: the notes and the pedals. The note information is comprised of
the six parameters involved in producing a tone: “Key” (note), “Velo”(velocity), “Gate”, “Step”, “Bar” and “Time”. “Velo” is the dynamics,given by the value of 1–127. “Gate” is the duration of the note inmilliseconds. “Step” is the interval of time between notes, and it also
Corresponding author. Email: ezaki@mmcs.mse.kyutech.ac.jp*

2.3. Search System
As a result of the analysis, it was found that phrases of the same
pattern existing in the same tune are performed in a similar expression (Figure 3) [5]. This time, we used DP matching to search forsimilar phrases.
DP matching is a technique used widely in the field of speech recognition, bioinformatics and so on. It has a feature that can calculate the similarity between two words that are different in a numberof characters from each other. In Figure 4, the route of minimum
cost in each point is taken, and the route with the lowest cost is
assumed finally to be the optimal path. The cost at that time is74

M. Ezaki and E. Hayashi / Journal of Robotics, Networking and Artificial Life 7(1) 73–76

we found by doing DP matching from music data. Objective phrase
is phrase we want to inference in this time. Therefore, we need toselect the best phrases for objective phrases by doing the Selectionsystem. In this year’s system, three indicators were used from theviewpoint of music theory: Dynamic symbols, Beats and Steps, andMusical forms.

2.5. Dynamic Symbols
Figure 1 | The automatic piano.

If the Dynamic symbols are different between similar phrase andobjective phrase, the performance is affected even if the phrase is
the same, and the search phrase and the similar phrase that matches
the dynamics on the score are preferentially selected.

2.6. Beat and Step
Based on the musical grammar, it is known that strong beats areclosely related to Step [6]. In places considered to be strong beats,
similar phrases are selected using the property that the Step value islarger than in other places. The position of the strong beat depends
on the rhythms.
Figure 2 | The structure of the edit system.

2.7. Musical Forms
If the Selection system in “Subsections 2.5 and 2.6” does not narrow
down the number of similar phrases, the selection is performed
according to the Musical Forms. Musical Forms is a music format
such as the Rondo Forms or the Sonata Forms. For example, if different songs were in the sonata format, it was predicted that thepresentation part had a presentation part feature, the development
part had a development part feature, and each part had a uniquefeature. We decide same part has similar performance.

Figure 3 | The discovery about the relation of same tunes and similar
expression.

3. INFERENCE SYSTEM
3.1. System Overview
The Velo value that the performer is executing for the dynamic
symbol on the score is examined, and Velo suitable for the optimalphrase is inferred from the search phrase.

3.2. Inference by the Strength Symbol
Figure 4 | Result of dynamic programming matching method.

defined as the distance between patterns. In this system, this distance is handled as a threshold to judge whether the phrases aresimilar to each other.

2.4. Selection System
After doing the Search system, there are a lot of similar phrases
which has the same DP matching points. Similar phrases are phrases

In order to obtain the characteristic of the change of the Velo value
due to the influence of the intensity sign, the tendency was examined from the performance data of the pianist. The pianist of thesurveyed performance data is Gerhard Opitz, and its title is shownin Table 1 [7].
As a result, the definition of Velo with detailed strength varied
depending on the song, but the one with the high frequency of each
Velo value was adopted as the strength symbol, and the results aresummarized in Table 2.
Inference using Table 2 and Velo’s inference formula for the xth
sound are shown in Equations (1) and (2). SVelo refers to the optimal phrase Velo.

M. Ezaki and E. Hayashi / Journal of Robotics, Networking and Artificial Life 7(1) 73–76

Table 1 | The studied title of musical compositions

75

Table 4 | The Velo value for a composite dynamic markTitle of musical compositions①②③④

Sonate fur Klavier Nr.8 c-moll Op.13-1 “Grande Sonate Pathetique”
Sonate fur Klavier Nr.8 c-moll Op.13-2 “Grande Sonate Pathetique”
Sonate fur Klavier Nr.8 c-moll Op.13-3 “Grande Sonate Pathetique”
Sonate fur Klavier Nr.23 f-moll Op.57-1 “Appassionata”

Table 2 | The Velo value for a composite dynamic markDynamicmarkppppppmpmf

Velo

15–20

20–45

45–65

65–75

75–80

fff

80–95 95–115
Figure 6 | Comparing select phrase to pianist’s performance data.

Figure 5 | Search phrase.
Table 3 | Selecting method by using Dynamics symbols

Figure 7 | Comparing select phrase to inferred select phrase to pianist’sperformance data.

4.3. Consideration
 x = 1 ∩ strong beat, chose maximum value of range Velo(x ) = (1) x = 1 ∩ weak beat, chose median of range(2)
Velo(x ) = Velo (x − 1) − SVelo ’s value (x ≥ 2) 4. INFERENCE EXPERIMENT
4.1. Experimental Method
An experiment was conducted to compare the music reproduced using the editing support system with the performanceof the pianist. The target song is Beethoven Piano Sonata No.14,
Op.27-2(“Moonlight Sonata”). Figure 5 shows an example of asearch phrase.

4.2. Experimental Result
Using graphs and correlations, we compared inferred phrases
with actual performance. The Search system found 18 similarphrases. Table 3 shows the 10 data selected by Dynamics symbols.
○ indicates that the numerical value is 80% or more of the maximum value, and × indicates that the value is 40% or less, and △indicates neither of them. From Table 3, the optimal phrase wasselected as the 1st number 226. For confirmation, Table 4 shows
the correlation coefficient between each similar phrase and actualperformance.

Table 4 shows that the phrase selected by the selection system
is closer to the actual performance than other similar phrases(Figure 6). From Figure 7, it was confirmed that the Velo was closer
to the actual performance. Thus, the selection system and inference
system newly added this year are considered to be effective.

5. CONCLUSION
This year, we introduced a selection system that selects the mostsuitable phrases and an inference system that infers the Velo values
that are likely to be performed by the pianist, to the editing support
system. The inference experiment showed that the performance
expression of the unedited piece could be inferred. This selection
system is based on concepts that are faithful to music theory. In
reality, however, not all pianists play faithfully to the strength andweakness symbols on the score.
Therefore, the current system of inference based on indicators suchas strength and weakness symbols cannot be used to infer the case ofperformers who do not have strength and weakness symbols in theirscores, such as Bach. In order to solve this problem, we need to consider
human playing from a new perspective. After that, we will develop anew system that can select the best phrases for classical music of all era.

CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

M. Ezaki and E. Hayashi / Journal of Robotics, Networking and Artificial Life 7(1) 73–76

76

REFERENCES
[1]
[2]
[3]

[4]

[5]

E. Hayashi, M. Yamane, H. Mori, Development of a moving coil
actuator for an automatic piano, Int. J. Japan Soc. Prec. Eng. 28(1994), 164–169.
K. Asami, E. Hayashi, M. Yamane, H. Mori, T. Kitamura,
Behavior of piano-action in a grand piano for an automatic
piano, IEEE Proceeding of RO-MAN'98, 1999, pp. 672–677.
E. Hayashi, M. Yamane, H. Mori, Behavior of piano-action
in a grand piano for an automatic piano, Proc.1st Pioneering
International Symposium on Motion and Vibration Control in
Mechatronics, 1999, pp. 80–85.
Y. Isomichi, T. Ogawa, Pattern matching by using dynamic programming, J. Inf. Process Soc. Japan 16 (1975), 15–22 (in Japanese).

[6]
[7]

K. Koga, E. Hayashi, Interactive musical editing system to support human errors and offer personal preferences for an automatic piano: a method for inferring performance expression
with the best alignment of DP matching, The Japan Society ofMechanical Engineers Kyushu Branch, The 63rd Term, 2010.
S. Jo, Analise about harmony and musical style, Ongaku-NoTomo Sha Corp., Tokyo, Japan, 1964.
L. van Beethoven, Sonata fur Klavier Nr.14 cis-moll Op.27-2“Sonata quasi una Fantasia” (Mondscheinsonate), in: B.A.
Wallner, ed. Klaviersonaten Band I, Germany, G. Henle Verlag,
2009.

AUTHORS INTRODUCTION
Ms. Mami Ezaki

Prof. Eiji Hayashi
She is a Master course student in at theDepartment of Intelligent and Control
Systems Kyushu Institute of Technology,
Japan. She received her Bachelor’s degreefrom the Department of Mechanical
Information Science and Technology
Kyushu Institute of Technology, Japan in
2019.

He is a Professor in the Department ofIntelligent and Control Systems at Kyushu
Institute of Technology. He received thePhD (Dr. Eng.) degree from Waseda
University in 1996. His research interests
include Intelligent mechanics, Mechanicalsystems and Perceptual information processing. He is a member of The Instituteof Electrical and Electronics Engineers (IEEE) and The Japan
Society of Mechanical Engineers (JSME).