Journal of Robotics, Networking and Artificial Life
Vol. 8(2); September (2021), pp. 99–103
DOI: https://doi.org/10.2991/jrnal.k.210713.006; ISSN 2405-9021; eISSN 2352-6386
https://www.atlantis-press.com/journals/jrnal

Research Article

Performance Information Editing System for Player Piano­Aiming at Human-like Performance
Ryo Kinoshita*, Eiji Hayashi
Department of Intelligent and Control Systems, Kyushu Institute of Technology, Hayashi Lab, 680-4, kawazu, Iizuka-City, Fukuoka, 820-8502, Japan

ARTICLE INFOABSTRACTArticle History

We have developed a system that enables automatic piano performance. To make this piano play like a living pianist, we haveto add expression to the performance. Accordingly, we have developed a musical editing system that utilizes a database to edit
music more efficiently.

Received 25 November 2020
Accepted 10 May 2021
Keywords
Automatic pianoknowledge database
computer music
music interface© 2021 The Authors. Published by Atlantis Press International B.V.
This is an open access article distributed under the CC BY-NC 4.0 license (http://creativecommons.org/licenses/by-nc/4.0/).

1. INTRODUCTION
We have developed a system that enables automatic piano performance [1]. Eighty-eight actuators are attached to a grand piano’skeys, and two actuators are attached to the pedals. These actuators
execute keystrokes and pedal movements to govern the piano’s performance, e.g., “Figure 1” [2].
In our research, we are trying to bring human-like skill and expression to player piano performance. Playing the piano expressiveby player piano, adjusting the volume, length, and music timingis necessary. We aimed to develop a system that can infer performance based on skilled pianist’s features. Therefore, we developeda system that can estimates the performance expression of unedited
music automatically by using edited performance data and score
data. In this paper, we described a phrase search using Dynamic
Programming (DP) matching [3]. In addition, we mention amethod for selecting an optimal phrase and how to infer parameters of notes. Finally, we evaluated an inferred entire song playedby both-handed.

2. EDITING SUPPORT SYSTEM
2.1. Performance Information
The automatic piano that we have developed uses a musical data
structure similar to musical instrument digital interface. Theediting support system edits four parameters contained in a tone:“Velo” (velocity), “Gate,” “Step”, and “Time”. “Velo” is the volume of

Corresponding author. Email: kinoshita@mmcs.mse.kyutech.ac.jp*a sound, given as a value from 1 to 127. “Gate” is the duration of thenote in milliseconds. “Step” is the interval of time between notes.
“Time” indicates when the sound will start after the music started.
“Step” and “Time” units are also milliseconds.

2.2. How to Make the Data for Player Piano
The structure of the editing support system for making the data forplayer piano is shown in Figure 2. The system extracts a pianist’s
features from performance data that the pianist has played. Next,
the performance data in the unperformed song is inferred from thepianist’s features and score information of a musical piece that thepianist has never played before.

2.3. Search System
The results of the analysis showed that phrases with the same pattern that exists in the same song are played with similar expressions (Figure 3) [4]. Equally, it was predicted that the same pattern’s
expression would be the same at other songs. Therefore, DP matching was adopted, which allows us to search for similar phrases forarbitrarily determining search phrases.
Dynamic programming matching is a technique used widely in thefield of speech recognition, bioinformatics, and so on. It has a featurethat can calculate the similarity between two words that are different
in a number of characters from each other. In Figure 4, the minimum
cost route in each point is taken, and the route with the lowest costis finally assumed to be the optimal path. The cost at that time isdefined as the distance between patterns. In this system, this distanceis handled as a threshold to judge whether the phrases are similar.

100R. Kinoshita and E. Hayashi / Journal of Robotics, Networking and Artificial Life 8(2) 99–103

2.4. Select System
After done the Search system, there are many phrases which has thesame DP matching points. Such phrases are called similar phrases.
An objective phrase is a phrase for which inference will be made. Inaddition, an optimal phrase is a phrase that is considered to be thebest fit for an objective phrase among similar phrases. It is required
to select the optimal phrases for objective phrases. Because ofthis, the select system is used to determine an optimal phrase. In
this system, five indicators are used from the viewpoint of music
theory: Dynamic symbols, Beats and Steps, Similarity in changesof the interval, Dynamic symbols and Velo, and Musical Forms.
Figure 1 | The automatic piano.

2.4.1. Dynamic symbols
If the Dynamic symbols are different between a similar phrase andan objective phrase, the performance is affected even if the phraseis the same. Then, the search phrase and the similar phrase thatmatches the dynamics on the score are preferentially selected.

2.4.2. Beat and step

Figure 2 | Editing system structure.

Based on the musical grammar, it is known that strong beats areclosely related to Step [5]. In places considered to be strong beats,
similar phrases are selected using the property that the Step value islarger than in other places. The position of the strong beat depends
on the rhythms.

2.4.3. Similarity in changes of the interval
If the interval changes of the phrases are similar, the performanceexpression will also be similar. Thus, the system selects phrases
with high similarity in a change of interval.

2.4.4. Dynamic symbols and VeloFigure 3 | The discovery about the relation of same tunes and similar
expressions.

It is likely that there is a trend in the Velo values depending on
the dynamic symbol. Hence, the range of Velo values for each
dynamic symbol is examined from performance data. Next, thesystem selects a phrase in which the Velo value corresponds to thedynamic symbol of the search phrase.

2.4.5. Musical forms
Finally, a selection is conducted using musical forms because it waspredicted that the same part has a similar expression.

2.4.6. How to select left-hand phrases
Figure 4 | Example of DP matching method.

When selecting left-hand phrases, the system narrows downphrases by phrase category before making a selection based on theR. Kinoshita and E. Hayashi / Journal of Robotics, Networking and Artificial Life 8(2) 99–103

five indicators. It is due to the same category likely have a similarexpression. Left-hand phrases can divide four categories: “Maintheme”, “Broken chord”, “Single”, and “Chord”. If there are no similar phrases in the same category, the system selects an optimalphrase from other categories.

2.5. Infer System
If the expression of an optimal phrase is used directly to a searchphrase, it will be an unnatural expression. Thus, this system infers
appropriate “Step”, “Velo”, and “Gate” values for the search phrasefrom the optimal phrase.

2.5.1. Data used for inference
For proper inference, investigation from performance data is necessary. This year, data for the four songs shown in Table 1. wereused. All songs were composed by W.A. Mozart and performed by
Maria Joao Pires.

2.5.2. Inference about step
The tempo investigation and the calculated normalization factor
using the music data in Table 1 are shown in Table 2. “Normalization
Factor” is a number that “All Tempo Average” divide by “TempoAverage”.
The inference equation for Step is Equation (1). “PStep” represents
the provisional Step value which calculates by optimal phrase and“NF” represents the normalization factor.
Step = PStep ´ NF ´ All Tempo Average(1)When inferring left-hand phrases, adjust the timing based on theresults of right-hand inference in the case of the note is the sametiming in the musical score.

101

Similar interval phrases are those phrases that have the highest similarity of interval change among similar phrases. A similar intervalphrase is used to infer the Velo value.
The inference equations for Velo are Equations (2–4). “n” represents
the number of notes in an objective phrase and “SVelo” represents
a similar interval phrase’s value of Velo. Equation (2) is used when
the first note in the phrase and the dynamic symbol is different
from the previous note. Depending on the results of DP matching,
the search phrase may have to be split. In this case, the first note of
the split phrase is calculated by Equation (3) except in the case ofEquation (2). Other notes are calculated by Equation (4).	  Velo (n) = average in dynamics symbol

(2)	  Velo (n) = Velo (n – 1)

(3)	  Velo (n) = Velo (n – 1)–SVelo’s value(4)

2.5.4. Inference about gate
The investigation of factors for each musical symbol that gives aneffect on the length of a note is shown in Table 4 by using music
data in Table 1.
The inference equation for Gate is Equation (5).
Gate = Inferred step ´ factor		(5)3. EXPERIMENT
An experiment was conducted to compare the music reproducedby the editing support system with the pianist’s performance. The
target music is Mozart Piano Sonata No.11 in A major, K.311 1st
movement “Theme”.
The score of the part showing the inference results is shown inFigure 5. All inference results are shown as graphs. Figures 6 and 7
are results about “Step”, and Figure 6 is right-hand, Figure 7 is

2.5.3. Inference about Velo

Table 3 | Average of Velo value for each Dynamic symbol

The average Velo value’s investigation using the music data inTable 1 is shown in Table 3.

Dynamic symbol

Table 1 | Music title of performance data
Music title
Piano Sonata No.11 in A major, K.311 3rd movement “Turkish March”
Piano Sonata No.15 in C major, K.545 1st movement “Allegro”
Piano Sonata No.15 in C major, K.545 2nd movement “Andante”
Piano Sonata No.15 in C major, K.545 3rd movement “Rondo”

pmffAverage (right-hand)

Average (left-hand)

6462
64

443645

Table 4 | Factors of musical symbols affect thenote length
Musical symbols staccato

noneslurFactor

0.9

10.4

Table 2 | Investigate results about tempo
Music numberTempo average
All tempo average
Normalization factor

0.80

0.82
1.00
0.89
1.07
0.89

0.92

1.11

0.96

Figure 5 | The score of the inferred part.

102

R. Kinoshita and E. Hayashi / Journal of Robotics, Networking and Artificial Life 8(2) 99–103­left-hand. Figures 8 and 9 are results about “Gate”, and Figure 8
is right-hand, Figure 9 is left-hand. Figures 10 and 11 are results
about “Velo”, and Figure 10 is right-hand, Figure 11 is left-hand.

Figure 10 | Compare graph about right-hand Velo.

Figure 6 | Compare graph about right-hand Step.

Figure 11 | Compare graph about left-hand Velo.

4. CONSIDERATION
Figure 7 | Compare graph about left-hand Step.

Figure 8 | Compare graph about right-hand Gate.

We can confirm that inferred Step value is closer to the performance of the pianist than unedited data. It is common to bothhands. We can also confirm the same trend in Gate. Hence, theresults suggest that the infer system about Step and Gate is effective. However, the inference result for the phrase containing rests
is a bit far from the performance data like numbers 13–17 in
Figure 9. So, it is necessary to consider the infer method whenphrases include rests. Also, the inferred value of Velo is not similar to performance data. Accordingly, infer system about Velo isneeds to improve. Moreover, the current system can not take into
account features such as the last phrase of a passage. It is also
necessary to establish an inference method that can consider theposition of phrases.

5. CONCLUSION
In the past few years, a method of using change of interval wasestablished [6], and a select system was established. [7] This year,
we introduced an inference system about left-hand and improvedsome systems. The inference experiment showed that we were able
to infer a performance expression in an unperformed song fromsome performance data. The next outlook is inferring pedal information and improve inference accuracy.

Figure 9 | Compare graph about left-hand Gate.

The Editing system of performance information is the systemthat infers a phrase from similar phrases in other songs. In thecase of the same phrases are repeated, the inferred result will be
precisely the same. By contrast, pianists should subtly change
inflection and timing for the same phrase. Moreover, the currentsystem can not infer performance information if there are noR. Kinoshita and E. Hayashi / Journal of Robotics, Networking and Artificial Life 8(2) 99–103

dynamics symbols in a score. To solve these problems, we needto develop a new ­versatile system that adds other methods to thecurrent system.

[3]
[4]

CONFLICTS OF INTEREST
The authors declare they have no conflicts of interest.

REFERENCES
[1]
[2]

[5]
[6]

E. Hayashi, M. Yamane, H. Mori, Development of a moving coil
actuator for an automatic piano, Int. J. Japan Soc. Prec. Eng. 28(1994), 164–169.
K. Asami, E. Hayashi, M. Yamane, H. Mori, T. Kitamura, Behaviorof piano-action in a grand piano for an automatic piano, IEEE
Proceeding of RO-MAN’98, 1999, pp. 672–677.

[7]

103

Y. Isomichi, T. Ogawa, Pattern matching by using dynamic
programming, J. Inf. Process Soc. Japan 16 (1975), 15–22 (inJapanese).
K. Koga, E. Hayashi, Interactive musical editing system to support
human errors and offer personal preferences for an automatic
piano: a method for inferring performance expression with thebest alignment of DP matching, The 63rd Term, The Japan Societyof Mechanical Engineers, Kyushu Branch, 2010, pp. 79–80.
S. Jo, Analise about Harmony and Musical Style, Ongaku-No
Tomo Sha Corp., Tokyo, Japan, 1964.
M. Ushio, E. Hayashi, Interactive musical editing system to support human errors and offer personal preferences for an automatic piano -Inferring performance expression by consideringchange of pitch-, The 71st Term, The Japan Society of MechanicalEngineers, Kyushu Branch, 2018, pp. 295–296.
M. Ezaki, E. Hayashi, The research about editing system of performance information for player piano, J. Robot. Netw. Artif. Life
7 (2020), 73–76.

AUTHORS INTRODUCTION
Mr. Ryo Kinoshita

Prof. Eiji Hayashi
He is a Master’s course student at theDepartment of Intelligent and Control
Systems Kyushu Institute of Technology,
Japan. He received his Bachelor’s degreefrom the Department of Mechanical
Information Science and Technology Kyushu
Institute of Technology, Japan, in 2020.

He is a Professor in the Department ofIntelligent and Control Systems at Kyushu
Institute of Technology. He received a PhD(Dr. Eng.) degree from Waseda Universityin 1996. His research interests include
Intelligent mechanics, Mechanical Systems,and Perceptual information processing. He
is a member of The Institute of Electricaland Electronics Engineers (IEEE) and The Japan Society of
Mechanical Engineers (JSME).