SUGISAKAMASANORI

Journal of Robotics, Networking and Artificial Life
Vol. 10(3); December (2023), pp. 282–285
ISSN (Online): 2352-6386; ISSN (Print): 2405-9021
https://alife-robotics.org/jrnal.html

Research Article

A Research on a System Using Deep Learning for Inferring Piano
Performance
Yoshiki Hori, Eiji Hayashi

Kyushu Institute of Technology, 680-4 Kawazu, Iizuka-shi, Fukuoka, 820-8502, Japan

ARTICLE INFO
Article History
Received 30 November 2023
Accepted 05 April 2024
Keywords
Automatic piano
Computer music
Deep learning

ABSTRACT

Achieving expressive and human-like automated piano performances has proven challenging.
This study proposes a deep learning system to infer expressive nuances from musical scores,
addressing the limitations of traditional rule-based approaches. By leveraging neural networks tolearn the mapping between scores and expert performances, the system automates the inference
process, improving accuracy while enhancing efficiency. This novel application of deep learningshows promise for advancing automated music performance and enabling more artistically
expressive renditions. The insights gained could have broader implications for computer-aided
musical interpretation and synthesis.

© 2022 The Author. Published by Sugisaka Masanori at ALife Robotics Corporation Ltd.
This is an open access article distributed under the CC BY-NC 4.0 license(http://creativecommons.org/licenses/by-nc/4.0/).

1. Introduction

The advent of deep learning techniques has opened up
new avenues for modeling complex patterns in data. Byleveraging the power of neural networks, this research
investigates the potential of deep learning algorithms tocapture the subtle variations in timing, dynamics, andarticulation that characterize expressive pianoperformances.
Unlike conventional rule-based or parametricapproaches, deep learning offers a data-driven approachto learn the intricate relationships between musical scoresand expressive performances directly from data. This
novel application of deep learning to piano performanceinference promises to bridge the gap between mechanicalprecision and artistic expression.

Automated piano performance has been a longstanding
challenge in the field of computer music. While previoussystems (Fig. 1) have achieved precise control overkeystroke mechanics, replicating the nuanced andexpressive qualities of human performances remains anelusive goal. This study aims to develop a system capableof inferring the intricate musical nuances from score data,
enabling natural and human-like piano renditions.

The proposed system not only advances the state of theart in automated piano performance but also holds
broader implications for computer-aided music
interpretation and synthesis. By enabling more naturaland human-like renditions, this research paves the wayfor enhancing the artistic capabilities of automated musicsystems.
Fig. 1 Automatic piano playing device [1]Corresponding author E-mail: hori.yoshiki311@mail.kyutech.jp, haya@mse.kyutech.ac.jp282

Conventional systems could replicate expert
performances by interpreting pitch and dynamics
information encoded in MIDI data. However, a humanlike expressive rendition could not be achieved fromscore data alone. To address this limitation, efforts havebeen made to develop systems that can generate
expressive performances directly from musical scores.
As an initial step, attempts were made to replicate theperformances of renowned pianists.

we established four parameters to characterize theperformance information. The details of these
performance parameters are presented in Table 1.
Table 1 The performance information format

However, previous approaches required manual
inference of the expressive nuances for each individual
note, a process that was exceedingly time-consuming andlabor-intensive, even for short musical pieces containing
thousands of notes. Therefore, in this study, weconstructed a system that automates the inference process
by leveraging deep learning techniques.

Parameter
VeloGate
Step

Units
None
msmsTimems

About
Sound intensity
Length of note
Interval between the nextnoteTime of soundIn this study, Time is not an object of deduction as it
can be derived from the Gate and Step values.
2.4. Score information
Score information consists of performance data
featuring musical notations, including notes and symbols.
In this study, we defined five parameters based on thescore information. These parameters are detailed in Table
2.

This novel approach, employing deep learning forautomated inference, aims to overcome the limitations ofconventional methods that relied on manual annotation.
By enabling neural networks to directly learn themapping between musical scores and expressive nuances,efficient and scalable inference can be achieved without
the need for human intervention.

Table 2 The score information format

2. Inference system with Deep Learning
2.1. Dataset
In our study, we analyzed performance metrics from therenowned pianist Vladimir Davidovich Ashkenazy,
captured using the MIDI protocol.

ParameterKeyBar
Dyn
Tgate

Units
None
None
None
ms

Tstep

msAbout
Sound heightBar number
Dynamics markLength of note on thescore
Interval between the next
note in the score

2.5. Neural network configuration

2.2. Performance data

Existing inference systems have predominantly reliedon traditional methods, with deep learning architectures
remaining largely unexplored. Nevertheless, whencreating a predictive model with deep learning,
identifying the most effective network architecture forinterpreting performance information is crucial.
Consequently, we opted to develop and test a system

Given the inherent complexity of raw performance data
for straightforward use in machine learning, this study
segmented the data into distinct categories: performancedetails and score particulars. By establishing specificparameters, we were able to refine and structure the data,
rendering it more accessible for machine learningalgorithms. This methodology not only clarifies andsystematizes the data but also optimizes it for algorithmic
learning, thereby improving the applicability andefficiency of machine learning techniques in analyzing
piano performance interpretation.
In Sections 2.3 and 2.4, the research details theparameters set for both performance information andscore information.
2.3. Performance information
In this research, the performance informationencompasses the expressive qualities demonstrated by
the pianist in the collected data. For the purpose ofanalysis, we established four specific parameters todescribe this performance information. In this research,

Fig. 2 System overview
283

featuring a straightforward network design. Fig. 2illustrates the schematic of the network we constructed.

3. Inference Experiment and Results
3.1. Data splitting
In this study, the performance data was segmented intosets for training, validation, and testing. K-fold crossvalidation, a method used to assess the model’s ability togeneralize, involves splitting the data into K segments.
Each segment serves as validation data in turn, while theremaining K-1 segments are used for training. This
process ensures each data segment is used for validation
once. Fig. 3 illustrates the concept of K-fold crossvalidation. For the testing phase, “Prelude Op.28-7” byFryderyk Franciszek Chopin was selected as the test data.

Fig. 4 Pattern 2 VeloFig. 3 Overview of K-fold Cross-Validation
3,2. Validated network structure
In this experiment, the number of neurons in the firstand second intermediate layers was adjusted based on thepatterns outlined in Table 3.

Fig. 5 Pattern 3 Gate

Table 3 Combination of neurons in intermediate layers 1and 2Patternnumber1234Layer 1Layer 250501001005010010050

3.3. Results
The inference system generated graphs for each of Velo,Gate, and Step, comparing the deduced performance
information with that of the pianist’s actual performance.
Fig. 4, Fig. 5 and Fig. 6 display the graphs for the patterns
exhibiting the strongest correlation coefficients. Table 4
details the correlation coefficients between theperformance information predicted by the system and theactual performance data from the pianist.

Fig. 6 Pattern 3 Step284

more accurate and robust inference. Future work will
investigate the incorporation of additional neurons, layers,and gating mechanisms that take into account previous
inputs. By iteratively refining the network architecture
and leveraging more advanced deep learning techniques,
we aim to develop a comprehensive system capable ofgenerating truly human-like and artistically expressive
piano performances from score data.

Table 4 Performance information’s correlation
coefficient
Patternnumber12
3
4

Layer 1

Layer 2

VeloGate

Step50
50
100
100

50
100
100
50

0.763
0.776
0.770
0.763

0.720
0.741
0.743
0.691

0.876
0.880
0.885
0.832

This line of research not only advances the state of theart in automated piano performance but also holds
broader implications for computer-aided music
interpretation and synthesis, potentially enabling a wide
range of applications that bridge the gap between
mechanical precision and artistic expression.
References

4. Consideration
Table 4 indicates that the correlation coefficient for
Velo remains largely unchanged across all patterns.
Despite Velo’s high correlation coefficient, the graphwaveforms display dissimilarity, as observed in Fig. 4.
This discrepancy arises from the model interpreting
points with significant value shifts as anomalies, leading
to reduced variability in predicted values.

1.

In contrast, the correlation coefficient for Stepsurpasses those of Velo and Gate, attributable to the Step
waveform’s more consistent variations, facilitating easierlearning, as depicted in Fig. 6.

E. Hayashi, M. Yamane, H. Mori, Development of amoving coil actuator for an automatic piano, Int. J. Japan
Soc. Prec. Eng. 28 (1994), 164–169.

Authors Introduction
Mr. Yoshiki Hori

Moreover, an examination of the correlation
coefficients for patterns 2 and 3 reveals high values for
Velo, Step, and Gate alike. This implies that a network
configuration incorporating a greater number of neurons
might be more effective in capturing the nuances of theseparameters. This points towards the potential benefits ofemploying a more intricate network structure to improve
the fidelity of performance information inference,
ultimately leading to more nuanced and precise modelingof musical expression.

Dr. Eiji Hayashi

He received bachelor degree inEngineering in 2023 from mechanicalsystem engineering, Kyushu Instituteof Technology in Japan. He iscurrently a Master student at Kyushu
Institute of Technology and conducts
research at Hayashi Laboratory.

He is a professor in the Department

of Intelligent and Control Systems atKyushu Institute of Technology. He
received the Ph.D. (Dr. Eng.) degreefrom Waseda University in 1996. His
research interests include Intelligent
mechanics, Mechanical systems andPerceptual information processing.
He is a member of The Institute ofElectrical and Electronics Engineers (IEEE) and The Japan
Society of Mechanical Engineers (JSME).

5. Conclusion
This study explored the application of deep learningtechniques for inferring expressive performance
information from musical scores. We constructed andevaluated an inference system with a simple neuralnetwork architecture, focusing on identifying the optimal
configuration for accurately capturing the nuances in
piano performances.
The experimental results demonstrated that a network
structure with a larger number of neurons in the layers
closer to the output layer exhibited superior performancein inferring the intricate expressive details. This finding
suggests that increasing the representational capacity ofthe deeper layers can effectively model the complex
relationships between score data and expressiverenditions.
While the proposed system represents a promising step
forward, further enhancements are necessary to achieve285