Computational Analysis of Audio Recordings ofPiano Performance for Automatic Evaluation ⋆Norihiro Kato1 , Eita Nakamura2 , Kyoko Mine3 , Orie Doeda1 , andMasanao Yamada1
1National Institute of Technology, Kushiro College, Japanym@kushiro-ct.ac.jp
2
Kyoto University, Japan
3
Osaka Ohtani University, Japan

Abstract. We developed a computational evaluation method for piano
performance with the goal of building a practice support system for beginners. We recorded students’ performances as audio data and applied
several recent methods for audio-to-MIDI transcription based on deepneural networks to extract the pitch, onset time, and offset time of musical notes. To determine the correctness of the performance, we aligned
the extracted MIDI data with the musical score using a hidden Markovmodel (HMM). We compared the audio-to-MIDI transcription methods
and optimized the weight on different types of performance errors to conform to teacher’s assessment. Our experiments showed a strong correlation between the rate of performance errors obtained from the alignment
and the evaluation by a teacher who listened to the performance. Theresults that indicate performance errors and tempo stability can be usedin a practice support system that provides feedback to learners.
Keywords: Computational performance evaluation · Hidden Markov
model · Audio-to-MIDI transcription.

1Introduction

Effective performance evaluation plays a crucial role in music education, as it enables learners to receive valuable feedback for improving their skills [7]. However,traditional methods often fall short in providing timely and personalized feedback, particularly in large classrooms or for remote learning situations. In recentyears, there has been a growing interest in applying technology to evaluate theperformance of musical instruments [2,7]. Research efforts have explored various
approaches, aiming to support learners in their musical journey. One of thesemethods includes visualization of performances [8], which has seen advancements
in incorporating augmented reality technology [1, 4].
Among the diverse range of musical instruments, the piano has many performers at varying skill levels, which results in differing performance evaluation⋆This work was supported by JSPS KAKENHI Grant Numbers 21K02846, 21K12187,
22H03661.

2Authors Suppressed Due to Excessive Length

criteria across studies — from basic elements such as pitch and rhythm in beginners, to musical expression in advanced players. In previous research [12],an overall evaluation was performed for each full piece performed by beginners
using acoustic data and support vector regression (SVR) models. Despite theseadvancements in performance evaluation techniques, challenges still remain inleveraging technology for effective formative assessment. Formative assessment,
an evaluation of learning achievement during the instructional process, allows
for adjustments in learning activities and teaching methods. As each individual
has different performance aspects that need to be improved, it can be difficult
for teachers to provide sufficient advice to every student in a class.
In response to these challenges, our study aims to enhance formative assessment in piano performance education for beginners by contrasting score datawith transcriptions of audio recordings in MIDI (Musical Instrument Digital Interface) format. Our focus is on individual elements of performance such as wrong
touch, re-playing, and tempo consistency for personalized evaluation. With these
considerations in mind, our central research question is: How closely can computationally analyzed performance errors align with teachers’ evaluations? This
investigation could pave the way for a feedback system that utilizes computationally analyzed errors for formative assessment.

2Related work

Active research on performance evaluation of piano has explored various methods
to provide feedback to learners within the music education context [5]. Fukuda
et al. [3] developed a system that detects mistakes in the user’s performanceand simplifies challenging sections of a musical score based on the user’s skilllevel. Wang et al. [11] proposed two audio-based piano performance evaluation
systems for beginners using deep neural networks (DNN). DNN-based piano performance evaluations use end-to-end processing, eliminating the need for sheetmusic. However, for formative evaluation, which requires assessing each note,musical note information needs to be transformed from the audio signal to compare with sheet music.
Extracting musical note information from audio performance data is a challenging problem in the area of music information retrieval, especially for polyphonic music such as piano music. Recently, accurate methods have been realized using machine learning techniques, including PovNet [10] based on a dual
DNN architecture for pitch estimation and onset and velocity (intensity) estimation, and the high-resolution MIDI transcription (HighReso) method [6]based on convolutional-recurrent neural network. While these MIDI transcription methods are usually evaluated on relatively “clean” piano recording data,
their performance for low-quality recordings common in practical situations ofmusic education has not been investigated well in the literature. In this study,
we use both two methods as a part of the automatic evaluation method andcompare the results.

Title Suppressed Due to Excessive Length33Method

The data we used in this study comes from the final exam of group piano lessons
conducted in the Early Childhood Education major at a certain university’s Faculty of Education. In the exam, the students perform one piece each from Beyer
Op. 101 and children’s songs, and one instructor evaluated their performances.
The criteria for this exam were (a) wrong touch and re-playing, (b) rhythm, and(c) dynamics on a scale from A to C (A+, A, A–, B+, B, B–, C).
The performances in the exam were recorded as audio data using an iPad,which was placed in a concert hall with a total area of 1,741 square meters and217 seats, positioned 3.23 meters in front of the grand piano and at a heightof 0.65 meters. To convert the audio data into MIDI, we employed two recent
methods for MIDI transcription, PovNet [10] and HighReso [6].
After the transcription, we prepared the MIDI data of the sheet music manually, and an HMM was employed to align the MIDI data from both the performance and the sheet music. Benefitting from the HMM’s ability to discern
correctly and incorrectly played notes [9], we calculated their proportions relative
to the total number of played notes. Since the wrong touches and re-playing ofevaluation criterion (a) corresponds to pitch errors and extra notes, respectively,
we define the evaluation score E as
E = wp Ep + (1 − wp )Ee ,(1)where Ep denotes the pitch error rate, Ee the extra note rate, and wp the weightfor pitch errors. We measured the correspondence between the teacher’s evaluation and the evaluation score E using the Spearman’s rank correlation coefficients
while varying the weight wp .
For the variation in rhythm, we calculated the relative speed of each notein the performance locally by dividing the aligned sheet music’s Inter-Onset
Interval (IOI), by the performance IOI. IOI is the time interval between theonsets of consecutive musical notes. We took the median of this local tempo list
as the reference tempo for the performance, and used the root mean square of therelative errors between the local tempo and the reference tempo as an indicatorof playing speed consistency. Extra notes that did not have corresponding sheetmusic notes were excluded from this calculation.

4Results and Discussion

A total of 29 students from three classes took the exam. The results of comparison between the transcription methods, PovNet and HighReso, are shown inFig. 1. The results for PovNet consistently showed a high correlation with teacher
evaluations, with the correlation coefficient reaching its maximum value of 0.86
when wp = 0.58. For HighReso, the maximum value was 0.77 when wp = 0.905.
In the following discussions, PovNet with wp = 0.58 is used.
The relationship between the evaluation score E and the teacher evaluations
is shown in Fig. 2. Overall, performances with a lower score have higher teacher

4Authors Suppressed Due to Excessive Length

Fig. 1. Comparison of audio-to-MIDI transcription methods

Fig. 2. Relationship between incorrect notes and teacher’s evaluation

evaluations, and vice versa. Below, we discuss some of the performances thatdeviate from this trend.
The performance at the right end of the B evaluation (Bayer Op.101, No.78)has a higher evaluation than some of C-graded performances, despite having
more performance errors indicated by the score E. To examine the cause, Fig. 3
displays the progression of the performance compared to the sheet music. Consecutive extra notes indicate re-plays, and blue plus signs scattered among black
dots are due to pitch detection errors by PovNet. Extra notes that cannot be
associated with a position in the sheet music are plotted at the zero point ofscore time. In the figure, a significant number of extra notes are detected because of re-playing at approximately 23 seconds of score time. It is likely thatthe teacher considered this as a single mistake, and despite the large number ofdetected errors, still awarded a B evaluation.
The rightmost data point in the C evaluation of Fig. 2, representing thechildren’s song ’Crickets’, is another intriguing case. Although the performanceerrors are comparable to the average of B-rated performances, it falls into theC rating category. Fig. 4 illustrates the progression of this performance, whichTitle Suppressed Due to Excessive Length5Fig. 3. Performance progression of Bayer Op.101, No.78

Fig. 4. Performance progression of ”Crickets”

Fig. 5. Tempo deviation of ”Crickets”

contains hardly any extra notes, but reveals gaps starting at approximately the7-second and 23-second in the performance time. To illustrate this brief pause
clearly, Fig. 5 shows the rhythmic variability calculated by the method describedin the previous section, for the same performance. The values below 1 indicatinga slower performance than the sheet music. From this figure, it is easy to discern
the points where the hands have momentarily stopped.

5Conclusion

In this research, we explored a novel approach to piano performance evaluation
by utilizing advanced machine learning methods, applying PovNet and HighReso

6Authors Suppressed Due to Excessive Length

for audio-to-MIDI transcription in real-world classroom settings. We found that
PovNet demonstrated superior performance and a significant correlation withteacher evaluations. We also observed deviations through the data analysis, andwere able to identify and visualize contributing factors to these deviations, suchas re-plays and brief pauses. Our findings demonstrate the potential of machinelearning in performance evaluation to provide personalized and timely feedback.
Future research directions include developing an interactive feedback system
for music practice, aiding learners in error identification and correction. Additionally, conducting a multi-year longitudinal study can assess the long-term
effects of technology-enhanced formative assessment on students’ musical development, achievement, motivation, self-efficacy, and engagement.

References
1. Deja, J.A.: Piano learning and improvisation through adaptive visualisation anddigital augmentation. Companion Proceedings of the 2022 Conference on Interactive Surfaces and Spaces pp. 41–45 (2022)
2. Dorfman, J.: Theory and Practice of Technology-based Music Instruction. Oxford
University Press (2022)
3. Fukuda, T., Ikemiya, Y., Itoyama, K., Yoshii, K.: A score-informed piano tutoring
system with mistake detection and score simplification” within the music education
contexts. Proceedings of the 12th Sound and Music Computing Conference (SMC)
1, 105–110 (2015)
4. Heyen, F., Ngo, Q.Q., Kurzhals, K., Sedlmair, M.: Data-driven visual reflection onmusic instrument practice. ACM CHI Conference on Human Factors in Computing
Systems (2022)
5. Kim, H., Ramoneda, P., Miron, M., Serra, X.: An overview of automatic piano
performance assessment within the music education contexts. Proceedings of theInternational Society for Music Information Retrieval 1, 465–474 (2017)
6. Kong, Q., Li, B., Song, X., Wan, Y., Wang, Y.: High-resolution piano transcription
with pedals by regressing onset and offset times. IEEE/ACM Transactions onAudio, Speech, and Language Processing 29, 3707–3717 (2021)
7. Lerch, A., Arthur, C., Pati, A., Gururani, S.: An interdisciplinary review of music
performance analysis. Transactions of the International Society for Music Information Retrieval 3(1), 221–245 (2021)
8. Lima, H.B., Santos, C.G.R.D., Meiguins, B.S.: A survey of music visualizationtechniques. ACM Computing Surveys (CSUR) 57(7), 1–29 (2022)
9. Nakamura, E., Yoshii, K., Katayose, H.: Performance error detection and postprocessing for fast and accurate symbolic music alignment. Proceedings of theInternational Society for Music Information Retrieval pp. 347–353 (2017)10. Shibata, K., Nakamura, E., Yoshi, K.: Non-local musical statistics as guides foraudio-to-score piano transcription. Information Sciences 566, 262–280 (2021)
11. Wang, W., Pan, J., Yi, H., Song, Z., Li, M.: Audio-based piano performance evaluation for beginners with convolutional neural network and attention mechanism.
IEEE/ACM Transactions on Audio, Speech, and Language Processing 29, 1119–1133 (2021)
12. Wu, C.W., Gururani, S., Pati, A., Vidwans, A.: Towards the objective assessment ofmusic performances. International Conference on Music Perception and Cognition(ICMPC) pp. 99–103 (2016)