# Generating Fingerings for Piano Music with Model-Based Reinforcement Learning
## 1 Introduction  
Piano composers and arrangers add fingering to sheet music to indicate which finger should be used to play each note. However, most piano sheet music only provides detailed notation of the musical notes themselves, without specific fingering instructions. Human performers rely on their professional knowledge, personal physical conditions, the actual sequence of notes, and multiple rehearsals to determine the fingering. This process involves a significant amount of decision-making. A scientifically and logically arranged fingering not only reduces the difficulty of performance and relieves mental burden for the pianist but also enhances the musicality and fluidity of the performance. Generally, when novice performers receive a piece of music they are about to practice, their first task is to annotate their own fingering on the sheet music. In theory, any key on the piano can be played with any finger. However, the number of possible fingering paths increases exponentially with the length of the note sequence, and the majority of these fingerings are difficult to execute or even physically impossible. The fingering annotation task is mainly approached through two categories: rule-based and data-driven methods. Rule-based methods [1; 2; 3; 4; 5; 6; 7] aim to summarize the anatomical and kinematic principles of piano playing, model the piano keyboard and human hand as realistically as possible, and scientifically quantify the motion cost of fingering transitions. They make use of the state information in simulated performances to decide on fingerings. Data-driven methods [8; 9; 10; 11; 12; 13], on the other hand, utilize piano scores annotated with humanfingerings to extract implicit patterns of how humans arrange fingerings. These patterns are then used to predict fingerings for new music scores. In earlier works, due to the lack of human expert annotated dataset, rule-based methods were often used. However, in recent years, the availability of the PIG dataset [10] has shifted the research focus towards data-driven methods. Nevertheless, in recent research [12], we have found that the accuracy of the PIG dataset needs improvement, as it contains fingerings like \"crossed chords\" or \"thumbless cross\" that are not realistically playable by humans. This directly affects the effectiveness of data-driven methods, often resulting in the generation of fingerings that include such unplayable fingerings. In this research, we employed rule-based approaches. We model the generation of piano fingerings as a Markov decision process (MDP). The agent interacts with its environment and evaluates the fingerings using a reward function. We introduce the prioritized sweeping algorithms [14] to implement model-based reinforcement learning. In order to execute the algorithm without enumerating the state space and action space of the piano fingering problem, we replaced the traditional Q-table used in tabular reinforcement learning with a hash table based on key-value storage. Additionally, we propose a method for segmenting the music score to enable the parallel execution of the algorithm. The completed processing steps are shown in Figure 1.  
The main contributions are summarized as follows:  
* We constructed a more realistic environment model for piano performance, and invalid action masking is used to maximally constrain physically impossible fingerings. * We used model-based reinforcement learning to address the fingering annotation task, which has improved the sampling efficiency compared to the previous model-free approach. * We introduce a Q-table based on key-value storage to achieve tabular reinforcement learning without enumerating action and state spaces. * We propose a piano sheet music segmentation method to parallelize problem-solving. This paper is divided into seven sections. Section 1 provides a brief overview of the problem. Section 2 presents related work in the field. Section 3 describes the approach used to model the environment for the problem. Section 4 describes learning algorithm we used.