# Generating Fingerings for Piano Music with Model-Based Reinforcement Learning
## 2 Related Work  
Piano fingering automatic annotation can be broadly categorized into two methods: rule-based and data-driven.  
The earliest rule-based method for piano fingering annotation was proposed by [PERSON] and others [1]. [PERSON] [2] introduced a method that utilizes dynamic programming to compute optimal fingerings for single-note melodies. [PERSON] [3; 4] applied two different search algorithms to tackle this problem. [PERSON] [5] and [PERSON] [6] explored model-free reinforcement learning algorithms to find fingerings with minimal hand movement; however, since it is based on a very simple environment model, it can only handle single-note melodies. [PERSON] [7] conducted research on the application of reinforcement learning to the piano performance of humanoid robotic hands with only four fingers. Applying reinforcement learning to other combinatorial optimization problems has also been the subject of extensive research [15; 16; 17]. [PERSON] et al. [18] conducted research on invalid action masking in reinforcement learning.  
[PERSON] and [PERSON] [8; 9] proposed hidden Markov models (HMMs) for piano fingering, respectively, for single-handed and double-handed outputs, laying the foundation for subsequent data-driven approaches using statistical learning to predict fingerings. In their subsequent work [10], [PERSON] formalized the task as a statistical learning problem and achieved good fingering consistency by modeling fingerings with a third-order HMM and a long short-term memory (LSTM) network, which is similar to the part-of-speech tagging task in natural language processing. They also introduced the chord hidden Markov model (CHMM) for chord fingering modeling and published the PIG dataset. [PERSON] et al. [11] proposed the pitch difference model, which efficiently utilized the PIG dataset by using relative pitch instead of absolute pitch. [PERSON] et al. [12] proposed checklist models, which introduced soft constraints and used the REINFORCE algorithm to optimize evaluation metrics, improving the overall fluency of fingerings, but the experimental results show that the soft constraint rules used in this work do not fundamentally eliminate unplayable fingerings such as crossed chord, and the optimization objectives of the reinforcement learning part also need to be improved. [PERSON] et al. [13] treated the fingering annotation as an information retrieval problem and used the Czerny corpus as the dataset.  
The quality of data determines the upper limit of data-driven approaches. Existing data-driven methods often struggle to directly eliminate physically impossible fingering predictions. Additionally, during data preprocessing, they often create unrealistic fingering, for example, due to limited data, data-driven methods adopt a technique of flipping left-hand finger labels and pitch to create a second \"right-hand portion\" in order to augment the training data. However, although the human left and right hands are mirror images of each other, the arrangement of black and white keys on the keyboard is not a mirror-symmetric structure. Simply flipping the left-hand pitch and finger labels cannot accurately represent a right-hand performance. Rule-based methods, on the other hand, have difficulties describing the performance process with fixed paradigms, and their modeling accuracy needs improvement, as many studies are limited to fingering for single-note melodies.  
As a continuation of rule-based method research, we aim to formalize the piano performance process through MDP and propose a quantifiable method for calculating motion to enhance the accuracy of the environmental model. Our research aims to achieve a certain level of matching with human fingerings while maximizing the limitation of physically impossible fingerings and minimizing the motion, even seeking fingerings superior to human annotations. Our action constraints and modeling methods can also be applied to future research combining data-driven methods.