At Your Fingertips: Automatic Piano Fingering Detection

1 INTRODUCTION . Learning to play the piano is a hard task taking years to master . One of the challenging aspects when learning a new piece is the fingering choice in which to play each note . While beginner booklets contain many fingering suggestions , advanced pieces often contain none or a select few . Automatic prediction of PIANO-FINGERING can be a useful addition to new piano learners , to ease the learning process of new pieces . As manually labeling fingering for different sheet music is an exhausting and expensive task1 , In practice previous work ( Parncutt et al. , 1997 ; Hart et al. , 2000 ; Jacobs , 2001 ; Kasimi et al. , 2007 ; Nakamura et al. , 2019 ) used very few tagged pieces for evaluation , with minimal or no training data . In this paper , we propose an automatic , low-cost method for detecting PIANO-FINGERING from piano playing performances captured on videos which allows training modern - data-hungry - neural networks . We introduce a novel pipeline that adapts and combines several deep learning methods which lead to an automatic labeled PIANO-FINGERING dataset . Our method can serve two purposes : ( 1 ) an automatic “ transcript ” method that detects PIANO-FINGERING from video and MIDI files , when these are available , and ( 2 ) serve as a dataset for training models and then generalize to new pieces . Given a video and a MIDI file , our system produces a probability distribution over the fingers for each played . Running this process on large corpora of piano pieces played by different artists , yields a total of 90 automatically finger-tagged pieces ( containing 155,107 notes in total ) and results in the first public large scale PIANO-FINGERING dataset , which we name APFD . This dataset will grow over time , as more videos are uploaded to YouTube . We provide empirical evidence that APFD is valuable , both by evaluating a model trained on it over manually labeled videos , as well as its usefulness by fine-tuning the model on a manually created dataset , which achieves state-of-the-art results . The process of extracting PIANO-FINGERING from videos alone is a hard task as it needs to detect keyboard presses , which are often subtle even for the human eye . We , therefore , turn to MIDI files to obtain this information . The extraction steps are as follows : We begin by locating the keyboard and identify each key on the keyboard ( §3.2 ) . Then , we identify the playing hands on top of the keyboard 1Nakamura et al . ( 2019 ) privately reported labeling time of 3-12 seconds per note . ( §3.3 ) , and detect the fingers given the hands bounding boxes ( §3.4 ) . Next , we align between the MIDI file and its corresponding video ( §3.6 ) and finally assign for every pressed note , the finger which was most likely used to play it ( §3.5 ) . Albeit the expectation from steps like hand detection and pose estimation , which were extensively studied in the computer-vision literature , we find that in practice , state-of-the-art models do not excel in these tasks for our scenario . We therefore address these weaknesses by fine-tuning an object detection model §3.3 on a new dataset we introduce and train a CycleGAN ( Zhu et al. , 2017 ) to address the different lighting scenarios with the pose estimation model §3.4 . 2 BACKGROUND . PIANO-FINGERING was previously studied in multiple disciplines , such as music theory and computer animation ( Parncutt et al. , 1997 ; Hart et al. , 2000 ; Jacobs , 2001 ; Kasimi et al. , 2007 ; Zhu et al. , 2013 ; Nakamura et al. , 2019 ) . The fingering prediction task is formalized as follows : Given a sequence of notes , associate each note with a finger from the set { 1 , 2 , 3 , 4 , 5 } × { L , R } . This is subject to constraints such as the positions of each hand , anatomical plausibility of transitioning between two fingers , the hands ’ size , etc . Each fingering sequence has a cost , which is derived from a transition cost between two fingers . Early work modeled fingering prediction as a search problem , where the objective is to find the optimal fingering sequence to play all the notes with . A naive approach to finding the best sequence is to exhaustively evaluate all possible transitions between one note to another which is not computationally feasible . By defining a transition matrix corresponding to the probability or “ difficulty ” of transitioning from one note to another - one can calculate a cost function , which defines the predicted sequence likelihood . Using a search algorithm on top of the transitions allows finding a globally optimal solution . This solution is not practical as well , due to the exponential complexity , and therefore heuristics or pruning are employed to reduce the space complexity . The transition matrix can be manually defined by heuristics or personal estimation ( Parncutt et al. , 1997 ; Hart et al. , 2000 ) , or instead , not relying on a pre-defined set of rules , and use a Hidden Markov Model ( HMM ) to learn the transitions ( Yonebayashi et al. , 2007 ; Nakamura et al. , 2014 ) . In practice , ( Yonebayashi et al. , 2007 ) leaves the parameter learning to future work , and instead they manually fine-tune the transition matrix . On top of the transition matrix , practitioners suggested using dynamic programming algorithms to solve the search ( Hart et al. , 2000 ) . Another option to solve the huge search space is to use a search algorithm such as Tabu search ( Glover & Laguna , 1998 ) or variable neighborhood search ( Mladenović & Hansen , 1997 ) , to find a global plausible solution ( Balliauw et al. , 2015 ; 2017 ) . These works are either limited by the defined transition rules , or by making different assumptions to facilitate the search space . Such assumptions come in the form of limiting the predictions to a single hand , limiting the evaluation pieces to contain no chords , rests or substantial lengths during which player can take their hand off the keyboard . Furthermore , all of these works have very small evaluation sets , which in practice makes it hard to compare different approaches , and does not allow to use more advanced models such as neural networks . In this work , we continue the transition of search-based methods that optimize a set of constraints with learning methods that try to imitate human behavior by the use of large datasets . In practice , these methods require lots of training data to achieve good performance , and fingering labeling is particularly expensive and hard to obtain . One way to automatically gather rich fingering data with full hand pose estimation is by using motion capture ( MOCAP ) gloves when playing the piano . Zhu et al . ( 2013 ) suggests a rule-based and data-based hybrid method , initially estimating fingering decisions using a Directed Acyclic Graph ( DAG ) based on rule-based comfort constraints which are smoothed using data recorded from limited playing sessions with motion capture gloves . As MOCAP requires special equipment and may affect the comfort of the player , other work , ( Takegawa et al. , 2006 ) tried to automatically detect piano fingering from video and MIDI files . The pianist ’ s fingernails were laid with colorful markers , which were detected by a computer vision program . As some occlusions can occur , they used some rules to correct the detected fingering . In practice , they implemented the system with a camera capturing only 2 octaves ( out of 8 ) and performed a very limited evaluation . The rules they used are simple ( such as : restricting one finger per played note , two successive notes can not be played with the same finger ) , but far from capturing real-world scenarios . Previous methods for automatically collecting data ( Takegawa et al. , 2006 ; Zhu et al. , 2013 ) were costly , as apart of the equipment needed during the piece playing , and the data-collectors had to pay the participating pianists . In our work , we rely solely on videos from YouTube , meaning costs remain minimal with the ability to scale up to new videos . Recently , Nakamura et al . ( 2019 ) released a relatively large dataset of manually labeled PIANOFINGERING by one to six annotators , consisting of 150 pieces , with partially annotated scores ( 324 notes per piece on average ) with a total of 48,726 notes matched with 100,044 tags from multiple annotators . This is the largest annotated PIANO-FINGERING corpus to date and a valuable resource for this field . The authors propose multiple methods for modeling the task of PIANO-FINGERING , including HMMs and neural networks , and report the best performance with an HMM-based model . In this work , we use their dataset as a gold dataset for comparison and adapt their model to compare to our automatically generated dataset . 3 OUR APPROACH : EXTRACTING FINGERING FROM ONLINE VIDEOS . There is a genre of online videos in which people upload piano performances where both the piano and the hands are visible . On some channels , people not only include the video but also the MIDI file recorded while playing the piece . We propose to use machine learning techniques to extract fingering information from such videos , enabling the creation of a large dataset of pieces and their fingering information . This requires the orchestration and adaptation of several techniques , which we describe below . The final output we produce is demonstrated in Figure 1 , where we colored both the fingers and the played notes based on the pose-estimation model ( §3.4 ) and the predicted fingers that played them ( §3.5 ) . Note that the ring fingers of both hands as well as the index finger of the left hand and the middle finger of the right hand do not press any note in this particular frame , but may play a note in others . We get the information of played notes from the MIDI events . 3.1 DATA SOURCE . We extract videos from youtube.com , played by different piano players on a specific channel containing both video and MIDI files . In these videos , the piano is filmed in a horizontal angle directly to the keyboard , from which both the keyboard and hands are displayed ( as can be seen in Figure 1 ) . MIDI files A standard protocol for the interchange of musical information between musical instruments , synthesizers , and computers Musical Instrument Digital Interface ( MIDI ) is a standard format for the interchange of musical information between electronic musical instruments . It consists of a sequence of events describing actions to carry out , when , and allows for additional attributes . In the setup of piano recording , it records what note was played in what time for how long and its pressure strength ( velocity ) . We only use videos that come along with a MIDI file , and use it as the source for the played notes and their timestamp . 3.2 KEYBOARD AND BOUNDARIES DETECTION . To allow a correct fingering assignment , we first have to find the keyboard and the bounding boxes of the keys . We detect the keyboard as the largest continuous bright area in the video and identify key boundaries using standard image processing techniques , taking into account the expected number of keys and their predictable location and clear boundaries . For robustness and in order to handle the interfering hands that periodically hide parts of the piano , we combine information from multiple random frames by averaging the predictions from each frame .

Summary: The paper is a nice piece of works which clearly articulates the objective and the subsequent discussion. The focus of the paper--i.e. disclose the difficulties of piano fingering data annotation and the proposal of automating this process by automatically extracting fingerings from public videos and MIDI files, using  computer-vision DNN-based algorithms —although not really mainstream, it does provide some practical insights using a couple of experimental settings (piano fingering model and prediction) to help the readers.