At Your Fingertips: Automatic Piano Fingering Detection
Authors: Amit Moryossef
Yanai Elazar
Yoav Goldberg
Venue:  (2020)
Keywords: piano
fingering
dataset

Abstract: Automatic Piano Fingering is a hard task which computers can learn using data. As data collection is hard and expensive, we propose to automate this process by automatically extracting fingerings from public videos and MIDI files, using computer-vision techniques. Running this process on 90 videos results in the largest dataset for piano fingering with more than 150K notes. We show that when running a previously proposed model for automatic piano fingering on our dataset and then fine-tuning it on manually labeled piano fingering data, we achieve state-of-the-art results.
In addition to the fingering extraction method, we also introduce a novel method for transferring deep-learning computer-vision models to work on out-of-domain data, by fine-tuning it on out-of-domain augmentation proposed by a Generative Adversarial Network (GAN).

For demonstration, we anonymously release a visualization of the output of our process for a single video on https://youtu.be/Gfs1UWQhr5Q

Under review as a conference paper at ICLR 2020
AT YOUR FINGERTIPS:
AUTOMATIC PIANO FINGERING DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Automatic Piano Fingering a hard task which computers can learn using data. As data collection is hard and expensive, we propose to automate this process by automatically extracting fingerings from public videos and MIDI files, using computer-vision techniques. Running this process on 90 videos results in the largest dataset for PIANO-FINGERING with more than 150K notes. We show that when running previously proposed model for automatic PIANO-FINGERING on our dataset and then fine-tuning it on manually labeled piano fingering data, we achieve state-of-the-art results. In addition to the fingering extraction method, we also introduce novel method for transferring deep-learning computer-vision models work on out-of-domain data, by fine-tuning it on out-of-domain aug- mentation proposed by a Generative Adversarial Network (GAN).
For demonstration, we anonymously release visualization of the output of our process for a single video on httpps:/yyoutu.ee bew.:ettt.bbeeeeeee. 10
INTRODUCTION
Learning to play the piano is hard task taking years master. One of the challenging aspects when learning new piece is the fingering choice in which to play each note. While beginner booklets contain many fingering suggestions, advanced pieces often contain none or a select few. Automatic prediction of PIANEININGERING can be useful addition to new piano learners, to the learning process of new pieces. As manually labeling fingering for different sheet music is an exhausting and expensive task In practice previous work (Parneutt et al. 1997 Hart et al. 2000 Jacobs 2001
Kasimi et al.) 2007 Nakamura et al. (019)) used very few tagged pieces for evaluation, with minimal or training data.
In this paper, we propose an automatic, low-cost method for detecting PIANO-FINGERING from piano playing performances captured on videos which allows training modern data-hungry neural networks. We introduce a novel pipeline that adapts and combines several deep learning methods which lead to an automatic labeled PIANO-FINGERING dataset. Our method can serve two purposes: (1) an automatic "transcript" method that detects when these are available, and (2) serve as a dataset for training models and then generalize to new pieces.
from video and MIDI files,
Given a video and a MIDI file, our system produces a probability distribution over the fingers for each played. Running this process on large corpora of piano pieces played by different artists, yields a total of 90 automatically finger-tagged pieces (containing 155,107 notes in total) and results in the first public large scale PIANO-FINGERING over time, as more videos are uploaded to YouTube. We provide empirical evidence that APFD is valuable, both by evaluating model trained it over manually labeled videos, as well as its usefulness by fine-tuning the model on a manually created dataset, which achieves state-of-the-art results.
dataset, which we name APFD. This dataset will grow
The process of extracting PIANO-FINGEIS FINGERING from videos alone is a hard task as it needs to detect keyboard presses, which are often subtle even for the human eye. We, therefore, turn to MIDI files to obtain this information. The extraction steps are as follows: We begin by locating the keyboard and identify each key on the keyboard (.2).. Then, identify the playing hands on top of the keyboard
Nakamura al. 2019) privately reported labeling time of 3- 3- seconds per note.

---

Under review as a conference paper at ICLR 2020
(a) recreation of the piano, the hand pose estimation and the predicted fingering, which are colored in corre- sponding colors to the fingers.
(b) Overlay of what we perceive on the original frame.
Example for detection of the overall system. We manage to automatically detect what fingers pressed each keynote the piano.
Figure 1:
(3.3), and detect the fingers given the hands bounding boxes (3.4).. Next, we align between the MIDI file and its corresponding video (33.6) and finally assign for every pressed note, the finger which was most likely used to play it ($3.5). Albeit the expectation from steps like hand detection and pose estimation, which were extensively studied in the computer-vision literature, we find that in practice, state-of-the-art models do not excel in these tasks for our scenario. We therefore address these weaknesses by fine-tuning an object detection model [3.3] on new dataset we introduce and train CycleGAN (Zhu et al. 2017) to address the different lighting scenarios with the pose estimation model 3.4
2 BACKGROUND
HIANO-FINGERING was previously studied in multiple disciplines, such as music theory and com-
puter animation (Parncutt et al. 1997 Hart et al., 2000; Jacobs 2001 Kasimi et al. 2207 Zhu et al.
2013
Nakamura et al. 2019).
The fingering prediction task formalized as follows: Given a sequence of notes, associate each note with finger from the set {1, 3, 5, x {L, R). This is subject to constraints such as the positions of each hand, anatomical plausibility of transitioning between two fingers, the hands' size, etc. Each fingering sequence has cost, which is derived from transition cost between two fingers.
Early work modeled fingering prediction as a search problem, where the objective is to find the op- timal fingering sequence play all the notes with. naive approach finding the best sequence is to exhaustively evaluate all possible transitions between one note to another which is not compu- tationally feasible. By defining a transition matrix corresponding of transitioning from one note to another one can calculate a cost function, which defines the predicted sequence likelihood. Using a search algorithm on top of the transitions allows finding globally optimal solution. This solution is not practical as well, due to the exponential complexity, and therefore heuristies or pruning are employed to reduce the space complexity. The transition ma- trix can be manually defined by hcuristics personal estimation (Parncutt al. 1997] Hart et al. 2000), or instead, not relying on a pre-defined set rules, and use Hidden Markov Model (HMM) to learn the transitions (Yonebayashi al. 2007 Nakamura al. 2014). In practice, (Yonebayashi et al. 2007) leaves the parameter learning to future work, and instead they manually fine-tune the transition matrix.
the probability
"difficulty"
On top the transition matrix, practitioners suggested using dynamic programming algorithms solve the search (Hart al. 2000). Another option solve the huge search space is to
2

---

Under review as a conference paper at ICLR 2020
search algorithm such as Tabu search (Glover & Laguna 1998) variable neighborhood search MMldenović & Hansen 1997). to find a global plausible solution (Balliauw et al. 2015 2017). These works are either limited by the defined transition rules, by making different assumptions to facilitate the search space. Such assumptions come in the form of limiting the predictions to a single hand, limiting the evaluation pieces contain chords, rests or substantial lengths during which player can take their hand off the keyboard. Furthermore, all of these works have very small evaluation sets, which in practice makes hard to compare different approaches, and does not allow to use more advanced models such as neural networks.
In this work, we continue the transition of search-based methods that optimize a set of constraints with learning methods that try to imitate human behavior by the use of large datasets. In practice, these methods require lots of training data to achieve good performance, and fingering labeling is particularly expensive and hard to obtain. One way to automatically gather rich fingering data with full hand pose estimation is by using motion capture (MOCAP) gloves when playing the piano. Zhu et al. (2013) suggests rule-based and data-based hybrid method, initially estimating fingering decisions using Directed Acyclic Graph (DAG) based on rule-based comfort constraints which smoothed using data recorded from limited playing sessions with motion capture gloves. As MOCAP requires special equipment and may affect the comfort of the player, other work, (Takegawa al., 2006) tried to automatically detect piano fingering from video and MIDI files. The pianist's fingeraills were laid with colorful markers, which were detected by a computer vision program. As some occlusions can occur, they used some rules to correct the detected fingering. In practice, they implemented the system with camera capturing only 2 octaves (out of 8) and performed a very limited evaluation. The rules they used are simple (such as: restricting one finger per played note, two successive notes cannot be played with the same finger), but far from capturing real-world scenarios.
Previous methods for automatically collecting data (Takegawa et al., 2006; Zhu et al. 2013) were costly, as apart of the equipment needed during the piece playing, and the data-collectors had to pay the participating pianists. In our work, we rely solely on videos from remain minimal with the ability to scale up to new videos.
meaning costs
Recently, Nakamura al. (019)) released relatively large dataset of manually labeled PIANO- FINGERING by one to six annotators, consisting of 150 pieces, with partially annotated scores (324 notes per piece average) with total of 48,726 notes matched with 100,044 tags from multiple annotators. This is the largest annotated for this field. The authors propose multiple methods for modeling the task of PIANO-FINGERING including HMMs and neural networks, and report the best performance with an HMM-based model. In this work, use their dataset gold dataset for comparison and adapt their model to compare to our automatically generated dataset.
corpus to date and a valuable resource
3 OUR APPROACH: EXTRACTING FINGERING FROM ONLINE VIDEOS
There is genre of online videos in which people upload piano performances where both the piano and the hands are visible. On some channels, people not only include the video but also the MIDI file recorded while playing the piece. We propose to use machine learning techniques to extract fingering information from such videos, enabling the creation of large dataset of pieces and their fingering information. This requires the orchestration and adaptation of several techniques, which we describe below.
The final output we produce is demonstrated in Figure I where we colored both the fingers and the played notes based on the pose-estimation model ($3.4) and the predicted fingers that played them (B.5). Note that the ring fingers of both hands as well as the index finger of the left hand and the middle finger of the right hand do not press any note in this particular frame, but may play a note in others. We get the information of played notes from the MIDI events.
3.1 DATA SOURCE
We extract videos from youtube.. com played by different piano players on a specific channel containing both video and MIDI files. In these videos, the piano is filmed in horizontal angle
3

---

Under review as a conference paper at ICLR 2020
directly to the keyboard, from which both the keyboard and hands are displayed (as can be seen in Figure TT.
MIDI files A standard protocol for the interchange of musical information between musical instru- ments, synthesizers, and computers Musical Instrument Digital Interface (MIDI) a standard format for the interchange of musical information between electronic musical instruments. It consists of sequence of events describing actions to carry out, when, and allows for additional attributes. In the setup of piano recording, it records what note was played in what time for how long and its pressure strength (velocity). We only use videos that come along with a MIDI file, and use it as the source for the played notes and their timestamp.
3.2 KEYBOARD AND BOUNDARIES DETECTION
To allow a correct fingering assignment, we first have to find the keyboard and the bounding boxes of the keys. We detect the keyboard as the largest continuous bright area in the video and identify key boundaries using standard image processing techniques, taking into account the expected number of keys and their predictable location and clear boundaries. For robustness and in order to handle the interfering hands that periodically hide parts of the piano, we combine information from multiple random frames by averaging the predictions from each frame.
3.3 HAND DETECTION
A straightforward approach for getting fingers locations in an image is to use a pose estimation model directly on the entire image. In practice, common methods for full-body pose estimation such OpenPose (Cao et al. 2017) containing hand pose estimation (Simon et al., 2017), make assumptions about the wrist and elbow locations to automatically approximate the hands' locations. In the case of piano playing. the elbow does not appear in the videos, therefore these systems don't work. We instead, turn to a pipeline approach where we start by detecting the hands, cropping them, and passing the cropped frames to pose estimation model that expects the hand be in the middle of the frame
Object Detection (Viola & Jones 2001 Redmon et al. 2016 Lin et al. 2017a b). and specifically Hand Detection (Simon et al.) [207 Kölsch & Turk (0044) Sridhar et al. 2015) are well studied sub- jects. However, out of the published work providing source code, the code was either complicated to run (e.g. versioning, mobile-only builds, supporting specific GPU, etc.), containing private datasets, or only detecting hands with no distinction between left and right, which is important in our case.
We, therefore, created small dataset with random frames from different videos, corresponding to 476 hands total evenly split between left and righ(? We then fine-tuned a pre-trained object detection model (Inception v2 (Ioffe & Szegedy 2015), based on Faster R-CNN (Ren et al., 2015), trained on COCO challenge (Lin et al. 20144) on our new dataset. The fine-tuned model works reasonably well and some hand detection bounding boxes presented in Figure We release this new dataset and the trained model alongside the rest of the resources developed in this work.
Having working hand-detection model, we perform hand detection every frame in the video. If more than two hands are detected, we take the 2 highest probability defections as the correct bounding boxes. If two hands detected with the same label ("left-left" or "right-right"), discard the model's label, and instead choose the leftmost bounding box to have the label "left" and the other to have the label "right" which is the most common position of hands on the piano.
3.4 FINGER POSE ESTIMATION
Having the bounding box of each hand is not enough, as in order to assign fingers to notes we need the hand's pose. How can we detect fingers that pressed the different keys? We turn to pose estimation models, a well-studied subject in computer vision and use standard models (Wei et al.) 2016).
Using off-the-shelve pose estimation model, turned to often fail in our scenario. Some failure ex- ample are presented Figure 2c where the first pose is estimated correctly, but the rest either have
The data labeled by the first author using labellng.
4

---

Under review as a conference paper at ICLR 2020
Pre-trained model on CycleGAN checkpoints.
(b) Fine- sune-tuned model on CycleGAN checkpoints.
(c) Pre-trained model on real-world data.
(d) Fine-tuned model real-world data.
Figure 2:
Output of CycleGAN on 5 images, with 5 different checkpoints. The colors drawn on the hands depict the pose estimation of the hands.
wrong finger positions, shorter, or broken fingers. The videos we use contain visual effects (c.g. LED lights turned on every time key is pressed), and such the pose-estimation models exhibit two failures: (1) when the LED colors are warm (red, orange, yellow), the model sees the light as extension the finger, and as such poorly estimates where the keypoints are for each finger; (2) when the LED colors are cool (green, blue, white), the over-saturation seems to overwhelm the model, and it mistakenly estimates the hand's pose much shorter, considering the lit parts of the piano as part of the background. Some examples of these errors are presented in Figure 2c Further- more, the videos are usually very dark, high-contrast, and blurry due motion blur, which standard datasets TToppso et al. 2014 Yuan et al. 2017; Simon et al. 2017) and models trained on top of them rarely encounter.
Given the observation that pose-estimation works well on well lid-images, how can we augment the pose estimation model to other lighting situations? This scenario is similar to sim2real works (Sadeghi et al.] 2018 Gupta & Booher Gamrian & Goldberg 2018) where one wants to transfer a model from simulations to real-world. These works learn a mapping function G1 T S that transfer instances x, from the target domain T (the real-world) into the source domain S (the simulation), where the mapping is usually achieved by employing a CycleGan (Zhu et al.) 2017). Then, models which domain G1 (2;) and manage to generalize on the target domain. In our setup, we seek a mapping G2 S T' that transforms the source domain (i.e the well-lid videos) into the target data (i.e the challenging lighted scenarios). After obtaining the transformation function G2, we employ the pose estimation model on the source domain, the transformation separately, and align the prediction to the new representation. This novel setup benefits from performance boost as we only use the transformation function offline, before training and avoid using for every prediction. We also benefit of better generalization as we keep good performance on the source domain, and gain major performance on the target domain.
trained the source domain
used the transformation of the target
We manually detect videos and assign them into their group (well-lit poorly-lit). Then, we auto- matically detect and crop hands from random frames, resulting in 21,747 well-lit hands, and 12,832 poorly lit hands. We then trained CycleGAN for multiple epochs and chose 15 training check- points that produced different lighting scenarios (some examples can be seen in Figure 2). We then fine-tune pose-estingg eeteematiiitnnnnneee model on the original, well lit frames, and on the 15 transformed frames. This procedure results in model that robust to different lighting scenarios, as we show in Figures 2b ahaand 2d. demonstrating its performance different lighting scenarios.
5

---

Under review as a conference paper at ICLR 2020
3.5 PRESSED FINGER ESTIMATION
Given that we know which notes were pressed in any given frame (see (3.6 below), there is still uncertainty which finger pressed them. This uncertainty either comes from imperfect pose estimation, or multiple fingers located on top of a single note. We model the finger press estimation by calculating the probability of a specific finger to have been used, given the hand's 'ss pose and the pressed note:
argmaziy P m))
where i [] for the 5 fingers, h2 (4, hr stands for the hand being used (left or right) and n. E [1, 88) corresponding to the played key.
We chose estimate the pressed fingers as a Gaussian distribution N(p, o²), where μ and are defined as follows:
(mg) x X
u((n)) X. 0.5
J the center of the key on the x axis and a is its width. The score of each finger given note in specific frame is defined as:
g hy frame) =
The probability given finger to have played a note given a note and a frame: (normalizing g for all fingers)
off.. f
p(S, hii,hyle bylk frame)
>n= Eme((r)9(Jn In hm my. frame)
As most keyboard presses last more than one frame, we make use of multiple frames to overcome some the errors from previous steps and to estimate more accurate prediction. For this reason, aggregate the frames that were used during key press. We treat the first frame the main signal point, and assign each successive frame an exponentially declining weight
E 0.5' * p(.fs, hy frames))
p(fi, hylm
21 0.5
As finger changes can occur in later frames. Finally, we normalize the weighted sum of probabilities to achieve probability distribution for all frames.
In our dataset, we release all probabilities for each played note, along with the maximum likelihood finger estima- tion. We define the "confidence" score of the extraction from a single piece, as the product of the highest probabil- ity for finger for each note. FigureBsshowe the precision and recall of the predictions based on cutoff for the high- est probability between confidence threshold and precision, and nega- tive correlation between confidence and recall, meaning get relatively high precision for small number of notes, or relatively low precision for high number of notes.
the note. We see positive correlation
17
0.5
Figure 3: Precision and recall based on confidence for all 6,894 manually tagged notes
3.6 VIDEO AND MIDI ALIGNMENT
We consider the MIDI and video files to be complementary, they were recorded simultancously. The MIDI files are the source to which keys were pressed, what time, and for how long. The videos are the source for the piano, hands, and fingers locations. These two sources not syn- chronized, but as they depict the same piano performance, perfect alignment exist (up to the video frequency resolution).
We extract the audio track from the video, and treat the first audio peak as the beginning of the piece, clipping the prior part of the video and aligning with the first event from the MIDI file. In practice,

---

Under review as a conference paper at ICLR 2020
determine this starting point as the first point in the signal where the signal amplitude is higher than fifth of the mean absolute value of the entire signal.
This heuristic achieves reasonable alignment, but observe some alignment mismatch of 80- 200ms. We tackle the misalignment by using a signal from the final system confidence (Section 3.5 where every piece gets final score estimation after running the whole process, depicting the system confidence on the predicted notes. We look for an alignment that maximizes the system confidence over the entire piece:
alignment Video) arggmar,score(MMTD. Videot,)
where MI is the starting time of the MIDI file, and Video,, is the alignment time of the video. V Videor, is obtained by the heuristic alignment described in the previous paragraph. We use the confidence score a proxy of the alignment precision and search the alignment that maximizes the confidence score of the system. More specifically, given the initial offset from the audio-MIDI alignment, we take a window of second in frames (usually 25) for each side and compute the score of the final system on the entire piece. We choose the offset that results in the best confidence score as the alignment offset.
3.7 THE RESULTING DATASET: APFD
We follow the methods described in this section, and it to label piano pieces from 42 different composers with 155, 107 notes in total. On average, each piece contains 1,723 notes.
RESULTS
In this section, present multiple evaluations of our overall system. We begin by evaluating the entire process where we assess how the overall system performs on predicting the pressed fingering. Next, we use the dataset we collected and train PIANO-FINGERING model. We fine-tune this model on a previously manually-annotade dataset for PIANO-FINGERING and show using our data achieves better performance.
As piano pieces are usually played by two hands, we avoid modeling each hand separately, and instead use their symmetry property, and simply flip one hand's notes, matching it the piano scale, following previous work practice (Nakamura et al. 2019).
For evaluation, we use the match rate between the prediction and the ground truth. For cases where there is a single ground truth, this is equivalent to accuracy measurement. When more than one labeling is available, we simply average the accuracies with each labeling
4.1 FINGER PRESS ESTIMATION EVALUATION
As the pose estimation one of the major components directly affecting our system's performance, we isolate this part in order to estimate the gain from fine-tuning using the CycleGAN (Section
We manually annotated five random pieces from our dataset by marking the pressing finger for each played note in the video. Then, by using our system ($3.5). we estimate what finger was used for each key. We use the confidence score produced by the model as a threshold to use or discard the key prediction of the model, and report precision, recall and Fl scores of multiple thresholds. Moreover, we compare the scores of these picces with and without using the CycleGAN, We do so for the five annotated pieces and report these results in Table I When considering high confidence score (>90%) both the pre-trained and fine-tuned models correctly mark all considered notes (which consist of between 34-36% of the data). However, when considering decreasing confidences, the fine-tuned model manages to achieve higher precision and higher recall, contributing to an overall higher fl score. With no confidence threshold (i.e using all fingering predictions). the pre-trained model achieves 93% F1, while the fine-tuned one achieves 97% F1, a 57% error reduction.
4.2 AUTOMATIC PIANO FINGERING PREDICTION
"This matches the general match evaluation metric in (Nakamura al. 2019

---

Under review as a conference paper at ICLR 2020
Confidence
<>
0.9
0.75
0.5
Metrics
p
I
11
p
p
r
f1
P
r
Pretrained
1.0
0.34
0.51
0.99
0.49
0.65
0.95
0.78
0.86
0.88
1.0
0.93
FT Single Frame
0.99
0.36
0.53
0.99
0.52
0.68
0.97
0.8
0.88
0.9
1.0
0.95
FT Multiple Frames
1.0
0.35
0.52
0.99
0.5
0.67
0.98
0.8
0.88
0.94
1.0
0.97
Table Precision, Recall, Fl scores for 6,894 notes we manually tagged across different pieces, at different confidence points. The first row depict the performance of the system with the vanilla pose estimation. The second and the third rows present the results on the fine-tuned (FT) pose estimation model 3.4 on single and multiple frames accordingly.
In order to assess the value of APFD, we seek to show its usefulness on the end task: Automatic Piano Fingering. To this end, we train a standard sequence tagging neural model using our dataset, evaluating on the subset of the videos we manually annotated. Then, fine-tune this model on PIG (Nakamura et al., 2019), manually la- beled dataset, which we achieve superior results than simply training on that dataset alone.
Training/Dataset
APFD
PIG
Previous Neural
61.3
Previous SOTA
64.5
Human Agreement
71.4
PIG
49.9
64.1
+ Fine-tune APFD
73.6
54.4
APFD
73.2
55.2
Fine-tune PIG
62.9
66.8
We model the PIANO-FINGERING ing task, where given a sequence of notes n1, 122, Py need to predict sequence of fingering: 1/1 1/22- Vnn where E {1, 2, 3, 4, 5} corresponding to 5 fingers of one hand. We employ a standard sequence tagging technique, by embedding each note and using BiLSTM on top of it. On every contextual- ized note we then use culti-Layer-Percepptee
sequence label-
Table 2: Results of training and fine tun- ing on different dataset.
(MLP) to predict the label. The model is trained to
minimize cross-entropy loss. This is the same model used in (Nakamura al. 2019), referred as DNN (LSTM).
4.2.1 PIANO FINGERING MODEL
Nakamura et al. 2019) didn't use a development set, therefore in this work, we leave piece from the training set and make development set. Our we split into 75/10 for training and development sets respectively, and use the 5 manually annotated pieces as a test set. We note that the development set is silver data (automatically annotated) and probably contains mistakes. The results are summarized in Table [ We run the same architecture by (Nakamura et al., (209) with some different hyperparameters and achieve 71.4% 1% on our and PIG's s test set respectively. To evaluate the usefulness of our data to PIG's data, we use the model trained on our silver data and fine-tune on PIG. This results in 66.8% accuracy, 2.3% above the previous state-of-the-ar model which was achieved by an HMM (Nakamura et al. 2019). We attribute this gain in performance to our dataset, which both increases the number of training examples and allows to train bigger neural models which excel with more training examples. We also experiment in the opposite direction and fine-tune the model trained on PIG with our data, which result in 73.6% accuracy, which is better than training our data alone, achieving 73.2% accuracy.
composed of 90 pieces, which
DISCUSSION AND FUTURE WORK
In this work, we present an automatic method for detecting PIANO-NINGERING from MIDI and video files of a piano performance. We employ this method on large set of videos, and create the first large scale PIANO-FINGERING dataset, containing 90 unique pieces, with 155,107 notes in total. We show this dataset- although being noisy-is valuable, by training a neural network gold dataset, where achieve state-of-the-art results. In future work,
model on it, fine-tuning we intend to improve the data collection by improving the estimatiini model, better handling high speed movements and the proximity of the hands, which often cause errors in estimating their pose. Furthermore, we intend to design improved neural models that can take previous fingering predictions into account, in order to have better global fingering transition.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Matteo Balliauw, Dorien Herremans, Daniel Palhazi Cuervo, and Kenneth Sörensen. Generating fingerings for polyphonic piano music with a tabu search algorithm. In International Conference on Mathematics and Computation in Music, pp. 149-160. Springer, 2015.
Matteo Balliauw, Dorien Herremans, Daniel Palhazi Cuervo, and Kenneth Sörensen. A variable neighborhood search algorithm to generate piano fingerings for polyphonic sheet music. Interna- tional Transactions in Operational Research, 24(3):509-55 535, 2017.
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291-72999 2017.
Shani Gamrian and Yoav Goldberg. Transfer learning for related reinforcement learning tasks via image-to-ime image translation. arXiv preprint arXiv: 2018.
Fred Glover and Manuel Laguna. Tabu search. In Handbook of combinatorial optimization, pp- Springer, 1998.
Anchit Gupta and Jonathan Booher. Cyclegan for sim2real domain adaptation.
Melanie Hart, Robert Bosch, and Elbert Tsai. Finding optimal piano fingerings. The UMAP Journal, 21(2)1167-177, 2000.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03160 2015.
J Pieter Jacobs. Refinements the ergonomic model for keyboard fingering of parncutt, sloboda, clarke, rackallio, and desain. Music Perception: An Interdisciplinary Journal, 18(4):505-511 2001.
Alia Al Kasimi, Eric Nichols, and Christopher Raphael. A simple algorithm for automatic generation of polyphonic piano fingerings. In ISMIR, 2007.
Mathias Kölsch and Matthew Turk. Robust hand detection. In FGR, pp. 614-619, 2004.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017b.
Nenad Mladenović and Pierre Hansen. Variable neighborhood search. Computers & operations research, 24(11):1097-1100, 1997.
Eita Nakamura, Nobutaka Ono, and Shigeki Sagayama. Merged-output hmm for piano fingering of both hands. In ISMIR, pp. 531-536, 2014.
Eita Nakamura, Yasuyuki Saito, and Kazuyoshi Yoshii. Statistical learning and estimation of piano 2019.
fingering, arXiv preprint I904
Richard Parncutt, John A Sloboda, Eric F Clarke, Matti Raekallio, and Peter Desain. An ergonomic model of keyboard fingering for melodic fragments. Music Perception: An Interdisciplinary Journal, 14(4):341-382 1997.
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 779-788, 2016.

---

Under review as a conference paper at ICLR 2020
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91- 2015.
Fereshteh Sadeghi, Alexander Toshev, Eric Jang, and Sergey Levine. Sim2real viewpoint invariant visual servoing by recurrent control. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4691 16999, 2018.
Tomas Simon, Hanbyul Joo, lain Matthews, and Yaser Sheikh. Hand keypoint detection in single images using multiview bootstrapping. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 1145-1153, 2017.
Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, and Christian Theobalt. Fast and robust hand tracking using detection- guided optimization. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 3213-3221, 2015.
Yoshinari Takegawa, Tsutomu Terada, and Shojiro Nishio. Design and implementation of real-time fingering detection system for piano performance. In ICMC, 2006.
Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. Real-time continuous pose recovery of human hands using convolutional networks. ACM Transactions on Graphics, 33, August 2014.
Paul Viola and Michael Jones. Rapid object detection using a boosted cascade of simple features. CVPR (I), 1(511-518)33, 2001.
Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutional pose machines. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4724- 4732, 2016.
Yuichiro Yonebayashi, Hirokazu Kameoka, and Shigeki Sagayama. Automatic decision of piano fingering based on a hidden markov models. In IJCAI, pp. 2915-2921, 2007.
Shanxin Yuan, Qi Ye, Bjorn Stenger, Siddhant Jain, and Tae-Kyun Kim. Bighand2. 2m bench- mark: Hand pose dataset and state of the art analysis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4866-4874, 2017.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE Interna- tional Conference on, 2017.
Yuanfeng Zhu, Ajay Sundar Ramakrishnan, Bernd Hamann, and Michael Neff. A system for auto- matic animation of piano performances. Computer Animation and Virtual Worlds, 24(5):445-457, 2013.
Appendices
A DATASET SAMPLES
For every video our system extracts PIANO-FINGERING from it also outputs the video overlayed by the estimation of the piano keys, an indication which notes are played, and what fingers are being used to play them (the key's 's color), up two bounding boxes, for both hands (where light blue means left hand, and light green means right hand), and the pose estimation for each hand.
We include the output videos for all of the pieces manually annotated to visually demonstrate system and its accuracy on different playing cases. The videos were uploaded to a new, anonymous You channel, and each contain link to the original video in the description.
River Flows in You: foot:attt:/:/ourttte wttp://uwuuuueeeeeueeeeeaaeeaeee 1fwwhhhrrs00000h Faded:
be /12 bow6677bbbo007700
10

---

Under review as a conference paper at ICLR 2020
Moonlight Sonata 1st Movement: M Rondo Alla Turca: Nocturne in E Flat Major (Op. 9 No. 2): /ters////oattu.o////HUU2z2z2


Reviews:
[{"review_id": "ryeHZxXPYB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is a nice piece of works which clearly articulates the objective and the subsequent discussion. The focus of the paper--i.e. disclose the difficulties of piano fingering data annotation and the proposal of automating this process by automatically extracting fingerings from public videos and MIDI files, using  computer-vision DNN-based algorithms —although not really mainstream, it does provide some practical insights using a couple of experimental settings (piano fingering model and prediction) to help the readers. \n\nI really enjoyed reading this paper. I think that it can be considered a relevant and interesting piece of work, very well written and clear. Furthermore, providing new benchmarks/datasets/competitions for the AI community is always refreshing. Also, the results seem believable and solid, and potentially useful. \n\nMy only concern is that, although the rationale and utility of the paper is clear, the rest of the paper is somewhat incremental/engineering piece which depends somehow on previous works (see Nakamura,2019). I fail to see much novel scientific contribution to the area of research (apart from the dataset) and I’m not sure whether there are enough scientific technical advancements. Furthermore, the experimental setting is somewhat limited, and it is not clear whether results are statistically significant.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "N/A"}, "extra_text": {}}, {"review_id": "rkxDbgiWYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "<Strengths> \n+ This paper addresses an interesting and practically important problem: detection of piano fingering from videos and MIDI files. Fingering is a valuable source for piano learners and has to be manually annotated otherwise. The proposed approach is automatic and low-cost from playing videos. \n+ This paper collects a large-scale dataset for piano-fingering named APFD, including 90 finger-tagged pieces with 155K notes.\n+ The paper reads very well.\n\n\n<Weakness>\n1. One major weakness of this work is lack of technical novelty. \n- As described in section 3 in detail, the proposed approach consists of a sequence of well-known techniques (e.g. Faster R-CNN for hand detection and CycleGAN for finger pose estimation) and is largely based on lots of heuristics in every step of the procedure. \n- Thus, the method may be practically viable but bear little technical novelty.\n\n2. Experimental results are rather weak. \n- Only a single baseline is used in the existing PIG dataset, while no baseline method is compared for the new APFD dataset,  for which more baselines may need to be implemented and compared. \n- The proposed approach (64.1) is slightly worse than the previous SOTA (64.5) in the PIG dataset, although it is improved by fine-tuning with the APFD data that are not available for the previous SOTA. Thus, no experimental evidence is presented in the paper to convince that the proposed approach is better than existing ones.\n\n<Conclusion>\nAlthough this work is practically promising, my initial decision is ‘reject’ mainly due to lack of technical novelty and limited experiments. ", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "BJevc4E2tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed an automatic piano fingering algorithm, that accepts YouTube videos and corresponding MIDI files and outputs fingering prediction for each note. The claimed contribution is two-fold: First, they proposed the algorithm, and second, they claim that the algorithm can be used to automatically generate large datasets for piano fingering problems. The motivation is clearly stated and convincing. The overall algorithm is mainly described. \n\nHowever, I would like to reject this paper. Major issues:\n\n* Some key information is missing in Section 3.6, which is the only section that shows technical details: What is X_{n_k}? How is that related to the estimated finger poses? What is the function f in the definition of function g? (Also, it would be helpful to label the equations for clarification.) Are you doing Bayesian inference? With the key information missing, it is hard to fully understand the remaining technical details in this section. \n* Their experimental results cannot properly support their claims. In Section 4.2, the authors try to show the strength of their proposed piano fingering algorithm by comparing their automatically annotated dataset APFD with an existing manually annotated dataset PIG. The authors showed the evaluation results of models trained and fine-tuned with different datasets. However, this is not an acceptable comparison for me, due to several reasons.\nFirst, in order to show the strength of automatic piano fingering prediction, it is much better to directly run the prediction algorithm on datasets with known labels. According to the related work section, there is at least one existing work by Takegawa et al. that uses videos and MIDI files to detect piano fingering. Can you compare your algorithm with theirs? \nSecond, it is essentially unreliable to compare two datasets by comparing the performance of two prediction models, as there are too many implementation details that are almost impossible to control. \nThird, it is not clear how we should compare the testing errors in Table 2. Yes, a model initially trained on PIG and fine-tuned on APFD may perform better than a model trained merely on APFD, but does that suggest anything (and the advantage is just 0.4%)? Similarly, the experimental result that an MLP model initially trained on APFD then fine-tuned with PIG works better than an HMM model that is trained with PIG data alone cannot prove anything. There are too many possible reasons that may lead to this experimental result. \n* How is this method more attractable than the existing ones? There are neither experimental comparisons nor high-level justifications of why the existing algorithms are not applicable to the given scenario. In Section 2, although the authors described a good number of existing work on piano-fingering and their drawbacks, they failed to point out the strength of their paper as a comparison. As a result, the strength of this paper is still unclear after reading this section. How does this paper avoid the drawbacks of these previous papers? \n* The writing of this paper needs to be greatly improved. It takes a lot of effort to literally understand this paper: There may be missing parts, misplaced clauses, and broken logic between sentences. I have listed several examples in the minor issues part. \n\n\nMinor issues:\n* In the first paragraph of Section 1: The sentence before 'In practice ...' is incomplete. \n* In the last paragraph of Section 1: Missing brackets for \\textsection 3.3 and \\textsection 3.4. Also, 'on A new dataset we introduce' should be 'on THE new dataset we introduce'. \n* On page 3, the sentence 'In this work, we continue the transition of search-based methods that optimize a set of constraints with learning methods that ...' is not making sense to me. Do you mean that your work is an extension of search-based methods, or do you mean that your work is not a search-based method? Also, are you optimizing a set of constraints, or optimizing with a set of constraints? \n* On page 3, the last sentence in Section 2: '... and adapt their model to compare TO our ...' should be '... and adapt their model to compare WITH our ...'. The last part of this sentence is also a bit confusing: How do you compare a model with a dataset?\n* On page 4, the paragraph starting with 'MIDI files': The first two sentences are almost the same; the period between them is missing. I guess one of them should be deleted. The following sentences in this paragraph are also subject to grammatical errors. For example, the sentence 'It consists of a sequence of events ... to carry out, WHEN, and allows for ...' is not a complete sentence. 'We only use videos that come along with a MIDI file' -> 'We only use videos that come along with MIDI files'. \n* On page 5, last paragraph in Section 3.3: 'highest probability defections' -> 'highest probability detections'.\n* The last paragraph on Page 5: 'Using off-the-shelve ...' -> 'Use off-the-shelf ...'.\n* In Section 4.2.1, the corresponding result is Table 2, instead of Table 1. ", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows an automatic piano fingering algorithm. The idea is good. But the reviewers find that the novelty is limited and it is an incremental work. All the reivewers agree to reject.
Decision: Reject