JOINT ESTIMATION OF NOTE VALUES AND VOICESFOR AUDIO-TO-SCORE PIANO TRANSCRIPTION
Yuki Hiramatsu1
Eita Nakamura1,2
Kazuyoshi Yoshii1,3
1Graduate School of Informatics, Kyoto University, Japan
2
The Hakubi Center for Advanced Research, Kyoto University, Japan
3
PRESTO, Japan Science and Technology Agency (JST), Japan{hiramatsu, enakamura, yoshii}@sap.ist.i.kyoto-u.ac.jp

Piano recording

ABSTRACTThis paper describes an essential improvement of a stateof-the-art automatic piano transcription (APT) system that
can transcribe a human-readable symbolic musical scorefrom a piano recording. Whereas estimation of the pitches
and onset times of musical notes has been improved drastically thanks to the recent advances of deep learning, estimation of note values and voice labels, which is a crucial component of the APT system, still remains a challenging task. A previous study has revealed that (i) the
pitches and onset times of notes are useful but the performed note durations are less informative for estimating
the note values and that (ii) the note values and voices havemutual dependency. We thus propose a bidirectional longshort-term memory network that jointly estimates note values and voice labels from note pitches and onset times
estimated in advance. To improve the robustness against
tempo errors, extra notes, and missing notes included in
the input data, we investigate data augmentation. The experimental results show the efficacy of multi-task learningand data augmentation, and the proposed method achieved
better accuracies than existing methods.
1. INTRODUCTION
The ultimate goal of automatic piano transcription (APT)
is to convert a piano recording into a human-readable musical score that can be used for music analysis and performance [1]. This is a challenging task because of the polyphonic nature of piano music; musical notes form weaklysynchronous multiple streams called voices running in parallel. Much work on APT aims to estimate not a musicalscore but a piano roll from a music signal, i.e., estimate thequantized pitches and non-quantized onset times of musical notes [2–7]. Although noticeable research progress has
independently been made for multipitch detection [8–10]
and rhythm transcription [11, 12], estimation of note values and voice labels, which is crucial for score typesetting© Y. Hiramatsu, E. Nakamura, and K. Yoshii. Licensed under a Creative Commons Attribution 4.0 International License (CC BY
4.0). Attribution: Y. Hiramatsu, E. Nakamura, and K. Yoshii, “Joint
Estimation of Note Values and Voices for Audio-to-Score Piano Transcription”, in Proc. of the 22nd Int. Society for Music Information Retrieval Conf., Online, 2021.

278

Multipitch detection & onset time quantization [20]PitchesQuantized onset times
Meter (bar lines)
Note value estimation & voice separation
PitchesQuantized onset times
Meter (bar lines)
Note values
Hand parts and voices
Score typesetting (MuseScore 3)      


















 









Figure 1. Overview of proposed method that jointly estimates note values and voice labels from transcribed pitches
and onset times.
with high readability [13], still remains a challenging task.
Note that the note value represents the duration of a noteon a symbolic musical score, and the voice label of a note
specifies one of the voices of the upper or lower staff (rightor left-hand part) the note belongs to (Fig. 1).
Several attempts have been made for estimating voicelabels for piano scores having no voice labels [14–17].
Under an assumption that each voice has a strictly monophonic structure and note values are already transcribedwith a certain degree of accuracy, voice labels can be estimated accurately [18,19]. In practice, however, each voice
has a homophonic structure consisting of concurrent notes(chords) and accurate estimation of note values is still an
open problem. To deal with such a realistic situation, astate-of-the-art APT method uses a rule-based cost function with limited performance [20]. This calls for a principled statistical approach based on modern deep learning.
Note value estimation has relatively scarcely been studied [4, 21] and is still considered a challenging task [20].
Nakamura et al. [21] conducted a detailed statistical analysis and found that (i) the pitches and onset times of notes
are useful but the performed note durations are less informative for estimating the note values and that (ii) the noteProceedings of the 22nd ISMIR Conference, Online, November 7-12, 2021

values and voices have mutual dependency. The second
conclusion derives from the fact that in a voice stream, theoffset time of a note usually matches the onset time of thenext note, or equivalently, short rests are rarely inserted
between notes [4]. This indicates that joint estimation ofnote values and voice labels is an effective way of bringing
improvements on both tasks.
In this paper, we propose a deep neural network (DNN)that estimates note values and voice labels jointly from atranscribed sequence of pitches and onset times for audioto-score APT. Specifically, we train a bidirectional longshort-term memory (BiLSTM) network in multi-task learning. We also investigate data representation, network architecture, post-processing, and data augmentation, which
are considered to have an impact on the estimation performance. We report experimental evaluation conducted ondatasets of classical and popular music, to investigate theefficacy of the joint estimation framework.
Our main contribution is to propose joint estimation ofnote values and voice labels based on deep learning. Combined with the state-of-the-art methods for multipitch detection and rhythm transcription used in the latest pianotranscription system [20], we can achieve the state-of-theart performance of audio-to-score APT. Another contribution is to propose new evaluation metrics for the polyphonic music transcription task, extending the one proposed in [22] to deal with voice labels. The example transcription results and the source code for the evaluation tool
are available on the accompanying webpage 1 .
2. RELATED WORKThis section reviews methods for audio-to-score piano transcription, note value estimation, and voice separation.
2.1 Audio-to-Score Piano Transcription
Some piano transcription methods that can yield symbolic
piano scores have been proposed, and the methods consisting of multi-stage processing [13, 20] achieved high accuracies. Cogliati et al. [13] proposed a transcription methodthat performs rhythm quantization and voice estimation fora piano performance MIDI file and generates a piano score.
This method uses metrical, stream, and harmonic structures from the MIDI sequence estimated by a probabilistic model by Temperley [4]. Shibata et al. [20] proposeda state-of-the-art transcription method that can generate a
piano score from audio signals with multi-stage processing. The method first estimates from a piano recording
a performance MIDI sequence consisting of pitches, onset and offset times, and velocities using a convolutional
neural network (CNN). The onset times are then quantizedusing a hidden Markov model (HMM). After note values
and voice labels are separately estimated, piano scores are
generated using MuseScore 3. In this study, we jointly estimate note values and voice labels, and aim to improve theaccuracies that were lower than those of pitches and onset
times in the method.
1 https://nvvest.github.io

There are also end-to-end approaches that directly estimate musical scores from audio signals. Carvalho et al.
[23] proposed a seq2seq model that estimates from an audio signal a piano score represented in the Lilypond music
notation language. Román et al. [24] used a convolutional
recurrent neural network (CRNN) that estimates a musical score represented in the **kern format. The network is
trained with a connectionist temporal classification (CTC)
loss function. These end-to-end methods have been tested
only on very short or synthetic recordings, and there has
been no account in the literature describing how well they
perform in practice.
2.2 Note Value Estimation
Note value estimation is a difficult problem because notevalues do not always correspond to the performed durations [21]. Temperley [4] proposed a rhythm quantization
method based on a probabilistic model. The method quantizes onset times by estimating beat positions. After voicelabels are estimated, an offset time is set to the onset timeof the next note in the same voice. One of the problemsof the method is outputting no rests that are essential to
make scores easy to read. Nakamura et al. [21] proposeda method based on Markov random fields. The method
consists of a context model that represents a distributionof note values given pitches and onset times, and a performance model that generates actual performance durations
from note values. It was shown that the performance model
had a small impact on the estimation performance [21].
Therefore, we estimate note values only from pitches andonset times and do not use performed durations.
2.3 Voice Separation
Voice separation aims to divide musical notes into groups
of notes representing musical streams. Karydis et al. [15]proposed a rule-based voice separation method for symbolic piano scores. The method is based on vertical integration, which integrates notes with the same onset time
and the same duration, and horizontal integration, which
integrates notes close in time and pitch. While this method
can deal with homophonic voices, most other methods canonly estimate monophonic voices. McLeod et al. [18] proposed a voice separation method for MIDI data using an
HMM, and achieved high accuracy. Valk et al. [19] proposed a DNN-based voice separation method. The method
uses a deep feedforward neural network that classifies eachnote represented by 33 handcrafted features into five classes.
In piano transcription, these existing voice separation methods are not appropriate because voices often contain chords
and durations are not estimated precisely. Explicit handpart and voice labeling (rather than clustering) are also necessary for typesetting piano scores; for example, voice labels are used for determining the directions of note stems.
Shibata et al. [20] proposed a cost-function-based voice
separation method. Although this method is applicable tothe situation of piano transcription, there is room for improvement in accuracy. We attempt to develop an improved
DNN-based method.
279

voice1     ④③ (74, 0, 0), (36, 1)
(67, 0,0), (6, 2) (62, 0, 6), (6, 2)

Proceedings of the 22nd ISMIR Conference, Online, November 7-12, 2021

    ②
 ①
0

⑥

⑤

0, 0), (6, 5)  (57,(50, 0, 0), (6, 5)φn (X) = {φn (v; X)}8v=1 are separately calculated at eachtime step n after passing through fully connected layers
and softmax layers, where πn (d; X) denotes the probability that the n-th note has duration d and φn (v; X) denotes
the probability that the n-th note is in voice v.
We train the network by minimizing a cross-entropy
loss function given by  Note value
Voice label(45, 6, 6), (6, 5)

Tatum

6

Metrical onset position
Onset interval
Pitch

Figure 2. Data representation of the input and the outputof the BiLSTM network.

L = Ld + Lv ,where
Ld = −

3. PROPOSED METHOD

(2)

N
X

log πn (d∗n ; X),(3)

log φn (vn∗ ; X),

(4)

n=1

In this section, we propose BiLSTM networks that jointly
estimate note values and voice labels, post-processing methods that correct the estimated note values, and data augmentation methods.
3.1 Problem Specification
Each note zn = (pn , on , dn , vn ) in a musical score is represented by a pitch pn , an onset time on , a note value dn ,and a voice label vn . The pitch pn ∈ {0, . . . , 127} is represented by a MIDI note number (60 = C4). The onset timeon ≥ 0 and the note value dn ∈ {0, . . . , 479} are represented by integers (one measure is divided into 48 units); azero note value is used for a grace note. Note that different
meters have different tatum units: for example, a quarter
note is represented as 12 in 4/4 time and 16 in 3/4 time.
The maximum number of voices in each hand part is set
to 4, following the convention for score notation used inscore editing software such as MuseScore3 and Finale; labels vn = 1, 2, 3, 4 are used for the right-hand part andvn = 5, 6, 7, 8 for the left-hand part. We represent a piano score as a sequence of notes Z = {zn }N
n=1 , where
notes are arranged in the increasing order of onset times,
and notes with the same onset time are ordered according
to the pitches. Our goal is to estimate the note values andvoice labels {(dn , vn )}N
n=1 from a given set of pitches andonset times {(pn , on )}N
n=1 .
3.2 BiLSTM Network
We propose a BiLSTM network that estimates note values
and voice labels from pitches and onset times. We first
represent the onset time on as the interval from the previous onset in ∈ {0, . . . , 767} and the metrical positionbn ∈ {0, . . . , 47} calculated as follows:in = on − on−1 ,bn = on mod 48.

(1)

The input is then represented as X = {(pn , in , bn )}N
n=1
and the output is Y = {(dn , vn )}N
n=1 (Fig. 2).
The proposed network architecture is shown in Fig. 3(a).
Each musical note of input X is represented as a (128 ×
768 × 48)-dimensional one-hot vector. These one-hot vectors are first transformed to 25-dimensional feature vectors
by a fully connected layer. The resulting vectors are thentransformed to 50-dimensional vectors (latent representations) through a BiLSTM layer. Note value probabilities
π n (X) = {πn (d; X)}479
d=0 and voice label probabilities
280

Lv = −

NX
n=1

where d∗n and vn∗ are the correct note value and voice label,
respectively. In the inference step, note values and voicelabels are estimated from given pitches and onset times X
as follows:dˆn = arg max πn (d; X),(5)dv̂n = arg max φn (v; X),(6)v

where dˆn and v̂n indicate the estimated note value andvoice label, respectively.
3.3 Alternative Network Architectures
The network architecture in Fig. 3(a) is a simple joint network that equally treats the note value and voice label probabilities. We call this network SIM (simultaneous). We examine other network architectures shown in Fig. 3. As discussed in the Introduction, the voice structure has a strong
impact on determining note values. To reflect this dependency structure, we propose the second network architecture (VLF; voice label first). In this network (Fig. 3(b)),
voice labels are estimated first and note values are estimated with the latent representations used to estimate voicelabels. For comparison, we also consider the third networkarchitecture (NVF; note value first) that has a reverse structure (Fig. 3(c)). The networks SIM, VLF, and NVF aretrained in a multi-task learning framework by minimizing
the loss function L in Eq. (2). To confirm the efficacy ofmulti-task learning, we examine the fourth network architecture (IND; independent) that estimates note values andvoice labels independently (Fig. 3(d)). IND consists of two
BiLSTM networks and they are trained separately: by minimizing the loss functions Ld and Lv , respectively. In all
the network architectures, the first fully connected layer
outputs 25-dimensional vectors, and each BiLSTM layer
outputs a 50-dimensional hidden vector at each time step.
3.4 Post-Processing Methods
The note values and voice labels {(dˆn , v̂n )}N
n=1 estimatedby the network are sometimes inconsistent with the musical convention. As general rules, notes with the same onset
time and the same voice should have the same note values.
Also, the offset times of those notes should not be larger

Proceedings of the 22nd ISMIR Conference, Online, November 7-12, 2021
Pitches and onset times
𝑿

Pitches and onset times
𝑿

Pitches and onset times
𝑿

One-hotFC

One-hotFC

One-hotFC

One-hotFC

One-hot
FC

BiLSTM

BiLSTM

BiLSTM

BiLSTM

BiLSTM

ConcatConcat
BiLSTM
FC

FC

FC

FC

FC

Softmax

Softmax

Softmax

Softmax

Softmax

FC

FC

BiLSTM
FC

Softmax

Softmax

Softmax

Note value probabilities Voice label probabilities
𝝅! (𝑿)
𝝓! (𝑿)

Pitches and onset times Pitches and onset times
𝑿
𝑿

Note value probabilities Voice label probabilities Note value probabilities Voice label probabilities Note value probabilities Voice label probabilities
𝝅! (𝑿)
𝝓! (𝑿)
𝝅! (𝑿)
𝝓! (𝑿)
𝝅! (𝑿)
𝝓! (𝑿)(a) SIM (simultaneous)(b) VLF (voice label first)(c) NVF (note value first)(d) IND (independent)

Figure 3. Proposed BiLSTM networks for estimating note values and voice labels. The four architectures are explained in Section 3.3.
than the next onset time in that voice. For two notes n andm in the same voice, these constraints are represented asfollows:
on = om =⇒ dn = dm ,(7)

on < om =⇒ dn ≤ om − on .

(8)

To impose the constraints, we consider three possible
post-processing methods to adjust the estimated note valKues {dˆn }Nn=1 . Let {nk }k=1 index a set of notes with thesame onset time in the same voice. In the first method(PP1), the note values {dˆnk }K
k=1 are all set to the interval
to the next onset time as in [13]. Note that in this methodnote values are determined by the estimated voice labels
and the note value probabilities are not used. In the second
method (PP2), the note values are modified to their maximum value as follows:dˆ0 = max dˆn .
(9)nk

l=1,...,Kl

If the adjusted note values dˆ0nk are longer than the interval
to the next onset time, they are set to this interval. In thethird method (PP3), we calculate the note value with the
maximum probability from the candidate note values thatsatisfy the constraints as follows:dˆ0nk = arg max
d: d≤d0KYπnl (d; X),

(10)

l=1

where d0 indicates the interval to the next onset time.
3.5 Data Augmentation
In the situation under consideration, the pitches and onset times used as the input X are estimated in advance bysome pitch and rhythm transcription methods. As the result, the input contains tempo errors, extra notes, and missing notes. To make the networks robust to these errors, we
can apply data augmentation methods that add tempo errors, extra notes, and missing notes to the original training data D. Since rhythm transcription methods often produce half-tempo and double-tempo errors [20], we createa tempo-transformed dataset Dt by halving or doubling
the correct onset times and note values. Extra notes produced by multipitch detection methods often have a pitch
shifted by an octave from a correct note. We thus create
a dataset containing extra notes and missing notes Dem byrandomly deleting correct notes and adding notes whose

pitches differ from correct pitches by one octave. In addition, to increase the amount of the training data, we implement another data augmentation method. Assuming thattransposed piano scores are also musically valid, we train
the network using data obtained by transposing the original
data by an interval of δ semitones (δ = −12, −11, . . . , 12).

4. EVALUATION
We report experiments to evaluate the transcription accuracy of the proposed method.

4.1 Experimental Conditions
To evaluate the method in a practical condition, we incorporated it in an audio-to-score transcription system andgenerated transcriptions for test piano recordings. We firstestimated pitches and quantized onset times by the stateof-the-art methods for multipitch detection and rhythm transcription used in the transcription system in [20]. For the
results (called quantized MIDI data) we estimated note values and voice labels with the proposed method. We finally used public score editing software MuseScore 3 forscore typesetting and generated transcriptions in the MusicXML format (Fig. 1). For comparison, we also generated transcriptions by existing methods [13, 20] using
the same quantized MIDI data and with the same procedure for score typesetting. The CTD16 method [13] uses
the Melisma Analyzer [4] for estimating note values andvoice labels. The SNY21 method [20] is currently the bestperforming system and uses a statistical model for notevalue estimation and a dynamic-programming method forvoice separation.
As test data, we used 30 recordings of classical pianomusic in the MAPS-ENSTDkCl dataset [25] and 81 piano cover recordings of popular music used in [20]. The
ground-truth musical scores for these recordings were prepared in the MusicXML format and used for assessing thegenerated transcriptions. We used the musical scores of
80 classical music pieces and 763 popular music pieces
for training the BiLSTM networks; the same training data
were used in [20]. We applied the data augmentation in
Section 3.5 to these training samples.
281





Proceedings of the 22nd ISMIR Conference, Online, November 7-12, 2021

  



2
1





 
   

5
3

4

1
6





4



5








8

3

7

2

6

MethodSIM+DAVLF+DA
NVF+DAIND+DA
VLF

Eoff
33.3
32.2
32.9
32.9
33.1

Ev
39.1
39.0
40.7
40.5
39.1

Pv
63.9
65.2
63.1
64.1
64.3

Rv
64.9
65.7
62.6
63.8
64.3

Fv
64.0
65.1
62.5
63.6
64.0

Table 1. Error rates (%) and accuracies (%) of estimatednote values and voice labels for the MAPS dataset. DA
indicates that each network is trained with augmented data.

Figure 4. Voice structures represented by graphs.
4.2 Evaluation Metrics
We used the edit-distance-based error rates [22] for evaluating the quality of generated transcriptions. The error
rates are the pitch error rate Ep , the missing note rate Em ,the extra note rate Ee , the onset-time error rate Eon , and theoffset-time error rate Eoff . They are calculated after aligning an estimated score with a correct score. In particular,
we can use the offset-time error rate Eoff as a metric forevaluating the accuracy of estimated note values. Since
these metrics do not evaluate the accuracy of estimatedvoice labels, we consider the voice error rate Ev definedas the proportion of notes with incorrect voice labels. The
overall error rate Eall is defined as the mean of these six
error rates.
The edit-distance-based metrics have a clear interpretation: they count how many notes or score elements shouldbe edited to obtain the correct score. This is an advantage
over other metrics for music transcription [26, 27]. We callthe above defined metrics MUSTER (MUsic Score Transcription Error Rates) and the evaluation tool is made available online 2 .
We also used an F-measure [17] for assessing the quality of estimated voice labels; this metric is conventionallyused in studies on voice separation. The original metric [17] is formulated for monophonic voices and we here
extend it for homophonic voices. A voice structure can be
represented by a graph, where notes of consecutive chords
in a voice are connected by an edge (Fig. 4). The graphcan be represented by an adjacency matrix (aij ), where
aij = 1 when the i-th note is in a chord and the j-th note isin the next chord of the same voice, and otherwise aij = 0.
We use the notation (aij ) for a correct score and (âij ) foran estimated score. The precision Pv , the recall Rv , andthe F-measure Fv are defined as follows:PP
i<j aij âij /ŵii<j aij âij /wi
Pv = P, Rv = P, (11)
i<j âij /ŵii<j aij /wi
Fv =2Pv Rv.
Pv + Rv(12)

P
Here, i<j signifies a summation over all notes i and allnotes j that appear after i, and we have defined the weight
for each note i asXX
wi =aij , ŵi =
âij
(13)
j>i

j>i

in order to normalize the contribution of each chord nomatter how many notes it contains.
2 https://amtevaluation.github.io/

282

MethodSIM+DAVLF+DA
NVF+DAIND+DA
VLF

Eoff
17.9
17.2
18.1
18.7
17.5

Ev
12.2
11.4
12.4
12.5
11.4

Pv
87.1
87.7
87.4
86.8
87.5

Rv
87.3
87.7
87.2
86.4
87.8

Fv
87.1
87.6
87.2
86.5
87.6

Table 2. Error rates (%) and accuracies (%) of estimatednote values and voice labels for the J-pop dataset.
MethodVLF+DAVLF+DA+PP1
VLF+DA+PP2
VLF+DA+PP3

MAPS
32.2
28.0
31.4
32.2

J-pop
17.2
15.3
16.3
16.8

Table 3. Error rates Eoff (%) of estimated note values withdifferent post-processing methods.
4.3 Experimental Results
We first compare the four network architectures (SIM, VLF,
NVF, and IND) with or without the application of data
augmentation (DA). The evaluation results are listed in Tables 1 and 2 for the MAPS dataset and the J-pop dataset, respectively. Among the four architectures trained with data
augmentation, VLF achieved the best accuracy in both notevalues and voice labels. The higher accuracy of VLF compared to NVF indicates that it is better to estimate voicelabels first. A comparison between VLF and IND confirms the efficacy of multi-task learning. By comparing
the results for VLF with and without data augmentation,
we found a positive effect of data augmentation. Similar
results were obtained for the other network architectures.
We next compare the three post-processing methods (Table 3). The first method (PP1) achieved the lowest error
rates. The second one (PP2) slightly reduced the offset
error rates for both datasets. Before and after the third
method (PP3), the error rates were almost the same. In
the first post-processing method, note values are calculated
from estimated voice labels and note value probabilities
estimated by the network are not used. Importantly, thisdoes not mean that note value estimation was useless inthe present method: estimating note values by the network
was effective for improving the voice estimation throughthe multi-task learning, which in turn led to more accurate
note value estimations.
The first method also has a limitation that it cannot estimate rests. Rests are used to express articulations andto make scores easier to read. An example of the tran-Proceedings of the 22nd ISMIR Conference, Online, November 7-12, 2021

MethodProposed (VLF+DA+PP1)
SNY21 [20]CTD16 [13]Proposed (VLF+DA+PP1)
SNY21 [20]CTD16 [13]

Test
MAPS
MAPS
MAPS
J-pop
J-pop
J-pop

Ep
0.67
0.67
0.88
0.61
0.61
0.82

Em
8.11
8.11
13.5
4.03
4.03
12.8

Ee
6.23
6.23
6.33
7.29
7.29
7.21

Eon
11.6
11.5
16.8
2.67
2.69
8.48

Eoff
28.0
28.3
44.0
15.3
20.9
55.7

Ev
39.1
44.6
74.3
11.4
18.0
65.8

Eall
15.6
16.6
26.0
6.89
8.92
25.1

Pv
65.2
62.4
56.0
87.6
78.6
51.3

Rv
65.7
59.4
42.5
87.7
77.0
38.8

Fv
65.1
60.6
47.9
87.6
77.7
44.0

Table 4. Error rates (%) and accuracies (%) of transcription. The CTD16 method could output results for 27 (72)pieces in the MAPS (J-pop)
dataset; the metrics are calculated from these pieces.
ismir2021_09_gt
ismir_2021_16_gt
 
 

Ground truth                  ismir_2021_16_2













Proposed (VLF+DA+PP1)
          


 









 


        

Proposed (VLF+DA+PP2)


 

 


SNY21[20]

 





          

 










                  















                             ismir2021_09_tr







 
       







                       
 


       






 
  

SNY21[20]                               

 




 
  





Proposed (VLF+DA+PP1)

     











      









Figure 5. Example transcription results. The proposed
method improved the accuracy of note values.
scription results is shown in Fig. 5, where the proposed
method with the second post-processing method correctly
estimated rests. In the future, it is important to estimate
rests in order to improve the average accuracy of note values.
We finally compare the proposed method with existingtranscription methods [13, 20]. The full set of MUSTER
metrics and the voice F measure for the MAPS and J-pop
datasets are listed in Table 4. The present method achieved
the best accuracy for both datasets. The transcription accuracies for the MAPS dataset were lower than those for
the J-pop dataset because the former has more complicatedvoice structures and there were a small number of classicalmusic pieces in the training data.
To compare the performance of the voice estimation by
our method with a recent method focusing on voice separation, we also evaluated the HMM-based voice separation
method (MS16) [18]. Since this method requires as input
pitches, onset times, and offset times, we used the notevalues estimated by the network IND. The F-measures Fvof the voice separation results by MS16 were 55.7% and66.5% for the MAPS dataset and the J-pop dataset, respectively. It is confirmed that the proposed method significantly outperformed the MS16 method.
An example of the transcription results is shown in Fig.
6, for the proposed method and the SNY21 method. The
proposed method estimated voice labels close to the groundtruth, and made the piano score easier to read than the
one estimated by the SNY21 method. Other examples areshown on the supplemental web page 3 .
3 https://nvvest.github.io

Ground truth        







           





 


           

 




         

 



Figure 6. Example transcription results. The proposed
method improved the accuracy of voice labels and improved the readability.
5. CONCLUSION
This paper presented a neural method that jointly estimatesnote values and voice labels from transcribed pitches andonset times. Since note values and voices are interrelated,
we constructed a BiLSTM network in a multi-task learning framework. We demonstrated through experiments thatthe proposed method achieved the state-of-the-art performance of audio-to-score APT when combined with the latest methods for multipitch detection and onset time quantization.
The error rates of note values and voice labels are stillhigh compared to the other metrics. In future work, weplan to further investigate the data representation and network architecture to increase the consistency between estimated note values and voice labels. To correctly estimate
rests and improve the readability of transcribed scores, wewill develop a more sophisticated post-processing method
and study the effective use of performed durations.
Although we focused on the estimation of note values
and voices in this study, we found that the result is affectedby errors made by the onset time quantization method. It
is thus important to develop a method that integrates onset
time quantization. As the fully end-to-end approaches still
have difficulties in practical applications [23, 24], it is also
considered effective to unify the multiple stages in Fig. 1
one step after another.
6. ACKNOWLEDGEMENT
This work is supported in part by JST PRESTO No. JPMJPR20CB, JSPS KAKENHI Nos. 19H04137, 19K20340,
20K21813, and 21K12187, and ISHIZUE 2021 of Kyoto
University Research Development Program.
283

Proceedings of the 22nd ISMIR Conference, Online, November 7-12, 2021

7. REFERENCES[1] E. Benetos, S. Dixon, Z. Duan, and S. Ewert, “Automatic music transcription: An overview,” IEEE Signal
Processing Magazine, vol. 36, no. 1, pp. 20–30, 2018.
[2] P. Desain and H. Honing, “The quantization of musical time: A connectionist approach,” Computer Music
Journal, vol. 13, no. 3, pp. 56–66, 1989.
[3] C. Raphael, “A hybrid graphical model for rhythmic
parsing,” Artificial Intelligence, vol. 137, pp. 217–238,
2002.
[4] D. Temperley, “A unified probabilistic model for polyphonic music analysis,” Journal of New Music Research, vol. 38, no. 1, pp. 3–18, 2009.
[5] E. Vincent, N. Bertin, and R. Badeau, “Adaptive harmonic spectral decomposition for multiple pitch estimation,” IEEE TASLP, vol. 18, no. 3, pp. 528–537,
2010.
[6] E. Benetos and T. Weyde, “An efficient temporallyconstrained probabilistic model for multipleinstrument music transcription,” in ISMIR, 2015,
pp. 701–707.
[7] S. Sigtia, E. Benetos, and S. Dixon, “An end-to-end
neural network for polyphonic piano music transcription,” IEEE/ACM TASLP, vol. 24, no. 5, pp. 927–939,
2016.
[8] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon,
C. Raffel, J. Engel, S. Oore, and D. Eck, “Onsets andframes: Dual-objective piano transcription,” in ISMIR,
2018, pp. 50–57.
[9] Y.-T. Wu, B. Chen, and L. Su, “Polyphonic musictranscription with semantic segmentation,” in ICASSP,
2019, pp. 166–170.
[10] Q. Kong, B. Li, J. Chen, and Y. Wang, “GiantMIDIPiano: A large-scale MIDI dataset for classical pianomusic,” arXiv preprint arXiv:2010.07061, 2020.
[11] E. Nakamura, K. Yoshii, and S. Sagayama, “Rhythm
transcription of polyphonic piano music based onmerged-output HMM for multiple voices,” IEEE/ACM
TASLP, vol. 25, no. 4, pp. 794–806, 2017.
[12] E. Nakamura and K. Yoshii, “Music transcription
based on Bayesian piece-specific score models capturing repetitions,” arXiv preprint arXiv:1908.06969,
2019.
[13] A. Cogliati, D. Temperley, and Z. Duan, “Transcribing human piano performances into music notation.” in
ISMIR, 2016, pp. 758–764.
[14] J. Kilian and H. H. Hoos, “Voice separation-A local
optimization approach,” in ISMIR, 2002, pp. 39–46.
284[15] I. Karydis, A. Nanopoulos, A. Papadopoulos, E. Cambouropoulos, and Y. Manolopoulos, “Horizontal andvertical integration/segregation in auditory streaming:
A voice separation algorithm for symbolic musicaldata,” in Proc. of Sound and Music Computing Conference, 2007, pp. 299–306.
[16] E. Cambouropoulos, “Voice and stream: Perceptualand computational modeling of voice separation,” Music Perception, vol. 26, no. 1, pp. 75–94, 2008.
[17] B. Duane and B. Pardo, “Streaming from MIDI using constraint satisfaction optimization and sequencealignment.” in Proc. of International Computer Music
Conference, 2009.
[18] A. McLeod and M. Steedman, “HMM-based voice
separation of MIDI performance,” Journal of New Music Research, vol. 45, no. 1, pp. 17–26, 2016.
[19] R. de Valk and T. Weyde, “Deep neural networks withvoice entry estimation heuristics for voice separation insymbolic music representations,” in ISMIR, 2018.
[20] K. Shibata, E. Nakamura, and K. Yoshii, “Non-local
musical statistics as guides for audio-to-score pianotranscription,” Information Sciences, vol. 566, pp.
262–280, 2021.
[21] E. Nakamura, K. Yoshii, and S. Dixon, “Note value
recognition for piano transcription using Markov random fields,” IEEE/ACM TASLP, vol. 25, no. 9, pp.
1846–1858, 2017.
[22] E. Nakamura, E. Benetos, K. Yoshii, and S. Dixon,“Towards complete polyphonic music transcription:
Integrating multi-pitch detection and rhythm quantization,” in ICASSP, 2018, pp. 101–105.
[23] R. G. C. Carvalho and P. Smaragdis, “Towards end-toend polyphonic music transcription: Transforming music audio directly to a score,” in Proc. of IEEE Workshop on Applications of Signal Processing to Audioand Acoustics, 2017, pp. 151–155.
[24] M. A. Román, A. Pertusa, and J. Calvo-Zaragoza, “Aholistic approach to polyphonic music transcription
with neural networks,” in ISMIR, 2019, pp. 731–737.
[25] V. Emiya, R. Badeau, and B. David, “Multipitch estimation of piano sounds using a new probabilistic spectral smoothness principle,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 6,
pp. 1643–1654, 2009.
[26] A. Cogliati and Z. Duan, “A metric for music notationtranscription accuracy.” in ISMIR, 2017, pp. 407–413.
[27] A. McLeod and M. Steedman, “Evaluating automatic
polyphonic music transcription.” in ISMIR, 2018, pp.
42–49.