2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

October 20-23, 2019, New Paltz, NY

END-TO-END MELODY NOTE TRANSCRIPTIONBASED ON A BEAT-SYNCHRONOUS ATTENTION MECHANISM
Ryo Nishikimi1

Eita Nakamura1

Masataka Goto2

Kazuyoshi Yoshii1

1

2

Graduate School of Informatics, Kyoto University, Japan
National Institute of Advanced Industrial Science and Technology (AIST), Japan{nishikimi, enakamura, yoshii}@sap.ist.i.kyoto-u.ac.jp, m.goto@aist.go.jp

6LQJLQJYRLFH ‫ ܆‬୧ୱ୭This paper describes an end-to-end audio-to-symbolic singing transcription method for mixtures of vocal and accompaniment parts.
Given audio signals with non-aligned melody scores, we aim totrain a recurrent neural network that takes as input a magnitude
spectrogram and outputs a sequence of melody notes represented
by pairs of pitches and note values (durations). A promising approach to such sequence-to-sequence learning (joint input-to-output
alignment and mapping) is to use an encoder-decoder model withan attention mechanism. This approach, however, cannot be usedstraightforwardly for singing transcription because a note-level decoder fails to estimate note values from latent representations obtained by a frame-level encoder that is good at extracting instantaneous features, but poor at extracting temporal features. To solve
this problem, we focus on tatums instead of notes as output units
and propose a tatum-level decoder that sequentially outputs tatumlevel score segments represented by note pitches, note onset frags,
and beat and downbeat flags. We then propose a beat-synchronous
attention mechanism constrained in order to monotonically aligntatum-level scores with input audio signals with a steady increment. The experimental results showed that the proposed method
can be trained successfully from non-aligned data thanks to thebeat-synchronous attention mechanism.

(QFRGHU$XGLRPL[WXUH‫ ܆‬୫୧୶

ABSTRACT(QFRGHU

0RQRWRQLFLW\$WWHQWLRQZHLJKWVહ&HQWURLGV

5HJXODULW\'HFRGHU

3LWFK
(
(
(
(
(
(
*
*
*
*
*
*
UHVW
UHVW
(
&

2QVHW

















%HDW

















'%

















5HFRQVWUXFWLRQ

Figure 1: The proposed neural encoder-decoder model with a beatsynchronous attention mechanism for end-to-end singing transcription. DB = ‘downbeat’. New loss functions for the centroids of attention weights are introduced to align them with equally-spannedbeat times.

Index Terms— Automatic singing transcription, end-to-end learning, sequence-to-sequence learning, encoder-decoder recurrent neural networks, attention mechanism
1. INTRODUCTION
Automatic singing transcription (AST), i.e., estimating a sequenceof musical notes corresponding to a sung melody from a music audio signal, is a challenging task that generally consists of multiple sequential sub-tasks. Singing voice separation [1, 2] and vocal
F0 estimation (a.k.a. melody extraction) [3–6] are used for estimating an F0 trajectory of singing voice, which is then quantized in
the frequency and temporal directions to estimate the pitches, onset
times, and durations of notes by using note-tracking methods [7–10]
and rhythm transcription methods [11, 12]. This cascading process,however, suffers from the error propagation problem.
Since AST is a typical sequence-to-sequence task that maps asequence of acoustic features to a sequence of musical note symbols, the end-to-end approach based on a neural encoder-decoder
model with an attention mechanism is considered to be promising in theory. In general, an encoder is used for converting an inThis work was supported in part by JST ACCEL No. JPMJAC1602,
JSPS KAKENHI No. 16H01744, No. 19H04137, and No. 19K20340

26

put sequence into a sequence of latent representations of the samelength. A decoder is then used for sequentially predicting an output
sequence of an appropriate length from the latent sequence whileassociating each output symbol with input symbols (frames) in theattention mechanism. While this approach has been investigated intensively and made a big success in machine translation [13,14] andautomatic speech recognition (AST) [15–17], only a few attempts
have been conducted for automatic music transcription [18].
If we limit our focus to monophonic music transcription, thefundamental difficulty of attention-based AST lies in estimation of
temporal information of musical notes (metrical positions of note
onsets and note values). Previously, a standard attention-based model
consisting of a frame-level encoder and a note-level decoder wasused for AST [18]. This model was found to be able to estimate
note pitches, but often failed to estimate note values. This is because that the encoder is good at extracting instantaneous features(pitch and timbral information) from an input sequence, but poorat extracting temporal features (duration information). Even longshort-term memory networks (LSTMs) cannot propagate temporal
information through several tens or hundreds of frames that a noteduration would have.
To solve this problem, we consider tatums (16th-note-level beattimes) as output symbols instead of musical notes and propose a
new attention-based model consisting of two frame-level encoders
followed by a tatum-level decoder (Fig. 1). The two encoders independently extract latent representations about notes and beats fromsinging signals (assumed to be separated in advance) and music signals, respectively. These representations are used jointly for calculating the attention weights for an output symbol at each step.
The decoder sequentially predicts output symbols, i.e., tatum-level2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

October 20-23, 2019, New Paltz, NY

on connectionist temporal classification (CTC) [31] that estimates
the sequence of music symbols such as keys, pitches, note values,
and time signatures from the magnitude spectrograms of synthesized piano signals. Nishikimi et al. [18] estimated a sequence ofpitches and note values from an isolated solo singing voice basedon an encoder-decoder model with a weakly-supervised attention
mechanism. The mechanism guides the attention weights into ideal
values by using ground-truth onset times of musical notes and enables the model to estimate pitches and note values correctly fromvague onset times of singing voice. One difference between the
proposed method and the conventional one [18] is the joint estimation of notes (pitches and onsets) and metrical structure (beats anddownbeats). Another difference lies in the new loss functions basedon the unsupervised constraints of attention weights.

score segments consisting of note pitches, note onset flags, and beatand downbeat flags. This architecture is thus considered to be able
to make pitch- and beat-aware accurate input-output alignment byleveraging the metrical structures of notes and beats.
We further propose a beat-synchronous attention mechanismthat imposes constraints on the attention weights in terms of monotonicity and regularity. Since popular songs typically have steady
tempo and regular beats, these constraints guide together the attention centroids of output symbols to line up in ascending orderwith an almost equal interval. Conventional methods implement themonotonicity constraint by modifying network architectures [19] ordesigning a special architecture of calculating attention weights [20],whereas we implement both monotonicity and regularity constraints
in the loss functions that are minimized jointly with the cross-entropy
loss for output symbols.
The main contribution of this paper is to propose a new attention model with a tatum-level decoder for popular music having regular metrical structure. Our model can jointly perform note estimation with beat and downbeat tracking in a unified framework. We
experimentally investigate the effectiveness of the monotonicity andregularity constraints.

3. PROPOSED METHOD
The proposed method of AST is based on an encoder-decoder modelwith an attention mechanism (Fig. 1). The proposed model has two
encoders that take as inputs singing and mixture spectrograms, respectively, and output latent vectors for each frame. These latentvectors are then input to a decoder with a beat-synchronous attention mechanism to output tatum-level score segments.

2. RELATED WORKThis section reviews related work on automatic music transcription.

3.1. Problem specification

2.1. Piano-roll estimation

The input is a music audio signal and the output is a symbolic scoreof the vocal part. Let T , F , and N be the number of time frames,that of frequency bins, and that of the 16th-note-level time indices,
respectively. The inputs for the proposed network are the mel-scaleiso
spectrogram of an isolated singing voice Xiso = [xiso1 , . . . , xT ] ∈F ×TR+and that of a mixture of singing voice and accompaniment
F ×T
mix
mix
, where xiso
indicate
Xmix = [xmixt and xt
1 , . . . , xT ] ∈ R+
the mel-scale spectra at frame t of isolated and mixture signals. The
isolated singing voice Xiso is used as an input to simplify the problem because it is difficult to directly estimate the pitch and onset flagfrom the mixture signal that includes accompaniment sounds. The
output of the network is a sequence of symbols Y = [y1 , . . . , yN ].
Each symbol yn = (pn , on , bn , dn )n consists of four symbols:
a semitone-level pitch pn ∈ {1, . . . , K, ⟨sos⟩, ⟨eos⟩} = V p ,where K represents the size of the pitch vocabulary (including therest), an onset flag on ∈ {0, 1, ⟨sos⟩, ⟨eos⟩} = V o , a beat flagbn ∈ {0, 1, ⟨sos⟩, ⟨eos⟩} = V b , and a downbeat flag dn ∈{0, 1, ⟨sos⟩, ⟨eos⟩} = V d . We can reconstruct a score fromthe information contained in Y. The special elements, ⟨sos⟩ and⟨eos⟩, in the vocabulary of each symbol represent the start and endof an output sequence.

Many studies have attempted to estimate a piano-roll representation in which pitches are quantized in semitones but onset times
and durations are not quantized and are represented in seconds (orframe indices) [7–10, 21, 22]. The piano-roll estimation for singing
voice is usually performed by note tracking (i.e., pitch quantizationand note region detection) for F0 trajectories estimated in advance.
Note tracking methods are usually based on hand-crafted rules andfilters [7, 8] or hidden Markov model (HMM) [9, 10]. Some note
tracking methods for other musical instruments (e.g., piano) directly
deal with not F0s but spectrograms because pitches of the instruments are stabler and onset times are clearer than those of singing
voice. Spectrogram factorization techniques like probabilistic latent component analysis (PLCA) [21] are employed to estimate discrete pitches of each time frame, followed by note tracking based on
HMMs. Hawthorne et al. [22] proposed a method based on LSTMs
that estimate semitone-level pitches and onset frames directly froman input spectrogram.
2.2. Musical score transcription
There are several attempts to estimate not a piano roll but a complete score [12,18,23–28]. Nakamura et al. [12] proposed a rhythm
transcription method based on a metrical HMM [23, 24] that takes
the piano-roll representation including performance fluctuations in
the time direction and outputs quantized onset times and note values
in tatums. Nishikimi et al. [25] and Nakamura et al. [26] formulated
hidden semi-Markov models (HSMMs) that quantize an F0 trajectory of singing voice to estimate semitone-level pitches, tatumlevel onset times, and note values by using beat times estimatedin advance. End-to-end approaches to audio-to-score transcription
based on deep neural networks (DNNs) have recently been studied.
Carvalho et al. [27] proposed a method based on the sequence-tosequence model [29] that estimates the symbols of Lilypond format [30] from features extracted from an audio signal of synthesized piano sound by using a one-dimensional convolutional neural network (CNN). Román et al. [28] proposed a method based

3.2. Frame-level encoders
The proposed model has two encoders for Xiso and Xmix . The
encoders transform the spectrograms into sequences of intermediiso
E×T
ate representation vectors Hiso = [hisoand
1 , . . . , hT ] ∈ R
mix
mix
mix
E×T
H
= [h1 , . . . , hT ] ∈ R, where E is the dimension of
the intermediate representation vectors. Since the length of the input spectrogram is variable, we use as each of the encoders a recurrent neural network (RNN), specifically a multi-layer bidirectional
LSTM network.
3.3. Tatum-level decoder with an attention mechanism
The decoder predicts a sequence Y from the latent vectors H =h1:T ∈ R2E×T , where ht is a concatenation of the intermediate27

2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

vectors hiso
and hmix. The decoder consists of a unidirectionaltt
LSTM and is defined as follows:
αn = Attend(sn−1 , αn−1 , H),
gn =

T
∑

αnt ht ,for estimating the pitches and onsets that are components of musicalnotes of a sung melody and gnmix is used for estimating the beats anddownbeats from the percussive sounds in the mixture sound.
(4) represents the recursive calculation of the next state sn . We
adopt the teacher forcing for training. In short, the concatenation of
one-hot vectors separately converted from the ground-truth pitch,
onset, beat, and downbeat is used as yn . The proposed model is
optimized by minimizing the sum of the cross entropies for the elements of each yn and the additional losses described in the next
subsection. In the inference phase, the output symbols obtained by(10), (12), (14), and (16) at the previous step are converted into onehot vectors and used for the concatenation of the one-hot vectors
for predicting the current symbol. This recursive process is stopped
when the output sequence reaches a predefined maximum length orwhen ⟨eos⟩ symbol is generated as pn , on , bn , or dn .

(1)
(2)

t=1

yn = Generate(sn−1 , gn ),sn = Recurrency(sn−1 , gn , yn ),

(3)
(4)

where αn ∈ RT is a set of attention weights, sn ∈ RD denotes then-th hidden state of the decoder, and the functions Attend, Generate,and Recurrency are explained in the following .
(1) and (2) represent the attention mechanism. αn ∈ RT is avector of normalized probabilities representing the degrees of relevance between the latent vectors H and each hidden state sn . Eachelement of αn is calculated as follows:exp(ent ),′′t =1 exp(ent )αnt = ∑T(5)

ent = Score(sn−1 , ht , αn−1 ),(6)

3.4. Loss functions for attention weights
We introduce new loss functions for the attention weights α1:N ∈RN ×T to satisfy the monotonicity and regularity constraints mentioned in Section 1. In appropriate input-output alignment, the attention weights of each αn are known to be biased toward a narrow
region in the time axis. As a representative point of each αn , weuse the centroid given by

where Score is a function that calculates a raw weight. We calculatea shared matrix of attention weights α = α1:N ∈ RN ×T from Hisoand Hmix so that the attention mechanism can put emphasis on theintermediate representations that are important in terms of both noteand beat structures. We use as the Score function a convolutional
function [15] given by
fn = F ∗ αn−1 ,October 20-23, 2019, New Paltz, NY

cn =ent = w tanh (Wsn−1 + Vht + Ufnt + b) ,Lmono =

(8)

Lreg =

N−2∑
1|∆cn+1 − ∆cn |2 .
N − 2 n=1

(19)

4. EVALUATIONThis section reports the results of comparative evaluations on theperformance of the proposed method.

(11)
(12)

4.1. Data
To evaluate our model, we used 54 popular songs with reliable annotations from the RWC Music Database [32]. We split the input
audio signals and the corresponding tatum-level scores into segments of 8 secs with an overlap of 1 sec. When we generated thetatum-level scores and split them, we referred to the annotated musical scores and beat times [33]. If the tatum crossed the start boundary of the segment, the tatum was removed from the segment. The
tatum crossing the end boundary of the segment remained in thesegment. We did not use segments including only rests.
All songs were sampled at 44.1 kHz, and we used a short-time
Fourier transform (STFT) with a Hann window of 2048 points anda shifting interval of 441 points (10 ms) for calculating magnitude

(13)
(14)
(15)
(16)

d∈V d

⋆

(18)

This function makes the centroids be arranged at almost equal intervals that do not suddenly change over time.

b∈V b

(
)
ξ(n) = softmax Pd sn−1 + Qd gnmix + bd ,()(n)
dn = argmax ξd
,

N
−1
∑
1
ReLU (−∆cn ) ,N − 1 n=1

where ∆cn = cn+1 − cn is the difference of the consecutive centroids, and ReLU is a rectified linear function given by ReLU (x) =max (0, x). (18) prevents the order of the centroids from being reversed by imposing the positive cost only if the order of adjacentcentroids is reversed.
The loss function regarding regularity is given by

p∈V p

(
)
ψ
= softmax Po sn−1 + Qo gniso + bo ,()
on = argmax ψo(n) ,
o
o∈V
(
)
(n)
η
= softmax Pb sn−1 + Qb gnmix + bb ,()(n)
bn = argmax ηb,(17)

The loss function regarding monotonicity is given by

where “∗” means the 1-dimensional convolution operation, F ∈RC×I is a set of convolution kernels, fn ∈ RC×T is the result of
the convolution, and C and I indicate the number of kernels andthe size of each kernel. w ∈ RA is a weight vector, W ∈ RA×D ,V ∈ RA×2E , and U ∈ RA×C are weight matrices, b ∈ RA is abias vector, and A is the number of rows of W, V, and U, as wellas the number of elements of b.
Let gniso = [gn1 , . . . , gnE ]⊤ and gnmix = [gn,E+1 , . . . , gn,2E ]⊤denote the parts of gn calculated from hiso and hmix in (2). The
generation process of yn in (3) is given by()
ϕ(n) = softmax Pp sn−1 + Qp gniso + bp ,
(9)
(
)
,
(10)
pn = argmax ϕ(n)p(n)

t · αnt .

t=1

(7)

⊤

T
∑

⋆

where P⋆ ∈ R|V |×D , Q⋆ ∈ R|V |×E are weight matrices, and⋆b⋆ ∈ R|V | is a bias parameter. Here, “⋆” represents “p” (pitch),“o” (onset), “b” (beat), or “d” (downbeat). Note that gniso is used28

2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

MethodOctober 20-23, 2019, New Paltz, NY

Table 1: Error rates [%] in tatum and note levels.
Attention loss
Tatum-level error rate
Note-level error rate [12]N
N
N
Mono. Reg. EpT
EoT
EbT
EdT
EaT
EpN
Em
EeN
Eon
Eoﬀ

Proposed

✓
✓

✓

67.5
42.6
34.6

29.4
26.0
22.5

15.7
16.4
15.4

18.1
18.8
17.8

77.2
58.2
48.3

19.5
20.1
18.3

43.7
20.9
11.4

67.5
42.2
31.7

61.1
55.8
46.3

48.8
42.0
34.1

Majority-vote

n/a

n/a

34.8

25.1

n/a

n/a

n/a

20.4

11.3

51.3

55.5

51.7

2. If pn−1 = pn and on = 1, then the (n−1)-th and n-th
tatums are included in different notes having the same pitch.
3. If pn−1 = pn and on = 0, then the (n−1)-th and n-th
tatums are included in the same note.

spectrograms, which were normalized to make the maximum value
equal to 1, followed by the calculation of the mel-scale spectrograms with 229 channels. The mel-scale spectrograms of training
data were standardized for each frequency bin, and the mel-scale
specgtrograms of evaluation data were standardized for each frequency bin by using means and standard deviations calculated fromthe training data.

Note that the note-level metrics do not take into account rests, beatsand downbeats. We used the parameters that minimize an epoch
average of EpT calculated using some of the evaluation data.

4.2. Setup
The vocabulary of pitches consisted of rest, 40 semitone-level pitchesfrom E2 to G5 (K = 41), and the special elements ⟨sos⟩ and⟨eos⟩. The pitch vocabulary covered pitches contained in both thetrain and test data. We assumed that an input sung melody can be
represented as a monophonic sequence of musical notes.
Each encoder consisted of a three-layer bidirectional LSTM
with 100 × 2 cells with the dropout rate of 0.2. The decoder consisted of one-layer LSTM with 100 cells. A padding size and astride of convolution in the attention mechanism were set to 50 and
1, respectively, and the parameters were C = 10, I = A = 100.
Adam [34] was used to optimize the parameters of the proposedmodel, and a weight decay (L2 regularization) with a controllable
hyperparameter 10−5 and a gradient clipping with a threshold of 5.0
were applied for training. All weight parameters of fully-connected
layers were initialized with random values drawn from the uniform
distribution U(−0.1, −0.1), and all bias parameters were initializedwith zeros. The kernels of the CNN in the attention mechanism andthe weight parameters of the encoder and decoder were initialized
by He’s method [35]. The maximum lengths in the inference phase
was set to 200. The batch size and the number of epochs were 150
and 100, respectively. PyTorch v1.0.1 was used for implementation.

4.4. Results
Experimental results in Table 1 showed that both tatum- and notelevel error rates became worse by using the proposed loss functions
for attention weights. The reason for this result is that the constraintof the loss functions is too strong to prevent the attention mechanism from finding appropriate position in the input sequence. In theearly stages of training, the centroids usually line up at the roughlysame positions, and Lmono and Lreg are almost zero. The centroids
cannot escape from the initial positions because the values of Lmono
and Lreg increase when the centroids change.
We also compared the proposed method to the majority-votemethod used in [25] using ground-truth F0 trajectory with voice activity detection and tatum times. A boundary of pitch changes was
regarded as note onset positions and the successive tatums having
the same pitch were included in one note. Since the method does
not estimate beats and downbeats, EbT , EdT , and EaT are not used. In
most metrics, the proposed method without the attention losses attained better error rates than the majority-vote method. Especially,
we obtained an improvement on EeN .

4.3. Metrics
The performance of tatum-level transcription was measured using
tatum-level error rate defined as:
NS + ND + NI
× 100 [%],
(20)
N
where the numerator represents the Levenshtein distance between
the ground-truth and prediced scores: NS , ND , and NI are the minimum number of substitutions, deletions, and insertions required tochange the predicted sequence into the ground-truth N is the number of tatums in the ground-truth. We used the error rate for evaluating each of a pitch EpT , an onset flag EoT , a beat flag EbT , and a
downbeat flag EdT and all of them EaT .
To measure the note-level performance, we used the metrics
proposed in [12] that calculate the following five values: pitch errorNrate EpN , missing note error rate Em, extra note rate EeN , onset-timeNN
error rate Eon , and offset-time error rate Eoﬀ. In constructing of thescore from the predicted symbols, we applied the following rules:

5. CONCLUSION
This paper presented the method for musical note transcription ofa sung melody based on the encoder-decoder model with the beatsynchronous attention mechanism. We extended the standard encoderdecoder model to simultaneously predict a pitch, an onset flag, abeat flag, and a downbeat flag that are needed to construct a musicalscore. We also proposed the loss functions to induce the attention
weights to have proper structure. We experimentally investigatedwhether those loss functions are effective or not.
For future work, we can incorporate other tasks of music analysis in the proposed model. We can integrate a model for separating
isolated singing voice from an audio mixture, which are separately
fed to the proposed model. Owing to the tatum-level representationof output symbols, other symbols (e.g., chords) in musical score canbe easily added into the output of the proposed method. These integrations would improve the transcription accuracy and facilitate thesimultaneous analysis of music.

1. If pn−1 ̸= pn , then the (n−1)-th and n-th tatums are included in different notes.

29

2019 IEEE Workshop on Applications of Signal Processing to Audio and Acoustics

October 20-23, 2019, New Paltz, NY[17] R. Prabhavalkar, T. N. Sainath, B. Li, K. Rao, and N. Jaitly,“An analysis of ”attention” in sequence-to-sequence models,”in INTERSPEECH, 2017, pp. 3702–3706.
[18] R. Nishikimi, E. Nakamura, S. Fukayama, M. Goto, andK. Yoshii, “Automatic singing transcription based on encoderdecoder recurrent neural networks with a weakly-supervised
attention mechanism,” in ICASSP, 2019, pp. 161–165.
[19] C. Chiu and C. Raffel, “Monotonic chunkwise attention,” in
ICLR, 2018.
[20] C. Raffel, M.-T. Luong, P. J. Liu, R. J. Weiss, and D. Eck,
“Online and linear-time attention by enforcing monotonicalignments,” in ICML, 2017, pp. 2837–2846.
[21] E. Benetos and S. Dixon, “Multiple-instrument polyphonicmusic transcription using a temporally constrained shiftinvariant model,” The Journal of the Acoustical Society ofAmerica, vol. 133, no. 3, pp. 1727–1741, 2013.
[22] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck, “Onsets and frames: Dualobjective piano transcription,” in ISMIR, 2018, pp. 50–57.
[23] C. Raphael, “A hybrid graphical model for rhythmic parsing,”
Artificial Intelligence, vol. 137, no. 1-2, pp. 217–238, 2002.
[24] M. Hamanaka, M. Goto, H. Asoh, and N. Otsu, “A learningbased quantization: Unsupervised estimation of the model parameters,” in ICMI, 2003, pp. 369–372.
[25] R. Nishikimi, E. Nakamura, M. Goto, K. Itoyama, andK. Yoshii, “Scale- and rhythm-aware musical note estimationfor vocal F0 trajectories based on a semi-tatum-synchronous
hierarchical hidden semi-markov model,” in ISMIR, 2017, pp.
376–382.
[26] E. Nakamura, R. Nishikimi, S. Dixon, and K. Yoshii, “Probabilistic sequential patterns for singing transcription,” in APSIPA ASC, 2018, pp. 1905–1912.
[27] R. G. C. Carvalho and P. Smaragdis, “Towards end-to-end
polyphonic music transcription: Transforming music audio directly to a score,” in WASPAA, 2017, pp. 151–155.
[28] M. A. Román, A. Pertusa, and J. Calvo-Zaragoza, “An endto-end framework for audio-to-score music transcription on
monophonic excerpts,” in ISMIR, 2018, pp. 34–41.
[29] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequencelearning with neural networks,” in NIPS, 2014, pp. 3104–
3112.
[30] H.-W. Nienhuys and J. Nieuwenhuizen, “Lilypond, a system
for automated music engraving,” in CIM, 2003, pp. 167–171.
[31] A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber,“Connectionist temporal classification : Labelling unsegmented sequence data with recurrent neural networks,” in
ICML, 2006, pp. 369–376.
[32] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC
Music Database: Popular, classical and jazz music databases,”in ISMIR, 2002, pp. 287–288.
[33] M. Goto, “AIST annotation for the RWC music database.” in
ISMIR, 2006, pp. 359–360.
[34] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in arXiv preprint arXiv:1412.6980, 2014, pp. 1–15.
[35] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in ICCV, 2015, pp. 1026–1034.

6. REFERENCES[1] P.-S. Huang, S. D. Chen, P. Smaragdis, and M. HasegawaJohnson, “Singing-voice separation from monaural recordings
using robust principal component analysis,” in ICASSP, 2012,
pp. 57–60.
[2] A. Jansson, E. Humphrey, N. Montecchio, R. Bittner, A. Kumar, and T. Weyde, “Singing voice separation with deep u-net
convolutional networks,” in ISMIR, 2017, pp. 23–27.
[3] M. Mauch and S. Dixon, “pYIN: A fundamental frequency estimator using probabilistic threshold distributions,”in ICASSP, 2014, pp. 659–663.
[4] Y. Ikemiya, K. Yoshii, and K. Itoyama, “Singing voice analysis and editing based on mutually dependent F0 estimationand source separation,” in ICASSP, 2015, pp. 574–578.
[5] R. M. Bittner, B. McFee, J. Salamon, P. Li, and J. P. Bello,“Deep salience representations for f0 estimation in polyphonicmusic.” in ISMIR, 2017, pp. 63–70.
[6] J. W. Kim, J. Salamon, P. Li, and J. P. Bello, “CREPE: A
convolutional representation for pitch estimation,” in ICASSP,
2018, pp. 161–165.
[7] E. Molina, L. J. Tardón, A. M. Barbancho, and I. Barbancho, “SiPTH: Singing transcription based on hysteresis defined on the pitch-time curve,” IEEE/ACM Transactions on
Audio, Speech and Language Processing, vol. 23, no. 2, pp.
252–263, 2015.
[8] N. Kroher and E. Gómez, “Automatic transcription offlamenco singing from polyphonic music recordings,”
IEEE/ACM Transactions on Audio, Speech and Language
Processing, vol. 24, no. 5, pp. 901–913, 2016.
[9] M. Mauch, C. Cannam, R. Bittner, G. Fazekas, J. Salamon,
J. Dai, J. Bello, and S. Dixon, “Computer-aided melody notetranscription using the Tony software: Accuracy and efficiency,” in TENOR, 2015, pp. 23–30.
[10] L. Yang, A. Maezawa, J. B. L. Smith, and E. Chew, “Probabilistic transcription of sung melody using a pitch dynamic
model,” in ICASSP, 2017, pp. 301–305.
[11] H. Takeda, N. Saito, T. Otsuki, M. Nakai, H. Shimodaira, and
S. Sagayama, “Hidden markov model for automatic transcription of MIDI signals,” in MMSP, 2002, pp. 428–431.
[12] E. Nakamura, K. Yoshii, and S. Sagayama, “Rhythm transcription of polyphonic piano music based on merged-output
HMM for multiple voices,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25, no. 4, pp.
794–806, 2017.
[13] M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” in
EMNLP, 2015, pp. 1412–1421.
[14] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in ICLR,
2015, pp. 1–15.
[15] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and
Y. Bengio, “Attention-based models for speech recognition,”in NIPS, 2015, pp. 577–585.
[16] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend andspell: A neural network for large vocabulary conversational
speech recognition,” in ICASSP, 2016, pp. 4960–4964.

30