AUTOMATIC SINGING TRANSCRIPTION BASED ON ENCODER-DECODER RECURRENTNEURAL NETWORKS WITH A WEAKLY-SUPERVISED ATTENTION MECHANISM
Ryo Nishikimi1

Eita Nakamura1

Satoru Fukayama2

Masataka Goto2

Kazuyoshi Yoshii1

1

Graduate School of Informatics, Kyoto University, Japan
National Institute of Advanced Industrial Science and Technology (AIST), Japan
ABSTRACTThis paper describes neural singing transcription that estimates a sequence of musical notes directly from the audio signal of singing
voice in an end-to-end manner without time-aligned training data.
A conventional approach to singing transcription is to perform vocal F0 estimation followed by musical note estimation. The performance of this approach, however, is severely limited because theF0 estimation errors propagate to the note estimation step and richacoustic information cannot be used. In addition, it is difficult andtime-consuming to split continuous signals of singing voices into
segments corresponding to musical notes for making precise timealigned transcriptions. To solve these problems, we use an encoderdecoder model with an attention mechanism that can automaticallylearn an input-output alignment and mapping, even from non-aligned
training data. The main challenge of our study is to estimate temporal categories (note values) in addition to instantaneous categories(pitches). We thus propose a novel loss function for the attention
weights of time-aligned notes for semi-supervised alignment training. By gradually reducing the weight of the loss function, a better input-output alignment can be learned much more quickly. We
showed that our method performed well for isolated singing voice inpopular music.
Index Terms— Automatic singing transcription, end-to-end
learning, sequence-to-sequence learning, encoder-decoder recurrent
neural networks, attention mechanism
1. INTRODUCTION
Automatic singing transcription (AST) refers to estimating a sequence of musical notes of a sung melody from music audio signals,
and forms a basis for music information retrieval (MIR) because
the melody is the most prominent part of popular music that influences the impression of a song. The estimated musical notes canbe used for singing voice generation [1], musical grammar analysis,
query-by-humming, and active music listening [2].
Many studies have been conducted on AST. A naive approachto AST is to sequentially use a singing voice separation method thatextracts singing voice from music audio signals [3–5] and an F0 estimation method that estimates F0 trajectories from singing voice[5–11]. This approach requires an additional step to estimate thesemitone-level pitches and note values of musical notes by quantizing F0 trajectories in the time and frequency domains. Most studies,
however, have focused on only pitch quantization for estimating piano rolls [12–14], while note-value quantization (a.k.a. rhythm tranThis work was supported in part by JST ACCEL No. JPMJAC1602,
JSPS KAKENHI No. 16H01744, No. 16J05486, Kayamori Foundation, andthe Kyoto University Foundation.

Mel spectrogram ‫܆‬Musical Notes ‫܇‬
Pitch

Value

<SOS>

<SOS>

Rest
C4#
D4#
E4
F4#
<EOS>

Onset frames ofmusical notes ‫܈‬

Encoder (bidirectional LSTM × 5)

Decoder (LSTM × 1)

2

Attention weights હGround-truth weights ઺<EOS>‫୔ܮ‬୧୲ୡ୦ ‫ܮ‬୚ୟ୪୳ୣ

‫ܮ‬୅୲୲

Fig. 1. Our encoder-decoder model with an attention mechanism forend-to-end AST. This model is trained by minimizing the weighted
sum of loss functions for ground-truth pitches and note values, aswell as alignment information (onset times) if available.
scription) has been investigated independently [15–17]. Some studies have tried to jointly estimate the pitches and note values of musical notes from F0 trajectories using a musical language model that
represents the pitches, rhythms, and scales of musical notes [18, 19].
The major problem of the aforementioned methods are that the errors
occurred in the F0 estimation adversely affect the note estimation.
In this paper, we propose an end-to-end approach for AST.
AST is similar to automatic speech recognition (ASR) in that audio spectra are converted into a sequence of meaningful symbols(musical notes or characters). Inspired by the success of sequenceto-sequence or end-to-end learning in ASR [20–22] and machinetranslation [23, 24], we focus on an encoder-decoder model withan attention mechanism for directly estimating a sequence of musical notes from a sequence of vocal spectra. The encoder-decoder
model is a deep neural network (DNN) that can convert an input
sequence into an output sequence of different lengths. The attention
mechanism is an efficient method for dealing with long sequencesby setting attention weights (weak alignment) between output andinput sequences.
A key difference between AST and ASR is that temporal categories (note values) must be estimated in addition to instantaneous
categories (pitches). To deal with such fine temporal structure as
rhythms, it is crucial to train attention weights precisely. Althougha simple solution is to apply supervised learning for the attention
weights, instead of the common unsupervised learning, there is aproblem of the limitation of available training data for AST.
To solve this, we propose a novel framework based on semisupervised learning for the attention weights by using a limited
amount of partially inaccurate time-aligned data as guiding information (Fig. 1). Specifically, we propose a novel loss functionthat evaluates the input-output alignment estimated by the attention mechanism. If time-aligned data are available, we calculate
the cross-entropy loss between the estimated normalized attention
weights and the corresponding ground-truth weights for each note.
We find that when the model is trained successfully in an unsupervised manner, the attention weights of each note tend to concentratearound its onset frame. As the ground-truth weights for each note,
therefore, we choose a one-hot vector that is peaked at the onset
frame of the note. Although the onset time annotations are often unreliable, the model can learn correct alignment by gradually
reducing the weight of the attention loss function. The main contribution of this study is an easy-to-implement general technique for
effectively avoiding bad local optima and accelerating convergencein the framework of semi-supervised end-to-end learning when bothaligned and non-aligned data are available.
2. ATTENTION-BASED ENCODER-DECODER MODEL
This section reviews the standard encoder-decoder model with anattention mechanism for sequence-to-sequence learning [21].
2.1. Encoder

where F ∈ RC×F is a convolutional filter, fn ∈ RT ×C is the resultof the convolution, and C and F indicate the number of channels
and the size of the filter. w ∈ RA indicates a weight vector, W ∈RA×D , V ∈ RA×E , and U ∈ RA×C represent weight matrices,and bAtt ∈ RA represents a bias vector. Here, A is the number ofrows of W, V, and U, as well as the number of elements of bAtt .
Eq. (3) represents the generation of yn from the previous hidden
state sn−1 and the weighted sum gn as folows:
π = Softmax Psn−1 + Qgn + bGen ,(9)yn = argmax (πyn ) ,(10)yn ∈{1,...,K}

where P ∈ RI×D , Q ∈ RI×E represent weight matrices, andbGen ∈ RI is a bias vector.
Eq. (4) represents the calculation of the next state sn . Note thatthe ground-truth symbol is used as yn in the training phase, whereas
in the inference phase, yn is predicted by the decoder at the previous step and the symbol prediction stops when the output sequence
reaches a specified maximum length or when heosi is generated.
3. PROPOSED METHOD

The encoder transforms a sequence of feature vectors (input data)
X = [x1 , . . . , xT ] ∈ RF ×T into a sequence of latent representations H = [h1 , . . . , hT ] ∈ RE×T , where T , F , and E indicate thelength of the input sequence, the dimension of the feature vectors,
and the dimension of the latent vectors, respectively. The encoder
is usually formulated as a recurrent neural network (RNN). In thisstudy, we use a multi-layer bidirectional long short-term memory
unit (LSTM).
2.2. Decoder with Attention Mechanism
The decoder predicts a sequence of symbols Y = [y1 , . . . , yN ] fromthe latent vectors H, where N indicates the number of symbols predicted by the decoder. yn ∈ {1, . . ., I} indicates the n-th predicted
symbol, where I indicates the vocabulary size of the decoder. The
vocabulary includes two special symbols: hsosi and heosi. The
attention-based decoder consists of a unidirectional RNN and recursively calculates the following steps:
αn = Attend(sn−1 , αn−1 , H),
gn =

T
X

αnt ht ,

(1)
(2)

t=1

yn = Generate(sn−1 , gn ),sn = Recurrency(sn−1 , gn , yn ),

(3)
(4)

where sn ∈ RD indicates the n-th hidden state of the decoder, andAttend, Generate, and Recurrency are functions that perform operations on vectors and matrices.
Eq. (2) represents the attention mechanism. αn ∈ RT is a vectorof normalized weights representing the degrees of relevance between
the input sequence X and a note yn . Each element of αn is given by
αnt = PT

exp(ent ),exp(ent′ )(5)This section explains the proposed method of AST with a modified
attention-based encoder-decoder model (Fig. 1).
3.1. Problem Specification
Our goal is to train an encoder-decoder model that takes as input
a mel-scale spectrogram X = x1:T ∈ RF ×T of an isolated solo
singing voice and outputs a sequence of musical notes Y = y1:N =(pn , vn )1:N by using the onset frames of those notes Z = z1:N ifavailable, where T , F , and N indicate the number of time frames,that of frequency bins, and that of musical notes, respectively. xt ∈RF indicates the mel-scale spectrum at frame t. Each note yn isrepresented as a pair of a semitone-level pitch pn ∈ {1, . . ., K} anda 16th-note-level duration vn ∈ {1, . . ., L}. K and L indicate thesize of the pitch vocabulary (including the rest) and that of the notevalue vocabulary, respectively. zn ∈ {1, . . ., T } indicates the onset
frame of the musical note yn in the spectrogram X.
3.2. Pitch and Note Value Decoder
In AST, it is necessary to output two symbols: a pitch and a notevalue. One possibility is to regard the pair of a pitch and a note value
as one symbol. Such a model, however, is difficult to train from alimited amount of training data because the joint vocabulary size is
I = K × L. To reduce the vocabulary size to I = K + L, we extend
the decoder to separately output both symbols at the same time asfollows:

φ = Softmax P̂sn−1 + Q̂gn + b̂ ,(11)pn = argmax (φpn ) ,(12)pn ∈{1,...,K}
ψ = Softmax P̄sn−1 + Q̄gn + b̄ ,vn = argmax (ψvn ) ,

(13)
(14)

vn ∈{1,...,L}t′ =1

ent = Score(sn−1 , ht , αn−1 ),(6)where Score is a function that calculates a raw weight. In this paper,
we use a convolutional function [21] given by
fn = F ∗ αn−1 ,
ent = w⊤ tanh Wsn−1 + Vht + Ufnt + bAtt ,

(7)
(8)

where P̂ ∈ RK×D , Q̂ ∈ RK×E , P̄ ∈ RL×D , and Q̄ ∈ RL×E
indicate weight matrices, and b̂ ∈ RK and b̄ ∈ RL indicate bias
parameters. To treat a pitch and a note value as separate symbols, we
use two different hsosi and heosi symbols for the pitch and notevalue prediction, In short, the pitch vocabulary includes hsos pi
and heos pi, and the note-value vocabulary includes hsos vi andheos vi.

3.3. Loss Function for Attention Weights
We define a loss LAtt for attention weights α= [α1 , . . . , αN ]⊤ ∈RN ×T to guide them to ideal values. To calculate LAtt , the guiding
ground-truth weights β for α are introduced by using the optional
data Z. We find that when the model is successfully trained without
calculating LAtt , the attention weights of each note tend to concentrate around its onset frame. A one-hot vector is thus considered to
be a good choice for β as follows:(
1 (zn = t)
,
(15)
βnt =
ε (otherwise)
where ε is a small positive number. After each row of β is normalized, LAtt is given by the cross entropy as follows:
N

LAtt = −T

λ XX
βnt log αnt ,N n=1 t=1

(16)where λ is a hyperparameter to scale loss LAtt , which is gradually
decreased during training to learn optimal input-output alignmentthat is considered to be different from hard alignment β.
3.4. Training and Inference Algorithms
In the training phase, the pitch and note value are separately converted to one-hot vectors, and then input to the decoder. The loss
LPitch for the output pitches and the loss LValue for the output notevalues are given by their cross entropies. The sum of LPitch , LValue ,and LAtt is defined as the total loss function to be minimized. In theinference phase, the pitch and note value obtained by Eqs. (12) and(14) at the previous time step are converted into one-hot vectors andused for predicting the current symbol. This process stops when
the output sequence reaches a specified maximum length or whenheos pi or heos vi is generated.
4. EVALUATIONThis section describes experiments conducted to evaluate the performance of the proposed model for AST.
4.1. Experimental Data
To evaluate our model, we used 54 popular songs with reliable annotations from the RWC Music Database [25]. We split the audio
signals of isolated singing voice and the corresponding time-aligned
musical scores [26] into all possible segments ranging from 1 measure to 8 measures with an overlap of 1 measure. When a notecrossed a bar line, it was included in the precedent measure. Rests ina measure were concatenated into a single rest. Musical notes longer
than a whole note were discarded. We used 44, 5, and 5 songs as
training, validation, and test data, respectively.
All songs were sampled at 44.1 kHz, and we used an STFT witha Hann window of 2048 points and a shifting interval of 441 points(10 ms) for calculating magnitude spectrograms, which we normalized to make the maximum value 1. Since a tempo determines notevalues, which would be difficult to correctly estimate for the proposed method without any mechanism to estimate tempos, all songs
were modified to a BPM of 150 using a phase vocoder for the training, validation, and test data. We standardized the spectrograms foreach frequency bin, and then calculated the mel-scale spectrogramswith 229 channels.
4.2. Configurations
The vocabulary of pitches consisted of a rest, hsos pi, heos pi,
and 40 semitone-level pitches from E2 to G5 (K = 43). The vo-

Table 1. Word error rates on the test data.
Configuration of λNER [%]PER [%]VER [%]λ=1
λ=0
λ is gradually reduced

48.8
119.8
43.0

31.6
90.0
24.7

34.8
96.8
29.6

cabulary of note values consisted of hsos vi, heos vi, and 16
values which were integer multiples of a 16th note up to a wholenote (L = 18). We discarded any data containing out-of-vocabulary
notes, and it was also assumed that an input sung melody could be
represented as a monophonic sequence of musical notes.
We applied frame stacking [27] with a stack size of 5 and a skip
size of 1 to the mel-scale spectrograms, and added zero frames at theboth ends to align with hsosis and heosis. The encoder consistedof five-layers of bidirectional LSTMs with 300 × 2 cells. We set
the dropout rate to 0.2 for each layer. The decoder consisted of onelayer LSTM with 200 cells. The number of channels and the filter
size were C = 10 and F = 100. We used a padding size and a strideof convolution in the attention mechanism of 50 and 1, respectively.
Adam [28] with a standard setting was used to optimize the proposed model. The hyperparameter A was 200. To avoid overfitting,a weight decay (L2 regularization) with a controllable hyperparameter of 10−5 was used. To prevent weight parameters from diverging,
gradient clipping with a threshold of 5.0 was also used. All weightparameters of fully-connected layers were initialized with random
values drawn from the uniform distribution U(−0.1, 0.1). The filter of the one dimensional CNN in the attention mechanism and theweight parameters of the encoder and decoder were initialized byHe’s method [29]. All bias parameters were initialized with zeros.
The small positive value ε was set to 10−4 . The batch size and thenumber of epochs were 20 and 15. Pytorch v0.4.1 [30] was used for
implementation.
To verify the effectiveness of LAtt , we compared the following
three configurations of the hyperparameter λ:• λ is fixed to 1.
• λ is fixed to 0.
• λ is initialized to 1 and reduced by 10−4 at each iteration.
We also examined the effectiveness of the semi-supervised learningwhen only a part of Z is available. We randomly selected data from
Z at a rate of 5%, 10%, 25%, 50%, and 75%, and only the selected
data were used to calculate the guiding ground-truth weights β andthe loss function LAtt with a gradually reduced λ. For evaluation,
we used the model which minimized the average of validation losses
per an epoch during training.
4.3. Evaluation Metrics
Performance was measured using word error rate (WER) defined as
NS + ND + NI
WER =× 100 [%],
(17)
N
where the numerator represents the Levenshtein distance between
ground-truth and estimated note sequences. NS , ND , and NI indicate the minimum number of substitutions, deletions, and insertions
required to change the estimated sequence into the ground-truth. N
indicates the number of ground-truth musical notes. We used threevariants of WER: note error rate (NER) for evaluating both pitches
and note values, pitch error rate (PER) for evaluating only pitches,and value error rate (VER) for evaluating only note values.

4
2
[NER]
08

2000

4000

ߣൌͳ

ߣൌͲ

6000

ߣ is reduced.

Freq. bin6

ߣ is reduced.

Note
number8

ߣൌͲ[Iteration]

6[NER]
140 2. NERs calculated on the validation data during the training.
Fig.
NER
PER
VER
1204
Grey
lines indicate NERs of each iteration, and colored lines indicate
1002
theaverage values of the NERs for the past 100 iterations.
80
060[Iteration]
2000
4000
6000
40
[NER]
140
20
119.8
NER
PER
VER
120
0
96.8
100
0
5
10
25
50
75
100
80
Usage rates
90
50
60
48
46.5
43
44.1
45.7
36
32.9
40
34.1
32.7
31.8
29.6
20
30.4
27.8
27
27.3
24.8
24.7
0
0
5
10
25
50
75
100
Usage rates

0

100

200

300

400

500

600

Frame

70
65
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60
68 units
16th64
-note

10
0

100

200

300

400

500

600

700
Frame

70
65
0

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60
68 units
16th64
-note

20
0

Note
number

200

300

400

500

600

700
Frame

65

20

4.4. Experimental Results

100

70

0

Fig. 3. WERs with different usage rates of training data Z.

Experimental results are shown in Table 1. The models obtained using the proposed loss function clearly outperformed those obtainedwith λ = 0. Fig. 2 shows NERs calculated on the validation data
during the training of each proposed model. It indicates that the proposed loss function was effective in accelerating the convergence ofNERs and finding a better solution with lower validation loss. Aninteresting fact is that better WERs were obtained by gradually reducing the value of λ than by fixing λ to 1. Since the time-aligned
onset annotations and guiding ground-truth weights β were not always accurate, gradually reducing λ enables the proposed model toautomatically find better time alignment.
Examples of attention weights and musical notes estimatedby the proposed model are illustrated in Fig. 4. The yellow score
estimated with λ = 0 included many 8th notes that were not in
the ground-truth score, and vague attention weights were learned.
On the other hand, the blue and green scores estimated using the
proposed loss function were nearly correct, and peaked attention
weights were learned. Given that musical note estimation failed
with softly-learned attention weights, it seems to be reasonable touse the peaked ground-truth weights. Although a rest that was not in
the ground-truth score was estimated near the center of the bottom
score, the input spectrogram certainly had an unvoiced interval nearthe 300th frame. The annotated data regards the silence as a continuation of the musical note, whereas the proposed method estimateda rest for the silent section of the spectrogram. Offset detection isstill an open problem in music transcription, and we will consider
offsets further in future work.
Experimental results with different usage rates of Z are shownin Fig. 3. Note that the cases of using 0% and 100% of Z in Fig. 3
correspond to the second and third rows of Table 1, respectively. As
shown in Fig. 3, the WERs were almost monotonically reduced as
the usage rate of Z was increased. Especially, the WERs were drastically reduced even if only 5% of Z was used. This indicates thatthe loss function LAtt is very effective even when a small amount ofsupervised data is available.

200

0

Note
numberߣൌͳNotenumber[NER]

0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60
68 units
16th64
-note

0

100

200

300

400

500

600

700
Frame

70
65
0 4 8 12 16 20 24 28 32 36 40 44 48 52 56 60
68 units
16th64-note

Fig. 4. Examples of attention weights and musical notes estimatedby the proposed method. Red, blue, yellow, and green horizontal lines indicate musical notes, grey lines indicate rests, and blacksquares indicate the onset positions of the musical notes. The top two
figures are the input spectrogram and the ground-truth musical notes.
The subsequent figures are attention weights and musical notes for
λ = 1, λ = 0, and the gradual reduction of λ from top to bottom.

5. CONCLUSION
This paper has presented the method for estimating the musical notesof a sung melody with an attention-based encoder-decoder model.
We extended the standard model to simultaneously predict a pitch
and a note value at each step. We also proposed a new loss function for attention weights and a semi-supervised training method forbetter performance and faster convergence. The experimental results
showed that the proposed encoder-decoder model has great potential
for end-to-end AST, and that the performance of AST was improvedby using the attention loss function.
One of the most important research directions for future work
is to integrate the proposed model with an end-to-end RNN for estimating time-varying tempo and beats/downbeats from audio signals. Although tempo changes often occur in popular songs, theproposed model assumes that the tempo of a song is constant. Suchintegration should improve the robustness of the note value estimation. We would also like to extend the proposed method to directly
deal with polyphonic music signals without singing voice separation.
This would enable us to leverage a huge amount of music recordings
with non-aligned melody transcriptions.

6. REFERENCES[1] M. Blaauw and J. Bonada, “A neural parametric singing synthesizer modeling timbre and expression from natural songs,”
Applied Sciences, vol. 7, no. 12, 2017.
[2] M. Goto, K. Yoshii, H. Fujihara, M. Mauch, and T Nakano,“Songle: A web service for active music listening improved byuser contributions,” in International Society for Music Information Retrieval Conference (ISMIR), 2011, pp. 311–316.
[3] Y. Li and D. Wang, “Separation of singing voice from music
accompaniment for monaural recordings,” IEEE Transactions
on Audio, Speech, and Language Processing, vol. 15, no. 4, pp.
1475–1487, 2007.
[4] P.-S. Huang, S. D. Chen, P. Smaragdis, and M. HasegawaJohnson, “Singing-voice separation from monaural recordings
using robust principal component analysis,” in IEEE ICASSP,
2012, pp. 57–60.
[5] Y. Ikemiya, K. Yoshii, and K. Itoyama, “Singing voice analysis
and editing based on mutually dependent F0 estimation andsource separation,” in IEEE ICASSP, 2015, pp. 574–578.
[6] D. J. Hermes, “Measurement of pitch by subharmonic summation,” The Journal of the Acoustical Society of America, vol.
83, no. 1, pp. 257–264, 1988.
[7] M. Goto,“A real-time music-scene-description system:
Predominant-f0 estimation for detecting melody and bass lines
in real-world audio signals,” Speech Communication (ISCA
Journal), vol. 43, no. 4, pp. 311–329, 2004.
[8] J. Salamon and E. Gómez, “Melody extraction from polyphonic music signals using pitch contour characteristics,”
IEEE Transactions on Audio, Speech, and Language Processing, vol. 20, no. 6, pp. 1759–1770, 2012.
[9] J.-L. Durrieu, G. Richard, B. David, and C. Févotte,“Source/filter model for unsupervised main melody extraction
from polyphonic audio signals,” IEEE Transactions on Audio,
Speech, and Language Processing, vol. 18, no. 3, pp. 564–575,
2010.
[10] M. Mauch and S. Dixon, “pYIN: A fundamental frequency
estimator using probabilistic threshold distributions,” in IEEEICASSP, 2014, pp. 659–663.
[11] R. Schramm, A. Mcleod, M. Steedman, and E. Benetos,“Multi-Pitch Detection and Voice Assignment for a Cappella
Recordings of Multiple Singers,” in International Society forMusic Information Retrieval Conference (ISMIR), 2017, pp.
552–559.
[12] E. Molina, L. J. Tardón, A. M. Barbancho, and I. Barbancho, “Sipth: Singing transcription based on hysteresis definedon the pitch-time curve,” IEEE/ACM Transactions on Audio,
Speech and Language Processing, vol. 23, no. 2, pp. 252–263,
2015.

[15] H. Takeda, N. Saito, T. Otsuki, M. Nakai, H. Shimodaira, and
S. Sagayama, “Hidden markov model for automatic transcription of MIDI signals,” in IEEE Workshop on Multimedia Signal
Processing (MMSP), 2002, pp. 428–431.
[16] C. Raphael, “A hybrid graphical model for rhythmic parsing,”
Artificial Intelligence, vol. 137, no. 1-2, pp. 217–238, 2002.
[17] M. Hamanaka, M. Goto, H. Asoh, and N. Otsu, “A learningbased quantization: Unsupervised estimation of the model parameters,” in International Conference on Multimodal Interfaces (ICMI), 2003, pp. 369–372.
[18] R. Nishikimi, E. Nakamura, M. Goto, K. Itoyama, andK. Yoshii, “Scale- and rhythm-aware musical note estimationfor vocal F0 trajectories based on a semi-tatum-synchronous
hierarchical hidden semi-markov model,” in International Society for Music Information Retrieval Conference, (ISMIR),
2017, pp. 376–382.
[19] E. Nakamura, R. Nishikimi, S. Dixon, and K. Yoshii, “Probabilistic sequential patterns for singing transcription,” in AsiaPacific Signal and Information Processing Association Annual
Summit and Conference (APSIPA ASC), 2018.
[20] W. Chan, N. Jaitly, Q. Le, and O. Vinyals, “Listen, attend andspell: A neural network for large vocabulary conversational
speech recognition,” in IEEE ICASSP, 2016, pp. 4960–4964.
[21] J. K. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and
Y. Bengio, “Attention-based models for speech recognition,”in Advances in Neural Information Processing Systems (NIPS),
2015, pp. 577–585.
[22] R. Prabhavalkar, T. N. Sainath, B. Li, K. Rao, and N. Jaitly,“An analysis of ”attention” in sequence-to-sequence models,”in INTERSPEECH, 2017, pp. 3702–3706.
[23] M.-T. Luong, H. Pham, and C. D. Manning, “Effective approaches to attention-based neural machine translation,” in
Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015, pp. 1412–1421.
[24] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” in International
Conference for Learning Representations (ICLR), 2015, pp. 1–
15.
[25] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC
Music Database: Popular, classical and jazz music databases,”in International Conference on Music Information Retrieval(ISMIR), 2002, pp. 287–288.
[26] M. Goto, “AIST annotation for the RWC music database.,” inInternational Conference on Music Information Retrieval (ISMIR), 2006, pp. 359–360.
[27] H. Sak, A. Senior, K. Rao, and F. Beaufays, “Fast and accurate
recurrent neural network acoustic models for speech recognition,” in INTERSPEECH, 2015, pp. 1468–1472.
[28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in arXiv preprint arXiv:1412.6980, 2014, pp. 1–15.

[13] L. Yang, A. Maezawa, J. B. L. Smith, and E. Chew, “Probabilistic transcription of sung melody using a pitch dynamic
model,” in IEEE ICASSP, 2017, pp. 301–305.

[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun,
“Delving deep into rectifiers: Surpassing human-level performance on imagenet classification,” in IEEE International Conference on Computer Vision (ICCV), 2015, pp. 1026–1034.

[14] N. Kroher and E. Gómez, “Automatic transcription of flamenco
singing from polyphonic music recordings,” IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 24,
no. 5, pp. 901–913, 2016.

[30] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer, “Automatic differentiation in pytorch,” 2017.