SIP (2021), vol. 10, e7, page 1 of 13 © The Author(s), 2021. Published by Cambridge University Press in association with Asia Pacific Signal and Information Processing
Association. This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence (http://creativecommons.org/licenses/by/4.0/),
which permits unrestricted re-use, distribution, and reproduction in any medium, provided the original work is properly cited.
doi:10.1017/ATSIP.2021.4

original paper

Audio-to-score singing transcription based on a
CRNN-HSMM hybrid modelryo nishikimi,1eita nakamura,1,2

masataka goto3

and kazuyoshi yoshii1,4This paper describes an automatic singing transcription (AST) method that estimates a human-readable musical score of asung melody from an input music signal. Because of the considerable pitch and temporal variation of a singing voice, a naivecascading approach that estimates an F0 contour and quantizes it with estimated tatum times cannot avoid many pitch andrhythm errors. To solve this problem, we formulate a unified generative model of a music signal that consists of a semi-Markov
language model representing the generative process of latent musical notes conditioned on musical keys and an acoustic model
based on a convolutional recurrent neural network (CRNN) representing the generative process of an observed music signal fromthe notes. The resulting CRNN-HSMM hybrid model enables us to estimate the most-likely musical notes from a music signal
with the Viterbi algorithm, while leveraging both the grammatical knowledge about musical notes and the expressive power of
the CRNN. The experimental results showed that the proposed method outperformed the conventional state-of-the-art method
and the integration of the musical language model with the acoustic model has a positive effect on the AST performance.

Keywords: Automatic singing transcription, Convolutional recurrent neural network, Hidden semi-Markov model
Received 25 November 2020; Revised 4 March 2021

I.

INTRODUCTION

The aim of automatic singing transcription (AST) is to estimate a human-readable musical score of singing voice froma given music signal. Since the melody line is usually themost salient part of music that influences the impressionof a song, transcribed scores are useful for music information retrieval (MIR) tasks such as query-by-humming,musical grammar analysis [1], and singing voice generation [2]. In this paper, we study statistical audio-to-score(wave-to-MusicXML) AST for audio recordings of popular music consisting of monophonic singing voice andaccompaniment sounds (Fig. 1).
To estimate the semitone-level pitches and tatum-level
onset and offset times of musical notes from music signals, one may estimate a singing F0 trajectory [3–6] andthen quantize it on the semitone and tatum grids obtainedby a beat-tracking method [7], where the tatum (e.g. 16thnote level) refers to the smallest meaningful subdivision of
the main beat (e.g. fourth-note level). This approach, however, has no mechanism that avoids out-of-scale pitches
1

Graduate School of Informatics, Kyoto University, Kyoto 606-8501, JapanThe Hakubi Center for Advanced Research, Kyoto University, Kyoto 606-8501, Japan
3
National Institute of Advanced Industrial Science and Technology (AIST), Ibaraki
305-8568, Japan
4
PRESTO, Japan Science and Technology Agency, Tokyo, 102-0076, Japan
2

Corresponding author:
Ryo Nishikimi
Email: nishikimi@sap.ist.i.kyoto-u.ac.jp

and irregular rhythms caused by the considerable pitch andtemporal variation of the singing voice.
An effective way of overcoming this problem is to use amusical language model that incorporates prior knowledge
about symbolic musical scores. Graphical models [8–11]
have been proposed for integrating such a language modelwith an acoustic model describing the generative processof acoustic features or F0s. In particular, the current stateof-the-art method of audio-to-score AST [11] is based on ahidden semi-Markov model (HSMM) consisting of a semiMarkov language model describing the generative processof a note sequence and a Cauchy acoustic model describing
the generative process of an F0 contour from the musicalnotes. The semi-Markov model (SMM) is an extension of
the Markov model that can explicitly represent the duration probability of each hidden state (e.g. note). While
being more accurate than other methods, the output scores
include errors caused by the preceding F0 estimation step,and repeated notes of the same pitch cannot be detectedfrom only F0 information. An alternative approach to AST
is to use an end-to-end DNN framework to directly convert a sequence of acoustic features into a sequence ofmusical symbols. At present, however, this approach covers
only constrained conditions (e.g. the use of synthetic soundsignals) and has only limited success [12–15].
To solve this problem, we propose an AST method that
integrates a language model with a DNN-based acoustic model. This approach can utilize both the statisticalknowledge about music notes and the capacity of DNNs
1

2

ryo nishikimi, et al.

Fig. 1. The problem of automatic singing transcription. The proposed method takes as input a spectrogram of a target music signal and tatum times and estimatesa musical score of a sung melody.

for describing complex data distributions of input musicsignals. This is known as the hybrid approach, which has
been one of the major approaches to automatic speech
recognition (ASR) [16]. To our knowledge, the hybrid
approach has not been attempted for audio-to-score AST
in the literature. The language model describing the generative process of local keys, note pitches, and onset times is
implemented with a SMM. The acoustic model describing
the generative process of a music audio signal from musicalnotes is implemented with a convolutional recurrent neuralnetwork (CRNN) estimating the pitch and onset probabilities for each tatum. Since the accuracy of beat tracking isalready high [7], we assume that beat and downbeat times(i.e. tatum times and their relative metrical positions inmeasures) are estimated in advance. In this paper, we focus
on typical popular songs with 4/4 time. We also investigate
how the application of singing voice separation for an input
signal affects the transcription results.
The main contributions of this study are as follows.
We propose the first DNN-HMM-type hybrid model foraudio-to-score AST. The key difference from the HSMMbased method [11] is that the acoustic model can directly
describe complex data distributions of music signals byleveraging the potential of the CRNN. Despite the active
research on AST-related tasks like singing voice separation and F0 estimation, a full AST system that can output
musical scores in a human-readable form has scarcely beenstudied. Our system can deal with polyphonic music signals and output symbolic musical scores in the MusicXML
format. We found that the proposed method outperformed
the HSMM-based method [11] by a large margin. We also
confirmed that the language model significantly improves
the AST performance, especially in the rhythm aspects.
Finally, we found that the application of singing voice separation to the input music signals can further improve theperformance1 .
The rest of this paper is as follows: Section II explains
backgrounds for the proposed method. Section III describes

1 We respect the reproducibility of research and are now working forallowing anyone to easily test our technique on arbitrary songs. The sourcecode is available upon request.

our approach to AST. Section IV reports the experimental
results. Section V concludes the paper.
II.

BACKGROUNDS

Before describing the proposed method in the next section,
we here explain the backgrounds by reviewing previous
studies. Input signal representations have been studied
for music information processing, including the shorttime Fourier transform (STFT) [17, 18], the constant-Q
transform (CQT) [6], and the log Mel-scale filter-bank[19]. Recently, the harmonic CQT (HCQT) representation, which is obtained by stacking pitch-shifted (upshiftedand downshifted) CQT spectrograms, has been proposed[3]. This representation was designed to better capture thestructure of harmonic partials in music audio signals. Since
the HCQT representation is considered to be especiallyeffective for extracting pitch features, we use a similar input
representation in our method.
Markov models have widely been used for musical language modeling. To characterize the musical scales, forexample, the statistical characteristics of pitch transitions
can be learned from musical scores transposed to the C
major key [20]. In automatic music transcription, musicalkeys are often treated as latent variables instead of referring
to key annotations [11]. As to musical rhythms, SMMs suchas the duration-based Markov model [21] and the metrical
Markov model [22, 23] have been proposed. The latter canbe used for effectively regularizing the metrical structuresof the estimated scores from the rhythmic viewpoint [24].
In this study, we construct a Markov model of latent note
pitches conditioned by latent musical keys and that of latent
note positions.
III.

PROPOSED METHOD

We specify the audio-to-score AST problem in Section
III-A) and describe the proposed generative modelingapproach to this problem in Section III-B). We formulate
the CRNN-HSMM model in Sections III-C) and III-E). We
explain how to train the model parameters in Section III-F)
and the transcription algorithm in Section III-G).

audio-to-score singing transcription based on a crnn-hsmm hybrid modelFig. 2. The proposed hierarchical probabilistic model that consists of a SMM-based language model representing the generative process of musical notes from local
keys and a CRNN-based acoustic model representing the generative process of an observed spectrogram from the musical notes. We aim to infer the latent notes
and keys from the observed spectrogram.

A) Problem specification
We formulate the audio-to-score AST problem under two
simplified but practical conditions; (1) The time signatureof a target song is 4/4; and (2) the tatum times of thesong, which form the 16th note-level grids, are estimated inadvance (e.g. [7]).
The input data consist of the audio spectrogram of atarget song with tatum times and their relative metrical
positions in measures. Similarly to the HCQT representation [3], the audio spectrogram is obtained by stacking H
pitch-shifted (upshifted and downshifted) versions of a logfrequency magnitude spectrogram obtained by warping thelinear frequency axis of the STFT spectrogram into the logfrequency axis. Thus, the input audio spectrogram can be
represented as a tensor X ∈ RH×F×T , where H, F, and T represent the number of channels, that of frequency bins, andthat of time frames, respectively.
We use the notation “i:j ” to represent a sequence of integer indices from i to j. The tatum times can be represented
by a sequence of frame indices t1:N+1 , where n s label tatums,
N is the number of tatums in the input song, and tN+1
indicates a “sentinel” frame of the song. By trimming off
unimportant frames before the first tatum and after the last
tatum if necessary, we can assume that t1 = 1 and tN+1 =T + 1. The relative position of tatum n in a measure is called
the metrical position and denoted by ln ∈ {1, . . . , L} (L = 16
is the number of tatums in each measure); ln = 1 means thattatum n is a downbeat (the first tatum of a measure). In
general, we have ln+1 − ln ≡ 1 (mod L); however, we do notassume l1 = 1. We use the symbol m ∈ {1, . . . , M} to labelmeasures (M is the number of measures).
The output of the proposed method is a sequence ofmusical notes represented by pitches p1:J and onset (score)time in tatum units n1:J+1 . We use the symbol j to labelmusical notes, and J represents the number of estimatedmusical notes. The pitch pj of the j th note takes a valuein {0, 1, . . . , K}; pj = 0 means that it is a rest and pj > 0means that it is a pitched note (K be the number of uniquesemitone-level pitches considered). The onset time nj of thej th note takes a value in {1, . . . , N + 1} and, for convenience,
we assume that n1 = 1 and nJ+1 = N + 1. The (J + 1) th note
onset is introduced only for defining the length of the J thnote and is not used in the output transcribed score.

For musical language modeling, we introduce musicalkey variables, which are also estimated in the transcription
process. To allow modulations (key changes) within a song,
we introduce a local key sm for each measure m. Each variable sm takes a value in {1 = C, 2 = C, . . . , 12 = B}. Since
similar musical scales are used in relative major and minor
keys, they are not distinguished here. For example, sm = 0
means that measure m is in the C major key or the A minor
key.

B) Generative modeling approach
We propose a generative modeling approach to the audioto-score AST problem (Fig. 2). We formulate a hierarchicalgenerative model of the local keys S = s1:M , the pitches P =p1:J and onset times N = n1:J+1 of the musical notes, and thespectrogram X asp(X, P, N, S) = p(X|P, N)p(P, N, S).

(1)Here, all the probabilities are implicitly dependent on thetatum information t1:N+1 and l1:N+1 . p(P, N, S) represents a
language model that describes the generative process of themusical notes and keys. p(X|P, N) represents an acoustic
model that describes the generative process of the spectrogram given the musical notes.
Given the generative model, the transcription problem
can be formulated as a statistical inference problem of estimating the musical scores (P, N) and the keys S that maximize the left-hand side of equation (1) for the given spectrogram X (as explained later). In this step, the acoustic model
evaluates the fitness of a musical score to the spectrogram
while the language model evaluates the prior probability ofthe musical score. The proposed method is therefore consistent with our intuition that both of these viewpoints are
essential for transcription.

C) Language model
We construct a generative model where the pitches P = p1:Jand the onset times N = n1:J+1 are independently generated and the pitches are generated depending on the local

3

4

ryo nishikimi, et al.

keys S = s1:M . The generative process can be mathematically
expressed asp(P, N, S) = p(P|S)p(N)p(S),(2)

where p(P|S), p(N), and p(S) represent the pitch transition model, the onset time transition model, and the key
transition model, respectively.
In the key transition model, to represent the sequential
dependency between the keys of consecutive measures, thekeys S = s1:M are generated by a Markov model asp(S) = p(s1 )Mp(sm |sm−1 ).

p(s1 = s) = πsini ,(4)p(sm = s | sm−1 = s ) = π(s−s ) mod 12+1 ,(5)where we have assumed that the transition probabilities are
symmetric under transpositions. For example, the transition probability from C major to D major is assumed tobe the same as that from D major to E major. We define
π ini = (πsini ), π = (πs ) ∈ R12≥0 .
In the pitch transition model, to represent the dependency of adjacent pitches and the dependency of pitcheson the local keys, the pitches P = p1:J+1 are generated by aMarkov model conditioned on keys S = s1:M asp(pj |pj−1 , sm(j) ),

(6)

j=2

where m(j) indicates the measure to which the j th note
onset belongs. The initial and transition probabilities areparameterized asini,
p(p1 = p | s1 = s) = φsp(7)p(pj = p | pj−1 = p , sm(j) = s) = φsp p .

(8)

We assume that these probabilities are key-transpositioninvariant so that the following relations hold:ini
ini∝ φ̄deg(s,p)
,
φsp

(9)

φspp ∝ φ̄deg(s,p)deg(s,p ) ,

(10)

where

p(nj |nj−1 ),(12)where the initial and transition probabilities are given byp(n1 ) = δ1,n1 ,(13)p(nj = n | nj−1 = n ) = ψln ln .

(14)Here, δ denotes the Kronecker’s symbol and the firstequation expresses the assumption n1 = 1. In the second
equation, ψ = (ψl l ) ∈ RL×L
≥0 represents the transition probabilities between metrical positions.

D) Tatum-level language model formulation
In the language model presented in Section III-C), the transitions of keys and transitions of pitches and onset times
are not synchronized. To enable the integration with theacoustic model and the inference for AST, we here formulate an equivalent language model where the variables aredefined at the tatum level. For this purpose, we introduce
tatum-level key variables s̄n , pitch variables p̄n , and counter
variables c̄n (Fig. 3). The first two sets of variables are constructed from the keys s1:M and the pitches p1:J so that s̄n =sm when tatum n is in measure m and p̄n = pj when tatum
n satisfies nj ≤ n < nj+1 . The counter variable c̄n represents
the residual duration of the current musical note in tatum
units and takes a value in {1, . . . , 2L}, where 2L is the maximum length of a musical note. This variable is gradually
decremented tatum by tatum until the next note begins; anote onset at tatum n is indicated by c̄n−1 = 1. In this way,
we can construct variables S = s̄1:N , P = p̄1:N , and C = c̄1:Nfrom variables S = s1:M , P = p1:J , and N = n1:J+1 , and viceversa.
The generative models for the tatum-level keys, pitches,and counters can be derived from the language model in
Section III-C) as follows. The keys S = s̄1:N obey the following Markov model:p(S̄) = p(s̄1 )Np(s̄n |s̄n−1 ),

(15)

n=2

where


deg(s, p) =

J+1

j=2

(3)

The initial and transition probabilities are parameterized asJp(N) = p(n1 )

m=2

p(P|S) = p(p1 |s1 )In the onset time transition model, to represent therhythmic patterns of musical notes, the onset times N =n1:J+1 are generated by the metrical Markov model [22, 23]
as(p − s) mod 12 + 1
0(p > 0),(p = 0)(11)

represents the degree (key-relative pitch class) of pitch
p in key s (e.g. deg(s, p) = 1 corresponds to C on theini
C major scale). We define φ̄ = (φ̄dini ) ∈ R13≥0 and φ̄ =13×13
(φ̄dd ) ∈ R≥0 .

p(s̄1 ) = πs̄ini,
1π(s̄n −s̄n−1 ) mod 12+1p(s̄n |s̄n−1 ) =δs̄n−1 ,s̄n(16)(ln = 1),(ln > 1).

(17)

The second equation says that a key transition occurs onlyat the beginning of a measure.

audio-to-score singing transcription based on a crnn-hsmm hybrid modelFig. 3. Representation of a melody note sequence and variables of the language model.

The counters C = c̄1:N obey the following Markov model:p(C) = p(c̄1 )Np(c̄n |c̄n−1 ),

(18)

n=2

where
p(c̄1 = c̄) = ψl1 l1+c̄ ,

(19)



p(c̄n = c̄ | c̄n−1 = c̄ ) =ψln ln+c̄δc̄ −1,c̄(c̄ = 1),(c̄ > 1).

(20)

p(P|S, C) = p(p̄1 |s̄1 )p(p̄n |p̄n−1 , c̄n−1 , s̄n ),

(21)

n=2

where
,
p(p̄1 |s̄1 ) = φs̄ini
1 p̄1p(p̄n |p̄n−1 , c̄n−1 , s̄n ) =

(22)



φs̄n p̄n−1 p̄nδp̄n−1 p̄n(c̄n−1 = 1),(c̄n−1 > 1).

(23)

The second equation expresses the constraint that a pitchtransition occurs only at a note onset.
Putting equations (15), (18), and (21) together, we havep(P, N, S) = p(P, C, S) = p(P|S, C)p(C)p(S).

We formulate an acoustic model p(X|P, N) = p(X|P, C)
that gives the probability of spectrogram X given a pitch
sequence P and a counter sequence C representing onset
times. We define the tatum-level spectra Xn as a segmentof spectrogram X in the span of tatum n. As in the standard HMM, we assume the conditional independence of theprobabilities of tatum-level spectra asp(X|P, C) =Np(Xn |p̄n , c̄n−1 ).

(25)

n=1

This is a kind of SMM called the residential-time Markov
model [25]. As shown in Fig. 3, at the onset tatums of musical notes, the counter variables change to the corresponding
note values. Otherwise, the counter variables are decremented by one. The former case is represented by ψln ln+c̄ andthe latter case is represented by δc̄ −1,c̄ in equation (20).
The pitches P = p̄1:N obey the following Markov model
conditioned on the keys and counters:NE) Acoustic model(24)

That is, the language model in Section III-C) and the tatumlevel language model defined here are equivalent probabilistic models. We use this tatum-level SMM in what follows.

Using Bayes’ theorem, the individual factors in the righthand side of equation (25) can be written asp(Xn |p̄n , c̄n−1 ) =∝p(p̄n , c̄n−1 |Xn )p(Xn )p(p̄n , c̄n−1 )(26)p(p̄n , c̄n−1 |Xn ),p(p̄n , c̄n−1 )(27)

where p(p̄n , c̄n−1 ) is the prior probability of pitch p̄n andcounter c̄n−1 and p(p̄n , c̄n−1 |Xn ) is the posterior probabilityof p̄n and c̄n−1 .
We use a CRNN for estimating the probability p(p̄n ,c̄n−1 |Xn ) (Fig. 4). Since it is considered to be difficult to
directly estimate the counter variables describing the durations of musical notes from the locally observed quantity
Xn , we train the CRNN to predict the probability whether anote onset occurs at each tatum. A similar DNN for joint
estimation of pitch and onset probabilities has been successfully applied to piano transcription [19]. For reliable
estimation, we estimate the pitch and onset probabilities
independently. Therefore, the CRNN takes the spectra Xn
as input and outputs the following probabilities:ξnk = p(p̄n = k|Xn ),(28)

ζn = p(ōn = 1|Xn ),(29)

where ōn ∈ {0, 1} is an onset flag and ζn is the (posterior)onset probability at tatum n (ōn = 1 if there is a note onset
at tatum n and ōn = 0 otherwise). The counter probabilities5

6

ryo nishikimi, et al.

Fig. 4. The acoustic model p(X|P, C) representing the generative process of the spectrogram X from note pitches P and residual durations C.

are assigned using the onset probability as(c̄n−1 = 1),ζnp(c̄n−1 |Xn ) =(1 − ζn )/(L − 1) (c̄n−1 = 1),

F) Training model parameters
ini

(30)

and the probability p(p̄n , c̄n−1 |Xn ) is then given as the product of equations (28) and (30).
In practice, it is reasonable to use spectra of a longer segment than a tatum as input of the CRNN since each tatum(16th note) spans a short time interval. In addition, it is computationally efficient to jointly estimate the pitch and onset
probabilities of all tatums in the wide segment. Therefore,
we use the whole spectrogram X (or its part of a sufficientduration) as the input of the CRNN and train it so as to
output all the tatum-level pitch and onset probabilities.
The CRNN consists of a frame-level CNN and a tatumlevel RNN linked through a max-pooling layer (Fig. 4). The
CNN extracts latent features e1:T (et = [et1 , . . . , etF ] ∈ RF )from the spectrogram X of length T:
e1:T = CNN(X).

(31)

Using the tatum times t1:N+1 , the max-pooling layer summarizes the frame-level features e1:T into the tatum-levelfeatures ē1:N (ēn = [ēn1 , . . . , ēnF ] ∈ RF ) asēnf = max etf .
tn ≤t<tn+1(32)

The RNN then converts the tatum-level features ē1:N into
intermediate features g1:N (gn ∈ RD is a D-dimensional
vector) through a bidirectional long short-term memory(BLSTM) layer and predicts the pitch and onset probabilities ξ n = (ξnk ) ∈ RK+1 and ζn through softmax and sigmoid
layers as follows:
g1:N = BLSTM(ē1:N ),ξ n = Softmax Wp gn + bp ,ζn = Sigmoid Wo gn + bo ,The parameters ψ, φ̄ , φ̄, π ini , and π of the language
model are learned from training data of musical scores. The
metrical transition probabilities ψll are estimated asψll ∝ max(all − κ, 0) + ,(36)

where all is the number of transitions from metrical positions l to l appear in the training data, κ is a discountparameter, and is a small value to avoid the zero count.
ini
The initial and transition probabilities of pitches (φ̄ andφ̄) are estimated in the same way, by using the key signatures. Although the initial and transition probabilities of
local keys (π ini and π ) can be trained in the same way inprinciple, a large amount of musical scores are necessary forreliable estimation since modulations are rare. Therefore, inthis study, we manually set π ini to the uniform distributionand π to [0.9, 0.1/11, . . . , 0.1/11] such that the self-transition
probability π1 has a large value.
The parameters of the CRNN are trained by using paired
data of audio spectrograms and corresponding musicalscores. After converting the pitches and onset times into
the form P = p̄1:N and O = ō1:N , we apply the following
cross-entropy loss functions to train the CRNN:

Lpitch = −

N 
K


δp̄n ,k log ξnk ,(37)

n=1 k=0

Lonset = −

N




ōn log ζn + (1−ōn ) log (1−ζn ) .

(38)

n=1

(33)

G) Transcription algorithm(34)

We can derive a transcription algorithm based on the constructed generative model. Using the tatum-level formulation, equation (1) can be rewritten as(35)

where Wp ∈ R(K+1)×D and Wo ∈ R1×D are weight matrices,and bp ∈ RK+1 and bo ∈ R are bias parameters.

p(X, S, P, C) = p(X|P, C)p(P, C, S),(39)

audio-to-score singing transcription based on a crnn-hsmm hybrid model

where the first factor on the right-hand side is givenby the CRNN as in equation (25) and the second factor by the SMM as in equation (24). Therefore, the integrated generative model is a CRNN-HSMM hybrid model.
The most likely musical score can be estimated from theobserved spectrogram X by maximizing the probabilityp(S, P, C|X) ∝ p(X, S, P, C), where we have used Bayes’ for∗mula. In equation, we estimate the optimal keys S = s̄∗1:N ,∗∗∗pitches P = p̄∗1:N , and counters C = c̄1:Nat the tatum levelsuch that∗∗∗

S , P , C = argmax p(X, S, P, C).

(40)

The most likely pitches P∗ = p∗1:J and onset times N∗ = n∗1:J∗∗of musical notes can be obtained from P and C . The
number of notes J is determined in this inference process.

1) Viterbi Algorithm
We can use the Viterbi algorithm to solve equation (40).
step, Viterbi variables ωn (qn ), where qn =In the forwards̄n , p̄n , c̄n , are recursively calculated as follows:ω1 (q1 ) =p(p̄1 |X1 )βξ p(c̄0 = 1|X1 )βζp(p̄1 , c̄0 = 1)βχ× p(s̄1 )βπ p(c1 )βψ p(p̄1 |s̄1 )βφ ,βξp(p̄n |Xn ) p(c̄n−1 |Xn )ωn (qn ) = max
qn−1
p(p̄n , c̄n−1 )βχ

(41)
βζ

× p(s̄n |s̄n−1 )βπ p(c̄n |c̄n−1 )βψ× p(p̄n |p̄n−1 , c̄n−1 , s̄n )βφ ,(42)where c̄0 = 1 was formally introduced in the initialization.
In the above equations, we have introduced weighting factors βπ , βφ , βψ , βξ , βζ , and βχ to balance the language
model and the acoustic model. In the recursive calculation,qn−1 that maximizes the max operation is memorized asprev(qn ).
In the backward step, the optimal variables q∗1:N are recursively obtained as follows:q∗N = argmax ωN (qN ),(43)

q∗n = prev(q∗n+1 ).

(44)

qN

2) Refinements
Musical scores estimated by the CRNN-HSMM tend to havelong durations because the accumulative multiplication ofpitch and onset time transition probabilities decreases theposterior probability. This is known as a general problemof the HSMM [11]. To ameliorate this situation, we penalize
long notes by multiplying each of equations (41) and (42) by
the following penalty term:βexp(1/c̄n ) η (c̄n−1 = 1),(45)
f (c̄n−1 , c̄n ) =1(c̄n−1 = 1),
where βη ≥ 0 is a weighting factor.

To save the computational costs of the Viterbi algorithm
defined in the large product space qn = {s̄n , p̄n , c̄n } without
sacrifice of its global optimality, we limit the pitch space tobe searched as follows:
n+1

p̄n ∈

top3 (ξn p ),(46)n =n−1 0≤p≤Kwhere top30≤p≤K (ξnp ) represents the set of the indices p thatprovide the three largest elements in {ξn0 , . . . , ξnK }.

IV.

EVALUATION

We report comparative experiments conducted for evaluating the proposed AST method. We compared the proposed
method with existing methods and examined the effectiveness of the language model (Section IV-C)). We then investigated the AST performance of the proposed method formusic and singing signals with different complexities andexamined the influence of the beat tracking performance on
the AST performance (Section IV-D)).

A) Data
We used 61 popular songs with reliable melody annotations [26] from the RWC Music Database [27]. We split thedata into a training dataset (37 songs), a validation dataset(12 songs), and a test dataset (12 songs), where the singersof these datasets are disjoint. We also used 20 synthesized
singing signals obtained by a singing synthesis software
called CeVIO [28]; 12, 4, and 4 signals are added to thetraining, validation, and test datasets, respectively.
To augment the training data for the acoustic model, we
added the separated singing signals obtained by Spleeter[29] and the clean isolated singing signals. To cover a wide
range of pitches and tempos, we pitch-shifted the original music signals by L semitones (−12 ≤ L ≤ 12) and randomly time-stretched each of those signals with a ratio of R
(0.5 ≤ R < 1.5). The total number of songs in the training
data was 37 × 25 × 3 (real) + 49 × 25 (synthetic) = 4000.
Since the initial and transition probabilities of pitches
are key-transposition-invariant, the pitch shifting does notaffect the training of those probabilities. Therefore, we did
not apply data augmentations for training the language
model.
For each signal sampled at 22.05 kHz, we used a STFT
with a Hann window of 2048 points and a shifting intervalof 256 points (11.6 ms) for calculating the amplitude spectrogram on the logarithmic frequency axis having 5 bins
per semitone (i.e. 1 bin per 20 cents) between 32.7 Hz (C1)and 2093 Hz (C7) [30]. We then computed the HCQT-like
spectrogram X by stacking the h-harmonic-shifted versionsof the original spectrogram, where h ∈ {1/2, 1, 2, 3, 4, 5} (i.e.
H = 6), and the lowest and highest frequencies of the hharmonic-shifted spectrogram are h × 32.7 Hz and h ×
2093 Hz, respectively.

7

8

ryo nishikimi, et al.

Fig. 5. Architecture of the CNN. Three numbers in the parentheses in each layer
indicate the channel size, height, and width of the kernel.

B) Setup
Inspired by the CNN proposed for frame-level melody F0
estimation [3], the frame-level CNN of the acoustic model(Fig. 5) was designed to have six convolution layers with theoutput channels of 128, 64, 64, 64, 8, and 1 and the kernel
sizes of (5, 5), (5, 5), (3, 3), (3, 3), (70, 3), and (1, 1), respectively, where the instance normalization [31] and the Mishfunction [32] are used. The output dimension of the tatumlevel BLSTM was set to D = 130 × 2. The vocabulary ofpitches consisted of a rest and 128 semitone-level pitches
specified by the MIDI note numbers (K = 128).
To optimize the proposed CRNN, we used RAdam withthe parameters α = 0.001 (learning rate), β1 = 0.9, β2 =0.999, and ε = 10−8 . A weight decay (L2 regularization)with a hyperparameter 10−5 and a gradient clipping with athreshold of 5.0 were used for training. The weight parameters Wp and Wo were initialized to random values between−0.1 and 0.1. The kernel filters of the frame-level CNN
and the weight parameters of the tatum-level BLSTM were
initialized by He’s method [33]. All bias parameters were
initialized with zero.
Because of the limited memory capacity, we split the
spectrogram of each song into 80-tatum segments. The
CRNN’s outputs of those segments were concatenated forthe note estimation based on the Viterbi decoding. The
weighting factors βπ , βφ , βψ , βξ , βζ , and βη were optimized for the validation data by using Optuna [34]. Consequently, βπ = 0.541, βφ = 0.769, βψ = 0.619, βξ = 0.917,βζ = 0.852, and βη = 0.609. We manually set the weighting factor βχ to 0 based on the results of preliminary
experiments. The discounting value κ and small value inequation (36) were set to 0.7 and 0.1, respectively.
The accuracy of estimated musical notes was measured
with the edit-distance-based metrics proposed in [24] consisting of pitch error rate Ep , missing note rate Em , extra noterate Ee , onset error rate Eon , offset error rate Eoff , and overall(average) error rate Eall .

proposed CRNN-HSMM hybrid model, the HHSMMbased method [11], and the majority-vote method. The
majority-vote method quantizes an input F0 trajectory insemitone units, then determines tatum-level pitches by taking the majority of the quantized pitches at each tatum.
Since the majority-vote method does not estimate note
onsets, we concatenated successive tatums with the same
pitch to obtain a single musical note. The HHSMM-based
method does not estimate rests because it is difficult tomodel the unvoiced frames in an F0 contour. To obtain restsfrom the musical score estimated by the HHSMM-based
method, we removed the estimated musical notes if theunvoiced frames occupied 90 and more of each musicalnote.
To examine the effect of the language model, we also run
a method using only the CRNN as follows:p∗n = argmax ξnp ,

(47)

1≤p≤K


o∗n =

0 (ζn < 0.5),
1 (ζn ≥ 0.5).

(48)

To construct a musical score from the predicted symbols p∗nand o∗n , we applied the following rules:(i) If p∗n−1 = p∗n , then the (n − 1) th and n th tatums areincluded in different notes.
(ii) If p∗n−1 = p∗n and o∗n = 1, then the (n − 1) th and n thtatums are included in different notes having the same
pitch.
(iii) If p∗n−1 = p∗n and o∗n = 0, then the (n − 1) th and n thtatums are included in the same notes.

C) Method comparison

To evaluate the methods in a realistic situation, only the
mixture signals and the separated signals were used as test
data, and the tatum times were estimated by [7].
Results are shown in Table 1. For both the mixture andseparated signals, the proposed method and the CRNN
method outperformed the majority-vote method and the
HHSMM-based method in the overall error rate Eall by large
margins. This result confirms the effectiveness of using the
CRNN as the acoustic model. Comparing the Eall metrics
for the proposed method and the CRNN method, there wasa decrease of 2.33 percentage points (PP) for the mixture
signals and 1.62 PP for the separated signals. This result
indicates the positive effect of the language model. Especially, the significant decreases of the Eon and Eoff metrics
indicate that the language model is particularly effective
for reducing rhythm errors. The proposed method andthe CRNN method achieved better performances for the
separated signals than the mixture signals.
Transcription examples obtained by the different models are shown in Fig. 6 2 . The musical score estimated by
the majority-vote method, which did not use a language
model, had many errors. In the musical score estimatedby the HHSMM-based method, whereas most notes had
pitches on the musical scale, repeated note onsets with the

To confirm the AST performance of the proposed method,
we compared the transcription results obtained by the

2 Other examples are available in the accompanying webpage: http://
sap.ist.i.kyoto-u.ac.jp/members/nishikimi/demo/apsipa-tsip-2020/

audio-to-score singing transcription based on a crnn-hsmm hybrid model
Table 1. The AST performances of the different methods.
MethodCRNN-HSMM (proposed)
”
CRNN
”
HHSMM
”
Majority vote”

SignalF0

TatumsEp ()Em ()

Ee ()

Eon ()Eoff ()Eall ()mixtureseparated [29]mixtureseparated [29]mixtureseparated [29]mixtureseparated [29]

–

[7]

”
–

”
[7]

”
[5]

”
[7]

”
[5]

”
[7]

”

”

8.34
7.85
7.81
8.56
8.74
9.81
20.52
20.55

13.50
9.63
12.07
8.60
33.07
31.80
7.02
7.69

13.70
14.45
18.00
17.57
16.96
15.79
32.23
32.90

24.45
22.46
29.35
31.12
53.02
52.68
58.46
59.38

23.06
21.64
28.05
27.21
33.29
31.79
49.36
50.63

16.61
15.21
19.06
18.61
29.02
28.38
33.52
34.23

Fig. 6. Examples of musical scores estimated by the proposed method, the CRNN method, the HHSMM-based method, and the majority-vote method from theseparated audio signals and the estimated F0 contours and tatum times. Transcription errors are indicated by the red squares. Capital letters attached to the redsquares represent the following error types: pitch error (P), rhythm error (R), deletion error (D), and insertion error (I). Error labels are not shown in the transcription
result by the majority-vote method, which contains too many errors.

same pitches were not detected. In the result by the CRNN
method, which did not use a language model, we can seethat most pitches are on the musical scale unlike in the result
by the majority-vote method. This indicates the capacityof the CRNN that some sequential constraints on musicalnotes can be learned by the RNN. However, there were someerrors in rhythms, which suggests the difficulty of learning rhythmic constraints by a simple RNN. Finally, in the
result by the proposed CRNN-HSMM method, there weremuch fewer rhythm errors than the CRNN method, which
demonstrates the effect of the language model. The transition probabilities φ̄ and ψ are shown in Fig. 7. Figure 7(a)shows that the transitions to the seven pitch classes on the C
major scale tend to occur frequently. Figure 7(b) shows thatthe transitions to the 8th-note-level metrical positions tend
to occur frequently.
The end-to-end approaches to AST based on sequenceto-sequence (seq2seq) learning have been studied [12, 13].
The RNN-based method [12] and the CTC-based method[13] achieved low error rates (e.g. P(sub) = 0.006 and
Ep = 0.99) for synthetic signals. Similarly, as shown in
Table 2, the proposed method also achieved low error rates(e.g. Ep = 0.42) for synthetic singing voices. Note thatthese methods were not evaluated on the same real data weused.

D) Influences of voice separation and beattracking methods
A voice separation method [29] and a beat-tracking method[7] are used in the preprocessing step of the proposed

method, and errors made in this step can propagate to thefinal transcription results. Here, we investigate the influences of those methods used in the preprocessing step.
We used the ground-truth tatum times obtained fromthe beat annotations [26] to examine the influence ofthe beat-tracking method. We used the isolated signals
for the songs in the test data to examine the influenceof the voice separation method. In addition, as a reference, we also evaluated the proposed method with thesynthetic singing voices. When tatum times were estimated by the beat-tracking method [7] for the real signals, the mixture signals are used as input and the results
are used for the mixture, separated, and isolated signals. Since the synthesized signals are not synchronized tothe mixture signals, the beat-tracking method is directly
applied to the vocal signals to obtain estimated tatum
times.
Results are shown in Table 2. As for the influence of thebeat-tracking method, using the ground-truth tatum times
decreased the overall error rate Eall by 1.1 PP for the separated signals and 1.3 PP for the isolated signals. This result
indicates that the influence of the beat-tracking method issmall for the data used. As for the influence of the voice
separation method, in both conditions with estimated andground-truth tatum times, Eall for the isolated signals wereapproximately 3 PP smaller than that for the separated
signals. This result indicates that further refinements on
the voice separation method can improve the transcription
results by the proposed method.
Although the singing voice separation had both thepositive and negative impacts, as a whole, it improved9

10

ryo nishikimi, et al.

Fig. 7. The transition probabilities φ̄ and ψ trained from the existing musical scores. The triangles indicate (a) the seven pitch classes on the C major scale and (b)the eighth-note-level metrical positions.

Table 2. The AST performances obtained from the different input data.
MethodCRNN-HSMM

SignalTatumsEp ()Em ()

Ee ()

Eon ()Eoff ()Eall ()mixtureseparated [29]isolatedsynthesized
mixtureseparated [29]isolatedsynthesized

ground-truth
”
”
”
[7]
”
”
”

7.30
7.89
6.68
0.00
8.34
7.85
6.83
0.42

13.81
8.42
6.64
0.10
13.50
9.63
6.55
2.87

14.76
13.18
8.56
0.47
13.70
14.45
10.31
1.52

24.46
21.81
16.81
0.34
24.45
22.46
19.17
11.25

22.80
20.24
16.47
1.69
23.06
21.64
18.79
6.97

16.62
14.31
11.03
0.52
16.61
15.21
12.33
4.60

Fig. 8. Examples of musical scores obtained with and without singing voice separation when the ground-truth tatum times were used. The left and right figures
illustrate the positive and negative impacts of singing voice separation.

the transcription performances in most metrics. Especially, the missing note rates were decreased by 5.4 PP
and 3.9 PP when the ground-truth and estimated tatum
times were used, respectively. However, the pitch error
rates and the extra note rates were increased when the
ground-truth and estimated tatum times were used, respectively. In addition, the performance gain obtained for the
separated signals was smaller than that for the isolated
signals. Figure 8 shows transcription examples obtained
for mixture and separated signals with the ground-truth
tatum times. In the left figure, the extra notes were eliminated successfully by suppressing the accompanimentsounds. In the right figure, in contrast, the residual accompaniment sounds were wrongly recognized as melody
notes.

Finally, in both conditions with estimated and groundtruth tatum times, the transcription error rates for the synthesized signals were significantly smaller than those forthe real signals. This result confirms that the difficulty of
AST originates from the pitch and timing deviations in sung
melodies. The relatively large onset and offset error rates forthe case of using the estimated tatum times are due to thedifficulty of beat tracking for the signals containing only asinging voice.

E) Discussion
Our results provide an important insight that a simple RNN
has a weak effect in capturing the rhythmic structure andthe language model that explicitly incorporates a rhythm

audio-to-score singing transcription based on a crnn-hsmm hybrid model

model plays a significant role in improving the transcription
results. Whereas musical pitches can be inferred from local
acoustic features, in order to recognize musical rhythms, itis necessary to look at durations or intervals of onset times,
which have extended structures in time. This non-local
feature of rhythms characterizes the difficulty of music transcription, which cannot be solved by simply applying DNN
techniques used for other tasks such as ASR. This result may
also explain why end-to-end methods that were successful
at ASR have not been so successful at music transcription[12–15]. For example, the paper [13] reports low error rates
for monophonic transcription, but the method was only
applied to synthetic data without timing deviations.
To simplify the AST task, we imposed the following
restrictions on target songs in this study: the tatum unit
(minimal resolution of a beat) is a 16th-note length and themeter of a target song is 4/4 time. Theoretically, we can relax
these restrictions by modifying the language model andextend the present method for more general target songs. To
include shorter note lengths and triplets, we can introduce
a shorter tatum unit, for example, a tatum corresponding to
one-third of a 32nd note. To transcribe songs in other meterssuch as 3/4 time, we can construct one metrical Markov
model for each meter and estimate the meter of a given
song by the maximum likelihood estimation [24]. Althoughmost beat-tracking methods such as [7] assumes a constantmeter for each song, popular music songs often have mixed
meters (e.g. an insertion of a measure in 2/4 time), whichcalls for a more general rhythm model. A possible solution
is to introduce latent variables representing meters (one foreach measure) into the language model and estimate thevariables in the transcription step.
The language model based on the first-order Markov
model was used in this study and it is possible to apply
more refined language models. A simple direction is to use
higher-order Markov models or a neural language model
based on RNN. While most language models try to capture local sequential dependence of symbols, using a model
incorporating a global repetitive structure is effective formusic transcription. To incorporate the repetitive structure
in a computationally tractable way, it is considered to beeffective to use a Bayesian language model [35].
Another important direction for refining the method
would be to integrate the voice separation and/or the beattracking with the musical note estimation method. A voice
separation method and a beat-tracking method are usedin the preprocessing step in the present method, and we
observed that errors made in the preprocessing step can
propagate to the transcription results. To mitigate the problem, multi-task learning of the singing voice separation and
the AST can also be effective in obtaining the singing voices
appropriate for the AST [5]. A beat-tracking method typically estimates beat times in the accompaniment sounds,
which can be slightly shifted from the onset times of thesinging voice due to the asynchrony between the vocal andthe other parts [36]. Therefore, it would be effective for AST
to jointly estimate musical notes and tatum times that matchthe onset times of singing voices.

V.

CONCLUSION

This paper presented an audio-to-score AST method basedon a CRNN-HSMM hybrid model that integrates a language model with a DNN-based acoustic model. The proposed method outperformed the majority-vote method andthe previously state-of-the-art HHSMM-based method. We
also found that the language model has a positive effecton improving the AST performance, especially in rhythmic
aspects.
The proposed approach of integrating the SMM-based
language model with the DNN-based acoustic model isa general framework that can be applied to other tasksof music transcription such as chord estimation, music
structure analysis, and instrumental music transcription.
It would be interesting to investigate how the proposed
method works on genres other than popular music. Anotherinteresting possibility is to integrate language models[37, 38] and acoustic models [19, 39] that can deal withchords for polyphonic piano transcription. Eventually,
based on the proposed framework, we aim to build a unified audio-to-score transcription system that can estimate
musical scores of multiple parts of popular music.
FINANCIAL SUPPORTThis work was partially supported by JSPS KAKENHI Nos.
16H01744, 19H04137, 19K20340, 19J15255, and 20K21813,
and JST ACCEL No. JPMJAC1602 and PRESTO No.
JPMJPR20CB.
CONFLICT OF INTERESTNone.
REFERENCES[1] Hamanaka, M.; Hirata, K.; Tojo, S.: deepGTTM-III: simultaneous
learning of grouping and metrical structures, in Int. Symp. on Computer Music Multidisciplinary Research, Matosinhos, 2017.
[2] Blaauw, M.; Bonada, J.: A neural parametric singing synthesizer modeling timbre and expression from natural songs. Appl. Sci., 7 (2017),
1313.
[3] Bittner, R.M.; McFee, B.; Salamon, J.; Li, P.; Bello, J.P.: Deep salience
representations for f0 estimation in polyphonic music, in Int. Society
for Music Information Retrieval Conf., Suzhou, 2017.
[4] Kim, J.W.; Salamon, J.; Li, P.; Bello, J.P.: CREPE: a convolutional representation for pitch estimation, in Int. Conf. on Acoustics, Speech, andSignal Processing, Calgary, 2018.
[5] Nakano, T.; Yoshii, K.; Wu, Y.; Nishikimi, R.; Edward Lin, K.W.; Goto,
M.: Joint singing pitch estimation and voice separation based on aneural harmonic structure renderer, in IEEE Workshop on Applications of Signal Processing to Audio and Acoustics, New Paltz, 2019.
[6] Gfeller, B.; Frank, C.; Roblek, D.; Sharifi, M.; Tagliasacchi, M.;
Velimirović, M.: SPICE: self-supervised pitch estimation. IEEE/ACM.
Trans. Audio. Speech. Lang. Process., 28 (2020), 1118–1128.
[7] Böck, S.; Korzeniowski, F.; Schlüter, J.; Krebs, F.; Widmer, G.: madmom: a new Python audio and music signal processing library, in
ACM Int. Conf. on Multimedia, Amsterdam, 2016.

11

12

ryo nishikimi, et al.
[8] Kapanci, E.; Pfeffer, A.: Signal-to-score music transcription usinggraphical models, in Int. Joint Conf. on Artificial Intelligence, Edinburgh, 2005.

[27] Goto, M.; Hashiguchi, H.; Nishimura, T.; Oka, R.: RWC Music
Database: popular, classical and jazz music databases, in Int. Conf. on
Music Information Retrieval, Paris, 2002.

[9] Raphael, C.: A graphical model for recognizing sung melodies, in Int.
Conf. on Music Information Retrieval, London, 2005.

[28] CeVIO.: http://cevio.jp.

[10] Ryynänen, M.P.; Klapuri, A.P.: Automatic transcription of melody,
bass line, and chords in polyphonic music. Comput. Music J., 32(2008), 72–86.
[11] Nishikimi, R.; Nakamura, E.; Goto, M.; Itoyama, K.; Yoshii, K.:
Bayesian singing transcription based on a hierarchical generativemodel of keys, musical notes, and f0 trajectories. IEEE/ACM. Trans.
Audio Speech Lang. Process., 28 (2020), 1678–1691.
[12] Carvalho, R.G.C.; Smaragdis, P.: Towards end-to-end polyphonicmusic transcription: transforming music audio directly to a score,
in IEEE Workshop on Applications of Signal Processing to Audio andAcoustics, New Paltz, 2017.
[13] Román, M.A.; Pertusa, A.; Calvo-Zaragoza, J.: An end-to-end
framework for audio-to-score music transcription on monophonicexcerpts, in Int. Society for Music Information Retrieval Conf., Paris,
2018.
[14] Nishikimi, R.; Nakamura, E.; Fukayama, S.; Goto, M.; Yoshii, K.:
Automatic singing transcription based on encoder-decoder recurrent
neural networks with a weakly-supervised attention mechanism, inInt. Conf. on Acoustics, Speech, and Signal Processing, Brighton, 2019.
[15] Nishikimi, R.; Nakamura, E.; Goto, M.; Yoshii, K.: End-to-end
melody note transcription based on a beat-synchronous attention
mechanism, in IEEE Workshop on Applications of Signal Processing
to Audio and Acoustics, New Paltz, 2019.

[29] Hennequin, R.; Khlif, A.; Voituret, F.; Moussallam, M.: Spleeter: a fastand efficient music source separation tool with pre-trained models. J.
Open Source Softw., 5 (2020), 2154.
[30] Cheuk, K.W.; Anderson, H.; Agres, K.; Herremans, D.: nnAudio: anon-the-fly GPU audio to spectrogram conversion toolbox using 1D
convolutional neural networks. IEEE Access, 8 (2020), 161981–162003.
[31] Ulyanov, D.; Vedaldi, A.; Lempitsky, V.: Instance normalization:
the missing ingredient for fast stylization, in arXiv preprint
arXiv:1607.08022, 2017.
[32] Misra, D.: Mish: a self regularized non-monotonic neural activation
function, in arXiv preprint arXiv:1908.08681, 2019.
[33] He, K.; Zhang, X.; Ren, S.; Sun, J.: Delving deep into rectifiers: surpassing human-level performance on imagenet classification, in IEEE
Int. Conf. on Computer Vision, Santiago, 2015.
[34] Akiba, T.; Sano, S.; Yanase, T.; Ohta, T.; Koyama, M.: Optuna: anext-generation hyperparameter optimization framework, in ACM
SIGKDD Int. Conf. on Knowledge Discovery & Data Mining, Anchorage, 2019.
[35] Nakamura, E.; Itoyama, K.; Yoshii, K.: Rhythm transcription of midi
performances based on hierarchical bayesian modelling of repetition and modification of musical note patterns, in European Signal
Processing Conf., Budapest, 2016.

[16] Dahl, G.E.; Yu, D.; Deng, L.; Acero, A.: Context-dependent pretrained deep neural networks for large-vocabulary speech recognition. IEEE Trans. Audio. Speech. Lang. Process., 20 (2012), 30–42.

[36] Nishikimi, R.; Nakamura, E.; Goto, M.; Itoyama, K.; Yoshii, K.:
Scale- and rhythm-aware musical note estimation for vocal F0 trajectories based on a semi-tatum-synchronous hierarchical hidden
semi-markov model, in Int. Society for Music Information Retrieval
Conf., Suzhou, 2017.

[17] Jansson, A.; Humphrey, E.; Montecchio, N.; Bittner, R.; Kumar,
A.; Weyde, T.: Singing voice separation with deep u-net convolutional networks, in Int. Society for Music Information Retrieval Conf.,
Suzhou, 2017.

[37] Ojima, Y.; Nakamura, E.; Itoyama, K.; Yoshii, K.: Chord-aware automatic music transcription based on hierarchical Bayesian integrationof acoustic and language models, APSIPA Trans. Signal Inf. Process.,
7 (2018), e14.

[18] Kelz, R.; Dorfer, M.; Korzeniowski, F.; Böck, S.; Arzt, A.; Widmer, G.:
On the potential of simple framewise approaches to piano transcription, in Int. Society for Music Information Retrieval Conf., 2016.

[38] Nakamura, E.; Yoshii, K.: Statistical piano reduction controlling performance difficulty, APSIPA Trans. Signal Inf. Process, 7 (2018), e13.

[19] Hawthorne, C., et al.: Onsets and frames: dual-objective piano transcription, in Int. Society for Music Information Retrieval Conf., 2018.
[20] Brooks, F.P.; Hopkins, A.L.; Neumann, P.G.; Wright, W.V.: An experiment in musical composition. IRE Trans. Electron. Comput., EC-6
(1957), 175–182.
[21] Takeda, H.; Saito, N.; Otsuki, T.; Nakai, M.; Shimodaira, H.;
Sagayama, S.: Hidden markov model for automatic transcription of
MIDI signals, in IEEE Workshop on Multimedia Signal Processing,
2002.
[22] Raphael, C.: A hybrid graphical model for rhythmic parsing. Artif.
Intell., 137 (2002), 217–238.
[23] Hamanaka, M.; Goto, M.; Asoh, H.; Otsu, N.: A learning-based quantization: unsupervised estimation of the model parameters, in Int.
Conf. on Multimodal Interfaces, Vancouver, 2003.
[24] Nakamura, E.; Yoshii, K.; Sagayama, S.: Rhythm transcription ofpolyphonic piano music based on merged-output HMM for multiple voices. IEEE/ACM. Trans. Audio Speech Lang. Process., 25 (2017),
794–806.
[25] Yu, S.-Z.: Hidden semi-Markov models. Artif. Intell., 174 (2010),
215–243.
[26] Goto, M.: AIST annotation for the RWC Music Database., in Int. Conf.
on Music Information Retrieval, Victoria, 2006.

[39] Kim, J.W.; Bello, J.P.: Adversarial learning for improved onsets andframes music transcription, in Int. Society for Music Information
Retrieval Conf., Delft, 2019.

Ryo Nishikimi received the B.E. and M.S. degrees from Kyoto
University, Kyoto, Japan, in 2016 and 2018, respectively. He iscurrently working toward the Ph.D. degree in Kyoto University,and has been a Research Fellow of the Japan Society for the Promotion of Science (DC2). His research interests include music
informatics and machine learning. He is a student member ofIEEE and IPSJ.
Eita Nakamura received a Ph.D. degree in physics from the
University of Tokyo in 2012. He has been a post-doctoral
researcher at the National Institute of Informatics, Meiji University, and Kyoto University. He is currently an AssistantProfessor at the Hakubi Center for Advanced Research andGraduate School of Informatics, Kyoto University. His researchinterests include music modelling and analysis, music information processing and statistical machine learning.
Masataka Goto received the Doctor of Engineering degreefrom Waseda University in 1998. He is currently a Prime SeniorResearcher at the National Institute of Advanced Industrialaudio-to-score singing transcription based on a crnn-hsmm hybrid model

Science and Technology (AIST), Japan. Over the past 28 years
he has published more than 270 papers in refereed journals andinternational conferences and has received 51 awards, including
several best paper awards, best presentation awards, the TenthJapan Academy Medal, and the Tenth JSPS PRIZE. In 2016, as
the Research Director he began OngaACCEL Project, a 5-year
JST-funded research project (ACCEL) on music technologies.
Kazuyoshi Yoshii received the M.S. and Ph.D. degrees in
informatics from Kyoto University, Kyoto, Japan, in 2005 and2008, respectively. He is an Associate Professor at the Graduate School of Informatics, Kyoto University, and concurrently
the Leader of the Sound Scene Understanding Team, Center for Advanced Intelligence Project (AIP), RIKEN, Tokyo,
Japan, and the Researcher of PRESTO, Japan Science andTechnolocy Agency (JST), Tokyo, Japan. His research interests include music informatics, audio signal processing, andstatistical machine learning.

13