IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. XX, NO. XX, XX 20191

Bayesian Singing Transcription
Based on a Hierarchical Generative Modelof Keys, Musical Notes, and F0 Trajectories
Ryo Nishikimi, Eita Nakamura, Masataka Goto, Katsutoshi Itoyama, Member, IEEE
and Kazuyoshi Yoshii Member, IEEE

Abstract— This paper describes automatic singing transcription (AST) that estimates a human-readable musical score of asung melody represented with quantized pitches and durations
from a given music audio signal. To achieve the goal, we propose a statistical method for estimating the musical score byquantizing a trajectory of vocal fundamental frequencies (F0s)in the time and frequency directions. Since vocal F0 trajectories
considerably deviate from the pitches and onset times of musicalnotes specified in musical scores, the local keys and rhythms
of musical notes should be taken into account. In this paper
we propose a Bayesian hierarchical hidden semi-Markov model(HHSMM) that integrates a musical score model describing the
local keys and rhythms of musical notes with an F0 trajectorymodel describing the temporal and frequency deviations of an F0
trajectory. Given an F0 trajectory, a sequence of musical notes,that of local keys, and the temporal and frequency deviations
can be estimated jointly by using a Markov chain Monte Carlo(MCMC) method. We investigated the effect of each componentof the proposed model and showed that the musical score model
improves the performance of AST.

*OQVU
NVTJDBVEJPTJHOBM
7PDBM'USBKFDUPSZ

1SFQSPDFTTJOHGPSFYUSBDUJOHGFBUVSFT

FH<>

5BUVNUJNFT
FH<>

0VUQVUNVTJDBMTDPSF

5JNFGSFRVFODZRVBOUJ[BUJPO QSPQPTFENFUIPE

Fig. 1. The problem of automatic singing transcription. The proposed method
takes as input a vocal F0 trajectory and tatum times, and estimates a sequenceof musical notes by quantizing the F0 trajectory in the time and frequency
directions.

fundamental frequencies (F0s) in both time and frequency
directions. Since a lot of effort has been devoted to estimating
Index Terms—Automatic singing transcription, hierarchical hid- vocal F0 trajectories from music audio signals (see Section II),
we focus on AST from vocal F0 trajectories.
den semi-Markov model
One of the major difficulties of AST is that continuous F0
trajectories include temporal and frequency deviations fromI. I NTRODUCTION
straight pitch trajectories indicated in scores. This prohibits a
Automatic singing transcription (AST) refers to estimating simple quantization method (called majority-vote method) thata musical score of a sung melody from a music audio signal estimates a pitch as the majority of F0s in each tatum interval.
(Fig. 1). It forms the basis of music information retrieval A promising way to obtain a natural score is to integrate a(MIR) because a melody is the most salient part of popular musical score model that describes the organization of notes inmusic that affects the impression of a musical piece. Generally, scores with an F0 trajectory model representing the temporala sung melody is represented as a sequence of musical notes and frequency deviations. This framework is similar to thewith pitches quantized in semitones and onset and offset times statistical speech recognition approach based on a language
quantized in certain time units (tatums). To realize AST, one model and an acoustic model [1]. Recent studies have applied
needs to quantize continuous physical quantities such as vocal musical score models for music transcription in the frameworkof probabilistic modeling [2], [3] and deep learning [4], [5].
Manuscript received XXXX XX, 2019; revised XXXX XX, 2019; accepted
XXXX XX, 2019. Date of publication XXXX XX, 2019; date of current
To build a musical score model, we focus on how pitches
version XXXX XX, 2019. This work was partially supported by JSPS
and rhythms of musical notes are structured in a sung melody.
KAKENHI Grant Numbers No. 19H04137, No. 19K20340, No. 16H01744,
In tonal music, pitches have sequential interdependence andNo. 16J05486, No. 19J15255, and No. 19K12017, JST ACCEL No. JPMJAC1602, and Kayamoni Foundation. The associate editor coordinating theare controlled by underlying musical keys or scales. Onset
review of this manuscript and approving it for publication is Prof. XXX YYYtimes in scores also have sequential interdependence and are(Corresponding author: Ryo Nishikimi).
controlled by underlying metrical structure. To represent such
R. Nishikimi, E. Nakamura, and K. Yoshii are with Graduate School ofInformatics, Kyoto University, Kyoto 606-8501, Japan, email: {nishikimi,
characteristics, it is necessary to formulate a musical scoreenakamura, yoshii}@sap.ist.i.kyoto-u.ac.jp. E. Nakamura is also with themodel in the musical-note level, instead of in time-frame level
Hakubi Center for Advanced Research, Kyoto University, Kyoto 606-8501,[4], [5]. On the other hand, a vocal F0 trajectory is repreJapan.
K. Itoyama is with School of Engineering, Tokyo Institute of Technology, sented in the time-frame level, or possibly in the tatum level
Tokyo 152-8552, Japan, email: itoyama@ra.sc.e.titech.ac.jp.
after applying beat tracking. Because of the mismatch of time
M. Goto is with National Institute of Advanced Industrial Science andscales, integration of a note-level musical score model and a
Technology (AIST), Ibaraki 305-8568, Japan, email: m.goto@aist.go.jp.
Digital Object Identifier XXXX
frame- or tatum-level F0/acoustic model poses a challenge inIEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. XX, NO. XX, XX 2019probabilistic modeling, which is still open [6].
For key- and rhythm-aware AST, we previously proposeda hierarchical hidden semi-Markov model (HHSMM) [7] thatconsists of a musical score model and an F0 trajectory model
under an condition that the tatum times are given in advance or
estimated by a beat detection method [8] (Fig. 2). The musicalscore model generates a note sequence and consists of threesub-models describing local keys, pitches, and onset score
times (Section III-B). The local keys are sequentially generated
by a Markov model and the pitches of musical notes are thensequentially generated by another Markov model conditionedon the local keys. The onset score times are sequentially generated by a metrical Markov model [9], [10] defined on thetatum grid. The F0 trajectory model describes the temporal andfrequency deviations added to a step-function-like pitch trajectory corresponding to the generated score (Section III-C). To
stably learn the musical characteristics unique to each musicalpiece from a small amount of piece-specific data, the HHSMM
is formulated in a Bayesian manner (Section III-D).
To estimate a latent sequence of musical notes with decent
durations from an observed vocal F0 trajectory by using the
HHSMM, in this paper we propose a combination of an iterative Gibbs sampler and a modified Viterbi algorithm that is
penalized for intensely favoring longer notes with less frequenttransitions (Section IV-C). The whole model can be estimatedin an unsupervised or semi-supervised manner (Sections IV-A
and IV-B) by optimizing on the fly or pretraining the musicalscore model, respectively. Since putting more emphasis on themusical score model was shown to be effective in our previous
work [7], in this paper we carefully optimize the weighting
factors on the individual components of the musical score andF0 trajectory models and the note duration penalization with
Bayesian optimization [11] or grid search (Section V-A2).
The main contributions of this study are as follows. First,
we provide a full description of the HHSMM (Section III)
that is used for transcribing a human-readable score consisting of quantized pitches and onset times from a music audio
signal (monophonic F0 trajectory) via the improved learning
methods (Section IV). This is a principled statistical approachto a well-known open problem of how to integrate a notelevel language model with a tatum- or frame-level acoustic
model in automatic music transcription. Second, we foundthat the rhythm and key models of the musical score modeland the note duration penalization were particularly effective,by conducting comprehensive comparative experiments for investigating the performances of the unsupervised and semisupervised learning methods (Section V-B1) and evaluating themusical score model (Section V-B2), the F0 trajectory model(Section V-B3), and the note duration penalization (Section
V-B4).
Section II introduces related work on AST. Sections III and
IV describe our statistical approach to AST (generative modeling and posterior inference). Section V reports the results ofcomparative experiments. Section VI summarizes the paper.
II. R ELATED W ORK
A typical approach to AST consists of three steps: melody
extraction, piano-roll estimation, and rhythm transcription. The.VTJDBMTDPSFNPEFM-PDBMLFZT

NFBTVSF$NBKPS'USBKFDUPSZNPEFMQJUDIFTBOEPOTFU UJNFT1$NBKPSNFBTVSF

2" NJOPS

5FNQPSBMEFWJBUJPOT5FNQPSBMMZEFWJBUFEOPUFT
7PDBM'USBKFDUPSZNFBTVSF

%FQFOEPOLFZT.VTJDBMOPUFT

'SFR

2

'SFRVFODZEFWJBUJPOT
5JNF

Fig. 2. The generative process of a vocal F0 trajectory based on the proposedmodel consisting of a musical score model and an F0 trajectory model.
The musical score model represents the generative process of musical notes(pitches and onset score times) based on local keys assigned to measures. In
the figure of musical notes, the black vertical lines represent a tatum grid
given as input. The F0 trajectory model represents the generative process ofa vocal F0 trajectory from the musical notes by adding the frequency andtemporal deviations. In the figure of temporally deviated notes, the arrows
represent temporal deviations of onset times from tatum times.

melody extraction refers to estimating predominant F0 trajectories from music audio signals. The piano-roll estimationrefers to converting the vocal F0s to a piano-roll representation, i.e., a sequence of note events represented by onset
times, durations, and quantized pitches. While the pitches takediscrete values, the onset times and durations take continuous
values. To estimate a complete musical score, i.e., a sequenceof musical notes, the rhythm transcription is performed toquantize onset and offset times in units of tatums. Given an
estimated F0 trajectory with tatum positions, the proposed
method simultaneously performs the piano-roll estimation andrhythm transcription by modeling the generative process of a
vocal F0 trajectory.
A. Melody Extraction
Estimation of vocal F0 trajectories for music audio signals,
a.k.a. melody extraction, has been actively studied [12]–[17].
One of the most basic methods is subharmonic summation(SHS) [12], which calculates the sum of the harmonic components of each candidate F0. Ikemiya et al. [13] improved
F0 estimation based on the combination of SHS and robust
principle component analysis (RPCA) [18]. Salamon et al. [14]estimated several F0 contours according to a salience function
and then recursively removed non-vocal contours by focusing
on the singing characteristics. Durrieu et al. [15] extracted themain melody by representing the melody part with a sourcefilter model and accompaniments with a model inspired by
non-negative matrix factorization (NMF). Mauch et al. [17]modified the YIN [16] in a probabilistic manner so that it
would determine multiple candidate F0s.
B. Piano-roll Estimation
Piano-roll estimation has also been studied [19]–[26]. A
straightforward quantization method that takes the majority of
vocal F0s in each tatum interval is often used [19]. Paiva et al.
[20] proposed a cascading method that performs multipitch
estimation, pitch trajectory construction and segmentation, andR. NISHIKIMI et al.: BAYESIAN SINGING TRANSCRIPTION

extraction of musical notes that form a main melody. Laaksonen et al. [22] divided audio signals into segments corresponding to note events by using the boundaries of chords
as input, and then estimated the pitch of each note event byusing a scoring function based on chord and key information.
Ryynänen et al. [23] proposed a hierarchical HMM (HHMM)
that categorizes the F0s of each note region into attack, sustain,
and release states. This model represents the pitch transitionsin an upper-level Markov chain and the transitions between
the vocal fluctuation types in a lower-level Markov chain.
Molina et al. [24] focused on the hysteresis characteristics of
vocal F0s. Yang et al. [25] proposed an HHMM that represents
the generative process of an F0 trajectory in a two-dimensional
phase plane spanned by F0s and their first derivations (∆F0s).
Mauch et al. [26] developed a software called Tony that estimates a vocal F0 trajectory from a music audio signal using
probabilistic YIN (pYIN) [17] and then estimates note eventsusing a modified version of Ryynänen’s method [23].
C. Rhythm Transcription
The goal of rhythm transcription is to quantize onset andoffset times of MIDI-like note events for appropriate score representation [9], [10], [27]–[30]. A major approach to this problem is to formulate an HMM [9], [10], [27]. Takeda et al. [27],for example, proposed a duration-based HMM that represents
note values as latent variables and actual note durations asobserved variables. Other studies have proposed onset-based
HMMs called metrical HMMs that represent the note onset
positions on the tatum grids as latent variables and the actual
onset times as observed variables [9], [10]. The rhythm andonset deviation models used by this method are similar to
those used by our method. Note that both types of HMM-based
methods can be extended for polyphonic rhythm transcription(e.g., for MIDI piano performances). A probabilistic contextfree grammar (PCFG) [28] and a connectionist approach [29]have also been studied for rhythm transcription. It has beenshown that HMMs are currently state of the art [30].
III. G ENERATIVE M ODELINGThis section defines the task of AST (Section III-A) and explains the hierarchical hidden semi-Markov model (HHSMM)
that consists of a musical score model and an F0 trajectorymodel (Fig. 2). The musical score model represents the generative process of sung notes in the tatum level (Section III-B)
and the F0 trajectory model represents the generative process of vocal F0s in the frame level from the note sequence(Section III-C). We introduce prior distributions to complete
Bayesian formulation. This is effective for estimating the reasonable parameters of the proposed model from a small amountof data (Section III-D). We define the meanings of several
terms regarding temporal information as follows:• Onset/offset times and duration: the start/end times andlength of a note represented in the frame level.
• Onset/offset score times and note value: the start/end times
and length of a note represented in the tatum level.
• Tatum position: the relative position of a tatum in a measure including the tatum.

3

4UBSUPGNVTJD/PUFT ‫܈‬'T ‫܆‬

Ͳ ͳ
‫ݖ‬଴

‫ݖ‬ଵ

ʹ

‫ݖ‬ଶ

‫ڮ‬

Ͳ ͳ ʹ ͵ Ͷ ͷ ͸ ͹ ‫ڮ‬‫ڮ‬‫ݖ‬௝ିଵ‫ܰ ڮ‬െ͸&OEPGNVTJD

‫ܬ‬

‫ݖ‬௃

ܰെͶ

‫ܬ‬൅ͳ /PUF݆

ܰെʹ

‫ڮ‬

ܰ

5BUVN݊

'SBNF‫ݐ‬Ͳ ͳ ʹ ͵ Ͷ ͷ ͸ ͹ ͺ ͻ ͳͲ ͳͳ ͳʹ ‫ܶ ڮ‬െͳ ܶ ܶ൅ͳ

Fig. 3. Relationships between different time indices j, n, and t. The upper
figure shows the start and end beat times of each musical note indexed by j.
The dotted lines between the upper and lower figures represent correspondence
between the tatum index n and the frame index t. The lower figure shows
the F0 value of each time frame. The onset of the first note z1 is the startof music and z0 is a supplementary note that is used only for calculating theslanted line representing the transient segment of z1 .

A. Problem Specification
Our problem is formalized as follows (Figs. 1 and 3):
Input:
A frame-level vocal F0 trajectory X = x0:T
and tatum times Y = y0:N = (tn , ln )0:N
Output:A sequence of musical notes Z = z0:J = (pj , oj )0:J
By-product:
A sequence of local keys S = s0:M
where x0:T = {x0 , . . ., xT } etc., and T , N , J, and M indicate
the number of frames, tatums, estimated notes, and measures,
respectively. The time-shifting interval is 10 ms in this study.
xt indicates a log F0 in cents at frame t, where unvoiced
frames are represented as xt = uv. tn indicates a frame corresponding to tatum n, where t0 = 0, t1 = 1, and tN = T + 1.
ln ∈ {1, . . ., L} indicates the tatum position, where L is thenumber of tatums included in a measure (L = 16 in this paper)and ln = 1 indicates the barline. Each note zj is represented
as a pair of a semitone-level pitch pj ∈ {1, . . ., K} and an
onset score time oj ∈ {0, . . ., N }, where K is the numberof unique pitches considered (e.g., K = 88 pitches from A0
to C8), o0 = 0, o1 = 1, oJ+1 = N . We introduce local
keys s0:M for each measure. The local key sm of measure mtakes a value in {C, C#, · · · , B}×{major, minor} (the tonic isrepresented as C=0, C#=1, . . . , B=11, and the local keys arenumbered from 1 to 24). We have introduced supplementary
variables x0 , y0 , z0 , and s0 in order to ease the handling oflatent variables at the beginning of music.
In this paper, we deal with songs in the pop music style.
It is assumed that a target piece is in 4/4 and that tatum
unit is 16th note. Rests, notes shorter than the tatum unit, andtriplets are not considered. Offset score times are not explicitly
modeled, i.e., the offset score time of each note corresponds tothe onset score time of the next note. It is also assumed thatthe maximum distance between successive onset score time(i.e., maximum note value) is L.

4

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. XX, NO. XX, XX 2019

-PDBMLFZT&RT  BOE $NBKPS$

1JUDIFT.VTJDBMOPUFT5BUVN
QPTJUJPOT

1JUDI

2VBTJTUBUJPOBSZ 5SBOTJFOU 2VBTJTUBUJPOBSZ
TFHNFOU
TFHNFOU
TFHNFOU" NJOPS&&RT  BOE %

5BUVNUJNF

$

0OTFU
EFWJBUJPO&R .VTJDBMOPUF

                     

5BUVN

Fig. 4. A musical score model that represents the generative process of local
keys, note pitches, and note onsets. The top row represents the Markov chainof local keys. The second row represents the Markov chain of the note pitches.
The third row represents a sequence of musical notes. The bottom represents
the Markov chain of the onset score times of musical notes. The vertical lines
represent tatum times and the bold ones represent bar lines.

B. Musical Score Model
The musical score model represents the generative processof local keys S and musical notes Z = {P, O}. More specifically, local keys S are sequentially generated by a Markov
model and pitches P are then sequentially generated by another Markov model conditioned on S (Fig. 4). With an independent process, onset score times O are sequentially generated by a metrical Markov model [9], [10]. We henceforthomit to explicitly write the dependency on Y for brevity.
1) Model for Local Keys: To consider the relevance ofadjacent local keys (e.g., the local keys are likely to change
infrequently), the local keys S are assumed to follow a firstorder Markov model as follows:s0 ∼ Categorical(π0 ),sm | sm−1 ∼ Categorical(πsm−1 ),

(1)
(2)

24
where π0 ∈ R24+ and πs ∈ R+ are initial and transition prob-

abilities. We write π = π0:24 . Given the similarities between
keys (e.g., relative transitions from C major would be similar
to those from D major), a hierarchical Dirichlet or PitmanYor language model [31] with a shared prior generating keyspecific priors and distributions would be useful for precise
key modeling.
2) Model for Pitches: The pitches P are assumed to follow
a first-order Markov model conditioned on the local keys S as
follows:
p0 | S ∼ Categorical(ϕs0 ,0 ),(3)pj | pj−1 , S ∼ Categorical(ϕsmj ,pj−1 ),(4)K
where ϕs0 ∈ RK
+ and ϕsp ∈ R+ are initial and transition probabilities for pitches in local key s, and mj denotes a measure
to which the onset of note j belongs. Let ϕ = ϕ1:24,0:K . We
assume that the initial and transition probabilities in different
local keys are related by a circular shift (change of tonic), andϕ are represented as follows:

ϕs0p′ ∝ ϕ̄type(s),0,deg(s,p′ ) ,ϕspp′∝ ϕ̄type(s),deg(s,p),deg(s,p′ ) ,

(5)
(6)

where type(s)∈{major, minor} indicates the type of key s,deg(s, p)∈ {1, . . ., 12} indicates the degree of pitch p in key

4UBSUGSBNF

1JUDI

7PDBM'T

7PDBM'T
5SBOTJFOUEVSBUJPO&OEGSBNF 5JNF

(a) Temporal deviations.VTJDBMOPUF'SFRVFODZEFWJBUJPO(b) Frequency deviations

5JNF

Fig. 5. Temporal and frequency deviations in vocal F0 trajectories. In both
figures, the black vertical lines represent tatum times. In Fig. (a), the blueand green vertical lines represent the start and end frames of the transient
segment of a vocal F0 trajectory. In Fig. (b), the arrows represent the frequency
deviations of a vocal F0 trajectory from the frequency of a musical note.

s (if the tonic of key s is [s], deg(s, p)=(p−[s] mod 12)+1),
12
and ϕ̄r0 ∈ R12+ and ϕ̄rh ∈ R+ indicate initial and transition
probabilities under a local key of type r ∈ {major, minor} andtonic C, where h ∈ {1, . . . , 12} is a subscript representing apitch degree. The proposed method learns only the probabilities of relative pitch degrees ϕ̄ in an unsupervised or semisupervised manner. The probabilities of absolute pitches ϕ arethen obtained by expanding ϕ̄ according to Eqs. (5) and (6)and used for estimating a sequence of musical notes. In otherwords, the same transition probabilities of pitch degrees areused for every octave range, and for pitch transitions beyond
an octave we use the probabilities of the corresponding pitchtransitions within an octave with the same pitch degrees.
3) Model for Rhythms: The onset score times O are assumed to follow a metrical Markov model [9], [10] as follows:
loj | loj−1 ∼ Categorical(λloj−1 ),

(7)

L×L
where λ = λ1:L ∈ R+
denotes transition probabilities fortatum positions, i.e., λl,l′ (l, l′ ∈ {1, . . ., L}) indicates thetransition probability from tatum position l to l′ . We interpretthat if loj−1 < loj the onsets of notes j − 1 and j are in
the same measure and if loj−1 ≥ loj they are in the adjacentmeasures.

C. F0 Trajectory Model
The F0 trajectory model represents the generative processof an F0 trajectory X from a note sequence Z. We consider
both temporal and frequency deviations (Fig. 5).
1) Model for Temporal Deviations: As shown in Fig. 5-(a),vocal F0s corresponding to each note are assumed to have atransient segment (e.g., portamento) and a quasi-stationary segment (e.g., vibrato). The actual onset time of note j, which isdefined as the first frame of the transient segment, can deviate
from the tatum time toj . Let ej ∈ [emin , emax ] be the deviationof the actual onset time from toj , where [emin , emax ] indicates
its range. The onset and offset time deviations at the start andend of the musical piece are fixed to zero (e1 = eJ+1 = 0),
and the onset time deviation of the supplementary note z0 is
also set to zero for convenience (e0 = 0). If ej < 0 (ej > 0),note j begins earlier (later) than toj . Because E = e0:J are

R. NISHIKIMI et al.: BAYESIAN SINGING TRANSCRIPTION

5

considered to be distributed according to a possibly multimodal distribution, in this paper we use a categorical distribution as the most basic distribution of discrete variables asfollows:
ej ∼ Categorical(ϵ),
e−e

4UBSUGSBNFTPGUIFUSBOTJFOUTFHNFOUT&OEGSBNFTPGUIFUSBOTJFOUTFHNFOUT
5BUVNUJNFTPGOPUFPOTFUT

1JUDI<DFOUT>(8)
5FNQPSBMMZEFWJBUFEQJUDIUSBKFDUPSZPGOPUF

+1

where ϵ ∈ R+max min is a set of deviation probabilities.
Let dj ∈ {1, . . ., dmax } be the duration of the transient
segment of note zj , where dmax is the maximum number, andwe set d0 = dJ+1 = 1. For the same reason as that for E, we
use a categorical distribution for D = d0:J as follows:
dj ∼ Categorical(δ),(9)

where δ ∈ Rd+max indicates a set of duration probabilities.
2) Model for Frequency Deviations: As shown in Fig. 5(b), the vocal F0 trajectory X is generated by imparting frequency deviations to a temporally deviated pitch trajectory
determined by the musical notes Z, the onset time deviations
E, and the transient durations D. Since vocal F0s can significantly deviate from score-indicated pitches, X are assumed tofollow Cauchy distributions, which are more robust to outliers
than Gaussian distributions, as follows:
xt | Z, E, D ∼ Cauchy(µt , σ),'SBNF

Fig. 6. Temporally deviated pitch trajectory used as Cauchy location parameters. The blue and green vertical boxes represent the start and end frames of
the transient segment and the grey vertical boxes represent the tatum times ofnote onsets. The red boxes represent the temporally deviated pitch trajectoryof note j and the grey boxes represent the temporally deviated pitch trajectoryof the other notes.

and the three terms of the F0 trajectory model are given by
Eq. (8), Eq. (9), and Eq. (10), respectively.
We put conjugate Dirichlet priors as follows:πs ∼ Dirichlet(γsπ ) (s ∈ {0, . . ., 24}),(r ∈ {major, minor}, h ∈ {0, . . ., 12}),(10)

where µt and σ are the location and scale parameters, respectively. Note that if xt = uv, xt is treated as missing data.
The related studies [6], [7] also used the Cauchy distribution
for frequency deviations as a better choice than the Gaussian
distribution. As shown in Fig. 6, the actual duration of note j isgiven by [toj +ej , toj+1 +ej+1 ) and the reference F0 trajectoryis modeled as a slanted line in the transient segment and ahorizontal line in the quasi-stable segment as follows (Fig. 6):([pj ]−[pj−1 ])(t−(toj +ej ))[pj−1 ] +
dj
(11)
µt =
(t
∈
[t
+
e
,
toj + ej + dj )),
o
j
j


[pj ] (t ∈ [toj + ej + dj , toj+1 + ej+1 )),(13)

ϕ̄rh ∼ Dirichlet(γrϕ̄ )λl ∼ Dirichlet(γ )λ(l ∈ {1, . . ., L}),

(14)
(15)

ϵ ∼ Dirichlet(γ ),(16)δ ∼ Dirichlet(γ ),

(17)

ϵ

δ

emax −emin +1
ϕ̄
12
λ
L
ϵ
,
where γsπ ∈ R24+ , γ r ∈ R+ , γ ∈ R+ , γ ∈ R+
and γ δ ∈ Rd+max are hyperparameters. We put a gamma prioron σ as follows:σ ∼ Gamma(γ0σ , γ1σ ),where γ0σ

(18)and γ1σare the shape and rate parameters of the
gamma distribution, which are also hyperparameters.

IV. P OSTERIOR I NFERENCE
where [pj ] indicates a log F0 [cents] corresponding to a semitoneThis
sectionexplains
posterior inference of the latent varilevel pitch pj . Although F0 transitions between different pitches
ables
and
parameters.
The
proposed HHSMM can be trained
have complicated dynamics in reality, in this paper we invesin
anunsupervisedmannerfrom the vocal F0 trajectory of atigate the feasibility of a simple linear transition model.
target musical piece by sampling the values of the parameters
and latent variables to approximate the posterior distribution
D. Bayesian Formulationof those variables (Section IV-A). The HHSMM can also be
Integrating the musical score model (prior distribution of trained in a semi-supervised manner by using the musical scorethe musical notes Z = {P, O}) with the F0 trajectory model model that is pretrained by using a large amount of musical(likelihood function of Z for the vocal F0s X), we formulate scores and that is expected to learn common musical grammar
an HHSMM with the parameters Θ = {π, ϕ, λ, ϵ, δ, σ} as and improve the musical appropriateness of the transcription
follows:
results. (Section IV-B). The musical notes are finally estimatedas the latent variables that maximize the posterior probabilityp(X, S, P, O, E, D|Θ) = p(S|π)p(P|S, ϕ)p(O|λ)
of them (Section IV-C). To obtain better results, the parameters{z}|Musical score model
are updated simultaneously with the latent variables.
· p(E|ϵ)p(D|δ)p(X|P, O, E, D, σ),|{z}F0 trajectory model
A. Unsupervised Learning(12)
Given an F0 trajectory X as observed data, our goal is towhere the three terms of the musical score model are given compute the posterior distribution p(S, P, O, E, D, Θ|X) of
by Eqs. (1) and (2), Eqs. (3) and (4), and Eq. (7), respectively, the latent variables (the pitches P, onset score times O, onset

6IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. XX, NO. XX, XX 2019deviations E, and transient durations D of musical notes withthe local key S) and the parameters Θ = {π, ϕ, λ, ϵ, δ, σ}.
Since the posterior distribution cannot be computed analytically, we use a Gibbs sampling method with efficient forwardbackward procedures and a Metropolis-Hastings (MH) step.
The initial values of Q = {P, O, E, D} are given by quantizing X on the semitone and tatum grids by the majority votemethod. The initial values of Θ are drawn from Eqs. (13), (14),(15), (16), (17), and (18). Then, the following three steps areiterated until the likelihood is fully maximized.
1) Obtain S from p(S|Q, Θ, X) with forward filteringbackward sampling.
2) Obtain Q from p(Q|S, Θ, X) with forward filteringbackward sampling.
3) Obtain Θ from p(Θ|S, Q, X) with Gibbs sampling andMH sampling.
Detailed algorithms for the steps 1), 2), and 3) are describedin Appendix A.
B. Semi-supervised Learning
An effective way of improving the performance of AST is
to estimate the parameters of the musical score model fromexisting musical scores (monophonic note sequences with key
annotations) in advance. Let Ŝ and Ẑ = {P̂, Ô} denote local
keys, pitches, and onset score times in training data, which
are defined in the same way as S and Z = {P, O} of a target
piece (Section III-B). Given Ŝ and Ẑ, the initial and transition
probabilities of pitch classes ϕ̄ are obtained by normalizing
the count data cϕ̄rh obtained from Ŝ and Ẑ. Similarly, the onset
transition probabilities λ are obtained from Ô. The initial andtransition probabilities of local keys π are not trained because
the key transitions tend to be unique to each musical piece
and are learned in an unsupervised manner. Keeping ϕ̄ and λfixed, the other parameters and latent variables are estimatedfor a target piece in a semi-supervised manner.
C. Posterior Maximization
Our final goal is to obtain the optimal values of S, Q, and Θthat maximize the posterior probability p(S, Q, Θ|X). First,
we choose the best samples of S, Q, and Θ that maximizep(S, Q, Θ|X) in the Gibbs sampling described in IV-A. Then,the following three steps are iterated until convergence.
1) Obtain S that maximizes p(S|Q, Θ, X) with Viterbi
decoding on the upper-level chain of S.
2) Obtain Q that maximizes p(Q|S, Θ, X) with Viterbi
decoding on the lower-level chain of Q.
3) Obtain Θ that maximizes p(Θ|S, Q, X).
We empirically confirmed that a few iterations are sufficient toreach convergence. In the Viterbi algorithm in step 2 above,
weighting factors β ϕ , β λ , β ϵ , β δ , and β χ are introduced in
the forward calculations to balance the individual sub-models.
A penalization term exp[β o /(oj+1 − oj )] for long durations
oj+1 − oj with a weighting factor β o is also introduced in theforward calculations to suppress the frequent occurrence oflong notes. Detailed algorithms for 1), 2), and 3) are describedin Appendix B.

V. E VALUATION
We conducted comparative experiments to evaluate the performance of the proposed method for AST. We investigatedthe effectiveness of the pretraining method in comparison withthe unsupervised and semi-supervised learning methods (Section V-B1) based on the learning configurations described in
Section V-A3. We also examined the contribution of the individual sub-models by ablating each of them (Sections V-B2,
V-B3, and V-B4) based on the model configurations describedin Section V-A4. To confirm the improvement of the performance of the proposed method, we conducted a comparative
experiment using conventional methods (Section V-C1). To
investigate the performance of the overall system that takes
a music signal as input and outputs a musical score, we also
tested the proposed method for F0 trajectories and tatum times
automatically estimated from music signals (Section V-C2).
A. Experimental Conditions
1) Datasets: From the RWC Music Database [32], we used
63 popular songs in 4/4 time and that satisfy the requirements mentioned in Section III-A. We verified the correctnessof ground-truth annotations [33] of musical notes and beattimes. For input vocal F0 trajectories, we used the groundtruth data in most experiments and estimated data obtainedby the method in [13] for some experiments. In both cases,
the ground-truth unvoiced regions were used to eliminate theinfluence of the performance of vocal activity detection (VAD).
Similarly, for the 16th-note-level tatum times, we used theground-truth data in most experiments and estimated data obtained by a neural beat tracking method [8] in some experiments. To prepare ground-truth scores used for evaluating theaccuracy of transcribed notes, we used MIDI files in which
the onset and offset times of each note are adjusted to corresponding ground-truth beat times.
2) Hyperparameters: The Dirichlet priors on the initial key
probabilities π0 , the onset transition probabilities λ, the onset
time deviation probabilities ϵ, and the transient duration probabilities δ given by Eqs. (13), (15), (16), and (17) were set touniform distributions, i.e., γ0π , γ λ , γ ϵ , and γ δ were set to allone vectors. The Dirichlet priors on the key transition probabilities πs (s ∈ {1, . . ., 24}) given by Eq. (13) were set as γsπ =[1, 1, · · · , 100, · · · , 1]T (only the s-th element takes 100) tofavor self-transition. The Dirichlet priors on the initial probabilities of pitch classes ϕ̄r0 given by Eq. (14) and the transition
probabilities of pitch classes ϕ̄rh (r ∈ {major, minor}, h ∈ϕ̄
{1, . . ., 12}) were set as γmajor= [10, 1, 10, 1, 10, 10, 1, 10, 1,
ϕ̄T
10, 1, 10] and γminor = [10, 1, 10, 10, 1, 10, 1, 10, 10, 1, 10, 1]T
to favor the seven pitch classes on the C major and minor
scales, respectively. The gamma prior on σ in Eq. (18) was
set as aσ0 = aσ1 = 1. Assuming that keys tend to change
infrequently, the s-th element of γsπ was set to a large valueϕ̄(100). Because non-diatonic notes are often used, γmajorandϕ̄
γminor were set to smaller values (10). Optimization of these
hyperparameters is left as future work.
For a model M3 in Table I the weighting factors β ϕ , β λ ,and β χ were determined by Bayesian optimization [11] as

R. NISHIKIMI et al.: BAYESIAN SINGING TRANSCRIPTION

7




















1SFWJPVTPOTFUQPTJUJPOTβ ϕ = 18.9, β λ = 49.6, and β χ = 5.1. The weighting factors
β ϵ and β δ were determined by grid search and set as β ϵ = 20.0
and β δ = 10.0. The weighting factor β o of the duration
penalty term was set to β o = 50, which was experimentally
selected from {1, 5, 10, 50, 100, 500, 1000} so that the performances of M3 and M4 were maximized. Since the forwardbackward algorithms (Appendices A-2 and B-2) are definedin a huge product space q̄n = {p̄n , ōn , ēn , d¯n }, the range ofpitches considered was limited as follows:
n+1
∪

{
}
pMaj
− 1, pMaj, pMaj
+1 ,
i
i
i

(19)

i=n−1

where pMaj
is the pitch estimated by the majority vote methodnbetween tatums n−1 and n. The pitch-range constraint might
prevent the proposed method from estimating some correct
notes. However, it is difficult to recover the correct notes froman erroneous F0 trajectory that is far from the ground-truth
pitch sequence. The pitch-range constraint is thus effective
for reducing the computational complexity of the proposed
method without much damaging the performance of note estimation.
3) Learning Configurations: The unsupervised and semisupervised schemes (Section IV-A and Section IV-B) wereused for estimating the initial and transition probabilities ofpitch classes ϕ̄ and the onset transition probabilities λ. In theunsupervised scheme ϕ̄ and/or λ were learned from only the
vocal F0 trajectory of a target song. In the semi-supervised
scheme ϕ̄ and/or λ were estimated as follows:[L1] Pitch transition learning: ϕ̄ were learned from 90 popular songs with no overlapped sung notes except for atarget song in the RWC Music Database [32].
[L2] Onset transition learning: λ were estimated in advancefrom a corpus of rock music [34].
4) Model Configurations: The four main components of the
proposed method, i.e., the local key model (Section III-B1),
rhythm model (Section III-B3), temporal deviation model (Section III-C1), and note duration penalty (Section IV-C) were
evaluated separately. More specifically, we examined the performance of AST when each component was included or notincluded as follows:[T1] Key modeling: When this function was enabled, the key
transition probabilities π were estimated. Otherwise, thenumber of local keys was set to 1, i.e., π0,1 = π1,1 = 1.
[T2] Rhythm modeling: When this function was enabled, theonset transition probabilities λ were estimated. Otherwise, λ were set to the same value, i.e., λl,l′ = 1.
[T3] Temporal deviation modeling: When this function was
enabled, the onset deviations E and the transient durations D were estimated and we set [emin , emax ] = [−5, 5]
and dmax = 10. Otherwise, E and D were not considered, i.e., ej = 0 and dj = 1.
[T4] Note duration penalization: When this function was enabled, the penalty term f (on ) was introduced in theViterbi decoding. Otherwise, f (on ) was not used.

 



         

/FYUPOTFUQPTJUJPOT(a) Transition probabilities

p̄n ∈







         (b) Occurrences of onset positions








         (c) Occurrences of note values

Fig. 7. Pretrained transition probabilities between the 16th-note-level tatum
positions.

5) Evaluation Measures: The performance of AST was evaluated in terms of the frame-level and note-level measures. The
frame-level accuracy, A, is defined as follows:A=

Tcorrect
,
Tgt

(20)

where Tgt is the total number of voiced frames in the beatadjusted MIDI files and Tcorrect is the total number of voiced
frames whose semitone-level pitches were correctly estimated.
The note-level measures including the precision rate P, therecall rate R, and the F-measure F are defined as follows:
P=

Jcorrect
2PR
Jcorrect
, R=
, F=
,
Joutput
Jgt
P +R

(21)

where Jgt is the total number of musical notes in the groundtruth scores, Joutput is the total number of musical notes in thetranscribed scores, and Jcorrect is the total number of correctlyestimated musical notes. We used three criteria for judging thata musical note is correctly estimated [35]:[C1] Pitch, onset, and offset match: The semitone-level pitch
and the onset and offset score times were all estimatedcorrectly.
[C2] Pitch and onset match: The semitone-level pitch and theonset score time were estimated correctly.
[C3] Onset match: The onset score time was estimated correctly.
When the tatum times were estimated with the beat tracking method [8], we used error tolerance for onset and offset
times to absorb slight differences between the estimated andground-truth tatum times. More specifically, an estimated onset
time was judged as correct if it was within 50 ms from theground-truth onset time. An estimated offset time was judged
as correct if it was within 50 ms or 20% of the duration of
the ground-truth note from the ground-truth offset time.

B. Experimental Results
The experimental results obtained with the different learning and model configurations are shown in Table I. In thisexperiment, the ground-truth vocal F0 trajectories and tatum
times were given as input to the proposed method.

8

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. XX, NO. XX, XX 2019

TABLE IP ERFORMANCE OF THE PROPOSED METHOD WITH DIFFERENT LEARNING AND MODEL CONFIGURATIONS .
L1
L2
T1
T2
T3
T4
Pitch trans. Onset trans.
Key
Rhythm Temp. dev. Note dur.
learning
learning
modeling modeling modeling penalization
M1
M2
M3
M4

✓

✓
✓

✓

M5
M6
M7
M8
M9
M10
M11

✓
✓
✓
✓
✓

✓

✓
✓
✓
✓

A

C1
Pitch, onset, offset
P
R
F

C2
Pitch, onset
P
R
F

C3
Onset
R

F

34.6 52.8 39.1
34.6 53.7 39.3
42.3 62.6 51.5
42.0 62.6 51.0

44.6
45.0
56.2
55.8

P

✓
✓
✓
✓

71.4 21.4 16.6
71.5 21.3 16.3
73.5 28.7 24.0
73.6 28.2 23.3

✓
✓
✓

✓
✓
✓
✓

21.7 19.2 20.2
22.2 19.8 20.8
27.7 22.9 24.9
27.3 23.8 25.3
28.4 26.5 27.3
27.5 26.4 26.8

39.9 35.1
40.8 36.1
46.2 37.6
45.5 39.2
44.6 41.3
43.8 41.9

37.1
38.0
41.2
41.9
42.6
42.6

55.2 48.8
55.0 49.0
62.5 50.8
61.2 52.8
61.3 57.0
60.5 58.1

51.5
51.5
55.6
56.3
58.7
58.9

✓

✓

27.1

44.1

42.0

60.8

58.1

✓

✓

✓
✓

71.7
72.9
72.6
73.5
73.7
73.1

✓

✓

73.1

✓

25.1

18.6 40.7 30.5
18.3 41.0 30.3
26.0 47.1 38.9
25.3 47.1 38.4

25.9

40.6

56.3

TABLE IIP ERFORMANCE OF THE CONVENTIONAL AND PROPOSED METHODS .

Majority-vote method
BS-HMM [36]M3 (proposed method)

A

P

C1
R

F

P

C2
R

F

P

C3
R

F

54.0
64.2
73.5

10.1
9.3
34.7

19.1
9.4
32.4

13.1
9.2
33.3

21.2
23.9
49.6

42.6
23.7
46.1

28.0
23.5
47.4

37.3
37.6
64.5

77.7
37.7
59.9

49.9
37.2
61.6

TABLE IIIP ERFORMANCE OF THE PROPOSED METHOD BASED ON E0 ESTIMATION AND / OR TATUM DETECTION .

M3
M3-1
M3-2
M3-3

F0 trajectory

Tatum times

A

P

C1
R

F

P

C2
R

F

P

C3
R

F

Ground-truth[13]
Ground-truth[13]

Ground-truth
Ground-truth
[8]
[8]

73.5
73.0
71.2
70.7

28.7
28.1
25.7
25.9

24.0
24.4
21.5
22.7

26.0
25.9
23.3
24.1

47.1
46.4
43.5
43.6

38.9
39.7
35.9
37.7

42.3
42.5
39.0
40.1

62.6
61.3
58.3
57.8

51.5
52.5
47.8
49.8

56.2
56.1
52.1
53.2

1) Learning Configurations: We evaluated the effectivenessof the semi-supervised learning strategy by comparing theperformances for the four different learning configurations M1,
M2, M3, and M4 in Table I. The accuracies for M3 (73.5%)
and M4 (73.7%) were better than those for M1 (71.4%) andM2 (71.5%). This indicates that the pretraining of the onset
transition probabilities λ was effective for improving the performance of AST. Examples of λ are shown in Fig. 7. The
probabilities of transitions to the 8th-note-level tatum grids
were larger than those to the other positions (Figs. 7-(a) and 7(b)) and the 8th notes frequently appeared in the date used forthe pretraining (Fig. 7-(c)). This enables the proposed method
to output scores with natural rhythms.
The accuracy for M1 (71.4%) was almost the same as thatfor M2 (71.5%) and that for M3 (73.5%) was almost the sameas that for M4 (73.6%). This indicates that the pretraining of
the transition probabilities of pitch classes ϕ̄ did not much
improve the performance of AST. The precision rates, recall
rates, and F-measures for M2 and M4 were slightly worse thanthose for M1 and M3 especially in terms of C1, respectively.
As shown in Fig. 8, when the transition probabilities of pitchclasses were pretrained, pitches were likely to continue orchange to near pitches. As shown in Figs. 9 and 10, in contrast,
there was no clear bias in the transition probabilities estimatedby the unsupervised learning method or the posterior maximization method because of the effect of the prior distribution.

Estimation errors caused by using the pretrained transition
probabilities are shown in Fig. 11, where the ground-truth key
was D minor in both examples. In the example 1, M3 failed
to estimate the correct key, but succeeded in estimating thecorrect notes on the scale of the estimated key. In contrast,
M4 failed to estimate the correct key and two estimated notes
were out of the scale of the estimated key. M4 also failed
to estimate even the note on the scale of the estimated key
because vocal F0s corresponding to the note were represented
as frequency deviations. In the example 2, M3 succeeded inestimating the correct note on the scale of the incorrectly
estimated key. M4 estimated a key as F major (a relative
major key of D minor), but incorrectly estimated the notebecause vocal F0s corresponding to the note were represented
as frequency deviations. As in these examples, incorrect key
estimation and/or incorrectly represented F0s sometimes led
to incorrect note estimation.
2) Key and Rhythm Modeling: We evaluated the effectiveness of the key and rhythm modeling by comparing the performances for M3, M5, M6, and M7 in Table I. Note that the
pretrained onset transition probabilities λ were not used forM5 and M6 that do not model rhythms. The accuracy for M3(73.6%) was better than those for M5 (71.7%), M6 (72.9%),
and M7 (72.6%). This indicates that the musical score model
including the key and rhythm models was effective for improving the performance of AST. While the accuracy for M6

R. NISHIKIMI et al.: BAYESIAN SINGING TRANSCRIPTION

1SFWJPVTQJUDIDMBTTFT

%
&
'




(
"

#


$

% & ' ( " #$VSSFOUQJUDIDMBTTFTD 7UDQVLWLRQSUREDELOLWLHVEHWZHHQSLWFKFODVVHVLQ&PDMRU



%
&
'



"


$

% & ' ( " #$VSSFOUQJUDIDMBTTFTE 7UDQVLWLRQSUREDELOLWLHVEHWZHHQSLWFKFODVVHVLQ&PLQRU

Fig. 8. Pretrained transition probabilities between the 12 pitch classes underthe major and minor diatonic scales.



1SFWJPVTQJUDIDMBTTFT

%
&
'




(
"

#


$

% & ' ( " #$VSSFOUQJUDIDMBTTFT(a) Transition probabilities between
pitch classes in C major



$



%

1SFWJPVTQJUDIDMBTTFT



$

&
'



.

"NJOPS

"
(
(
'
'
&
"
(
(
'
'
&

5JNF<TFD>&NJOPS

3FQSFTFOUFEBTUIFGSFREFW

0VUPGUIFTDBMFPGUIFFTUJNBUFELFZ

5JNF<TFD>

"


$

% & ' ( " #$VSSFOUQJUDIDMBTTFT(b) Transition probabilities between
pitch classes in C minor

Fig. 9. Transition probabilities between the 12 pitch classes estimated by theunsupervised learning method (Section IV-A).

'BJMVSF
.

(



(



1SFWJPVTQJUDIDMBTTFT

%
&
'



"

#


$

% & ' ( " #$VSSFOUQJUDIDMBTTFT(a) Transition probabilities between
pitch classes in C major



$

%



(



1SFWJPVTQJUDIDMBTTFT



&
'



"

#


$

% & ' ( " #$VSSFOUQJUDIDMBTTFT(b) Transition probabilities between
pitch classes in C minor

Fig. 10. Transition probabilities between the 12 pitch classes estimated by
the posterior maximization method (Section IV-C).

(72.9%) was almost the same as that for M7 (72.6%), theprecision rates, recall rates, and F-measures for M7 were muchbetter than those for M6. This indicates that the rhythm model
was more effective than the key model for AST.
3) Temporal Deviation Modeling: We evaluated the effectiveness of the temporal deviation modeling by comparing theperformances for M3 and M8 in Table I. In most criteria,
the performances for M8 were worse than those for M3. Thiswas mainly due to the large deviations of a vocal F0 trajectorythat were hard to represent precisely. The positive effect of thetemporal deviation modeling was shown in Fig. 12. In thesepieces, M8 successfully represented the transient segments ofvocal F0s. In the left piece, M8 correctly estimated a musicalnote regarded as frequency deviations by M3. The temporaldeviation model of M8, however, often degraded the perfor-&YBNQMF

'

'

&
(

4VDDFTT '
.
'

&

$

'
&
%
%
$
$
#
'
&
%
%
$
$
#

&YBNQMF

"NJOPS

5JNF<TFD>'NBKPS

3FQSFTFOUFEBTUIFGSFREFW"NJOPS'NBKPS

5JNF<TFD>

Fig. 11. Estimation errors caused by using the pretrained initial and transition
probabilities of pitch classes. The pale blue backgrounds indicate the diatonic
scales of estimated keys and the gray boxes indicate ground-truth musicalnotes. The blue and red lines indicate vocal F0s and estimated musical notes,
respectively. The orange dots indicate estimated note onsets. The gray grids
indicate tatum times and semitone-level pitches. The red balloons indicate theground-truth notes that the proposed method failed to estimate. The estimatedkeys are illustrated in the figure, and the ground-truth key in both examples
is D minor.



(
#

.



(
#

&YBNQMF

$

1SFWJPVTQJUDIDMBTTFT



$

9

5JNF<TFD>

5SBOTTFHNFOU

5JNF<TFD>

'
&
%
%
$
$
'
&
%
%
$
$

&YBNQMF

5JNF<TFD>

5SBOTTFHNFOU

5JNF<TFD>

Fig. 12. Positive effects of temporal deviation modeling (cf. Fig. 11). The
green lines estimated F0s with temporal deviations. The red arrows indicate
estimation errors and the green arrows and balloons indicate correct notes
obtained by modeling temporal deviations.

mance, as shown in Fig. 13. In the top pieces, the temporaldeviation model overfit overshoot and preparation in vocal
F0 trajectories. Although we had expected the overshoot andpreparation to be captured as frequency deviations following
the Cauchy distribution, the overfitting, in fact, caused inaccurate pitch estimation. In the bottom-left piece, an extra
note was inserted because the transient segment longer than
the tatum interval was represented by the temporal deviationsof the two successive tatums. In the lower-right piece, theproposed method failed to detect the correct onset because thetransition segment was represented as the temporal deviationof the previous tatum before the correct tatum. Treatment ofsuch slower F0 transitions should be included in future work.
As shown in Fig. 14, the categorical distribution of theonset time deviations E and that of the transient durations
D estimated by the unsupervised learning method (SectionIV-A) were found to have complicated shapes. In future work,
flexible parametric models (e.g., Poisson mixture models) thathave fewer parameters than the categorical distribution could
be used for improving the efficiency of training the distributions of the temporal deviations E and D.
4) Note Duration Penalization: We evaluated the effectiveness of introducing the penalty term f (on ) by comparing the

10

"
(
(
'
'
&
%
%
$
$
#
"

(
'
'
&
%
%
$
$
#

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. XX, NO. XX, XX 2019

&YBNQMF

$
#
"
"
(
(
'
'
&
%

0WFSTIPPU&YBNQMF

5JNF<TFD>

(
'
'
&
%
%
$
$
#
"
"
(

5SBOTTFHNFOU
5JNF<TFD>&YBNQMF

1SFQBSBUJPO&YBNQMF

5JNF<TFD>*ODPSSFDUPOTFUTDPSFUJNF

5JNF<TFD>

1SPCBCJMJUZ

1SPCBCJMJUZ










5JNF<TFD>

3FDPWFSFEOPUF*OTFSUJPOPG
FYUSBPOTFUT

3FDPWFSFEOPUFT

5JNF<TFD>

Fig. 15. Positive and negative effects of duration penalization (cf. Fig. 11).
The green and red balloons indicate improved and deteriorated parts.






.

$
$
#
"
"
(
(
'
'
&
%
%
$

0WFSTIPPU

Fig. 13. Negative effects of temporal deviation modeling (cf. Fig. 12). The
red balloons indicate estimation errors.


.

$
$
#
"
"
(
(
'
'
&
%
%
$



     











0OTFUUJNFEFWJBUJPO<GSBNF>




















 

5SBOTJFOUEVSBUJPO<GSBNF>

Fig. 14. The categorical distribution of the onset time deviations E and thatof the transient durations D estimated in the unsupervised learning method(Section IV-A).

performances for the two different models M3 and M9 inTable I. The accuracy for M9 (73.7%) was slightly better thanthat for M9 (73.5%). In terms of C1, the precision rate forM9 (28.4%) was slightly worse than that for M3 (28.7%),
whereas the recall rate for M9 (26.5%) was better than that forM3 (24.0%). As shown in Fig. 15, M9 successfully estimatedshorter notes that were not detected by M3. On the other hand,
M9 mistakenly split longer notes correctly estimated by M3
into several shorter notes. A possible way of mitigating thistrade-off problem is to extend the proposed model to deal
with both pitch and volume trajectories of singing voice. Thisis expected to improve the performance of onset detection.
We evaluated the effectiveness of jointly using the temporal deviation modeling and the note duration penalization bycomparing the performances for M9 an M10 in Table I. As
described in Section V-B3, the temporal deviation modelingdecreased the accuracy from 73.7% (M9) to 73.1% (M10). We
also checked the effectiveness of introducing both models into
M4. Although the accuracy for M11 and that for M10 weresame, the precision and recall rates and F-measures for M11
were slightly worse than those for M10. This indicates thatthe pretrained probabilities ϕ̄ have little effect when using the
temporal deviation modeling and note duration penalization.
C. Further Investigations
1) Comparison with Conventional Methods: We compared
the performance obtained by M3 with those obtained by thestraightforward majority-vote method and a statistical method
based on a beat-synchronous HMM (BS-HMM) [36]. Since
these conventional methods can only estimate a semitone-level
pitch at each tatum interval, successive tatum intervals wereconcatenated to form a single note if they had the same pitch.
Similarly, successive notes included in the ground-truth data
and those estimated by the proposed method were concate-nated to form a single note if they had the same pitch. To deal
with unvoiced regions in vocal F0 trajectories, the BS-HMM
was extended in the same way as the proposed method. Notethat the majority-vote method can handle the unvoiced regions
and estimate rest notes for those regions.
Experimental results are shown in Table II. The accuracy obtained by the proposed method (73.5%) was better than thoseobtained by the majority-vote method (54.0%) and the BSHMM (64.2%). Considering that the use of the musical scoremodel is the main difference between the proposed method
and the others, we can say that the musical score model iseffective for improving the performance of AST. In terms of
C3, only the recall rate obtained by the majority-vote method(77.7%) was better than those obtained by the other methods
because many extra short notes were obtained.
2) Impacts of F0 Estimation and Tatum Detection: To evaluate the impacts of vocal F0 estimation and tatum time detection on the performance of AST, we tested M3 by using the
ground-truth and estimated data as follows:• F0 trajectory: We used the ground truth data in [33] andestimated data obtained by [13].
• Tatum times: We used the ground truth data in [33] andestimated data detected by [8].
As shown in Table III, the accuracy obtained by M3 (73.5%)
using the ground-truth data was better than those obtained by
M3-1 (73.0%), M3-2 (71.2%), and M3-3 (70.7%) using theestimated data. The impact of tatum detection was larger thanthat of F0 estimation and the F0 estimation slightly degraded
the performance of the proposed method. To avoid such negative effects, we plan to develop a method that directly andjointly estimates a sequence of musical notes with tatum times
from music audio signals.
VI. C ONCLUSION
This paper presented a statistical AST method that estimatesa sequence of musical notes from a vocal F0 trajectory. Our
method is based on an HHSMM that combines a musical scoremodel with a vocal F0 model to represent the hierarchical generative process of a vocal F0 trajectory from a note sequence
determined by musical knowledge. We derived an efficient
Bayesian inference algorithm based on the Gibbs and MH
samplings for solving the inverse problem. The experimental results showed that the key and rhythm models included

R. NISHIKIMI et al.: BAYESIAN SINGING TRANSCRIPTION

11

in the musical score model were effective for improving theperformance of AST. The temporal deviation model was notalways effective because it was often difficult to discriminate
temporal deviations and frequency deviations.
One of future research directions is to extend the proposed
method for directly and jointly estimating musical notes andtatum times from music audio signals. This could be realized
by integrating the proposed method with an acoustic modelthat generates a music spectrogram from a vocal F0 trajectory
in a hierarchical Bayesian manner.
A PPENDIX AD ETAILED A LGORITHM FOR U NSUPERVISED L EARNING
1) Sampling Local Keys: In the forward step, a forward
message α(sm ) is calculated recursively as follows:α(s0 ) = p(p0 , s0 ) = p(p0 |s0 )p(s0 ) = ϕs0 ,0,p0 π0,s0 ,(22)α(sm ) = p(p0:jm+1 −1 , sm )=∑jm+1 −1

sm−1

=

∑

∏

p(sm |sm−1 )α(sm−1 )p(pj |pj−1 , sm )j=jm

7BSJBCMFTJOEFYFECZ݆ ‫ݍ‬௝ = {‫݌‬௝ , ‫݋‬௝ , ݁௝ , ݀௝ }
1JUDI

2
1
‫݋‬଴ = 0 ‫݋‬ଵ = 1‫݌‬଴݁଴ , ݀଴ ݁ଵ , ݀ଵ
‫ݐ‬௢బ
‫ݐ‬௢భ

3

‫݌‬ଵ

4 ˘ ݊െ5 ݊െ4 ݊െ3 ݊െ2 ݊െ1‫݋‬௝ = ݊ െ5‫݋‬௝ାଵ = ݊ െ3

݊ ˘ ܰെ2 ܰെ1‫݋‬௃ = ܰെ2

݁௝ , ݀௝
‫ݐ‬௢మ ˘ ‫ݐ‬௢ೕ݁௃ , ݀௃
‫ݐ‬௢ೕశమ˘ ‫ݐ‬௢಻‫݌‬௝‫݌‬௝ାଵ

݁௝ାଵ , ݀௝ାଵ‫ݐ‬௢ೕశభ

7BSJBCMFTJOEFYFECZ݊ ത‫ݍ‬௡ = {‫݌‬௡ҧ , ‫݋‬௡ҧ , ݁௡ҧ , ݀ҧ௡ }

1JUDI

2
1
‫݋‬ଵҧ = 0 ‫݋‬ସҧ = 1‫݌‬ଵҧ݁ଵҧ , ݀ଵҧ‫ݐ‬௢തభ
‫ݐ‬௢തర

3‫݌‬ସҧ

4 ˘ ݊െ5 ݊െ4 ݊െ3 ݊െ2 ݊െ1‫݋‬ҧ௡ିଷ = ‫݋‬ҧ௢ത೙ ‫݋‬௡ҧ = ݊ െ3
= ݊ െ5
‫݌‬௡ҧ
‫݌‬ҧ
௡ିଷ

݁ସҧ , ݀ҧସ݁௡ିଷҧ , ݀ҧ௡ିଷ‫ݐ‬ସ ˘ ‫ݐ‬௢ത೙షయ
‫ݐ‬௢ത೙

‫݌‬௃

ܰ

5BUVN

'SBNF

‫ݐ‬௢಻శభ݊ ˘ ܰെ2 ܰെ1 ܰ‫݋‬ே
ҧ = ܰെ2
5BUVN

݁௡ҧ , ݀ҧ௡‫ݐ‬௡ ˘ ‫ݐ‬௢തಿ

‫݌‬ே
ҧ

݁ே
ҧ , ݀ҧே 'SBNF‫ݐ‬ே

Fig. 16. A relationship between the variables qj = {pj , oj , ej , dj } andq̄n = {p̄n , ōn , ēn , d¯n }.

The variable q̄n is indexed by tatum n (unlike note j) toenable estimation by a hidden semi-Markov model whereby
the number of notes and the onset score time of each note are
obtained as a result [37].
In the forward step, a forward message ᾱ(q̄n ) is calculated
recursively as follows:

jm+1 −1∏

πsm−1 sm α(sm−1 )sm−1

ϕsm pj−1 pj ,

(23)

j=jm

where jm denotes the index of the first note in measure m.
In the backward step, the local keys S are sampled from a
conditional distribution given byp(S|P) = p(sM |P)M−1∏p(sm |sm+1:M , P).

m=0sM ∼ p(sM |P) ∝ α(sM ),(25)

sm ∼ p(sm |sm+1:M , P) ∝ πsm sm+1 α(sm ).

(26)2) Sampling Note-Level Variables: Given the local keys
S and the F0 trajectory X, we aim to jointly update Q ={P, O, E, D} by using a forward filtering-backward samplingalgorithm on the tatum grids. We define a forward messageᾱ(q̄n ) w.r.t. a tuple q̄n = {p̄n , ōn , ēn , d¯n } (Fig. 16), wherep̄n and ōn indicate the pitch and onset score time of thenote whose offset score time is given by n, and ēn and d¯nrespectively indicate the onset time deviation and transient
duration of the note whose onset score time is given by n.
The onset and offset times of the musical note whose offset
time is n are thus given by tōn + ēōn and tn + ēn − 1. Weformally write the emission probability of F0s in this timespan as follows:tn +ē
n −1∏Cauchy(xt |µt , σ),(27)

t=tōn +ēōnwhere µt is given by the piecewise linear trajectory given by
Eq. (11) as follows:¯[p̄ōn ] + ([p̄n ] − [p̄ōn ])(t − (tōn + ēōn ))/dōn
µt =
(28)
(t ∈ [tōn + ēōn , tōn + ēōn + d¯ōn )),¯[p̄n ] (t ∈ [tōn + ēōn + dōn , tn + ēn )).

(29)

q̄ōn

∑
= p(ēn )p(d¯n ) p(n|ōn )p(p̄n |p̄ōn , sōn )χ(q̄n )ᾱ(q̄ōn )(24)More specifically, local keys s0:M are sampled in the backward
order as follows:χ(q̄n ) =ᾱ(q̄1 ) = p(q̄1 |S, X) = ϕs0 ,0,p̄1 ,ᾱ(q̄n ) = p(x0:tn +ēn −1 , q̄n |S, X)∑=
p(x0:tn +ēn −1 , q̄ōn , q̄n |S, X)= ϵēn δd¯n

∑

q̄ōn

λlōn ,ln ϕsōn ,p̄ōn ,p̄n χ(q̄n )ᾱ(q̄ōn ),

(30)

q̄ōn

where sōn indicates the local key of a measure including thetatum ōn .
In the backward step, the variables Q are sampled from a
conditional distribution given byp(Q|S, X) = p(qJ |S, X)

J−1∏p(qj |qj+1:J , S, X),

(31)

j=0

where qj = {pj , oj , ej , dj } is a tuple of the semitone-level
pitch, onset score time, onset time deviation, and transient
duration of j-th note, respectively (Fig. 16). The variables Q,
however, cannot be sampled directly from Eq. (31) because thenumber of notes J is unknown before sampling notes. Insteadof sampling qj , the variable q̄n is recursively sampled in thereverse order as follows:q̄N ∼ p(q̄N |S, X) ∝ ᾱ(q̄N ),(32)

q̄ōn ∼ p(q̄ōn |q̄n , S, X)∝ ϵēn δd¯n λōōn ,ōn ϕsōn ,p̄ōn ,p̄n χ(q̄n )ᾱ(q̄ōn ).

(33)

As a result of the sampling, J is determined as the number ofthe sampled tuples.
3) Sampling Model Parameters: Given the latent variables
S and Q, the model parameters Θ except for σ are sampled12IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. XX, NO. XX, XX 2019from the conditional posterior distributions as follows:
πs | S ∼ Dirichlet(γiπ + cπi ),(34)

ϕ̄rh | S, P, O ∼ Dirichlet(γrϕ̄ + cϕ̄rh ),λl | O ∼ Dirichlet(γ λ + cλl ),
ϵ
ϵ

(35)

ϵ | E ∼ Dirichlet(γ + c ),(37)

δ | D ∼ Dirichlet(γ + c ),

(38)

δ

δ

(36)

ϕ̄
12
λ
L
ϵ
emax −emin +1
where cπs ∈ R24,+ , crh ∈ R+ , cl ∈ R+ , c ∈ Rdδmax
and c ∈ R+ are count data obtained from S and Q. Morespecifically, cπ0s indicates the number of times that s0 = s is
satisfied, cπss′ indicates the number of transitions from key s to
key s′ , cϕ̄r0h indicates the number of times that type(s0 ) = r
and deg(s0 , p0 ) = h are both satisfied, cϕ̄rhh′ indicates thenumber of transitions from a pitch degree h to a pitch degreeh′ under a key type r, cλll′ indicates the number of transitions
from a tatum position l to a tatum position l′ , cϵe indicates thenumber of onset time deviations taking e, and cδd indicates thenumber of transient durations taking d.
To update σ, we use a MH algorithm with a random-walk
proposal distribution as follows:q(σ ∗ |σ) = Gamma(σ ∗ |σ, 1),(39)

where L(σ) is the likelihood function of σ given by

Cauchy(xt |µt , σ).

(41)

t=toj +eoj

A PPENDIX B
D ETAILED A LGORITHM FOR P OSTERIOR M AXIMIZATION
1) Estimating Local Keys: In the forward step, a Viterbi
variable ω(sm ) is calculated recursively by replacing the sum
operation with the max operation in the recursion of α(sm )(Appendix A-1) as follows:ω(s0 ) = ϕs0 ,0,p0 π0,s0 ,(42)
jm+1 −1

ω(sm ) = max πsm−1 sm ω(sm−1 )sm−1∏

ϕsm pj−1 pj ,

(43)

j=jm

where an argument sm−1 that maximizes the max operationis memorized as prev(sm ) when calculating ω(sm ).
In the backward step, the local keys S are obtained in thereverse order as follows:sM = argmax ω(sM = i),

(44)

i

sm = prev(sm+1 ).

(46)

λ
ϵ
δ
ϕ
χ
ω̄(q̄n ) = ϵβēn δdβ¯ max λβlō ,ln ϕβsōn ,p̄ōn ,p̄n χ(q̄n )β ω̄(q̄ōn ),nn q̄ō
n(47)

where an argument q̄ōn that maximizes the max operation ismemorized as prev(q̄n ) when calculating ω̄(q̄n ), and β ϕ , β λ ,β ϵ , β δ , and β χ are weighting factors.
Our preliminary experiments show that the latent variables
estimated with an HSMM favor longer durations for reducing
the number of state transitions because the accumulated multiplication of transition probabilities reduces the likelihood.
As a possible solution for penalizing longer musical notes,
o1
we introduce an additional term f (ōn ) = {exp( n−ō)}β ton
Eq. (47) as follows:
ϵ

ϕ

λ

δ

(45)

χ

ω̄(q̄n ) = ϵβēn δdβ¯ max λβlō ,ln ϕβsōn ,p̄ōn ,p̄n χ(q̄n )β ω̄(q̄ōn )f (ōn ),
q̄ōn

n

(48)
where ōn and n indicate the onset and offset score times (i.e.,n − ōn indicates the note value) and β o is a weighting factor.
In the backward step, the musical notes Q are obtained in
the reverse order as follows:q̄N = argmax ω(q̄N = q̄),

(49)

q̄

q̄ōn = prev(q̄n ).

L(σ) = Gamma(σ|γ0σ , γ1σ )j=1ϕω̄(q̄1 ) = ϕβs0 ,0,p̄1 ,nwhere σ is a current sample and σ ∗ is a proposal. The proposal
σ ∗ is accepted as the next sample with the probability givenby)(L(σ ∗ )q(σ|σ ∗ )
,
1
,
(40)
A(σ ∗ , σ) = min
L(σ)q(σ ∗ |σ)oj+1 −1
J toj+1 +e∏∏2) Estimating Musical Notes: In the forward step, a Viterbi
variable ω̄(q̄n ) is calculated recursively by replacing the sum
operation with the max operation in the recursion of ᾱ(q̄n )(Appendix A-2). In practice, we can introduce weighting factors to balance the musical score model and the F0 trajectorymodel, as is usually done in statistical speech recognition [38].
The modified message is thus given by(50)3) Estimating Model Parameters: Given the latent variables
S and Q, the model parameters Θ except for σ are obtained as
the expectations of the posterior Dirichlet distributions given
in Appendix A-3. The Cauchy scale σ is updated to a proposal
given by Eq. (39) only when the posterior given by the productof Eqs. (12)–(18) is increased.
R EFERENCES[1] S. E. Levinson, L. R. Rabiner, and M. M. Sondhi, “An introduction to theapplication of the theory of probabilistic functions of a Markov process
to automatic speech recognition,” The Bell System Technical Journal,
vol. 62, no. 4, pp. 1035–1074, 1983.
[2] Y. Ojima, E. Nakamura, K. Itoyama, and K. Yoshii, “A hierarchicalbayesian model of chords, pitches, and spectrograms for multipitch
analysis,” in International Society for Music Information Retrieval
Conference, (ISMIR), 2016, pp. 309–315.
[3] A. McLeod, R. Schramm, M. Steedman, and E. Benetos, “Automatictranscription of polyphonic vocal music,” Applied Sciences, vol. 7,
no. 12, 2017.
[4] S. Sigtia, E. Benetos, N. Boulanger-Lewandowski, T. Weyde, A. S. d.
Garcez, and S. Dixon, “A hybrid recurrent neural network for musictranscription,” in IEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP), 2015, pp. 2061–2065.
[5] A. Ycart and E. Benetos, “A study on LSTM networks for polyphonicmusic sequence modelling,” in International Society for Music Information Retrieval Conference (ISMIR), 2017, pp. 421–427.
[6] E. Nakamura, R. Nishikimi, S. Dixon, and K. Yoshii, “Probabilistic
sequential patterns for singing transcription,” in Asia-Pacific Signal andInformation Processing Association Annual Summit and Conference(APSIPA ASC), 2018, pp. 1905–1912.

R. NISHIKIMI et al.: BAYESIAN SINGING TRANSCRIPTION[7] R. Nishikimi, E. Nakamura, M. Goto, K. Itoyama, and K. Yoshii, “Scaleand rhythm-aware musical note estimation for vocal F0 trajectories basedon a semi-tatum-synchronous hierarchical hidden semi-markov model,”in International Society for Music Information Retrieval Conference,(ISMIR), 2017, pp. 376–382.
[8] S. Böck, F. Korzeniowski, J. Schlüter, F. Krebs, and G. Widmer,“madmom: a new Python audio and music signal processing library,” inProceedings of the 24th ACM International Conference on Multimedia,
2016, pp. 1174–1178.
[9] C. Raphael, “A hybrid graphical model for rhythmic parsing,” Artificial
Intelligence, vol. 137, no. 1-2, pp. 217–238, 2002.
[10] M. Hamanaka, M. Goto, H. Asoh, and N. Otsu, “A learning-based
quantization: Unsupervised estimation of the model parameters,” inInternational Conference on Multimodal Interfaces (ICMI), 2003, pp.
369–372.
[11] B. Shahriari, K. Swersky, and Z. Wang, “Taking the human out of theloop: A review of Bayesian optimization,” Proceedings of the IEEE, vol.
104, no. 1, pp. 148–175, 2016.
[12] D. J. Hermes, “Measurement of pitch by subharmonic summation,” Thejournal of the acoustical society of America, vol. 83, no. 1, pp. 257–264,
1988.
[13] Y. Ikemiya, K. Yoshii, and K. Itoyama, “Singing voice analysis andediting based on mutually dependent F0 estimation and source separation,” in IEEE International Conference on Acoustics, Speech and Signal
Processing (ICASSP), 2015, pp. 574–578.
[14] J. Salamon and E. Gómez, “Melody extraction from polyphonic musicsignals using pitch contour characteristics,” IEEE Transactions on Audio,
Speech, and Language Processing, vol. 20, no. 6, pp. 1759–1770, 2012.
[15] J.-L. Durrieu, G. Richard, B. David, and C. Févotte, “Source/filter model
for unsupervised main melody extraction from polyphonic audio signals,” IEEE Transactions on Audio, Speech, and Language Processing,vol. 18, no. 3, pp. 564–575, 2010.
[16] A. de Cheveigné and H. Kawahara, “YIN, a fundamental frequency
estimator for speech and music,” The Journal of the Acoustical Societyof America, vol. 111, no. 4, pp. 1917–1930, 2002.
[17] M. Mauch and S. Dixon, “pYIN: A fundamental frequency estimatorusing probabilistic threshold distributions,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014,
pp. 659–663.
[18] P.-S. Huang, S. D. Chen, P. Smaragdis, and M. Hasegawa-Johnson,“Singing-voice separation from monaural recordings using robust principal component analysis,” in IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2012, pp. 57–60.
[19] M. Goto, K. Yoshii, H. Fujihara, M. Mauch, and T. Nakano, “Songle: A
web service for active music listening improved by user contributions,”in International Society for Music Information Retrieval Conference(ISMIR), 2011, pp. 311–316.
[20] R. P. Paiva, T. Mendes, and A. Cardoso, “On the detection of melody
notes in polyphonic audio,” in International Conference on Music
Information Retrieval (ISMIR), 2005, pp. 175–182.
[21] C. Raphael, “A graphical model for recognizing sung melodies,” inInternational Conference on Music Information Retrieval (ISMIR), 2005,
pp. 658–663.
[22] A. Laaksonen, “Automatic melody transcription based on chord transcription,” in International Society for Music Information Retrieval(ISMIR), 2014, pp. 119–124.
[23] M. P. Ryynänen and A. P. Klapuri, “Automatic transcription of melody,
bass line, and chords in polyphonic music,” Computer Music Journal,
vol. 32, no. 3, pp. 72–86, 2008.
[24] E. Molina, L. J. Tardón, A. M. Barbancho, and I. Barbancho, “SiPTH:
Singing transcription based on hysteresis defined on the pitch-time
curve,” IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP), vol. 23, no. 2, pp. 252–263, 2015.
[25] L. Yang, A. Maezawa, J. B. L. Smith, and E. Chew, “Probabilistictranscription of sung melody using a pitch dynamic model,” in IEEE
International Conference on Acoustics, Speech and Signal Processing(ICASSP), 2017, pp. 301–305.
[26] M. Mauch, C. Cannam, R. Bittner, G. Fazekas, J. Salamon, J. Dai,
J. Bello, and S. Dixon, “Computer-aided melody note transcription
using the Tony software: Accuracy and efficiency,” in Proc. of the
1st International Conference on Technologies for Music Notation and
Representation (TENOR), 2015, pp. 23–30.
[27] H. Takeda, N. Saito, T. Otsuki, M. Nakai, H. Shimodaira, and
S. Sagayama, “Hidden markov model for automatic transcription of
MIDI signals,” in IEEE Workshop on Multimedia Signal Processing(MMSP), 2002, pp. 428–431.

13[28] M. Tanji, D. Ando, and H. Iba, “Improving metrical grammar withgrammar expansion,” in Artificial Intelligence, 2008, pp. 180–191.
[29] P. Desain and H. Honing, “The quantization of musical time: A
connectionist approach,” Computer Music Journal, vol. 13, no. 3, pp.
56–66, 1989.
[30] E. Nakamura, K. Yoshii, and S. Sagayama, “Rhythm transcription ofpolyphonic piano music based on merged-output HMM for multiple
voices,” IEEE/ACM Transactions on Audio, Speech and Language
Processing (TASLP), vol. 25, no. 4, pp. 794–806, 2017.
[31] Y. W. Teh, “A hierarchical bayesian language model based on PitmanYor processes,” in International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational
Linguistics, 2006, pp. 985–992.
[32] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “RWC music
database: Popular, classical and jazz music databases,” in International
Conference on Music Information Retrieval (ISMIR), 2002, pp. 287–288.
[33] M. Goto, “AIST annotation for the RWC music database,” in International Conference on Music Information Retrieval (ISMIR), 2006, pp.
359–360.
[34] T. De Clercq and D. Temperley, “A corpus analysis of rock harmony,”
Popular Music, vol. 30, no. 01, pp. 47–70, 2011.
[35] E. Molina, A. M. Barbancho, L. J. Tardón, and I. Barbancho, “Evaluation
framework for automatic singing transcription,” in International Society
for Music Information Retrieval (ISMIR), 2014, pp. 567–572.
[36] R. Nishikimi, E. Nakamura, K. Itoyama, and K. Yoshii, “Musical note
estimation for F0 trajectories of singing voices based on a bayesian semibeat-synchronous HMM,” in International Society for Music Information
Retrieval Conference (ISMIR), 2016, pp. 461–467.
[37] S.-Z. Yu, “Hidden semi-Markov models,” Artificial Intelligence, vol.
174, no. 2, pp. 215–243, 2010.
[38] A. Lee, T. Kawahara, and K. Shikano, “Julius - an open source realtime large vocabulary recognition engine,” in European Conference on
Speech Communication and Technology (EUROSPEECH), 2001, pp.
1691–1694.

Ryo Nishikimi received the B.E. and M.S. degrees
from Kyoto University, Kyoto, Japan, in 2016 and2018, respectively. He is currently working toward
the Ph.D. degree in Kyoto University. His researchinterests include music informatics and machine learning. He is a member of IEEE and IPSJ.

Eita Nakamura received a Ph.D. degree in physics from the University ofTokyo in 2012. He has been a post-doctoral researcher at the National Instituteof Informatics, Meiji University, and Kyoto University. He is currently an
Assistant Professor at the Hakubi Center for Advanced Research and Graduate
School of Informatics, Kyoto University. His research interests include musicmodelling and analysis, music information processing and statistical machinelearning.

Masataka Goto received the Doctor of Engineering degree from Waseda University in 1998. He iscurrently a Prime Senior Researcher at the National
Institute of Advanced Industrial Science and Technology (AIST), Japan. Over the past 28 years he has
published more than 270 papers in refereed journals
and international conferences and has received 51
awards, including several best paper awards, best
presentation awards, the Tenth Japan Academy Medal,
and the Tenth JSPS PRIZE. In 2016, as the Research
Director he began OngaACCEL Project, a 5-year
JST-funded research project (ACCEL) on music technologies.

14

IEEE/ACM TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. XX, NO. XX, XX 2019

Katsutoshi Itoyama received the B.E. degree inengineering in 2006, the M.S. degree in informatics in 2008, and the Ph.D. degree in informatics in2011 all from Kyoto University. He worked with
Kyoto University from 2011 to 2018. After that, heis currently an Specially Appointed Associate Professor (Lecturer), Tokyo Institute of Technology. His
research interests include musical information processing, robot audition, and computational auditory
scene analysis based on statistical signal processing
and machine learning techniques. He is a memberof the IPSJ, ASJ, and IEEE.

Kazuyoshi Yoshii received the M.S. and Ph.D. degrees in informatics from Kyoto University, Kyoto,
Japan, in 2005 and 2008, respectively. He is an Associate Professor at the Graduate School of Informatics, Kyoto University, and concurrently the Leaderof the Sound Scene Understanding Team, Center
for Advanced Intelligence Project (AIP), RIKEN,Tokyo, Japan. His research interests include music
informatics, audio signal processing, and statisticalmachine learning.