JOINT TRANSCRIPTION OF LEAD, BASS, AND RHYTHM GUITARSBASED ON A FACTORIAL HIDDEN SEMI-MARKOV MODEL
Kentaro Shibata1 Ryo Nishikimi1 Satoru Fukayama2 Masataka Goto2
Eita Nakamura1 Katsutoshi Itoyama1 Kazuyoshi Yoshii1
Graduate School of Informatics, Kyoto University, Japan
National Institute of Advanced Industrial Science and Technology (AIST), Japan1

2

ABSTRACTThis paper describes a statistical method for estimating musicalscores for lead, bass, and rhythm guitars from polyphonic audio
signals of typical band-style music. To perform multi-instrumenttranscription involving multi-pitch detection and part assignment, itis crucial to formulate a musical language model that represents thecharacteristics of each part in order to solve the ambiguity of partassignment and estimate a musically-natural score. We propose afactorial hidden semi-Markov model that consists of three language
models corresponding to the three guitar parts (three latent chains)
and an acoustic model of a mixture spectrogram (emission model).
The language model for rhythm guitar represents a homophonicsequence of musical notes (chord sequence) and those for lead andbass guitars represent a monophonic sequence of musical notes in ahigher and lower frequency range respectively. The acoustic model
represents a spectrogram as a sum of low-rank spectrograms of thethree guitar parts approximated by NMF. Given a spectrogram, weestimate the note sequences using Gibbs sampling. We show that ourmodel outperforms a state-of-the-art multi-pitch detection methodin the accuracy and naturalness of the transcribed scores.

Rhythm guitar
‫ڮ‬
‫ڮ‬

E3

‫ڮ‬

Bm

‫ڮ ڮ‬

C

C3

Chord sequence

Lead guitar

ࢄࡸ

ࢄ

C6

E2

‫ڮ‬

‫ڮ‬

‫ڮ‬

C6

F2
E2

䠿
Note sequence

Bass guitar

ࢄ࡮

Music
spectrogram

G3

E1

‫ڮ‬

‫ڮ‬

‫ڮ‬

G3

Basis spectra

F1
E1

䠿
Note sequence
Activations

Low-rank
spectrograms

Fig. 1. A factorial hidden semi-Markov model (FHSMM) that represents the generative process of audio signals of three guitar partsof band music.
Since most of the previous methods rely on the timbral characteristics of instruments, they struggle with band music with multiple
instruments having similar timbre (e.g., two guitars). Recently, attempts have been made to incorporate a music language model representing a musical grammar (e.g., sequential dependency of chords
and musical notes) to complement the acoustic model in such difficult situations. Although Sigtia et al. [14] proposes an end-toend polyphonic transcription method based on a recurrent neuralnetwork (RNN)-based language model, the model’s effect is essentially smoothing since it is defined at the frame level. In order to
get well-formed transcriptions, as they mentioned, a beat-level language model is considered to be effective. Schramm et al. [16] combines a PLCA-based acoustic model with a hidden Markov model(HMM)-based musical language model (also defined at the framelevel), where multiple vocal parts were treated independently.
In this paper, we propose a transcription method for the accompaniment part of band music based on the beat-level sequential characteristics of accompanying instruments. We assume that three kindsof guitars—lead, bass, and rhythm—are used in band music andhave different sequential characteristics, as listed in Table 1. Our
method uses a factorial hidden semi-Markov model (FHSMM) [17–19] that consists of three language models (semi-Markov models),corresponding to the three guitar parts, and an NMF-based acoustic model of a mixture spectrogram (Fig. 1). The language modelfor the rhythm guitar represents a homophonic sequence of musicalnotes (chord sequence), while those for the lead and bass guitars represent a monophonic sequence of musical notes in a higher or lower
frequency range respectively. The acoustic model represents a spectrogram as a sum of the spectrograms of the three guitar parts, eachof which is represented as a low-rank matrix obtained by the product of an activation vector and basis spectra. A key feature of theIndex Terms— Automatic music transcription, multi-pitch estimation, multi-instrument transcription, HSMM, and NMF.
1. INTRODUCTION
Transcribing a musical score from an audio signal is a fundamental and challenging problem in music information processing [1].
From a practical viewpoint, it is important to develop a transcription method for band music to facilitate widely-enjoyed cover performances. A typical band playing popular music (e.g., The Beatles)
consists of vocal, several guitars, and drum parts. Transcription of
vocal and drum parts have each been studied extensively, and highaccuracies have been reported [2–7]. We therefore focus on accompaniment parts that are typically played by three guitars (lead, bass,
and rhythm guitars).
Accompaniment-part transcription for band music is a challenging task because multi-pitch detection and part assignment of eachnote are both required. A major approach to multi-pitch detection
is to use probabilistic latent component analysis (PLCA) or nonnegative matrix factorization (NMF) based on the sparseness andlow-rankness of source spectrograms. For part assignment, thesemethods have been extended to use pre-learned spectral templates ofmultiple instruments [8–13]. More recently, end-to-end neural networks have been applied successfully to executing multi-pitch detection and part assignment simultaneously [14, 15].
This work was supported in part by JST ACCEL No. JPMJAC1602,
JSPS KAKENHI No. 16H01744 and No. 16J05486, and the Kyoto University Foundation.

978-1-5386-4658-8/18/$31.00 ©2019 IEEE

ࢄࡾ
C4

‫ڮ‬

236

ICASSP 2019‫څݑ څ‬Transition between tatums : ܲ ‫ݑ‬௡௡ିଵTable 1. The characteristics of thee guitar parts.
Part
Lead
Bass
RhythmSequence
MonophonicMonophonic
HomophonicFrequency range
Middle - High
Low
Wide

Tatum U ‫څ‬RhythmComplex
Complex
Simple0Bases ܹ ‫څ‬࢝‫څ‬୉ଵ ‫څ࢝ ڮ‬େ଺

2

4

6

Select aෝ ‫څ‬௡‫څ‬basis ࢝at each tatum

8

10

12

C4

E4

14

0

Transition between pitches:Am‫څ‬‫ څ‬, ‫ୖݖ‬ܲ ‫ݖ‬௡‫ݖ| څ‬௡ିଵఘ‫( څ‬௡) = ߰୅୫,୅ଷ,େସ

A3

C4
‫څ‬
‫څ‬
‫څ‬
‫ݔ‬௙௧
~Poisson(‫ݓ‬ෝ௙ఎ‫ څ‬௧ ݄௧ )‫څ‬‫ݖ‬௡ିଷ

‫څ‬‫ݖ‬௡ିଶ

‫څ‬‫ݖ‬௡ିଵ

‫ݖ‬௡‫څ‬‫څ‬࢝େସ࢝‫څ‬୉ସ

‫څ‬
࢝୅ଷ

‫څ‬
࢝େସ

×

×

×

×

‫څ܆‬

Low-rank
spectrogram

Fig. 2. The generative process of spectrograms for lead/bass guitar
based on a conditional HSMM. ? indicates lead (L) or bass (B).
“•” will be used to represent “R” (rhythm guitar), “L”, or “B”.
The latent chain of the rhythm guitar is specified by a sequence ofRchord symbols ZR = {z1R , . . . , zN
R } with relative onset positions
R
R
R
U = {u1 , . . . , uN R }, where N R is the number of chords, znR
takes one of 24 values of {C, . . . , B} ⇥ {major, minor}, and uRntakes an integer from 0 to 15 indicating a position on the 16th-notelevel tatum grid in a measure. Likewise, the latent chain of the leador bass guitar’s HSMM is specified by a sequence of pitches Z? =?
?
?
?
{z1? , . . . , zN? } with relative onset positions U = {u1 , . . . , uN ? },?
L
where N is the number of musical notes zn takes one of 45 pitchesof {E2, . . . , C6}, znB takes one of 28 pitches of {E1, . . . , G3}, andu?n takes an integer from 0 to 15.
The sequence of chord symbols ZR and the sequence of pitches
Z? are represented by a Markov model as follows:⇣⌘⇣⌘
p z1R |⇡ R = ⇡zRR , p znR |znR 1 , R = zRR ,zR , (2)
1
n 1 n
⇣
⌘
?
R
R
?
?
p z1 |Z , U , ⇡ = ⇡zR ,z? ,
(3)
1
1
⇣
⌘
p zn? |zn? 1 , ZR , UR , ? = z?R? ,z? ,z? ,(4)

Our method jointly performs multi-pitch estimation and part assignment in a unified framework. We formulate a generative model ofa music spectrogram and the corresponding musical score and thensolve the inverse problem. That is, given a music spectrogram, weestimate the score described as latent variables in the model. The
problem is defined as follows:⇥T
Input: The magnitude spectrogram of a target signal X 2 RF+
and 16th-note-level tatum times
Output: Musical scores of the three guitar parts

Here, F is the number of frequency bins, T is the number of timeframes. In this paper we assume that the time signature of the target
signal is 4/4.

⇢ (n)

n

1

where ⇡aR is the initial probability of chord a,?

nRa,b is the transition

probability from chord a to chord b, ⇢ (n) indicates a chord to which?
musical note zn? belongs, ⇡c,ais the initial probability of pitch a?
given chord c, and c,a,b is the transition probability from pitch a topitch b given chord c. This formulation is based on the fact that thepitches of the lead and bass guitars are strongly correlated with thechords of the rhythm guitar.
The sequence of chord onset times UR and note onset times U?
are represented by metrical Markov models [20, 21] as follows:

2.1. Probabilistic Factorial Modeling⇥T
We represent a music spectrogram X 2 RF
as a sum of the spec+
F ⇥T
R
L
B
trograms X , X , and X 2 R+of the rhythm, lead, and bass

L

0

Activation ࡴ‫כ‬

2. PROPOSED METHOD

R

8Chord Label Z ୖ C

acoustic model is that only one of the basis spectra is allowed to be
activated in each tatum in each guitar part. Given a mixture spectrogram as observed data, the basis spectra, activations, and latent
chains are statistically estimated using Gibbs sampling. The musicalscore for each guitar is then obtained by using the Viterbi algorithm.
A major contribution of this study is to achieve joint transcription of multiple musical instruments that have similar timbral characteristics. More specifically, we propose an integrated language andacoustic model that can describe beat-level symbolic musical grammar and dependency between multiple instrument parts. We also
show that the initialization problem of the NMF-based model can be
improved by a support from a DNN-based model, which has strong
capability of expressing acoustic signals. We achieve an improvement on the accuracy, which is an important step towards developinga practical transcription method that can be used for popular music.

LB
xf t = xR
f t + xf t + xf t .

4

Tatum Index

Pitch Z ‫څ‬

guitars (Fig. 1):0‫څ‬= ߶଼,଴(1)
B

The generative processes of X , X , and X are represented byrhythm, lead, and bass guitar models, respectively. In each guitar
model, the corresponding musical score is represented as a latent
chain of variables, whose generative process is described by a semiMarkov model. Given the three chains, the spectrogram XR , XL ,and XB are generated by an NMF-based acoustic model conditionedon the musical score. Finally, the observed spectrogram X is generated according to Eq. (1). The generative process for each guitar partis described as a hidden semi-Markov model (HSMM) as explained
below. The total model is a factorial HSMM with three latent chains.

p(u•n |u•n 1 ,

•

)=

•
•,
u•
n 1 ,un

(5)

•
a,b

where
is the transition probability from tatum position a totatum position b. Note that if ab, the chord or note continues
over a bar line. The maximum duration of a chord or note is restricted to the measure length (16 tatums). When combined with themodels for pitches/chords, this is a semi-Markov model because thetime unit of state transition is different from the tatum unit and themetrical Markov model describes the distribution of sojourn timesof latent states.

2.1.1. Language Model
We formulate a rhythm guitar HSMM, representing the generative
process of XR , and lead and bass guitar conditional HSMMs, representing the generative process of XL and XB given the chord sequence specified by rhythm guitar (Fig. 2). For convenience, “?”will be used to represent “L” (lead guitar) or “B” (bass guitar) and2.1.2. Acoustic Model
The generative process of the spectrograms of three guitars are formulated in the same way. We use the probabilistic formulation of
NMF based on the Kullback-Leibler divergence (KL-NMF) [13].

237The spectrogram of the rhythm, lead, or bass guitar part X• is generated by using a set of basis spectra W• and an activation vectorh• 2 RT+ . The generative process of the mixture spectrogram is
described as follows:!
X••p(xf t |Z, U, W, ht ) = Poisson xf t
wf ⌘• (t) ht , (6)↵R (znR , uRn ) is calculated recursively as follows:⇣⌘⇣ ⌘↵R z1R , uR
= p z1R = ⇡zRR ,
1
1
⇣
⌘
⇣
⌘
R
R
R
↵ zn , un = p x⌧ R (n 1):⌧ R (n)+1 |znR , uR
n
⇣
⌘
X X R
R
·
↵ znR 1 , uRn 1 ,z R ,z R
uR ,uR•=R,L,Bnwhere ⌘ • (t) indicates the musical note to which frame t belongs andis determined by Z• and U• . Basis spectra W• are given by WR =
⇥24
⇥33
R
R
L
L
{wC
, . . . , wBm} 2 RF,WL = {wE2,. . ., wC6} 2 RF
or
+
+
F ⇥28
B
B
B
R
W = {wE1 ,. . ., wG3 } 2 R+ , where w indicates the basis
spectrum of chord and w? indicates the basis spectrum of note
. A key feature of our model is that only a single basis spectrum
specified by ⌘ • (t) is activated in each frame t for generating thespectrogram X• .

n

To integrate the sub-models described in Sections 2.1.1, and 2.1.2,
we formulate a semi-Bayesian model given by

(12)

n+1

n

n+1

Similarly as the Bayesian inference of NMF [22], W and H
are sampled directly from the conditional posterior distributionp(W, H|Z, U, X). Let •f t be an auxiliary variable, calculated
from the latest samples of W and H as•ft =wf•⌘(t) h•t
L
L
B
B
wfR⌘R (t) hR
t + wf ⌘ L (t) ht + wf ⌘ B (t) ht

Using , W and H are sampled as follows:wf•k ⇠ G

a•w +
0

X

h•t ⇠ G @a•h•t +xf t •f t ,

t

X

b•w +

X

h•t

t

xf t •f t ,fb•h•t +

X
f

!

.

(15)

(16)

,
1

wf•k A .

(17)

2.4. Musical Score Estimation
We obtain the most likely Z• and U• by the Viterbi algorithm andthe final estimate of H by taking the mean of the posterior distribution in Eq. (17). Musical scores for the three guitar parts are estimated with these variables. The musical score for the lead or bass
guitar can be obtained from the pitches Z? and onset times U? . Notethat a series of musical notes with the same pitch can be represented
by the self-transitions (e.g., znL 1 = znL = C3). On the other hand,for the rhythm guitar part, we need to determine the actual onsets(attack times) in each region of chord symbols ZR because the samechords can be repeated multiple times. To obtain a natural rhythmic pattern within each measure, we perform template matching byusing a dictionary of rhythmic patterns (16-dimensional binary vectors). The entries of the dictionary are obtained from a collection ofmusical pieces. We first obtain a tentative rhythmic pattern by detecting peaks in the activation HR and then use the closest pattern inthe dictionary based on the cosine distance for estimating the score.
The variables Z, U, and H are estimated as follows. First, we
generate a sufficient amount of samples of Z, U, W, and H byusing Gibbs sampling. Second, we fix the parameters W and H withthe last sample and the maximum-a-posteriori (MAP) estimates of Z
and U are obtained by using the Viterbi algorithm. Finally, the MAP
estimate of H is obtained from the posterior distribution in Eq. (17).

(9)

where a•ht and b•ht are hyperparameters.
2.3. Bayesian Inference
Given a music spectrogram X, we aim to calculate the posterior distribution according to Bayes’ theorem:p(Y|X, ⇥) = p(X, Y|⇥)/p(X|⇥).

1

2.3.2. Updating Basis Spectra W and Activations H(7)

where k 2 {C, . . . , Bm}, {E2, . . . , C6}, or {E1, . . . , G3} denotes
a chord or a pitch and aw and bw are the shape and rate hyperpaRrameters. We use weak priors on WR (aRw and bw are smaller thana?w and b?w ) to handle the large variance of the rhythm guitar. The
hyperparameters are set in a way that the prior expectation of W
matches the template spectra of chords and musical notes preparedin advance. Similarly, we use gamma priors for H as follows:ah•t , bh•t ,nLatent variables Z? and U? are updated similarly except that theR?
transition probability a,b
is replaced with c,a,b.

where ⇥ = {⇡ R , ⇡ L , ⇡ B , R , L , B , R , L , B } is parametersthat are trained in advance and Y = {Z, U, W, H} is randomvariables estimated for an observed spectrogram X during runtime.
Here, Z = {ZR , ZL , ZB } and U, W, and H are defined similarly.
We introduce prior distributions p(W) and p(H) to make W
close to the template bases and make H sparse. We use gamma
priors for W as follows:⇣⌘wf•k ⇠ G awf•k , bwf•k ,(8)h•t ⇠ Gn1where ⌧ R (n) is the last frame of the n-th chord and xa:b is{xa , . . . , xb }.
In the backward sampling step, znR and uR
n are sampled recursively by calculating a backward message R (znR , uRn ) as follows:
⇣
⌘
⇣
⌘
⇣
⌘
R
R
R
R
R
zN
, uR
= p zN
, uR
zN
, uR
(13)
N
N |X / ↵
N ,
⇣
⌘
⇣
⌘
R
R
R
znR , uR
= p znR , uRnn |zn+1:N , un+1:N , X⇣⌘
R/ zRR ,zR
↵ znR , uR(14)n .
uR ,uR2.2. Bayesian Formulationp(X, Y; ⇥)= p(X|Z, U, W, H)p(Z|⇡, )p(U| )p(W)p(H),

n

R
R
zn
1 un 1(11)(10)

Since this distribution is analytically intractable, we use Gibbs sampling for alternately and iteratively sampling the latent variables Z
and U, the basis spectra W, and the activations H. That is, we
get samples of G ⇢ Y from a conditional posterior distributionp(G|Y¬G , X, ⇥), where Y¬G indicates the subset of Y obtainedby removing G from Y. We henceforth do not write the dependencyon ⇥ for brevity.
2.3.1. Updating Latent Variables Z and U
To sample ZR and UR from p(ZR , UR |Y¬ZR ,UR , X), an efficient forward filtering-backward sampling algorithm can be used.
In forward filtering, a forward message of the rhythm guitar part238

3. EVALUATION

Table 2. Performance of transcription for three guitar parts.
MethodPart
P(%) R(%) F (%) Chord
Lead
32.4
19.0
22.8
–
Deep saliency [15]
Bass
57.0
61.4
58.8
–
Madmom [23]Rhythm
–
–
–
74.8
Lead
20.8
12.8
15.0
–
FHSMM-RND
Bass
34.3
36.6
35.3
–
Rhythm
–
–
–
48.2
Lead
34.7
19.8
24.1
–
FHSMM-DSM
Bass
57.7
62.2
60.0
–
Rhythm–––74.8

3.1. Experimental Setup
Ten songs by The Beatles in 4/4 time consisting of rhythm, lead,and bass guitars, vocals, and drums were used for evaluation. To
extract accompaniment sounds by suppressing the drum and vocal
sounds, we used a harmonic/percussive source separation method [2]and a singing voice separation method [5]. The tatum-level chordlabel accuracy was measured for the rhythm guitar and the tatumlevel recall and precision rates and F-measure were measured forthe lead and bass guitars. The chord vocabulary consisted of 24 laRRbels (wC, . . . , wBm) and the no chord regions were not considered.
Recall, precision, and F-measure were given by P = Ntp /Nsys ,R = Ntp /Nref , and F = 2RP/(R + P), where Ntp is the numberof correctly-estimated pitches, Nsys the number of detected pitches,
and Nref the number of ground-truth pitches.
For comparison, we tested a state-of-the-art method for melody
and bass line estimation based on convolutional neural networks(CNNs) [15] (called a deep saliency method) for transcribing thelead and bass guitar parts. For transcribing the rhythm guitar part,
we tested a state-of-the-art chord estimation method based on CNNs,(Madmom [23]). Since the three guitar parts are estimated at theframe level, the estimated results are quantized in the tatum level.
To investigate the sensitivity to initialization of the proposed model,
we compared random initialization (FHSMM-RND) and initialization using the results of the above methods (FHSMM-DSM).
The log-frequency magnitude spectrogram of a music signal was
obtained by the constant-Q transform [24] with 96 bins/octave, ashifting interval of 11 ms, and a frequency range from 32.7 Hz (C1)to 8372.0 Hz (C9). The tatum times were estimated in advance byRusing a beat tracking method [23]. The hyperparameters aR
w and bw
Rof the prior on W were determined using the spectra of 24 lowposition chords played by an acoustic guitar. The hyperparametersa?w and b?w of the prior on W? were determined in a similar way
by using the spectra of 45 pitches from E2 to C6 or 28 pitches from
E1 to G3. These reference spectra were made by MIDI synthesizers.
The hyperparameters of activations were set to ahR = ah?t = 1,tbhR = bh?t = 1. The uniform initial probabilities were used:t⇡ L = 1/45, ⇡ B = 1/28, and ⇡ R = 1/24. The language model parameters , , and a dictionary of rhythmic patterns for the rhythm
guitar were learned in advance from 206 pieces of Japanese popular
musicby the maximum-likelihood method.

MIDI note number

Deep saliency (Lead)
70

60

60

50

50

40

40
55

MIDI note number

FHSMM-DSM (Lead)

70

57

59

61

63

55

Deep saliency (Bass)

57

59

61

63

FHSMM-DSM (Bass)

58

Red: Insertion error
58 Green: Correct pitch
Blue: Correct pitch class Gray: Deletion error

48

48

38

38

28

28
55

57

59

61

63

55

57

Measure number

59

61

63

Measure number

Fig. 3. Musical scores of the lead and bass guitars estimated by the
deep saliency method and the proposed method initialized by the
deep saliency method for “Don’t Bother Me.”
Although we achieved an improvement on the transcription accuracy, there is still much room for improving the performance of thelead guitar transcription. First, it is necessary to deal with rests because the lead guitar often has rests during vocal parts. It is also necessary to relax the strong constraints that the lead guitar only plays
a monophonic melody and that the rhythm guitar is described with asingle set of bases. Another limitation is that the Markov model does
not have sufficient capability of expressing the long-term dynamics
of musical notes. We plan to extend a deep generative model suchas a variational autoencoder (VAE) [25] or a generative adversarial
network (GAN) [26] for dealing with time-series data.

3.2. Experimental Results
Table 2 shows the models’ performances on multi-guitar transcription. Note that P, R, and F can never be 100% because the lead andbass guitars were assumed to be monophonic here, but this assumption does not hold true in reality, e.g., the lead guitar sometimes plays
chords. FHSMM-DSM consistently outperformed the deep saliency
method in the transcription of the lead and bass guitars. As shownin Fig. 3, unnatural musical notes such as short insertion errors estimated by the deep saliency method were significantly reduced by the
proposed method. This shows the effectiveness of the language models, which induce musical naturalness of estimated scores. FHSMMRND, on the other hand, underperformed Madmom and the deep
saliency method because the NMF-based acoustic model was sensitive to initialization. We found that the proposed method of joint
guitar transcription can find more accurate scores than independenttranscription methods if it is appropriately initialized. The performance of chord estimation obtained by FHSMM-DSM were almost
same as that obtained by Madmom. This is probably because good
local optima were already found by Madmom.

4. CONCLUSION
We have described a statistical method to estimate the musical scoresof lead, bass, and rhythm guitars from a polyphonic audio signalof typical band-style music. Here, we proposed a unified Bayesianframework for integrating language and acoustic models of multiinstrument music based on a factorial hidden semi-Markov model.
We have shown that the unified model improves the accuracy andnaturalness of the transcribed scores. We focused on band-style popular music in this paper, but our framework can be applied to various
kinds of music transcription involving multi-pitch detection and partassignment. In piano transcription, for example, more reasonableestimations could be obtained by executing the multi-pitch detection and separation of right-hand and left-hand streams jointly [27].
In the future, we plan to develop a system that generates a complete score consisting of vocal, guitar, and drum parts by combining
the work described in this paper with automatic vocal (melody) anddrum transcription methods.

239

5. REFERENCES[14] Siddharth Sigtia, Emmanouil Benetos, and Simon Dixon, “An
end-to-end neural network for polyphonic piano music transcription,” IEEE/ACM Transactions on Audio, Speech andLanguage Processing, vol. 24, no. 5, pp. 927–939, 2016.
[15] Rachel M. Bittner, Brian McFee, Justin Salamon, Peter Li, andJuan P. Bello, “Deep salience representations for F0 estimation in polyphonic music,” in Proceedings of the International
Society for Music Information Retrieval Conference (ISMIR),
2017, pp. 63–70.
[16] Rodrigo Schramm, Andrew McLeod, Mark Steedman, andEmmanouil Benetos, “Multi-pitch detection and voice assignment for a cappella recordings of multiple singers,” in Proceedings of the International Society for Music Information
Retrieval Conference (ISMIR), 2017, pp. 552–559.
[17] Zoubin Ghahramani and Michael I. Jordan, “Factorial hidden
Markov models,” in Proceedings of the Advances in Neural
Information Processing Systems (NIPS), 1996, pp. 472–478.
[18] Gautham J. Mysore and Maneesh Sahani, “Variational inference in non-negative factorial hidden Markov models for efficient audio source separation,” in Proceedings of the International Conference on Machine Learning (ICML), 2012, pp.
1887–1894.
[19] Alexey Ozerov, Cédric Févotte, and Maurice Charbit, “Factorial scaled hidden Markov model for polyphonic audio representation and source separation,” in Proceedings of the IEEE
Workshop on Applications of Signal Processing to Audio andAcoustics (WASPAA), 2009, pp. 121–124.
[20] Christopher Raphael, “A hybrid graphical model for rhythmic
parsing,” Artificial Intelligence, vol. 137, pp. 217–238, 2002.
[21] Masatoshi Hamanaka, Masataka Goto, Hideki Asoh, andNobuyuki Otsu, “A learning-based quantization: Unsupervised
estimation of the model parameters,” in Proceedings of theInternational Computer Music Conference (ICMC), 2003, pp.
369–372.
[22] Ali Taylan Cemgil, “Bayesian inference for nonnegative matrix factorisation models,” Journal of Computational Intelligence and Neuroscience, , no. 785152, pp. 1–17, 2009.
[23] Sebastian Böck, Filip Korzeniowski, Jan Schlüter, Florian
Krebs, and Gerhard Widmer, “Madmom: A new python audio and music signal processing library,” in Proceedings of
the ACM International Conference on Multimedia, 2016, pp.
1174–1178.
[24] Christian Schörkhuber and Anssi Klapuri, “Constant-Q transform toolbox for music processing,” in Proceedings of theSound and Music Computing Conference (SMC), 2010, pp. 3–
64.
[25] Diederik P. Kingma and Max Welling, “Auto-encoding variational Bayes,” in Proceedings of the International Conference
on Learning Representations (ICLR), 2014.
[26] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing
Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, andYoshua Bengio, “Generative adversarial nets,” in Proceedings of the Advances in Neural Information Processing Systems(NIPS), 2014, pp. 2672–2680.
[27] Eita Nakamura, Kazuyoshi Yoshii, and Shigeki Sagayama,“Rhythm transcription of polyphonic piano music based onmerged-output HMM for multiple voices,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 25,
no. 4, pp. 794–806, 2017.

[1] Emmanouil Benetos, Simon Dixon, Dimitrios Giannoulis,
Holger Kirchhoff, and Anssi Klapuri, “Automatic music transcription: Challenges and future directions,” Journal of Intelligent Information Systems, vol. 41, no. 3, pp. 407–434, 2013.
[2] Derry Fitzgerald, “Harmonic/percussive separation using median filtering,” in Proceedings of the International Conference
on Digital Audio Effects (DAFX), 2010, pp. 1–4.
[3] Carl Southall, Ryan Stables, and Jason Hockman, “Automatic
drum transcription using bi-directional recurrent neural networks,” in Proceedings of the International Society for Music
Information Retrieval Conference (ISMIR), 2016, pp. 591–597.
[4] Chih-Wei Wu and Alexander Lerch, “Drum transcription using
partially fixed non-negative matrix factorization with template
adaptation,” in Proceedings of the International Society forMusic Information Retrieval Conference (ISMIR), 2015, pp.
257–263.
[5] Andreas Jansson, Eric Humphrey, Nicola Montecchio, Rachel
Bittner, Aparna Kumar, and Tillman Weyde, “Singing voice
separation with deep U-Net convolutional networks,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2017, pp. 745–751.
[6] Emilio Molina, Lorenzo J. Tardón, Ana M. Barbancho, and Isabel Barbancho, “SiPTH: Singing transcription based on hysteresis defined on the pitch-time curve,” IEEE/ACM Transactions on Audio, Speech and Language Processing, vol. 23, no.
2, pp. 252–263, 2015.
[7] Ryo Nishikimi, Eita Nakamura, Masataka Goto, Katsutoshi
Itoyama, and Kazuyoshi Yoshii, “Scale- and rhythm-aware
musical note estimation for vocal F0 trajectories based ona semi-tatum-synchronous hierarchical hidden semi-Markov
model,” in Proceedings of the International Society for Music
Information Retrieval Conference (ISMIR), 2017, pp. 376–382.
[8] Mert Bay, Andreas F. Ehmann, James W. Beauchamp, Paris
Smaragdis, and J. Stephen Downie, “Second fiddle is important too: Pitch tracking individual voices in polyphonic music.,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2012, pp. 319–324.
[9] Graham Grindlay and Daniel P.W. Ellis, “Transcribing multiinstrument polyphonic music with hierarchical eigeninstruments,” IEEE Journal of Selected Topics in Signal Processing,vol. 5, no. 6, pp. 1159–1169, 2011.
[10] Emmanouil Benetos, Roland Badeau, Tillman Weyde, andGaël Richard, “Template adaptation for improving automaticmusic transcription,” in Proceedings of the International Society for Music Information Retrieval Conference (ISMIR), 2014,
pp. 175–180.
[11] Emmanuel Vincent and Xavier Rodet, “Music transcription
with ISA and HMM,” in Proceedings of the International Conference on Independent Component Analysis and Signal Separation (ICA), 2004, pp. 1197–1204.
[12] Emmanuel Vincent, Nancy Bertin, and Roland Badeau, “Adaptive harmonic spectral decomposition for multiple pitch estimation,” IEEE Transactions on Audio, Speech, and Language
Processing, vol. 18, no. 3, pp. 528–537, 2010.
[13] Paris Smaragdis and Judith C. Brown, “Non-negative matrix
factorization for polyphonic music transcription,” in Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2003, pp. 177–180.

240