2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)

Joint Drum Transcription and Metrical Analysis
Based on Periodicity-Aware Multi-Task Learning
Daichi Kamakura, Eita Nanamura, Takehisa Oyama, and Kazuyoshi Yoshii
Graduate School of Informatics, Kyoto University, Japan
E-mail: {kamakura, nakamura, ooyama}@sap.ist.i.kyoto-u.ac.jp, yoshii@i.kyoto-u.ac.jp

Abstract—This paper describes a multi-task learning methodthat detects the onset times of drums and the beat and downbeat
times from a music signal. Since the drum part typically consistsof repetitive patterns synchronized with the metrical structure in
popular music, drum transcription and metrical analysis would
benefit each other. The basic approach is to train a deep neuralnetwork (DNN) with a branching architecture that extracts latentfeatures common to both tasks from a music signal and then feeds
them to task-specific networks separately. The estimated framelevel posterior probabilities of drum onsets, beats, and downbeats,
however, often have weak and inconsistent periodic structures. To
solve this problem, we propose a regularized training method thatencourages the three probability sequences to be highly auto- andcross-correlated. Specifically, our method aims to minimize theentropy of the auto-spectrum computed from each probability
sequence and that of the cross-spectrum computed from eachof the three sequence pairs, because a smaller entropy indicatesa stronger element-wise periodicity or pairwise consistency. The
experiment showed the mutual benefit of drum transcription andmetrical analysis and the effectiveness of the periodicity- andconsistency-based regularizations.

I. I NTRODUCTION
Automatic drum transcription (ADT) is one of the mostfundamental subtasks of automatic music transcription (AMT)
that aims to estimate the symbolic score of a music signal.
Since the drum part affects the rhythmic characteristics of thesong in popular music, drum transcription forms the basis ofmusic information retrieval (MIR). Although the ultimate goalof ADT is audio-to-score transcription, we focus on audio-toMIDI transcription that aims to estimate the onset times ofdrums in seconds as an intermediate goal.
In recent studies on audio-to-MIDI drum transcription, deepneural networks (DNNs) have successfully been used for estimating from a music spectrogram the posterior probabilities ofthe presence of drum onsets at the frame level [1]–[3]. Givena ground-truth binary sequence representing the presence orabsence of drum onsets, one can train a DNN such that theposterior probability of the binary sequence is maximized, i.e.,the cross-entropy loss is minimized. If only a limited amountof training data is available as is often the case with AMT [4],[5], however, the repetitive structures of the drum part observed
at multiple metrical levels might not be captured well.
In this paper, we take the multi-task learning approach tojoint drum transcription and metrical analysis (beat and downbeat detection). Both tasks are typically defined as binary classification problems that aim to estimate the presence or absenceBaseline

Proposed

BD
SD
HH
Beat
DownbeatFig. 1. The frame-level sequences of drum onset, beat, and downbeat probabilities estimated by the multi-task learning method without/with the periodicityaware regularization. The red and blue arrows indicate the element-wise
periodicity and pairwise consistency, respectively.

of drum onsets, beats, and downbeats at the frame level, andare expected to benefit each other by extracting common latentfeatures with respect to periodicity from a music signal [2]. In
this approach, one may train a DNN with a common feature
extractor followed by task-specific binary classifiers. In such abasic branching architecture, however, the posterior probabilities of drum onsets, beats, and downbeats are estimated in aconditionally independent manner and are thus not guaranteed
to be consistent in terms of periodicity.
To mitigate this problem, we propose a regularized training
method that encourages the sequences of drum onset, beat, anddownbeat probabilities to have periodic patterns synchronously(Fig. 1). We focus on the fact that the tempo tends to be kept
constant throughout the entire song in popular music. This
requests that each sequence should have clear periodicity solely
and that a sequence should share the same periodicity withanother sequence. We thus propose element-wise and pairwise
regularization terms described in detail below.
First, we focus on the auto-correlation function of each sequence. If the sequence of drum onset probabilities is repetitiveat the measure level, its auto-correlation function exhibits sharp
peaks spaced at the downbeat interval. The auto-spectrum obtained by applying the discrete Fourier transform (DFT) to theauto-correlation function exhibits a few peaks at the downbeat
frequency. The entropy of the auto-spectrum can thus be used
as a regularization term to be minimized. The same applies
to the sequences of beat and downbeat probabilities exhibiting
the periodicity at the beat and downbeat levels.
Second, we focus on the cross-correlation function for eachof the three sequence pairs. In general, every time the sequence

of drum onset probabilities is time-shifted by a downbeat interval, it matches well the sequence of downbeat probabilities interms of peak positions. The cross-correlation function exhibits
sharp peaks spaced at the downbeat interval. The entropy ofthe cross-spectrum can thus be another regularization term.
The same applies to the other two pairs.
We experimentally show that the multi-task learning approach is especially effective for improving the performanceof metrical analysis and the regularized training method has apotential of improving the performances of drum transcription
and metrical analysis.
II. R ELATED W ORK
Deep learning has been the central choice in modern ADT[2], [3], [5]–[10]. In general, a deep neural network (DNN)
is trained in a supervised manner using paired data consisting of music signals and drum onset annotations. Due to theimpulsive nature of drums sounds, CNNs have widely beenused as basic building blocks for extracting local features. In
particular, a CNN variant called the temporal convolutional
network (TCN) [11], [12] has gained much attention because
it can take into account both short- and long-term dependencies[13]. To consider long-term contextual information, a CNN isoften followed by a recurrent neural network (RNN), resulting
in a convolutional RNN (CRNN) [2].
To improve the performance with a limited amount of paired
data, one may use data synthesis [14], data augmentation [5],or unsupervised learning [15]. However, these methods are stillinsufficient for dealing with considerable variations in realworld drum sounds. Alternatively, regularized training techniques based on prior knowledge of drum patterns have been
explored [10] as well as phase consideration [16] and architectural innovations [7]. In this study, we focus on the periodicity
of drum onsets for regularization.
For metrical analysis, DNN-HMM combinations have often
been used [13], [17]–[20]. In this approach, a DNN is used
for estimating sequences of beat and downbeat probabilities,
and an HMM is then used for detecting beat and downbeat
times from these sequences. Instead of the standard HMM, ahidden semi-Markov model (HSMM) that explicitly represents
the duration of each state was proven to be effective for taking
into account the tempo and meter consistency of beat anddownbeat times. In this study, we use a similar HSMM-based
decoder to obtain consistent results.
III. P ROPOSED M ETHODThis section describes the multi-task learning of drum transcription and metrical analysis with a periodicity-aware regularized training method.

Drum Transcription

HH

Spectrograms

BLSTM

SD
BD
Prob. sequences
Elementwise loss
based onperiodicity

CNN
CNN
CNN

Pairwise loss
based onperiod consistency

Metrical Analysis

Supervised
loss based on
binary crossentropy

Downbeat

TCN

Beat
Prob. sequences

Fig. 2. The proposed joint drum transcription and metrical analysis based on
periodicity-aware regularized multi-task learning.

and that of downbeats at the frame level, respectively, where
F is the number of frequency bins, T is the number of frames,
and K is the number of drum instruments. We focus on thethree major drum classes (K = 3): bass drum (BD), snare
drum (SD), and hi-hats (HH). Let Y ≜ {YD , YB , YW }. In
the training phase, both X and Y are given as paired data. In
the test phase, Y is estimated from X.
B. Multi-Task Learning
We explain the loss function to be minimized for joint drumtranscription and metrical analysis. Inspired by [17], [20], we
use a CRNN with a branching architecture for jointly estimating a sequence of drum onset probabilities ϕD ∈ [0, 1]K×T ,that of beat probabilities ϕB ∈ [0, 1]T , and that of downbeat
probabilities ϕD ∈ [0, 1]T from the input X (Fig. 2).
1) Drum Transcription Loss: The drum transcription loss isgiven by the binary cross-entropy (BCE) of ϕD for the groundtruth data YD :K,TLD
BCE = −


1 X
D
D
D
Ykt
log ϕDkt +(1−Ykt ) log(1−ϕkt ) . (1)Kk,t=12) Metrical Analysis Losses: The metrical analysis losses
are given by the BCEs of ϕB and ϕW for the ground-truth data
YB and YW :TXLBBCE = −YtB log ϕBt +(1−YtB ) log(1−ϕBt ) ,
(2)
t=1

LW
BCE = −TXWW
YtW log ϕW
t +(1−Yt ) log(1−ϕt ) .

(3)

t=1

3) Total Loss: For joint training, we aim to minimize theweighted sum of the individual losses:
D
B
B
W
W
LBCE = λD
BCE LBCE + λBCE LBCE + λBCE LBCE ,(4)

A. Problem Specification

B
W
where λD
BCE , λBCE , and λBCE are adjustable weights.

We tackle drum transcription and beat and downbeat detection from a music signal. We use the power spectrograms of theleft and right channels of the audio signal X ∈ R2×F ×T as input. Let YD ∈ {0, 1}K×T , YB ∈ {0, 1}T , and YW ∈ {0, 1}T
denote the presence or absence of drum onsets, that of beats,

C. Periodicity-Aware Regularization
We explain regularization terms that encourage elementwise periodicity and pairwise consistency. We first explain thecore ideas underlying the regularization terms (Figs. 3(a) and3(b)) and describe the regularization terms.

Autocorrelation sequence

Probability sequence

Entropy

Spectrum

3.76

(a)
5.08

Probability sequences

andCross-correlation sequence

Entropy

Spectrum

3.93

(b)
5.11

Fig. 3. (a) Examples of probability sequences and the corresponding auto-correlation functions and auto-spectra with entropies. A more periodic probability
sequence results in the auto-spectrum with a smaller entropy. (b) Examples of probability sequence pairs and the corresponding cross-correlation functions andcross-spectra with entropies. A more consistent probability sequence pair results in the cross-spectrum with a smaller entropy.

1) Auto-Correlation Analysis: The auto-correlation function
for a probability sequence ξ ∈ [0, 1]T is given byTRξξ (τ ) =1X
ξt ξ(t+τ −1)%T +1 ,
T t=1

(5)

where % represents the modulo operation. Using the WienerKhinchin theorem, the spectrum of Rξξ is given bySξξ = F Rξξ = |F ξ|·2 ,(6)where a∗ represents the element-wise conjugate operation fora vector a, ⊙ represents the element-wise multiplication, and|a|· represents the element-wise operation that takes the absolute values of the elements of a vector a. The entropy of thespectrum Sξψ is given byHξξ = −T
X

S̄ξψ (t) log S̄ξψ (t),

(11)

t=1

S̄ξψ = Normalize(Sξψ ).

(12)

T ×T

where F ∈ C
represents the discrete Fourier transform(DFT) matrix and |a|·2 represents the element-wise operationthat takes the absolute square values of the elements of a vector
a. The entropy of the spectrum Sξξ is given byHξξ = −T
X

S̄ξξ (t) log S̄ξξ (t),

(7)

t=1

S̄ξξ = Normalize(Sξξ ),(8)

where Normalize(a) represents the normalization operationthat makes the sum of the elements of a vector a equal to
1. If the probability sequence ξ has clear periodicity, i.e., it
has regular periodic patterns and the peak intervals are integer
multiples of some basic time unit (e.g., tatum, beat, or downbeat), the auto-correlation function and the auto-spectrum haveequally-spaced sharp peaks. The negative entropy −Hξξ canbe used as an indicator of periodicity.
2) Cross-Correlation Analysis: The cross-correlation function for probability sequences ξ, ψ ∈ [0, 1]T is given byRξψ (τ ) =TX1ξt ψ(t+τ −1)%T +1 .
T t=1

(9)

K,T

1 X D
ϕ̄kt log ϕ̄D
kt ,
K

(13)

D ·2
ϕ̄D
k = Normalize(|F ϕk | ),

(14)

LD
ACE = −k,t=1

LBACE = −T
X

ϕ̄Bt log ϕ̄Bt ,

(15)

t=1

ϕ̄B = Normalize(|F ϕB |·2 ),LW
ACE = −TX

W
ϕ̄W
t log ϕ̄t ,

(16)
(17)

t=1

Using the Wiener-Khinchin theorem, the magnitude spectrumof Rξψ is given bySξψ = |F Rξψ |· = |(F ξ)∗ ⊙ (F ψ)|· ,If the probability sequences ξ and ψ are consistent in terms of
periodicity, i.e., they have synchronous periodic patterns, thecross-correlation function and the cross-spectrum have equallyspaced sharp peaks. The negative entropy −Hξψ can be used
as an indicator of consistency.
3) Element-Wise Regularization: We propose element-wiseBWregularization terms, denoted by LD
ACE , LACE , and LACE , basedon the auto-correlation entropy (ACE) of the drum onset probability sequence ϕD , that of the beat probability sequence ϕB ,and that of the downbeat probability sequence ϕW :(10)

ϕ̄W = Normalize(|F ϕW |·2 ),(18)

Twhere ϕDdenotes the drum onset probability sek ∈ [0, 1]
quence for drum class k.

We aim to minimize the weighted sum of the individual
regularization terms:
D
B
B
W
W
LACE = λD
ACE LACE + λACE LACE + λACE LACE ,

(19)

B
W
where λD
ACE , λACE , and λACE are adjustable weights.
4) Pairwise Regularization: We propose pairwise regularDW
BW
ization terms, denoted by LDB
CCE , LCCE , and LCCE , based on
the cross-correlation entropy (CCE) for each of the possible
pairs between the drum onset probability sequence ϕD , thebeat probability sequence ϕB , and the downbeat probability
sequence ϕW :K,T

1 X DBϕ̄kt log ϕ̄DB
kt ,
K

(20)

D ·
B ·
ϕ̄DB
k = Normalize(|F ϕk | ⊙ |F ϕ | ),

(21)

LDB
CCE = −k,t=1

LDW
CCE = −

1
K

K,T
X

DW
ϕ̄DW
kt log ϕ̄kt ,

D ·
W ·
ϕ̄DW
k = Normalize(|F ϕk | ⊙ |F ϕ | ),LBW
CCE = −

(22)

k,t=1

T
X

BW
ϕ̄BW
t log ϕ̄t ,

(23)
(24)

t=1

ϕ̄BW = Normalize(|F ϕB |· ⊙ |F ϕW |· ).

(25)

We aim to minimize the weighted sum of the individual
regularization terms:
DB
DW DW
BW BW
LCCE = λDB
CCE LCCE + λCCE LCCE + λCCE LCCE ,

(26)

DW
BW
where λDB
CCE , λCCE , and λCCE are adjustable weights.
5) Total Loss: For regularized joint training, we aim tominimize the sum of the basic supervised loss LBCE in (4),the element-wise regularization term LACE in (19), and the
pairwise regularization term LCCE in (26):

L = LBCE + LACE + LCCE .

(27)

The weights can be adjusted for controlling the relative importance of different tasks.
D. Decoding
In the test phase, given X, we jointly estimate the drum onset probability sequence ϕD , the beat probability sequence ϕB ,and the downbeat probability sequence ϕW using the trained
CRNN. For improved performance, we use different decoding
techniques for detecting the drum onsets YD and the beat anddownbeat times YD and YW .
1) Drum Transcription: The final output YD is obtained by
performing peak-picking and thresholding for ϕD as follows:
(
1 (ϕD
D
kt ≥ δk )
Ykt =
,
(28)
0 (otherwise)
where δk is a threshold configured for each drum instrument.

2) Metrical Analysis: Since the naive thresholding has alimitation in performance, the final outputs YB and YW are detected from from ϕB and ϕW with a metrical analysis method
based on a dynamic Bayesian network (DBN) [21].
We explain the decoding method for the beat probability sequence ϕB . The DBN is a hidden Markov model (HMM) withhidden states zt = [ct , ċt ] at frame t. Here, ct ∈ {1, 2, . . . , ċ}
is a discrete random variable representing the position withina beat at frame t, and ċt ∈ {ċmin , ċmin + 1, . . . , ċmax } is thetotal number of discretized positions within one beat, which
is a random variable representing the tempo at frame t, whereċmin and ċmax denote the minimum and maximum values of thetempo, respectively. Let ∆ be the frame length of the audio
signal and BPMt represent the tempo in beats per minute atframe t. Then, ċt can be expressed as follows:
4 × 60.
(29)ċt = round
BPMt ∗ ∆If we denote the observation sequence as {ot }Tt=1 , the probability model can be defined as follows:p(z1:T , o1:T ) = p(z1:T )p(o1:T |z1:T ).

(30)

Our goal is to estimate the hidden state sequence z∗1:T withthe maximum posterior probability as follows:z∗1:T = argmax p(z1:T |o1:T ),

(31)

z1:T

p(o1:T |z1:T ) ∝ p(z1 )TYp(zt |zt−1 )p(ot |zt ),

(32)

t=2

where p(ot |zt ) represents the observation model, and p(zt |zt−1 )represents the transition model. Equation (31) can be solved
efficiently using the Viterbi algorithm [22]. The term p(z1:T )in (30) represents the generative process of the hidden states
z1:T , which is given byp(z1:T ) = p(z1 )TYp(zt |zt−1 ),

(33)

t=2

where p(z1 ) represents the initial probability and p(zt |zt−1 )represents the transition probability given byp(zt |zt−1 ) = p(ct |ct−1 , ċt−1 )p(ċt |ċt−1 ),p(ct |ct−1 , ċt−1 ) = 1z .

(34)
(35)

where 1z is a function that takes 1 if ct = (ct−1 +1) mod ċt−1
and 0 otherwise. If ct−1 = ċt−1 , we haveċt
p(ċt |ċt−1 ) = exp −λ
−1 ,
(36)
ċt−1
where λ is a hyperparameter. Otherwise, we have(
1 (ct = ct−1 + 1)p(ċt |ċt−1 ) =.
0 otherwise(37)

Finally, for the hidden state sequence x∗1:T with the maximum posterior probability, we detect all t’s where ct = 1. Thissequence is the estimated beat time sequence.

TABLE I
T HE F- MEASURES OF DRUM TRANSCRIPTION AND METRICAL ANALYSIS .
VALUES IN BOLD ARE WITHIN 0.5 PTS OF THE BEST VALUE .
Multi-task
Learning
D B
W
✓ ✓
✓
✓
✓ ✓
✓ ✓
✓ ✓
✓ ✓
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓
✓ ✓ ✓

RegularizedTraining
ACE CCE

✓
✓
✓

✓
✓

✓
✓

✓
✓

Drum Transcription
BD
78.0
77.0
76.6
76.6
77.0
77.6
75.1
77.1
74.2
76.8

SD
71.2
70.1
69.6
71.2
71.3
73.5
68.7
74.1
70.7
72.2

HH
74.2
74.7
76.5
75.2
75.1
75.7
72.4
73.7
76.4
77.1

Ave.
74.5
73.9
74.2
74.3
74.5
75.6
72.1
75.0
73.8
75.4

MetricalAnalysisB
W
90.5 87.3
90.6 90.3 88.9 89.5 93.9 89.2
94.6 89.1
94.1 87.2
92.1 86.3

IV. E VALUATIONThis section reports a comparative experiment conducted forevaluating the proposed regularized multi-task learning method
through ablation study.
A. Experimental Conditions
The RWC Music Database: Popular Music [23] was used forevaluation. It consists of 100 Japanese popular songs (J-POP)
with human-performed and synthesized drum tracks. We used
64 songs with accurate drum onset annotations and 10 songs
without drum annotations. The stereo signals at a sampling rateof 44.1 kHz were processed using short-time Fourier transform(STFT) with a window size of 1024 points and a hop size of
441 points. The left and right channels were concatenated toform the input matrix X.
Our CRNN consists of common convolutional layers forfeature extraction followed by a bidirectional long short-term
memory (BLSTM) network for drum transcription and a TCN
for metrical analysis (Fig. 2). The convolutional layers, witha kernel size of 3 × 3, a padding size of 1 × 1, and a strideof 1, yielded a (512 × 4)-dimensional feature map, on whicha dropout of 30% was applied before feeding it into a linear
layer. The BLSTM network had three layers, each of which had
hidden states of 200 dimensions. The TCN had eleven layerswith a receptive field of about 80 [s]. The weights were set as
B
W
D
B
W
follows: λD
BCE =1, λBCE =λBCE =0.1, λACE =1, λACE =λACE =0,DB
DW
BWand λCCE =λCCE =λCCE =1. We used AdamW [24] with a learning rate of γ = 0.001, weight decays parameters of λ = 10−4 ,β1 = 0.9, β2 = 0.98, and ε = 10−9 for optimization.
To evaluate the effectiveness of the individual componentsof the proposed method, we conducted an ablation study. We
tackled drum transcription and metrical analysis jointly or separately with or without the element-wise and pairwise regularization terms. Five-fold cross-validation was conducted, where
the performance was evaluated on the 64 songs with drum
annotations. We also compared the proposed method with astate-of-the-art method [20].
The performances of drum transcription and metrical analysis were evaluated in terms of the F-measure with error tolerances of 30 [ms] and 70 [ms], respectively. The beat interval
stability was also evaluated in terms of the CMLt and AMLt

TABLE II
T HE PERFORMANCES OF METRICAL ANALYSIS FOR 100 SONGS OF RWC
P OPULAR M USIC DATABASE . B OLD FONTS INDICATE THE BEST VALUES .
F-meas.
Böck et al. [20]
89.5
92.8
Ours

Beat
CMLt
81.8
85.9

Downbeat
AMLt F-meas. CMLt AMLt
91.5
83.1
80.3
88.4
94.3
86.3
81.3
89.4

metrics [25] that consider double/half tempo ambiguity in beatinterpretation. Note that the drum onset annotations might beinconsistent due to the characteristics of annotators. In the
training phase, the ground-truth onset times were perturbed
by being convolved with a Gaussian distribution with a meanof 0 [ms] and a standard deviation of 12 [ms]. For data augmentation, the time stretch operation was applied to each songwith a scaling factor drawn from a Gaussian distribution witha mean of 1 and a standard deviation of 0.1.
B. Experimental Results
As shown in Table I, the multi-task learning approach improved the beat and downbeat detection performances by approximately 4 and 2 pts, respectively, compared with the singletask learning approach. Both the elementwise and pairwise regularizations improved the performance of drum transcription
by approximately 1–3 pts. For metrical analysis, in contrast,
no significant improvement was observed, mainly because the
DBN-based decoding had a dominant impact on the results.
If the naive thresholding was used instead, some performancegain was obtained (not reported in the table). As shown in
Table II, the configuration that achieved the best performanceof beat detection for all the 100 songs of the RWC PopularMusic Database (the ninth row in Table I) outperformed thestate-of-the-art method [20] by approximately 3 pts higher.
Fig. 4 shows the positive and negative effects of the multitask learning. In the left example, since the drum onsets werelocated on the beat and tatum grids, the snare drum onsets on
unnatural positions were suppressed successfully. In contrast,in the right example, although the drum onsets with swing
rhythm were deviated from the beat and tatum grids, they wereencouraged to be synchronized with the grids.
Fig. 5 shows the positive effect of the regularized training.
The histogram of the inter-onset intervals (IOIs) of the hi-hat
onsets had a single sharp peak, meaning that the estimated onsets were placed at a constant interval. Such periodicity-aware
regularization, however, might fail to estimate non-regular drum
patterns (e.g., fill-ins) and degrade the performance.
These results suggest the importance of the weight configuration according to the target data. One promising solution
worth investigation would be to use the self-attention mechanism for automatically adjusting the weights according to theregularity of drum patterns.
V. C ONCLUSION
In this paper, we presented a multi-task learning approach
to joint drum transcription and metrical analysis based on
the periodicity-aware element-wise and pairwise regularization
terms. Specifically, our method aims to minimize the entropy

Ground-truth

Only drumtranscription
Multi-task
learning

Fig. 4. Examples of drum transcription results obtained with the multi-task
and single-task learning approaches. The performance was improved in theleft example and was degraded in the right example.
Without the regularization

With the regularization

IOI (frames)

Fig. 5. Histograms of inter-onset intervals (IOIs) in frame units computedfrom hi-hat onsets detected without/with the regularization (RWC-MDB-P2001 No. 11).

of the auto-spectrum computed from each probability sequenceand that of the cross-spectrum computed from each of thethree sequence pairs, because a smaller entropy indicates astronger element-wise periodicity or pairwise consistency. We
experimentally showed the effectiveness of using the multitask learning and the regularized training for drum transcription. The performance of metrical analysis, however, was not
maximized by the concurrent use of these methods. Further
investigation is thus needed to determine the weights of theloss functions appropriately for each task, compare with otherdeep learning methods, and explore a peak detection methodthat maximizes the performance of the proposed method. In
addition, we plan to investigate a new regularization term basedon the fractal structure inherent in music.
ACKNOWLEDGMENT
This study was partially supported by JSPS KAKENHI Nos.
21K12187, 21K02846, 22H03661, 20H00602, and 21H03572,
JST PRESTO No. JPMJPR20CB, and JST FOREST No. JPMJPR226X.
R EFERENCES[1] C. W. Wu, C. Dittmar, C. Southall, R. Vogl, G. Widmer, J. Hockman,
M. Müller, and A. Lerch. A review of automatic drum transcription.
IEEE/ACM Transactions on Audio, Speech, and Language Processing,Vol. 26, No. 9, pp. 1457–1483, 2018.

[2] R. Vogl, M. Dorfer, G. Widmer, and P. Knees. Drum transcription
via joint beat and drum modeling using convolutional recurrent neuralnetworks. In International Society for Music Information Retrieval
Conference (ISMIR), pp. 150–157. IEEE, 2017.
[3] C. Jacques and A. Roebel. Automatic drum transcription with convolutional neural networks. In International Conference on Digital Audio
Effects (DAFx), pp. 80–86, 2018.
[4] E. Benetos and S. Dixon. Multiple-instrument polyphonic music transcription using a temporally constrained shift-invariant model. The
Journal of the Acoustical Society of America, Vol. 133, No. 3, pp. 1727–
1741, 2013.
[5] C. Jacques and A. Roebel. Data augmentation for drum transcription
with convolutional neural networks. In European Signal Processing
Conference (EUSIPCO), pp. 1–5. IEEE, 2019.
[6] R. Stables, J. Hockman, and C. Southall. Automatic drum transcription
using bi-directional recurrent neural networks. In International Society
for Music Information Retrieval Conference (ISMIR), pp. 591–597, 2016.
[7] S. Ueda, K. Shibata, Y. Wada, R. Nishikimi, E. Nakamura, and K. Yoshii.
Drum transcription using convolutional non-negative matrix factorization
based on deep drum score prior distribution (in japanese). IPSJ SIG
technical reports (EC), Vol. 2019, No. 26, pp. 1–6, 2019.
[8] R. Ishizuka, R. Nishikimi, E. Nakamura, and K. Yoshii. Tatum-level
drum transcription based on a convolutional recurrent neural networkwith language model-based regularized training. In Asia-Pacific Signal
and Information Processing Association Annual Summit and Conference(APSIPA ASC), 2020.
[9] Y. Wang, J. Salamon, M. Cartwright, Nicholas J. Bryan, and J. P. Bello.
Few-shot drum transcription in polyphonic music. In International
Society for Music Information Retrieval Conference (ISMIR), pp. 117–
124, 2020.
[10] R. Ishizuka, R. Nishikimi, and K. Yoshii. Global structure-aware drumtranscription based on self-attention mechanisms. Signals, Vol. 2, No. 3,
pp. 508–526, 2021.
[11] C. Lea, R. Vidal, A. Reiter, and Gregory D. Hager. Temporal convolutional networks: A unified approach to action segmentation. In European
Conference on Computer Vision (ECCV) Workshop, pp. 47–54, 2016.
[12] S. Bai, J. Zico Kolter, and V. Koltun. An empirical evaluation of genericconvolutional and recurrent networks for sequence modeling. In arXiv
preprint arXiv:1803.01271, 2018.
[13] M. E. P. Daveis and S. Böck. Temporal convolutional networks formusical audio beat tracking. In European Signal Processing Conference(EUSIPCO), pp. 1–5. IEEE, 2019.
[14] M. Cartwright and J. P. Bello. Increasing drum transcription vocabulary
using data synthesis. In International Conference on Digital Audio
Effects (DAFx), pp. 72–79, 2018.
[15] K. Choi and K. Cho. Deep unsupervised drum transcription. In International Society for Music Information Retrieval Conference (ISMIR), pp.
183–191, 2019.
[16] H. Kawata, T. Hori, and K. Nakamura. Automatic drum transcription
based on RNN considering phase information (in japanese). IPSJ SIG
technical reports (MUS), No. 27, pp. 1–4, 2019.
[17] T. Oyama, R. Ishizuka, and K. Yoshii. Phase-aware joint beat anddownbeat estimation based on periodicity of metrical structure. In International Society for Music Information Retrieval Conference (ISMIR),
pp. 493–499, 2021.
[18] S. Böck, F. Krebs, and G. Widmer. Joint beat and downbeat tracking withrecurrent neural networks. In International Society for Music Information
Retrieval Conference (ISMIR), pp. 255–261, 2016.
[19] S. Böck, M. E. P. Davies, and P. Knees. Multi-task learning of tempo
and beat: Learning one to improve the other. In International Society forMusic Information Retrieval Conference (ISMIR), pp. 486–493, 2019.
[20] S. Böck and M. E. P. Daveis. Deconstruct, analysis, reconstruct: How to
improve tempo, beat, and downbeat estimation. In International Society
for Music Information Retrieval Conference (ISMIR), pp. 574–582, 2020.
[21] F. Krebs, S. Böck, and G. Widmer. An efficient state-space model
for joint tempo and meter tracking. In International Society for Music
Information Retrieval Conference (ISMIR), pp. 72–78, 2015.
[22] L. R. Rabiner. A tutorial on hidden Markov models and selected
applications in speech recognition. In Proceedings of the IEEE., pp.
77(2):257–286, 1989.
[23] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka. RWC Music
Database: Popular, classical and jazz music databases. In International

Society for Music Information Retrieval Conference (ISMIR), pp. 287–
288, 2002.
[24] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In
International Conference on Learning Representations (ICLR), pp. 1–8,
2017.
[25] M. E. P. Davies, N. Degara, and M. D. Plumbley. Evaluation methods
for musical audio beat tracking algorithms. Technical report, Centre for
Digital Music, Queen Mary University of London, 2009.