2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)

CTC2: End-to-End Drum Transcription
Based on Connectionist Temporal Classification
With Constant Tempo Constraint
Daichi Kamakura, Eita Nakamura, and Kazuyoshi Yoshii
Graduate School of Informatics, Kyoto University, Japan
E-mail: {kamakura, enakamura}@sap.ist.i.kyoto-u.ac.jp, yoshii@i.kyoto-u.ac.jp

Abstract—This paper describes end-to-end automatic drumtranscription for directly estimating a drum score from an audio
signal of popular music using non-aligned paired data. We aim toconvert a sequence of frame-level acoustic features into a sequenceof tatum-level score fragments (three-dimensional multi-hot vectors) representing the presence or absence of the onsets of the bass
and snare drums and the hi-hats. The main challenge of this tasklies in estimating the correct number of inactive tatums having
no onset between active tatums. One may use the connectionist
temporal classification (CTC) for end-to-end training of a deepneural network (DNN) that infers a frame-level state sequence(alignment path) including the special “blank” states representing
the tatum boundaries. At run-time, a drum score is obtained byannexing repeated states and removing all blank states from themost likely frame-level state sequence. This approach, however,tends to yield a shortened drum score in which repeated inactive
tatums are annexed mistakenly because the blank state (tatumboundary) cannot be distinguished acoustically from the inactive
state (onset absence) at the frame level. In this paper, we propose asophisticated version of the CTC with constant tempo constraint,
CTC2 in short, that encourages each tatum to be aligned withalmost the same number of frames. Although the loss function canbe computed efficiently as in the basic CTC, the backpropagation
over the huge computation graph made through the forward algorithm is computationally prohibitive. To solve this problem, we
propose to perform the backpropagation with only an alignment
path stochastically drawn with Gibbs sampling. The experimentshowed that the proposed method worked well as expected.

I. I NTRODUCTION
Automatic drum transcription (ADT) is a fundamental taskthat aims to estimate a drum score (MusicXML format) from amusic signal. Although ADT plays a key role in computationalmusic understanding, it has been tackled only partially; moststudies aim to estimate a piano roll (MIDI format) by detecting
the onset times of drums in seconds [1], [2]. To estimate a drumscore, for example, one may estimate the beat and downbeat
times in advance [3], [4] and then perform quantized onset
detection [5]. Instead, one may take a more sophisticated approach in the same way as the state-of-the-art automatic pianotranscription [6] based on the combination of audio-to-MIDI
transcription [7] and MIDI-to-score transcription (rhythm transcription) [8]. Such a cascading approach, however, suffersfrom the error propagation problem.
Inspired by the great success of end-to-end automatic speech
recognition (ASR), end-to-end automatic music transcription

Audio signal

CTC
1

Posterior BD SD HH
probability No drums

01

Estimateddrum score
CTC2
1

Posterior BD SD HH
probability No drums

01

Estimateddrum score

Fig. 1. The comparison of the basic CTC and the proposed CTC with constanttempo constraint (CTC2). The repeated inactive tatums having no drum onsets
can be estimated correctly by encouraging the frames to be segmented into
the tatums with almost the same length.

(AMT) has recently been investigated [9]–[13]. This approach
can circumvent the error propagation problem and make effective use of non-aligned pairs of music signals and the corresponding scores as training data. Considering the monotonic
nature of the audio-to-score mapping, we focus on the connectionist temporal classification (CTC) [14] (cf. [9]) because
it is expected to work stably with a limited amount of training
data, compared with the encoder-decoder model (cf. [10]) andthat with the attention mechanism [15] (cf. [11]–[13]).
End-to-end AMT methods can also be characterized by theirscore representations: how to define symbols constituting theoutput sequence. The representation affects the difficulty uniqueto AMT in estimating the temporal information about notevalues (quantized durations) and metrical structures. The basic
way is to convert a sequence of frame-level acoustic features
into a sequence of notation-level score components (e.g., notes
and bars) [12]. The note values (temporal attribute), however,are considerably harder to estimate than the note pitches (instantaneous attribute) with the basic input-output alignment
mechanism. Another way is to estimate a sequence of tatumlevel score fragments [11], where the tatum is a basic time unit
on the score at the sub-beat level (typically at the sixteenthnote level). Instead of directly estimating the note values, thisapproach aims to estimate the presence or absence of the note
onsets (instantaneous attribute) at the tatum level.

モデル全体像

In this paper we tackle CTC-based ADT that aims to estimate a sequence of tatum-level score fragments (Fig. 1). Eachfragment (tatum) is represented as a three-dimensional multihot vector indicating the presence or absence of the onsets ofthe bass and snare drums and the hi-hats (eight states in total).
The main challenge unique to this task is to estimate the correct
number of inactive tatums (a state corresponding to the all-zero
vector) between active tatums (the other seven states). If thebasic CTC is used for end-to-end training, the state of eachtatum of the ground-truth sequence is associated with framesof the input sequence whose acoustic features are particularly
relevant to the state through forced alignment. Such frames are
typically only at the beginning of the tatum; the special blankstate (denoted by “_”) is associated with the other acousticallyirrelevant frames. At run-time, however, successive inactive
tatums are annexed mistakenly because the blank state (tatumboundary) and the inactive state (onset absence) are hard to
distinguish acoustically at the frame level.
To solve this problem, we propose a sophisticated versionof the CTC with constant tempo constraint, CTC2 in short,based on the reasonable assumption that the tempo is usually
kept constant throughout the song in popular music (Fig. 1).
In the basic CTC, the frame-to-tatum alignment is obtained
through unconstrained time stretch because only the order of
output symbols (states) matters and their durations are ignored,
as in the inference of the latent state sequence of the hidden
Markov model (HMM). Inspired by the hidden semi-Markov
model (HSMM) with explicit duration modeling [16], we instead encourage the tatum boundaries (transitions between themain eight states and the blank state) to occur regularly with aconstant interval, where adjacent tatums favor to have the same
duration [3], [17]. This makes successive inactive frames split
into an appropriate number of successive inactive tatums at theframes associated with the blank state.
We also propose an efficient training method that works with
the CTC and its variants. The objective function of the CTC2 to
be maximized is given by the sum of the posterior probabilitiesof all possible alignment paths as in the CTC. Although it
can thus be computed efficiently with dynamic programming(forward algorithm) in the analogy of the HMM to the HSMM,
the backpropagation over the huge computation graph is computationally prohibitive. We thus perform the backpropagation
along only an alignment path stochastically drawn according to
the posterior probability with Gibbs sampling. Technically, we
run the forward filtering-backward sampling in the evaluation
mode to draw a path and then make a compact computationgraph along the path in the training mode.
II. R ELATED W ORKThis section reviews related work on ADT and that on endto-end AMT.
A. Automatic Drum Transcription
Deep neural networks (DNNs) have intensively been used
for ADT [1], [2], [5], [18]–[22]. In general, the spectrogramof an audio signal is used as input, and the annotated drum

Insert blank labels

Spectrograms(8+1)-state classification
Ground-truth

CTC2 Loss

Minimize
Drum Score

Estimation

CNN

Fig. 2. The proposed ADT method based on the CTC2.

onset times are used as training data. For feature extraction,
convolutional neural networks (CNNs) have commonly beenused. Regularization based on prior information about drumscores [5] and refined network architectures [23] have also
been attempted, resulting in improved performance of transcription. For converting DNN outputs into musical scores, weuse beat information for quantization, and it has been reported
that using temporal convolutional networks (TCNs) [24], [25]for beat tracking is effective. However, one problem with thesemethods is that they require as training data a sufficient amountof music spectrograms with precise annotations of drum onset
times. To address this issue, the use of synthetic datasets [26],data augmentation [19], and unsupervised learning [27] havebeen proposed. Nevertheless, these approaches still face challenges in terms of robustness against real performances andthe ability to handle variations in drum sounds.
B. End-to-End Automatic Music Transcription
For end-to-end AMT that aims to directly estimate a musical score (a sequence of musical notations on the score), theattention mechanism [12] and the CTC [28], [29] have been
investigated. In singing transcription based on the attention
mechanism, the tempo consistency was taken into account [12],where the frame-to-tatum alignment is encouraged to be monotonic and regular by imposing a regularization term on theattention matrix. Such regularization, however, was found toprevent the initial progress of the training.
III. P ROPOSED M ETHODThis section describes the audio-to-score ADT method basedon the CTC2-based end-to-end training (Fig. 2).
A. Problem Specification
Our goal is to estimate a drum score from a sequence of thestereo power spectra of a target musical piece, X ≜ {xt }Tt=1(xt ∈ R2F ), where F is the number of frequency bins and T
is the number of frames. We aim to estimate the presence orabsence of the onsets of I drum instruments at the tatum level.
In this study, we focus on the three common drum instruments:
bass drum (BD), snare drum (SD), and hi-hats (HH) (I = 3).
Let yn ∈ {1, . . . , K} be the state of tatum n, which is definedas yn = 1+20 ⟨BD⟩n +21 ⟨SD⟩n +22 ⟨HH⟩n , where K = 2I = 8,
and ⟨DR⟩n is a binary value indicating the presence or absenceof an onset of DR at the tatum. Let Y ≜ {yn }N
n=1 , where N is
the number of tatums. The tatum is defined as the one-fourthof the beat, i.e., the tatum and beat correspond to the durationsof the sixteenth and quarter notes, respectively.

1

CTC：ドラム採譜
B. Training and Inference
We describe the basic flow of end-to-end ADT. In addition
to the main states indexed by {1, 2, . . . , K} (Section III-A),
we introduce the special blank state indexed by 0. Let π ≜{πt }Tt=1 be a redundant state sequence, where πt ∈ {0, . . . , K}
represents the state of frame t. We use a DNN with parametersθ that outputs the posterior probabilities of the (K + 1) statesat the frame level, denoted by ϕ ≜ {ϕk,t }K,T
k=0,t=1 , where ϕk,trepresents the probability of state k at frame t.
1) Training: Given a non-aligned pair of X and Y as training data, we train the DNN. Let B(π) be a one-to-one reducer
that returns Y by annexing repeated states and removing all
blank states from π, e.g., B(01110112) = B(01100122) =112. Let B −1 (Y) be a one-to-many expander that returns a
set of all possible redundant state sequences that reduce toY, e.g., B −1 (112) = {01110112, 1100122, . . . }. The DNN
parameters θ are optimized such that the following posterior
probability is maximized:
L = log p(Y|X, θ)X= logp(π|X, θ).

(1)2) Inference: Given X, we estimate Y using a trained DNN
with the parameters θ. The redundant state sequence π is
obtained by selecting the most likely state at each frame:(2)k

The final output Y is given by Y = B(π).
C. Conventional CTC
As a baseline, we briefly explain the training method basedon the basic CTC. The posterior probability of an alignment
path π in (1) is given by the product of the frame-wise posterior probabilities of the states as follows:p(π|X, θ) =TY

ϕπt ,t .

(3)

t=1

The loss function L or p(Y|X, θ) in (1) can be computed
efficiently with a dynamic programming technique called theforward algorithm as in the HMM. Let Y′ ≜ {ys′ }Ss=0 be an
expanded state sequence obtained by inserting the N −1 blankstates between the N states of Y and pushing the blank states′to the front and back of Y, i.e., S = 2N + 1, y2n−1= yn , and′ys = y⌊ 2s +1⌋ . Let αt (s) be the forward probability obtained byaccumulating the posterior probabilities of all possible paths
that align xt with ys′ :αt (s) =tYXϕπτ ,τ ,(4)

π1:t ∈B−1 (y1:⌊ s ⌋ ) τ =1
2

where a1:t ≜ {a1 , . . . , at }. Since an alignment path π is valid
if the last frame xT is aligned with the last non-blank symbol′
yS−1= yn or the last blank symbol yS′ , we havep(Y|X, θ) = αT (S − 1) + αT (S).

A path added to the forward probability

a)
Frame

Tatum

b)
Frame

Fig. 3. Examples of the alignment path π estimated by a) the CTC- and b)
CTC2-based training methods.

The forward probabilities can be computed recursively withthe time complexity of O(T S). First, we set the initial values:π∈B−1 (Y)πt = argmax ϕk,t .

Tatum

(5)

α1 (1) = ϕ0,1 ,(6)

α1 (2) = ϕy1 ,1 ,(7)

α1 (s) = 0, ∀s > 2.

(8)

We then use the following recursive formulas:′
if ys′ = blank or ys−2
[αt−1 (s) + αt−1 (s − 1)] ϕπs ,t ,(9)αt (s) =else[αt−1 (s) + αt−1 (s−1) + αt−1 (s−2)] ϕπs ,t .
In the inference stage, however, the optimal alignment path
π estimated by (2) tends to deviate considerably from the
diagonal line on the alignment map between the input sequence
X and the estimated sequence Y′ (Fig. 3-a). The durations of
the N symbols in Y = B(π) thus have a large variation,
meaning that the tempo is allowed to frequently change at thetatum level in a musically-unnatural manner.
D. Training with Proposed CTC2
We then explain the training method based on the proposed
CTC2 for explicit duration modeling with the constant tempo
constraint (Fig. 4). Let D = {dn }Nn=1 (Dmin ≤ dn ≤ Dmax )denote the durations of the N symbols of Y, where Dmin
and Dmax are the minimum and maximum durations to beconsidered. When the frame shift is 10 [ms], the local tempo attatum n is 1500/dn [bpm]. Let B(π) be redefined as a one-toone reducer that returns a pair (Y, D), e.g., B(01110112) =(112, 421), B(01100122) = (112, 412). Let B −1 (Y, D) be
redefined as a one-to-many expander that returns all possible sequences that reduce to (Y, D), e.g., B −1 (112, 412) ={01110112, 1100122, . . . }. Note that even if both Y and D
are given, B −1 (Y) may return multiple paths.
The posterior probability of Y in (1) is given by
X
p(Y|X, θ) =p(Y|X, θ, D)p(D),
(10)
D

4

CTC：ドラム採譜
Input spectrogram𝑦0′ = 0𝑦1′ = 6𝑦2′ = 0𝑦3′ = 1𝑦4′ = 0𝑦5′ = 5𝑦6′ = 0DNN

Target・・・

𝑦1 = 6𝑦2 = 1𝑦3 = 5𝑦4 = 1𝑦5 = 7

Frame

HH
SD
BD

Drums

𝑑1 = 2 𝑑2 = 3

𝑑3 = 3𝑝(𝝅|𝐗, 𝜃)

→

0.03

→

0.01

→

0.01

𝑑4 = 3 𝑑5 = 2

Maximize𝑝(𝐘|𝐗, 𝜃)

0.63

・・・

Fig. 4. The forward algorithm in the proposed CTC2-based training with
explicit duration modeling.

where p(Y|X, θ, D) is the posterior probability of Y conditioned by D and p(D) is the generative probability of D given
by a Markov model as follows:p(D) = p(d1 )NYp(dn |dn−1 ),

(11)

n=2

p(d1 ) ∝ 1,(12)

p(dn |dn−1 ) ∝ exp −λ

dn
−1
dn−1


.

(13)

The loss function L or p(Y|X, θ) in (1) can be computed
efficiently as in the HSMM. Let αt (s, d, c) be the forward
probability obtained by accumulating the posterior probabilities of all possible paths that align xt with ys′ :tYXαt (s, d, c) =ϕπτ ,τ · p(d1:⌊ 2s ⌋ ),π1:t ∈B−1 (y1:⌊ s ⌋ ,d1:⌊ s ⌋ ) τ =1
2

2

(14)
⌊ 2s ⌋

p(d1:⌊ 2s ⌋ ) = p(d1 )Yp(dn |dn−1 ),

(15)

n=2

where d represents the duration of the current main state y⌊ 2s ⌋and c is a counter variable that is set to d when a new state
starts and is then decremented until the state ends. We havep(Y|X, θ) =

S
X

D
max
X

d
X

αT (s, d, c).

(16)s=S−1 d=Dmin c=1

Note that if λ = 0 in (13), i.e., the durations of the output
symbols are allowed to take any values with the same probability, i.e., p(D) = 1, the CTC2 reduces to the CTC, i.e., (14)and (16) reduce to (4) and (5), respectively.
The forward probabilities can be computed recursively withthe time complexity of O(T SD2 ), where D ≜ Dmax −Dmin +1. First, we set the initial values:
α1 (1, d, c) = ϕ0,1 , ∀d, c,(17)

α1 (2, d, c) = ϕy1 ,1 , ∀d, c,(18)

α1 (s, d, c) = 0, ∀s > 2.

(19)

We then use the following recursive formulas:
if d = cif ys′ = blank




0,



′
′

else


Pif ys = ys−2

′
′

′

d′ αt−1 (s − 1, d , 1) · ϕys ,t · p(d|d ),



else


P

′
′
d′ ᾱt (s, d, c) · ϕys ,t · p(d|d ),αt (s, d, c) =(20)
else if d > c
if ys′ = blank¯ t (s, d, c) · ϕys′ ,t ,
ᾱ




else




αt−1 (s, d′ , c + 1) · ϕys′ ,t ,




else



0,
ᾱt (s, d, c) = αt−1 (s − 1, d, 1) + αt−1 (s − 2, d, 1),(21)¯ t (s, d, c) = αt−1 (s, d, c + 1) + αt−1 (s − 1, d, c + 1). (22)ᾱIn the inference stage, the optimal alignment path π estimated by (2) tends to roughly follow the diagonal line on
the alignment map between the input sequence X and theestimated sequence Y′ (Fig. 3-b). The durations D of the N
symbols of Y, which are given by (Y, D) = B(π), are thus
kept almost constant. This enables the active and inactive states(tatums) to have similar durations.
The CTC2, however, considers several orders of magnitudelarger number of possible paths than the CTC and thus makes
the exact backpropagation prohibitive in practice. Note that theforward computation in the inference mode without constructing the computation graph can still be performed quickly.
E. Computationally-Efficient Training
We propose an efficient training method that can be applied to the CTC2 (and the CTC) at a minimum sacrifice ofperformance. Instead of backpropagating the error through all
possible paths on the huge computation graph constructed by
the forward algorithm, we focus on only the most likely pathfound by the Viterbi algorithm or a path randomly selectedaccording to its posterior probability with the Gibbs sampling.
To determine such a path, the forward and backward recursions
can be executed faster in the inference mode. Since the Gibbs
sampling stochastically generates likely paths not limited tothe most likely one, it is expected to be more robust against
local optima than the Viterbi algorithm. Let Z ≜ {zt }Tt=1 bea latent state sequence, where zt ≜ (st , dt , ct ). An alignment
path π is uniquely determined by Z.
1) Viterbi Algorithm: In the forward recursion, the forward
probabilities {αt (zt )}Tt=1 are computed as described in Section III-D except that the sum operations over d′ of (20) andthe additions of (21) and (22) are replaced with the max operations, where the indices and terms that take the maximum
values are memorized. In the backward recursion, the statesof most likely Z are determined in the reverse order by backtracing the memorized path from zT such that αT (zT ) is maximized, where sT ∈ {S − 1, S}, dT ∈ {Dmin , . . . , Dmax }, andcT ∈ {1, . . . , dT }.

2) Gibbs Sampling: We use the forward filtering-backward
sampling algorithm, which was originally proposed for sampling the latent sequence of an HMM or HSMM [30]. In theforward filtering, the forward probabilities are computed as
described in Section III-D. In the backward sampling, the statesof Z are sampled in the reverse order according top(zT ) ∝ αT (zT ),TABLE IT HE TATUM - AND FRAME - LEVEL EVALUATION RESULTS .
MethodCTC [14]
CTC2 (ours)
Vogl et al. [2]

(23)
HH

where Azt zt+1 represents the transition probability from zt tozt+1 obtained by normalizing the following matrix: p(dt |dt+1 ) if condition 1 holds,
1
if condition 2 holds,
A′zt zt+1 =(25)
0otherwise,

BD
CTC2

HH
SD

BD
Ground-truth′ys′ ̸= blank and ys′ ̸= ys+2
and st+1 = st + 2),

HH

condition 2: dt > ct and dt = dt+1 and c = ct+1 + 1 andBDSD(ys′ = blank and st+1 = st + 1 or st+1 = st ).
This section reports a comparative experiment conducted forevaluating the performance of the proposed ADT method andthe effectiveness of the efficient training.
A. Experimental Conditions
The 100 songs of the RWC Music Database: Popular Music [31] were randomly split into 60 and 40 songs for training
and test data, respectively. The stereo signals of each song
sampled at 44.1 kHz were analyzed by short-time Fouriertransform (STFT) with a window size of 1024 pts (F = 513)
and a hop size of 441 pts (10 [ms]). The left and right channels
were concatenated to form the input data X. The tempo wasassumed to be between 50 and 250 [bpm], i.e., the duration ofa tatum was between Dmin = 6 to Dmax = 30.
We trained a convolutional recurrent neural network (CRNN)
with the basic CTC or the proposed CTC2. It has 11 convolutional layers with a kernel size of 3 × 3, a padding size of
1×1, and a stride of 1 yielding a (512×4)-dimensional feature
map, on which a dropout of 30% was applied before feeding it
into a linear layer, followed by three bi-directional long shortterm memory (BLSTM) layers with 200-dimensional hidden
states. We used AdamW [32] with a learning rate γ = 0.001, a
weight decay parameters λ = 10−4 , β1 = 0.9, β2 = 0.98, andε = 10−9 for optimization. When the Viterbi algorithm or the
Gibbs sampling (Section III-E) was used with the CTC, thebasic training (Section III-C) was performed for initialization.
For the CTC2, the basic training could not be tested due tothe huge computational cost.
For comparison, we implemented to test a recent audio-toMIDI transcription method based on multi-task learning thatjointly detects drum onset times and beat times at the framelevel [2]. This method was trained using music signals withframe-level onset annotations, whereas our method was trained
end-to-end using music signals with non-aligned scores. For

F-measure ↑Viterbi
Gibbs
78.6
76.1
79.2
79.0
83.1

SD

condition 1: ct = 1 and (ys′ = blank and st+1 = st + 1 or

IV. E VALUATION

Basic
79.4
-

CTC

(24)

p(zt |zt+1:T ) ∝ αt (zt )Azt zt+1 (t = {T − 1, . . . , 1}),

Tatum error rate ↓BasicViterbi
Gibbs
34.4
35.1
35.6
33.8
33.8
34.5

Tatum (16th note)Fig. 5. Examples of tatum-level drum score fragments estimated by the CRNN
trained with the basic CTC or the proposed CTC2. Whereas the CTC often
failed to detect inactive states, the CTC2 correctly detected them.

MIDI-to-score transcription, the detected drum onset times
were quantized on the tatum grid made of the beat times estimated by an HMM-based beat tracker [3] as post-processing.
Our end-to-end transcription method with the CTC or CTC2
and the cascading method mentioned above [2], [3] were evaluated in terms of the tatum- and frame-level accuracies. For
tatum-level evaluation of estimated scores, we used the tatum
error rate (TER), an edit distance similar in spirit to the word
error rate (WER) in speech recognition, as follows:#insersion + #deletion + #substitution, (26)
N
where #insersion, #deletion, #substitution represent the
numbers of insertion, deletion, and substitution errors, respectively, and N represents the total number of tatums in the
ground-truth data. For frame-level evaluation of estimated onset times, we set the error tolerance to 70 [ms]. Note that theinitial frames of the active states (tatums) determined by theestimated alignment path π were regarded as drum onset times
for convenience in our end-to-end method. Since those onset
times tend to have a constant offset from the ground-truth onset
times, the best offset was found for each song such that theperformance was maximized for fair comparison.
TER =B. Experimental Results
As shown in Table I, the CTC2 steadily outperformed the
CTC in both metrics. Although the quantitative difference wassmall, the CTC2 yielded considerably better drum scores in
the naturalness of rhythm despite the fact that the metricalstructure was not considered. As shown in Fig. 5, the CTC2
successfully estimated a correct number of inactive tatums,

BD SD HH

CTC

Tatum
Frame
BD SD HH

CTC2
Fig. 7. Comparion of convergence speeds achieved by the basic training, theViterbi algorithm, and the Gibbs learning.

Tatum
Frame

Fig. 6. Examples of alignment paths estimated by the CRNN trained with thebasic CTC or the proposed CTC2 in the training phase. Whereas the CTC
yielded a path with sudden tempo changes, the CTC2 yielded a diagonal pathwith an almost constant tempo.

which were missed by the CTC, between active tatums thanks
to the constant tempo constraint. As shown in Fig. 6, in the
training phase based on the forced alignment between the estimated and ground-truth state sequences, the CTC2 yielded analignment path with an almost constant grade (tempo) aroundthe diagonal line on the lattice. The CTC, in contrast, allowed
an alignment path to discontinuously and frequently change thegrade in a musically-unnatural manner. The Viterbi algorithm
and the Gibbs sampling worked comparably in performance(Table I) and were several orders of magnitude faster than thebasic backpropagation on the entire computation graph at aminimum sacrifice of performance (Fig. 7).
Compared with the conventional cascading method [2], ourend-to-end method with the CTC2 performed slightly betterby 0.7 pts in the TER, but worse by 6.0 pts in the F-measure.
Note that the frame-level metric is advantageous for audio-toMIDI transcription methods that use precise frame-level onset
annotations for training. Considering that our method can betrained end-to-end with non-aligned audio-score pairs and does
not aim at frame-level onset detection (the alignment path does
not necessarily indicate precise onset times), this result is still
considered to be promising.
V. C ONCLUSION
In this paper, we proposed an end-to-end ADT method basedon the CTC with constant tempo constraint, CTC2 in short,that estimates a sequence of tatum-level drum score fragmentsfrom a music signal. Although the loss function can be computed efficiently as in the vanilla CTC, the backpropagation

over the huge computation graph made by the forward algorithm is computationally prohibitive. To solve this problem, we
proposed a novel efficient training method that performs the
backpropagation through only an alignment path found by theViterbi algorithm or stochastically drawn with Gibbs sampling.
Since non-aligned drum scores are easier to collect than laborintensive frame-level onset annotations, the proposed end-toend ADT method has a large potential for performance improvement based on large-scale training.
The CTC2 is a general technique for explicit duration modeling in a monotonic sequence-to-sequence mapping task notlimited to ADT. It can be customized by formulating a duration
model according to the task. In handwritten character recognition [33], for example, the CTC with constant “size” constraint
would be useful for encouraging each character to have almost
the same size. The Viterbi algorithm and the Gibbs sampling
can also be applied to a wide range of tasks in various fields foraccelerated CTC-based training. We are currently investigating
how to effectively use the Viterbi algorithm in the inference
phase as well as the training phase.
ACKNOWLEDGMENT
This study was partially supported by JSPS KAKENHI Nos.
21K12187, 21K02846, 22H03661, 20H00602, and 21H03572,
JST PRESTO No. JPMJPR20CB, and JST FOREST No. JPMJPR226X.
R EFERENCES[1] R. Stables, J. Hockman, and C. Southall. Automatic drum transcription
using bi-directional recurrent neural networks. In International Society
for Music Information Retrieval Conference (ISMIR), 2016.
[2] R. Vogl, M. Dorfer, G. Widmer, and P. Knees. Drum transcription
via joint beat and drum modeling using convolutional recurrent neuralnetworks. In International Society for Music Information Retrieval
Conference (ISMIR), pp. 150–157, 2017.
[3] F. Krebs, S. Böck, and G. Widmer. An efficient state-space model
for joint tempo and meter tracking. In International Society for Music
Information Retrieval Conference (ISMIR), pp. 72–78, 2015.
[4] S. Böck and M. E. P. Daveis. Deconstruct, analysis, reconstruct: How to
improve tempo, beat, and downbeat estimation. In International Society
for Music Information Retrieval Conference (ISMIR), pp. 574–582, 2020.

[5] R. Ishizuka, R. Nishikimi, E. Nakamura, and K. Yoshii. Tatum-level
drum transcription based on a convolutional recurrent neural networkwith language model-based regularized training. In Asia-Pacific Signal
and Information Processing Association Annual Summit and Conference(APSIPA ASC), 2020.
[6] K. Shibata, E. Nakamura, and K. Yoshii. Non-local musical statistics as
guides for audio-to-score piano transcription. Information Sciences, Vol.
566, pp. 262–280, 2021.
[7] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel,
J. Engel, S. Oore, and D. Eck. Onsets and frames: Dual-objective pianotranscription. In International Society for Music Information Retrieval
Conference (ISMIR), pp. 50–57, 2018.
[8] E. Nakamura, E. Benetos, K. Yoshii, and S. Dixon. Towards complete
polyphonic music transcription: Integrating multi-pitch detection andrhythm quantization. In International Conference on Acoustics, Speech,
and Signal Processing (ICASSP), pp. 101–105. IEEE, 2018.
[9] M. A. Román, A. Pertusa, and J. Calvo-Zaragoza. An end-to-end framework for audio-to-score music transcription on monophonic excerpts.
In International Society for Music Information Retrieval Conference(ISMIR), pp. 34–41, 2018.
[10] R. C. G. Carvalho and P. Smaragdis. Towards end-to-end polyphonicmusic transcription: Transforming music audio directly to a score. In
IEEE Workshop on Applications of Signal Processing to Audio andAcoustics (WASPAA), pp. 151–155. IEEE, 2017.
[11] R. Nishikimi, E. Nakamura, M. Goto, and K. Yoshii. End-to-end melody
note transcription based on a beat-synchronous attention mechanism.
In IEEE Workshop on Applications of Signal Processing to Audio andAcoustics (WASPAA), pp. 26–30. IEEE, 2019.
[12] R. Nishikimi, E. Nakamura, S. Fukayama, M. Goto, and K. Yoshii.
Automatic singing transcription based on encoder-decoder recurrent
neural networks with a weakly-supervised attention mechanism. In
International Conference on Acoustics, Speech, and Signal Processing(ICASSP), pp. 161–165. IEEE, 2019.
[13] L. Liu, V. Morfi, and E. Benetos. Joint multi-pitch detection and scoretranscription for polyphonic piano music. In International Conference on
Acoustics, Speech and Signal Processing (ICASSP), pp. 281–285. IEEE,
2021.
[14] A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber. Connectionist
temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In International conference on Machine Learning(ICML), pp. 369–376, 2006.
[15] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by
jointly learning to align and translate. In International Conference on
Learning Representations (ICLR), pp. 1–15, 2015.
[16] M. Russell and R. Moore. Explicit modelling of state occupancy inhidden markov models for automatic speech recognition. In International
Conference on Acoustics, Speech and Signal Processing (ICASSP),
Vol. 10, pp. 5–8. IEEE, 1985.
[17] E. Nakamura, K. Yoshii, and S. Sagayama. Rhythm transcription ofpolyphonic piano music based on merged-output HMM for multiple
voices. IEEE/ACM Transactions on Audio, Speech, and Language
Processing, Vol. 25, No. 4, pp. 794–806, 2017.
[18] C. Jacques and A. Roebel. Automatic drum transcription with convolutional neural networks. In International Conference on Digital Audio
Effects (DAFx), 2018.
[19] C. Jacques and A. Roebel. Data augmentation for drum transcription
with convolutional neural networks. In European Signal Processing
Conference (EUSIPCO), pp. 1–5. IEEE, 2019.
[20] S. Ueda, K. Shibada, Y. Wada, R. Nishikimi, E. Nakamura, andK. Yoshii. Drum transcription using convolutional non-negative matrix
factorization based on deep drum score prior distribution (in japanese).
IPSJ SIG technical reports (EC), Vol. 2019, No. 26, pp. 1–6, 2019.
[21] Y. Wang, J. Salamon, M. Cartwright, Nicholas J. Bryan, and J. P. Bello.
Few-shot drum transcription in polyphonic music. In International
Society for Music Information Retrieval Conference (ISMIR), pp. 117–
124, 2020.
[22] R. Ishizuka, R. Nishikimi, and K. Yoshii. Global structure-aware drumtranscription based on self-attention mechanisms. Signals, Vol. 2, No. 3,
pp. 508–526, 2021.
[23] D. Kamakura, T. Oyama, and K. Yoshii. Drum transcription and beatstructure estimation based on multi-task learning (in japanese). The 84th
National Convention of IPSJ, Vol. 2022, No. 1, pp. 517–518, 2022.
[24] C. Lea, R. Vidal, A. Reiter, and G. D. Hager. Temporal convolutional

networks: A unified approach to action segmentation. In European
Conference on Computer Vision (ECCV) Workshop, pp. 47–54, 2016.
[25] M. E. P. Daveis and S. Böck. Temporal convolutional networks formusical audio beat tracking. In European Signal Processing Conference(EUSIPCO), pp. 1–5. IEEE, 2019.
[26] M. Cartwright and J. P. Bello. Increasing drum transcription vocabulary
using data synthesis. In International Conference on Digital Audio
Effects (DAFx), pp. 72–79, 2018.
[27] K. Choi and K. Cho. Deep unsupervised drum transcription. In International Society for Music Information Retrieval Conference (ISMIR), pp.
183–191, 2019.
[28] R. G. C. Carvalho and P. Smaragdis. Towards end-to-end polyphonicmusic transcription: Transforming music audio directly to a score. In
Workshop on Applications of Signal Processing to Audio and Acoustics(WASPAA), pp. 151–155. IEEE, 2017.
[29] M. A. Román, A. Pertusa, and J. Calvo-Zaragoza. An end-to-end framework for audio-to-score music transcription on monophonic excerpts.
In International Society for Music Information Retrieval Conference(ISMIR), pp. 34–41, 2018.
[30] J. Van Gael, Y. Saatci, Yee Whye Y. W. Teh, and Z. Ghahramani.
Beam sampling for the infinite hidden Markov model. In International
conference on Machine Learning (ICML), pp. 1088–1095, 2008.
[31] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka. RWC Music
Database: Popular, classical and jazz music databases. In International
Society for Music Information Retrieval Conference (ISMIR), pp. 287–
288, 2002.
[32] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In
International Conference on Learning Representations (ICLR), pp. 1–8,
2017.
[33] M. Ibrayim, W. Simayi, and A. Hamdulla. Unconstrained online
handwritten uyghur word recognition based on recurrent neural networks
and connectionist temporal classification. International Journal of
Biometrics, Vol. 13, No. 1, pp. 51–63, 2021.