2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)

Audio-to-Score Singing Transcription Based on Joint
Estimation of Pitches, Onsets, and Metrical Positions
With Tatum-Level CTC Loss
Tengyu Deng, Eita Nakamura, and Kazuyoshi Yoshii
Graduate School of Informatics, Kyoto University, Japan
E-mail: deng@sap.ist.i.kyoto-u.ac.jp, eita.nakamura@i.kyoto-u.ac.jp, yoshii@i.kyoto-u.ac.jp

Abstract—This paper describes an end-to-end singing transcription method that directly estimates a musical score (a sequence of sung notes with metrical positions) from a music audio
signal. The monotonicity of audio-to-score mapping naturally
calls for the use of connectionist temporal classification (CTC).
Inspired by the success of character-level automatic speech recognition, previous studies on CTC-based music transcription represent a musical score as a sequence of various kinds of symbols (e.g., note pitches and values and barlines) defined in somemusic notation. Such a naive notation-respecting representation,
however, does not fit the non-overlapping monotonic audio-tosymbol alignment and the positions of barlines and the durationsof beats tend to be incoherent in the estimated score. To solve thisproblem, we propose a tatum-level singing transcription methodthat jointly estimates the pitch (including rest), onset flag, andmetrical position at each tatum. Our approach enables the tatums
to be monotonically aligned with regularly-spanned intervals of
the music signal and the estimated notes are located on theestimated metrical positions that are encouraged to be periodic.
Experimental results clearly showed that our proposed model
reached comparable accuracy on score-level singing transcription
with only unaligned training data, and the proposed tatum-level
representation significantly improved the stability of the metricalstructures in the estimated scores.

(a) Musical score.

61 4
66
Pitch Duration

4

65

4

61

12

(b) Alignment with note-level representation.

MetricPitch
Onset

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 1 2 3 4 5 6 761 61 61 61 66 66 66 66 65 65 65 65 61 61 61 61 61 61 61 61 61 61 61 61
1

0

0

0

1

0

0

0

1

0

0

0

1

0

0

0

0

0

0

0

0

0

0

0

(c) Alignment with tatum-level representation.
Fig. 1. Comparison of (b) conventional and (c) proposed methods of CTCbased audio-to-score transcription.

I. I NTRODUCTION
Automatic singing transcription (AST) is one of the mostfundamental music analysis tasks in the field of music information retrieval (MIR). Its ultimate goal is to estimate a musicalscore of the vocal part (described in the MusicXML format)
from a music audio signal. The estimated scores can be used
for melody-based music search (e.g., query-by-humming [1],[2]), score-guided singing voice separation [3], [4], and interpretable emotion recognition [5].
Most studies on AST have attempted to estimate a pianoroll of the vocal part (described in the MIDI format) [6]–[10],where each sung note is represented by a semitone-level pitchand onset and offset times in seconds. The basic strategy is
to estimate a continuous contour of fundamental frequencies(F0s) in Hz (a.k.a., melody extraction) with a deep neuralnetwork (DNN) and then segment it to a series of sung andrest notes (discontinuous step function) with a note tracking
method [7], [8]. Note that the piano roll differs from the musical score in the time units of the note onsets and offsets.
The onsets and offsets are represented in frames or seconds inthe piano roll representation, whereas they are represented inquantized units such as beats or fractions of beats in the score
representation. To obtain a musical score from a piano roll,rhythm transcription methods [11] are required for quantizing
the onset and offset times of notes onto a grid represented inmusical metrics such as beats or tatums.
Only a few studies have tackled audio-to-score transcription[12], [13], which typically involve audio-to-MIDI transcription
and MIDI-to-audio transcription. Such a cascading approach,however, suffers from several problems. Since F0 annotationsaligned with music audio signals are limited both in size andvariation, the accuracy of supervised F0 estimation is hard to
improve. The complicated dynamics of the F0 contour (e.g.,
vibrato and glissando) make it hard to perform frequency andtemporal quantization without referring to the audio data [12].
This calls for the end-to-end (i.e., audio-to-score) approachto AST that can make effective use of non-aligned pairs ofmusical scores and audio signals.
Considering the monotonicity of audio-to-score alignment,
connectionist temporal classification (CTC) [14] has been used
for audio-to-score automatic music transcription (AMT) [15],

=1

8d

barline

4e

[8e

8eL]

tied note

8B

8d

beam

8eJ

transcribe the musical signals into note events have also bemade [9], using deep neural networks to overcome the error
accumulation problem in cascading methods. To improve theaccuracy of piano-roll transcription, the CTC loss was used as
an auxiliary loss function in addition to the standard framewise cross-entropy loss [10].
In the context of AST that aims to estimate musical scores,
rhythm transcription methods have been explored to quantize
piano-roll representations into musical scores [11], [22], [23].
A hybrid DNN-HSMM model [13], for example, was proposed
to avoid depending on the preprocessing step of F0 estimation,in the same way as the DNN-HMM approach to ASR. In
these studies, representations of rhythmic information for music transcription have been studied. In general, it is possible torepresent rhythmic information either at the tatum level using
metrical positions or at the note level using note values. Studies
using HMMs have shown that the tatum-level representation
generally performs better than the note-level representation
because certain logical structures of musical rhythms cannot
be represented in the latter [23].
Several recent studies have attempted end-to-end AMT toaddress the error accumulation issue in cascading methods.
Inspired by the success of end-to-end ASR with either theattention mechanisms or CTC, studies on end-to-end AMT
have employed both attention mechanism [24] and CTC [15],[16] to directly transcribe audio signals into musical scores.
To facilitate sequence-to-sequence learning in these methods,
it is necessary to represent a musical score as a sequence ofsymbols. Previous CTC-based end-to-end AMT methods [15],[16] used a note-level score representation, where quantized
durations of notes were directly estimated as output symbols.
However, these methods have been shown to work only on
synthetic audio signals made from MIDI-like data. Besides,
an encoder-decoder model with an attention mechanism and atatum-level score representation was also employed [24]. This
study proposed a regularized training method that encourages
the attention matrix to have monotonic and regular structures,aligning the tatums with the frames in a beat-synchronous
manner. However, this approach often failed to find correctalignments because the attention matrix does not necessarily
be monotonic in early epochs of non-regularized training.

==barline(a) The Kern representation.

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 1562 62 64 64 64 64 64 64 64 64 59 59 62 62 64 64
Onset 1 0 1 0 0 0 1 0 0 0 1 0 1 0 1 0MetricPitch(b) The proposed tatum-level representation.
Fig. 2. Comparison of score representations.

[16] (Fig. 1). In the analogy with CTC-based automatic speech
recognition (ASR) that converts a speech signal to a sequenceof characters [17], one may try to convert a music signal toa sequence of various kinds of symbols (e.g., note pitches,note values, and barlines) defined in some music notation (e.g.,
Lilypond [18] and Kern [19], Fig. 2 (a)). It is, however, unclear how such instantaneous and temporal symbols are monotonically aligned with the audio signal in a non-overlapping
manner (Fig. 1 (b)). Note that the basic CTC is intended for
instantaneous symbols (e.g., phonemes and note pitches) thatcan be estimated in local regions. To estimate temporal symbols (e.g., note values and barlines), acoustic features shouldbe aggregated from a nonlocal region. The mixture of thesemodes harms the stability of CTC-based training, leading toincoherent metrical structure in the estimated score. Indeed,
previous CTC-based methods were tested only on synthetic
data and evaluations on real audio data have not been reported
in the literature.
To solve this problem, we propose a CTC-based AST methodthat converts a music audio signal to a tatum-level score representation with a metrical structure. The tatum is a minimum
time unit on a score, which is assumed to be the sixteenth
note length in this study. Representing a musical score as asequence of tatums, each having a pitch (MIDI note number
or 128 (rest)), an onset flag (∈ [0, 1]), and a metrical position(∈ [0, 15]) (Fig. 2 (b)), we can achieve stable monotonic audioto-tatum alignment (Fig. 1 (c)). Specifically, a convolutional
recurrent neural network (CRNN) that predicts the probabilities of pitches, onset flag, and metrical positions from a musicsignal at the frame level is trained such that the CTC loss is
minimized. At run-time, the most likely tatum-level sequenceis estimated with a Viterbi algorithm that explicitly considers
the durations of sung notes and the metrical structure.

III. P ROPOSED M ETHODThis section describes the proposed CTC-based end-to-end
AST method that considers metrical structure.
A. Problem Specification
Given a music audio signal as input, the goal is to estimate
a musical score of sung notes. Let X ≜ {xt }Tt=1 be a framelevel sequence of acoustic features obtained by stacking themel-spectrogram of the music signal and that of the singing
voice separated with a source separation method called OpenUnmix [25], where T is the number of frames. The input toour system is X.
In this paper, the music signal is assumed to have a time signature of 4/4, and the duration of the sixteenth note is regarded

II. R ELATED W ORK
Conventional AST approaches combine frame-level F0 estimation and note tracking, where the piano-roll representation
is a transcription target [20], [21]. Recently, deep learningtechniques have been used for F0 extraction tasks, remarkably
improving the estimation accuracy [6]–[8]. A hidden semiMarkov model (HSMM) [12] was employed to quantize F0
contours into sequences of musical notes. Attempts to directly2bn

as the minimum time unit called a tatum, i.e., each measureincludes 16 metrical positions (tatums). Let Y ≜ {yn }N
n=1 bea tatum-level score representation, where N is the number oftatums and yn ≜ (bn , pn , on ) is a triplet of a metrical positionbn ∈ [0, 15] (relative position in the measure), a semitonelevel pitch pn ∈ [0, 128] represented by a MIDI note number(pn = 128 indicates the rest), and an onset flag on ∈ {0, 1}.
To reconstruct a musical score from Y, we first determine thepositions of barlines from B ≜ {bn }N
n=1 under the assumptionof the time signature and then the note pitches and values areNdetermined from P ≜ {pn }N
n=1 and O ≜ {on }n=1 .

*tatums

0
*

1
*

2
*

3
*

πt

*

*

0

0

0

1

1

1

*

*

*

frames

2

2

*

3

3

3

*

2
66
1

2
66
1

*

3
66
0

3
66
0

3
66
0

*

(a) Single-label CTC.

yn = (bn, pn, on)*

0tatums

0 61 1*

1 61 0*

B. End-to-End Training

2 66 1*In the training phase, given paired data of the input X andthe ground-truth output Y ≜ (B, P, O), we train a DNN with
the CTC loss in a supervised manner. The input length isassumed to be longer than the output length (T > N ), which
always holds in AST.
1) Single-Label CTC: We first explain the standard CTC.
Suppose we aim to estimate only B from X (P and O are
not considered here). Let K be the number of kinds of original
output symbols (metrical positions) (i.e., K = 16). The special
blank symbol “*” working as a wild card is introduced.
A DNN is trained such that the posterior probability of thetatum-level sequence B given as the ground-truth data, denotedby p(B|X), is maximized. Let π ≜ {πt }Tt=1 be a frame-level
sequence such that B = M(π), where πt ∈ [0, K − 1] ∪ {∗}represents the output symbol at frame t, and M(π) is a reduction operator that annexes repeated symbols and removes
all blank symbols from π. If B = {1, 2, 3}, for example,{1, 1, 1, ∗, ∗, 2, ∗, ∗, 3, ∗} is a possible example of π. The DNN
is used for inferring metrical positions in frames as follows:2) Multi-Label CTC: In our study on joint pitch, onset,
and metrical estimation, we use the multi-label CTC (MCTC)
loss [26]. Let I be the size of the pitch vocabulary (I = 129).
A DNN is trained such that the posterior probability p(Y|X)
of the tatum-level sequence Y given as the ground-truth data is
maximized. Let π ≜ {πt }Tt=1 be a frame-level sequence (alignment path) such that Y = M(π), where πt ≜ (bt , pt , ot ) ∈{(k, i, {0, 1})}K−1,I−1
k=1,i=1 ∪{∗} denotes the output symbol (triplet
or ∗) at frame t, and M(π) is a reduction operator. The DNN
is used for jointly inferring metrical positions, note pitches,and onset flags at the frame level as follows:(ϕB , ϕ∗ ) = DNN(X),(ϕB , ϕP , ϕO , ϕ∗ ) = DNN(X),

3 66 0*πt = (bt, pt, ot) *

(1)

0
61
1

0
61
1

1
61
0

1
61
0

1
61
0

*

*

*

(5)

T,I−1
P
O
O T
where ϕP ≜ {ϕPti }t=1,i=1 , ϕ ≜ {ϕt }t=1 , ϕti is the posteriorO
probability of pitch i when πt ̸= ∗, and ϕt is that of the onset

presence at frame t.
In the same ways as Eq. (2), p(Y|X) can be computed withdynamic programming (DP) as follows:
X
p(Y|X) =p(π|X),(6)π∈M−1 (Y)where M−1 (Y) is an expansion operator. Assuming the conditional independence over frames and labels, the posterior
probability p(π|X) can be computed in the same ways as
Eq. (3), where p(πt |X) is given by

where M−1 (B) is an expansion operator that maps B to aframe-level sequence and p(π|X) is the posterior probabilityof a sequence π. Assuming the conditional independence overframes, p(π|X) is given byp(πt |X),

0
61
1Fig. 3. Possible alignment paths in CTC.

π∈M−1 (B)

T
Y

0
61
1

(b) Multi-label CTC.

T,K−1
∗
∗ T
B
where ϕB ≜ {ϕBtk }t=1,k=1 , ϕ ≜ {ϕt }t=1 , ϕtk is the posterior
probability of metrical position k when πt ̸= ∗, and ϕ∗t is thatof πt = ∗ at frame t.
p(B|X) is obtained by accumulating the posterior probabilities over all possible π’s (frame-to-tatum alignments) that canbe reduced to B as follows (Fig. 3 (a)):
X
p(B|X) =p(π|X),(2)

p(π|X) =frames*p(πt |X)(P
O ot
O 1−ot(1 − ϕ∗t )ϕB
tbt ϕtpt (ϕt ) (1 − ϕt )
=
∗
ϕt

(3)

t=1

if πt ̸= ∗,(7)otherwise.

C. Network Architecture

where p(πt |X) is the local posterior probability given by((1 − ϕ∗t )ϕB
if πt ̸= ∗,tπtp(πt |X) =(4)∗
ϕtotherwise.

The DNN used in this study is based on an encoder-decoderarchitecture (Fig. 4). Since wide-band harmonic structures andlong-term temporal structures are considered to be useful clues
for pitch estimation and metrical analysis, respectively, two3EncoderBX
EncoderP,O

DecoderB

ϕB

Decoder*

ϕ*

ZB

P,O

Z

DecoderP,Oand encourages the durations of metrical positions to be stableby assuming


dn
(16)
−1 .
p(dn |dn−1 ) ∝ exp −λ
dn−1

ϕP

With the HSMM, one can efficiently decode the sequence of
metrical positions, pitch values and onset flags with the Viterbi
algorithm (see details in Appendix A).

ϕO

Fig. 4. Network architecture.

IV. E VALUATION

separate encoders are used to extract latent metrical features
ZB and latent pitch features ZP,O as follows:This section reports experiments conducted for evaluating
the effectiveness of the proposed method.

ZB = EncoderB (X),P,OZ

= Encoder

P,O

(X).

(8)

A. Experimental Conditions(9)1) Dataset: We made a dataset consisting of 343 Japanese
popular (JPOP) songs with the corresponding musical scoresin the MusicXML format transcribed by experts and real audio
recordings extracted from the original CDs. It was randomly
split into a training set of 308 songs and a test set of 35 songs.
Although this JPOP dataset is useful for evaluating the practical performance, it cannot be released due to copyright issues.
We thus also used the publicly available RWC Popular Music
dataset [27] for evaluation. In detail, we randomly selected 12
songs with the signature of 4/4 to form a test dataset (Nos. 7,
8, 13, 18, 20, 47, 63, 79, 80, 84, 90, and 100). The vocal scoresof these songs were transcribed by experts in the MusicXML
format as well.
Note that most common datasets (e.g., MIR-ST500 [28])consist of singing voices with annotations in the MIDI format,
where the onset and offset times of each note are given inseconds. These datasets have thus been used for audio-to-MIDI
AST, but are hard-to-use for audio-to-score AST (ours) due tothe lack of ground-truth musical scores.
To reduce the computational cost, all the audio recordings
were resampled at 22050 Hz and cut into segments of 8 seconds. To find the corresponding segments of scores, we synthesized waveforms from the scores and aligned them withthe original signals with dynamic time warping (DTW). Thesealignment results were then used to segment the ground truthscores. The spectrogram of a music signal was obtained withshort-time Fourier transform (STFT) with a window of 2048
samples and a hop length of 256 samples (11.61 ms). The mel
spectrogram was then obtained with 128 mel filter banks.
2) Network Configuration: The CRNN used for evaluation
was configured as follows. EncoderB and EncoderP,O were
implemented with CNNs of the same structure. Each CNN had
five layers and the numbers of channels for these layers were64, 32, 32, 32, and 32, respectively. The kernel sizes for thehidden layers were 5, 5, 3, 3, and 3, respectively. Following the
CNN, a fully-connected layer was used for reducing the hidden
dimension to 256 at each frame. DecoderB , DecoderP,O and
Decoder∗ were implemented with RNNs. Each RNN had twolayers of BLSTMs, and the number of channels in the hidden
layers was all 256. The CRNN was trained by an Adam optimizer with a learning rate of 1 × 10−4 for 20 epochs with thetraining set.

To estimate the pitch and onset presence at each frame, it
is sufficient to focus on the local region of the singing voicearound the frame. To estimate the metrical position, in contrast,
it is necessary to collect long-term periodic information overmultiple measures from the background music.
We then compute the posterior probabilities of the metrical
positions, pitches, onset presence, and blank symbol, denotedby ϕB , ϕP , ϕO , and ϕ∗ , as follows:
ϕB = DecoderB (ZB ),P

O(ϕ , ϕ ) = Decoder

P,O

(Z

P,O

(10)
),

ϕ∗ = Decoder∗ (ZB , ZP,O ).

(11)
(12)

Since the blank symbol tends to be selected at frames withsmall confidence about the pitch, onset, or metrical estimation,
all the latent features are considered.
D. Decoding
After the CTC-based training, one can estimate the frameand tatum-level sequences π̂ and Ŷ as follows:π̂ = arg max p(π|X),(13)

Ŷ = M(π̂).

(14)

π

According to this method, however, the estimated metrical
positions may not have a monotonically increasing order.
To impose the constraint of monotonically increasing metrical positions, we propose a refined decoding method basedon an HSMM. Generally in music performances including
singing, the durations of beats or tatums keep stable becauseof the regularity of metrical structure in music. To utilize
this temporal stability, we introduce the durations of tatums
D ≜ {dn }N
n=1 measured in frames. Consider the frame-level
input features and tatum-level output features described in Section. III-A, namely, input spectrograms X ≜ {xt }Tt=1 , andN
output metrical positions B ≜ {bn }N
n=1 , pitches P ≜ {pn }n=1
N
and onset flags O ≜ {on }n=1 . We formulate the generative
process of B, P, O, D and X as an HSMM, which only allows
the metrical positions to make periodic progress, that is,(bn−1 + 1 bn−1 < 15,
(15)
bn =
0
bn−1 = 15,
4

TABLE IC OMPARISON OF EDIT- DISTANCE - BASED ERROR RATES (%) OF DIFFERENT METHODS .
Test Dataset
JPOP-test

RWC-test

MethodVOCANO [9] + quantization [22]Note-level CTC (baseline)
Tatum-level CTC (proposed)
VOCANO [9] + quantization [22]Note-level CTC (baseline)
Tatum-level CTC (proposed)

Pitch Error
9.0
6.9
8.8
12.7
9.1
9.4

Miss Rate
47.8
24.6
23.4
16.5
12.1
13.3

Extra Rate
14.2
7.5
14.1
15.6
11.7
18.9

Onset Error
47.4
37.3
35.1
40.6
35.5
32.3

Offset Error
31.2
27.3
26.4
29.5
24.0
26.9

Mean Error
29.9
20.7
21.6
23.0
18.5
20.2

TABLE IIC OMPARISON OF BEAT AND DOWNBEAT F SCORES (%) FOR DIFFERENT METHODS .
Test Dataset
JPOP-test

RWC-test

MethodVOCANO [9] + quantization [22]Note-level CTC (baseline)
Tatum-level CTC (proposed)
VOCANO [9] + quantization [22]Note-level CTC (baseline)
Tatum-level CTC (proposed)

Precision
53.3
50.7
80.5
39.5
41.0
77.0

The HSMM used in the decoding phase was configured asfollows. The hyperparameter λ in the transition model was
set to 100, the minimum and maximum pitch values werepmin = 43, pmax = 79, and the minimum and maximum
tatum durations were dmin = 6, dmax = 30, corresponding totempos from 215 bpm to 43 bpm.

Beat
Recall
42.8
28.0
73.6
39.3
26.7
82.0

F score
45.7
34.6
75.2
39.0
31.5
79.3

Precision
15.3
13.0
67.9
24.7
15.4
66.5

Downbeat
Recall
15.7
6.9
68.8
25.2
8.1
70.2

F score
14.8
8.6
67.3
24.6
10.37
68.0

and then identifying five exclusive types of errors: pitch errors,
missing notes, extra notes, onset time errors, and offset timeerrors. The onset time and offset time errors are defined interms of score times in units of tatums, but they do not take into
account metrical positions. To evaluate the metrical structureof estimated scores, we additionally calculated the precision,
recall, and F-score by inspecting the beat and downbeat positions of the correctly aligned notes.

B. Compared Methods
To evaluate the efficiency of the proposed method, we compared it with VOCANO [9], a singing transcription system
which combined pitch extraction and note segmentation. The
original training data used for VOCANO have no overlap withthe test data in our experiments. Since VOCANO was designed
for estimating a pitch contour from singing voice, we processed
the frame-based result obtained by VOCANO with a rhythm
quantization method based on an HMM [22].
For comparison, we configured a baseline model based ona note-level score representation that estimates the note valueof each note instead of the metrical position. This model was
similar to ones used in the literature [15], [16] and was almost
the same as the proposed method, except that each output
symbol was given by yn ≜ (dn , pn , on ), where pn and on are
the same as ones described in Section III-A and dn ∈ [1, 16]
represents the note duration. Note that only the duration withina measure was considered, and notes longer than a measure
were split into different symbols. For example, a note with thepitch value of 64 and a duration of 18 tatums was represented
as (16, 64, 1), (2, 64, 0). The note-level baseline model wastrained with the same procedure and the same dataset as the
proposed CTC-based method and decoded with the greedy
algorithm as Eqs. 13,14.

D. Experimental Results
Tables I and II show the evaluation results. The multi-stepcombination of VOCANO and rhythm quantization had a low
overall accuracy in terms of both the edit-distance-based metrics and the beat and downbeat F scores. This shows the difficulty of quantizing the estimated frame-level F0 contours. The
CTC-based method with the note-level representation and its
variant with the tatum-level representation had similar meanerror rates in Table I, and the latter had higher beat and downbeat F scores (Table II). This shows the effectiveness of theend-to-end approach to AST based on CTC with the tatumlevel representation.
Fig. 5 shows examples of musical scores estimated for thesame piece of music from the JPOP-test dataset. As shownin Fig. 5(b), the score estimated by VOCANO contained extra
notes and missing notes as well as a number of rhythmic errors.
The score estimated by the baseline model with the notelevel representation had mostly correct pitches and note values,
but suffered from severe misalignment of metrical positions.
Since the note-level representation does not consider beat anddownbeat positions, rhythmic errors were accumulated whenestimated notes were concatenated together. Compared withthe baseline model, the proposed tatum-level model significantly reduced errors in metrical positions while keeping errors
in pitches and note values at the same level.
Fig. 6 shows the histograms of the durations of the estimatednotes obtained by the note-level model and the tatum-level
model on the JPOP-test dataset. We found that the CTC-based

C. Evaluation Measures
We evaluated the estimated results with edit-distance-based
metrics [29] in a way similar to the word error rate used for
ASR. These metrics were computed by performing alignment
between the estimated notes and ground-truth notes with DP
5(a) Ground truth.

(b) VOCANO + rhythm quantization.

(c) Note-level model.

(d) Tatum-level model.
Fig. 5. Examples of estimated scores.

Fig. 6. Histograms of the note values estimated by the note-level (above) andtatum-level (below) CTCs, where “gt” stands for the ground truth data.

model with the note-level representation failed to correctly predict note values longer than a half note. Although the method
was designed for dealing with note values longer than a wholenote, the model failed to learn this property. With the proposed
tatum-level representation, the histogram of the estimated notevalues was similar to that of the ground truth data. The proposed representation increased the rate of correctly estimatingnote values and contributed to correctly predicting metrical
positions.

A PPENDIX
A. HSMM-Based Decoding
1) Formulation of the HSMM: To impose the constraintof monotonically increasing metrical positions, we propose arefined decoding method based on an HSMM. Generally inmusic performances including singing, the durations of beats ortatums keep stable because of the regularity of metrical structure in music. To utilize this temporal stability, we introduce
the durations of tatums D ≜ {dn }N
n=1 measured in frames in
the proposed HSMM. Consider the frame-level input features
and tatum-level output features described in Section. III-A,
namely, input spectrograms X ≜ {xt }Tt=1 , and output metricalN
positions B ≜ {bn }N
n=1 , pitches P ≜ {pn }n=1 and onsetN
flags O ≜ {on }n=1 . We formulate the generative process of
B, P, O, D and X as

V. C ONCLUSION
In this paper, we proposed an end-to-end AST method basedon the CTC-based training with a tatum-level score representation consisting of positions of the tatums within measures, note
pitches, and onset flags. The experimental results showed thatthe proposed method achieved better accuracy on the estimation of metrical positions of notes, and a mean edit-distancebased error rate similar to the CTC-based baseline methodwith a note-level score representation. We also found that thecascade of F0 extraction and rhythm quantization yielded asignificantly worse result.
There are several issues left for future work. First, we assumed that the minimum time unit of a score is the sixteenth
note, and we thus need to extend the model for fine-grained
time units. Second, the proposed method can only transcribe
songs with the time signature of 4/4, and other common meters
should be properly dealt with. Additionally, popular music canhave multiple meters within a song, and such variable meters
should also be considered. Finally, in the decoding with the
HSMM, a note-level musical language model is expected tobe effective to improve the result, as in the case for ASR.

p(X, B, P, O, D) = p(X|B, P, O)p(B, P, O, D).

(17)

In Eq. 17, p(B, P, O, D) represents the language model ofthe hidden variables. We assume the distributions of pitches
P and onsets O to be uniform, and formulate the transition
probabilities asp(B, P, O, D) = p(b1 , p1 , o1 , d1 )×NY

p(bn , pn , on , dn |bn−1 , pn−1 , on−1 , dn−1 ).

n=2

(18)
We assume the initial distribution of B and D is uniform,
namelyACKNOWLEDGMENT

1 1
1
1
,
Kb Ko pmax − pmin + 2 dmax − dmin + 1(19)
where Kb = 16, Ko = 2 are the number of distinct metrical
positions and onset flags, pmin and pmax are assumed minimum and maximum pitch values other than the rest, and dminp(b1 , p1 , o1 , d1 ) =This study was partially supported by JSPS KAKENHI Nos.
21K12187, 21K02846, 22H03661, 20H00602, and 21H03572,
JST PRESTO No. JPMJPR20CB, and JST FOREST No. JPMJPR226X. We thank Ryo Nishikimi for the constructive discussion with him on this study.
6

TABLE III
A N EXAMPLE OF THE FRAME - LEVEL VARIABLES .
Metrical Position b̄t
Tatum Duration d¯tTatum Counter c̄tPitch p̄t
Onset ōt

15
4
3
62
1

15
4
2
62
1

15
4
1
62
1

15
4
0
62
1

0
5
4
62
0

0
5
3
62
0

and dmax are assumed minimum and maximum lengths of atatum. The transition probabilities of bn , dn from bn−1 , dn−1
are determined by

dn
−1
dn−1

(21)

u(n) ≜

0Pn

j=1 djn = 0,
n > 0,

1
4
2
59
1

1
4
1
59
1

p(X|B̄, P̄, Ō) =

1
4
0
59
1

...
...
...
...
...

T
Y

p(xt |b̄t , p̄t , ōt ),

(26)

t=1


.

(22)

According to Bayes’ theorem, the emission probability of aframe can be written asp(b̄t , p̄t , ōt |xt )p(xt )p(b̄t , p̄t , ōt )p(b̄t , p̄t , ōt |xt ).
∝p(b̄t , p̄t , ōt )2) Frame-Level Formulation of the HSMM: The probabilistic model proposed in Section. A1 contains temporally asynchronous variables, X on the frame level and B, P, O, D on
the tatum level. To formulate the model in a synchronous
way, we introduce the frame-synchronized versions of metrical positions B̄ ≜ {b̄t }Tt=1 , pitches P̄ ≜ {p̄t }Tt=1 , onset
flags Ō ≜ {ōt }Tt=1 , and tatum durations D̄ ≜ {d¯t }Tt=1 . Thesevariables can be constructed from the corresponding framelevel variables with b̄t = bn , p̄t = pn , ōt = on and d¯t = dn
when u(n − 1) + 1 ≤ t ≤ u(n), where
(

1
4
3
59
1

In Eq. 24, p(X|B̄, P̄, Ō) represents the emission probabilities of the observed spectrograms given the metrical positions,pitch values and onset flags. We assume the conditional independence of X given B̄, P̄, Ō, and we have

As suggested in [30], the transition probabilities of dn fromdn−1 are given byp(dn |dn−1 ) ∝ exp −λ

0
5
0
62
0

p(b̄t , p̄t , ōt , d¯t , c̄t |b̄t−1 , p̄t−1 , ōt−1 , d¯t−1 , c̄t−1 )δ(b̄t , b̄t−1 )δ(p̄t , p̄t−1 )δ(ōt , ōt−1 )δ(d¯t , d¯t−1 )δ(c̄t , c̄t−1 − 1),



c̄
t−1 > 0,=11δ (b̄ , b̄+ 1)p(d¯t |d¯t−1 )δ(c̄t , d¯t − 1), Ko pmax −pmin +2 16 t t−1c̄t−1 = 0.
(25)

where δ16 is the Kronecker’s function modulo 16, that is(mod 16),(mod 16).

0
5
1
62
0

except that the transition probabilities becomep(bn , pn , on , dn |bn−1 , pn−1 , on−1 , dn−1 )11=
δ16 (bn , bn−1 + 1)p(dn |dn−1 ), (20)
Ko pmax − pmin + 2(
1 if x ≡ y
δ16 (x, y) =0 if x ̸≡ y

0
5
2
62
0

p(xt |b̄t , p̄t , ōt ) =(27)

We assume that p(b̄t , p̄t , ōt ) is a uniform distribution and thatp(b̄t , p̄t , ōt |xt ) can be estimated with a DNN trained in an
end-to-end manner as described in Section III-B. Under theassumption of conditional independence of b̄t , p̄t , ōt given xt ,we put thatp(b̄t , p̄t , ōt |xt ) = p(b̄t , |xt )p(p̄t , |xt )p(ōt , |xt ),

(23)

P
O ōt
O 1−ōt= ϕB,
tb̄t ϕtp̄t (ϕt ) (1 − ϕt )p(bn |xt ) = ϕBtb̄t ,Pp(pn |xt ) = ϕtp̄t ,ōt
O 1−ōtp(on |xt ) = (ϕO.
t ) (1 − ϕt )is the index of the last frame for each tatum. In addition, atatum counter C̄ ≜ {c̄t }Tt=1 is used to represent the positionof the current frame within the tatum, counting down fromdn − 1 to 0. Table III shows an example of the frame-levelvariables.
With the tatum-level variables the HSMM can be rewritten
as

(28)
(29)
(30)
(31)

It is notable that only the outputs ϕB , ϕP and ϕO are used here
since they can be interpreted as the posterior probabilities when
the neural network judges a frame to be non-blank.
3) Viterbi Decoding: With the probabilistic model describedabove, we can estimate the optimal metrical positions B∗ , pitch
values P∗ and onset flags O∗ by first estimatingp(X, B, P, O, D) = p(X, B̄, P̄, Ō, D̄, C̄)= p(X|B̄, P̄, Ō)p(B̄, P̄, Ō, D̄, C̄). (24)B̄∗ , P̄∗ , Ō∗ , D̄∗ , C̄∗ = argmax p(X, B̄, P̄, Ō, D̄, C̄), (32)

In Eq. 17, p(B̄, P̄, Ō, D̄, C̄) represents the language modelof the frame-level hidden variables. It is similar to Eq. 18,

and then obtain B∗ , P∗ , O∗ from B̄∗ , P̄∗ , Ō∗ . Eq. 32 can besolved efficiently with the Viterbi algorithm.
7

R EFERENCES[25] F.-R. Stöter, S. Uhlich, A. Liutkus, and Y. Mitsufuji. Open-Unmix — areference implementation for music source separation. Journal of Open
Source Software, 2019.
[26] Curtis Wigington, Brian Price, and Scott Cohen. Multi-label connectionist temporal classification. In ICDAR, pp. 979–986, 2019.
[27] Masataka Goto, Hiroki Hashiguchi, Takuichi Nishimura, and Ryuichi
Oka. Rwc music database: Popular, classical and jazz music databases.
In ISMIR, 2002.
[28] Jun-You Wang and Jyh-Shing Roger Jang. On the preparation andvalidation of a large-scale dataset of singing transcription. In ICASSP,
pp. 276–280, 2021.
[29] Eita Nakamura, Emmanouil Benetos, Kazuyoshi Yoshii, and Simon
Dixon. Towards complete polyphonic music transcription: Integrating
multi-pitch detection and rhythm quantization. In ICASSP, pp. 101–105,
2018.
[30] Florian Krebs, Sebastian Böck, and Gerhard Widmer. An efficient statespace model for joint tempo and meter tracking. In ISMIR, pp. 72–78,
2015.

[1] Akinori Ito, Yu Kosugi, Shozo Makino, and Masashi Ito. A queryby-humming music information retrieval from audio signals based on
multiple F0 candidates. In ICALIP, pp. 1–5, 2010.
[2] Martín Rocamora, Pablo Cancela, and Alvaro Pardo. Query by humming:
Automatically building the database from music recordings. Pattern
Recognition Letters, Vol. 36, pp. 272–280, 2014.
[3] Zhiyao Duan and Bryan Pardo. Soundprism: An online system for scoreinformed source separation of music audio. IEEE Journal of Selected
Topics in Signal Processing, Vol. 5, No. 6, pp. 1205–1215, 2011.
[4] Sebastian Ewert, Bryan Pardo, Meinard Muller, and Mark D. Plumbley.
Score-informed source separation for musical audio recordings: An
overview. IEEE Signal Processing Magazine, Vol. 31, No. 3, pp. 116–
124, 2014.
[5] Shreyan Chowdhury, Andreu Vall Portabella, Verena Haunschmid, andGerhard Widmer. Towards explainable music emotion recognition: Theroute via mid-level features. In ISMIR, pp. 237–243, 2019.
[6] Jong Wook Kim, Justin Salamon, Peter Li, and Juan Pablo Bello. Crepe:
A convolutional representation for pitch estimation. In ICASSP, pp. 161–
165, 2018.
[7] Tsung-Han Hsieh, Li Su, and Yi-Hsuan Yang. A streamlined encoder/decoder architecture for melody extraction. In ICASSP, pp. 156–
160, 2019.
[8] Shuai Yu, Xiaoheng Sun, Yi Yu, and Wei Li. Frequency-temporalattention network for singing melody extraction. In ICASSP, pp. 251–
255, 2021.
[9] Jui-Yang Hsu and Li Su. VOCANO: A note transcription framework forsinging voice in polyphonic music. In ISMIR, 2021.
[10] Jun-You Wang and Jyh-Shing Roger Jang. Training a singing transcription model using connectionist temporal classification loss and crossentropy loss. TASLP, 2023.
[11] Eita Nakamura and Kazuyoshi Yoshii. Musical rhythm transcription
based on Bayesian piece-specific score models capturing repetitions.
Information Sciences, Vol. 572, pp. 482–500, 2021.
[12] Ryo Nishikimi, Eita Nakamura, Masataka Goto, Katsutoshi Itoyama, andKazuyoshi Yoshii. Scale- and rhythm-aware musical note estimation for
vocal F0 trajectories based on a semi-tatum-synchronous hierarchicalhidden semi-Markov model. In ISMIR, pp. 376–382, 2017.
[13] Ryo Nishikimi, Eita Nakamura, Masataka Goto, and Kazuyoshi Yoshii.
Audio-to-score singing transcription based on a CRNN-HSMM hybrid
model. APSIPA, Vol. 10, p. e7, 2021.
[14] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented
sequence data with recurrent neural networks. In ICML, pp. 369–376,
2006.
[15] Miguel A Román, Antonio Pertusa, and Jorge Calvo-Zaragoza. An endto-end framework for audio-to-score music transcription on monophonicexcerpts. In ISMIR, pp. 34–41, 2018.
[16] Victor Arroyo, Jose J. Valero-Mas, Jorge Calvo-Zaragoza, and Antonio
Pertusa. Neural audio-to-score music transcription for unconstrained
polyphony using compact output representations. In ICASSP, pp. 4603–
4607, 2022.
[17] Alex Graves and Navdeep Jaitly. Towards end-to-end speech recognition
with recurrent neural networks. In ICML, pp. 1764–1772, 2014.
[18] Han-Wen Nienhuys and Jan Nieuwenhuizen. Lilypond, a system for
automated music engraving. In CIM, pp. 167–171, 2003.
[19] Craig Stuart Sapp. Online database of scores in the humdrum file format.
In ISMIR, pp. 664–665, 2005.
[20] Matthias Mauch and Simon Dixon. Pyin: A fundamental frequency
estimator using probabilistic threshold distributions. In ICASSP, pp.
659–663, 2014.
[21] M. Mauch, C. Cannam, R. Bittner, G. Fazekas, J. Salamon, J. Dai,
J. Bello, and S. Dixon. Computer-aided melody note transcription using
the tony software: Accuracy and efficiency. In TENOR, 2015.
[22] Kentaro Shibata, Eita Nakamura, and Kazuyoshi Yoshii. Non-local musical statistics as guides for audio-to-score piano transcription. Information
Sciences, Vol. 566, pp. 262–280, 2021.
[23] Eita Nakamura, Kazuyoshi Yoshii, and Shigeki Sagayama. Rhythm
transcription of polyphonic piano music based on merged-output hmm
for multiple voices. TASLP, Vol. 25(4), pp. 794–806, 2017.
[24] Ryo Nishikimi, Eita Nakamura, Masataka Goto, and Kazuyoshi Yoshii.
End-to-end melody note transcription based on a beat-synchronous
attention mechanism. In WASPAA, pp. 26–30, 2019.

8