Tatum-Level Drum Transcription
Based on a Convolutional Recurrent Neural Networkwith Language Model-Based Regularized Training
Graduate School of Informatics, Kyoto University, Kyoto, Japan

Ryoto Ishizuka, Ryo Nishikimi, Eita Nakamura, and Kazuyoshi Yoshii

Overview
2 / 23

CHAPTER 1

CHAPTER 2

Background

Related works

CHAPTER 3

CHAPTER 4

Proposed method

Experiments

Overview
3 / 23

CHAPTER 1

CHAPTER 2

Background

Related works

CHAPTER 3

CHAPTER 4

Proposed method

Experiments

CHAPTER 1

Background
Automatic music transcription (AMT)
Aim： Estimate music scores from audio signals

Value： Help music composition and arrangement

4 / 23HHSD

Automatic drum transcription (ADT)
Role：

Rhythmic backbone of popular music

Inst：

Multiple

Pitch：

Different from instruments

Value：

Hard to adjust

Most works focus on onset times

which the main three parts are played atBD

CHAPTER 1

Background5 / 23

Conventional studies： Focusing only on acoustic features

DNN

Tatum level

Binarize
Quantize

Onset probability

Spectroram

Problem：Alignment

BD SD HH

Frame level

Drum score

CNN： Local fratures

Binarize： Threshold

RNN： Temporal dependencies

Quantize： Estimated beat times

Often estimates musically-unnatural drum patterns

CHAPTER 1

Background6 / 23

Idea： Tatum-level language model-based ADT method
Tatum level
Encoder-decoder
DNNLanguage model

How unnatural
the drum score is

Evaluator ofmusical naturalness

𝓛lang

Drum score

1. CNN-based encoder
2. GRU-based decoder

1. Repetition-aware bi-gram
2. GRU-based language modelOverview
7 / 23

CHAPTER 1

CHAPTER 2

Background

Related works

CHAPTER 3

CHAPTER 4

Proposed methodExperiments

CHAPTER 2

Related works

8 / 23

Markov model（HMM・n-gram） [Paulus+, 09]
What?

Statistical language model
considering temporal dependencies

Point

Simple architecture andgood performance

Deep language model (RNN) [Sigtia+, 15]
What?

Evaluate musical naturalness
with recurrent neural networks

Problem：

PointHigh expressive power andeasy implementation

It’s hard to learn tatum-level musical structure
due to the frame-level modelingCHAPTER 2

Related works

9 / 23

Cold fusion [Sriram+, 18]
What?

Logit output of a pretrained
language model is used in thetraining phase

Point

Easy implementation andfastness at run time

Point

Flexible prior distribution

Bayesian inerence [Ueda, 19]
What?

Point：A VAE-based language model isused as a prior of the NMF-basedtranscription model

There are few studies to integrate a DNN-based language model
into a DNN-based transcription model in ADT

Overview
10 / 23

CHAPTER 1

CHAPTER 2

Background

Related works

CHAPTER 3

CHAPTER 4

Proposed method

Experiments

CHAPTER 3

Proposed method

Point

1

Encoder-decoder
model

Point

2

Regularization in thetraining phase

Point

3

Language model

11 / 23

CHAPTER 3

Proposed method

Point

1

Encoder-decoder
model

Point

2

Regularization in thetraining phase

Point

3

Language model

12 / 23

CHAPTER 3

Proposed method

13 / 23

Point 1： Encoder-decoder model
Minimize

Spectrogram（Frame-level）

Encoder(Frame-level)

Decoder(Tatum-level)

CNN

GRU

Alignment
Max-pooling

Ground-truth

𝓛tran

Onset probabilities（Tatum-level）

CHAPTER 3

Proposed method

Point

1

Encoder-decoder
model

Point2

Regularization in the
training phase

Point

3

Language model

14 / 23

CHAPTER 3

Proposed method

15 / 23

Point 2： Regularization in the training phase
Minimize

𝓛lang

Ground-truth

Pretrained
language modelℒtran
Gumbel-sigmoidBinarization
Onset probabilities（Tatum-level）

Drum score（Tatum-level）ℒtotalThe musicalunnaturalness isevaluated in thedifferentiable
mannerCHAPTER 3

Proposed method

Point

1

Encoder-decoder
model

Point

2

Regularization in thetraining phase

Point

3

Language model

16 / 23

CHAPTER 3

Proposed method

17 / 23

Point 3： Design of the language models
Repetition-aware bi-gram

GRU

Skip type (16 tatums) transitions

Transitions from the first tatum𝐾

𝑝 𝑌:,𝑚 𝑌:,1:𝑚−1 = ෑ 𝑝 𝑌𝑘,𝑚 𝑌𝑘,𝑚−16
𝑘=1

𝑝 𝑌:,𝑚 𝑌:,1:𝑚−1

Overview
18 / 23

CHAPTER 1CHAPTER 2

Background

Related works

CHAPTER 3

CHAPTER 4

Proposed methodExperiments

CHAPTER 4Experiments
Dataset（Transcription）

RWC popular music database（65 songs）

Dataset（Language model）

Jpop & Beatles（512 songs）

Test / Validation

10-fold cross validation（test）
15％ of training data（valid）

Data augmentation

SpleeterArchitecture

CRNN as in the right figure

Audio features

Mel spectrogram（80bands）

Hyperparameters

OptunaMeasurement

Precision / Recall / F-measure

Beat estimation

Madmom（ℱ = 96.4%）

19 / 23=

CHAPTER 4Experiments

20 / 23

Repetition-aware bi-gram

GRU

Skip type (16 tatum) transitions

Transitions from the first tatum

GRU was better than bi-gram

CHAPTER 4Experiments

21 / 23

Weighting factor of the language modelState-of-the-art →Without regularization →This experiment showed that...
1. The frame-to-tatum outperformed the SoTA method by 8 points
2. The language model-based regularization outperformed the nonregularized method by 2 points
3. The GRU-based regularization had much improvement than the bi-grambased regularization[7] Vogl, Richard, et al. "Drum Transcription via Joint Beat and Drum Modeling Using Convolutional Recurrent Neural Networks." ISMIR. 2017.

CHAPTER 4Experiments

22 / 23

RWC-MDB-P-2001 No.88（ℱ = 58.3%→ 79.9 % ）LabelMusically-unnatural hihats
Non-regularizedCymbals
RegularizedFill-inSummary
23 / 23Points
Viewpoint

Frame-to-frame methods predict musically-unnatural drum patterns

Keypoint

Tatum-level language model-based regularized training

Experiments
F-measure：

The frame-to-tatum architecture improved about 8 points：

The regularization improved about 2 points：

GRU has much improvement than bi-gram

Future works
Work１：

Dealing with Fill-ins

Work２

：

Learn other than three main parts such as symbals and toms

Work３

：

Capturing global structure with self-attention mechanism