Dance Beat Tracking from Visual Information Alone
Fabrizio Pedersoli and Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST), Japan

5. Results

3. Classification

1. Introduction

key-points t−1

key-points t

Dance Information Retrieval (DIR)(xt−1, yt−1)(xt−1, yt−1)(2)(xt, yt)(xt, yt)(2)Extracting high-level semantics information from dance videos
DIR tasks similar to Music Information Retrieval (MIR)
DIR tasks typically solved by analyzing the visual informationŷ: Beat act.

x: Prep input(1)(1)

 P r(beat)1 


...




(1)
(13)
∆x,.
.
.
,∆x
11 

(13)  ∆x(1)2 , . . . , ∆x2 Dance Beat Tracking (without audio signals)



Unexplored fundamental topic in DIR research
Detection of musical beats by using visual information
Classify each video frame as “beat” or “non-beat” frame

...

(1)

(13)

∆xN , . . . , ∆xN

P r(beat)0

P r(beat)N

2. Prep

Automatic synchronization of dancing with music
Temporal alignment of videos (time stretching)
Identification of out-of-sync dance videos1   0  
. . .  

0

4. Post-proc

3.Classification

Important applications of dance beat tracking

y∗: Beats∆x := (xt − xt−1, yt − yt−1)

Step 2: Pre-process
Sequence of (x, y) absolute coordinates of body key-points is converted into frame-by-frame (∆x, ∆y)displacements

2. Approach

1D TCN as sequence to sequence classifier
Grid-search for best model specification: stack of 7 residual blocks with 128 units
Trained with weighted cross-entropy loss to account of sparsity of labels
Adam optimizer with default PyTorch parameters

P r(beat)fΘ(x)Step 4: Post-processt

Dance beat tracking is a challenging task
Performance results are lower if compared to music beat tracking
The performance on the “dance” split is higher than the performance on the “music” split

The periodicity loss achieves a considerable improvement of performance

Loss CMLc CMLt AMLc AMLt Cem
F
Lce 44.28 46.93 47.27 49.04 52.92 55.02
Lce +αLp 53.05 54.30 55.23 57.64 59.02 61.20
Table 1. Performance results on the “dancer” data split using the proposed loss with α = 0.05

6. Contributions

4. Improvementx

3. TCNClassifier

4. Beats

Step 1. Body key-points are extracted from video frames byOpenPose
Step 2. Body key-points are pre-processed
Step 3. Sequence of pre-processed key-points is classified bya Temporal Convolutional Neural Network (TCN)(output is the beat activation function)
Step 4. Beat activation is post-processed to get the final
beats positions

Loss CMLc CMLt AMLc AMLt Cem
FLce 40.14 39.71 44.84 47.53 47.43 53.02
Lce +αLp 46.50 48.33 48.27 50.87 54.27 58.25
Table 2. Performance results on the “music” data split using the proposed loss with α = 0.1

Off-the-shelf HMM post-process [1] to obtain the final beat positionst

2. Keypoints

Use the subset of videos recorded by the frontal camera and feature one dancer a time
Consider data splits based on “dancer” and “music”
Randomly split: 70% training, 20% validation, 10% test

Step 3: TCN Classification

y1. Frames

Test our algorithm on the AIST Dance Video Database [2]Baseline TCN is trained with weighted cross-entropy loss LcePropose a custom loss term Lp that improves performance
Idea: exploit the periodicity of output based on ground truth tempo
Beat probabilities at interval apart should be considered similar
Lp used only on the training set
Lp is mixed with a parameter α estimated by grid-search and summed to Lce

Propose the novel task of dance beat tracking using visual information
alone
Propose the periodicity loss term, which is scaled and added to thebaseline cross-entropy loss
Provide a baseline evaluation on the AIST Dance Video Databaseconsidering data splits based on music and dancerŷ
L = Lce + αLp

L(n)p =Nb −3X
k=00
k =k+1k 00=k 0+1

Lp → ŷ0:T ≈ ŷT :2T ≈ . . .

(n)(n)
ŷkT :k0T − ŷk0T :k00Tŷ0:T1

ŷT :2Tŷ2T :3Tt

T := inter-beat interval that corresponds
to the ground truth tempo

The 21st International Society for Music Information Retrieval Conference

7. References[1] F. Krebs, S. Böck and G. Widmer, “An Efficient State Space Model for Joint
Tempo and Meter Tracking”, ISMIR 2015[2] S. Tsuchida, S. Fukayama, M. Hamasaki and M. Goto, “AIST Dance Video
Database: Multi-genre, Multi-dancer, and Multi-camera Database for Dance
Information Processing”, ISMIR 2019