Raga Classification from Vocal Performances Using Multimodal Analysis
Martin Clayton1, Preeti Rao2, Nithya Sikharpur2, Sujoy Roychowdhury2 , Jin Li1
1Department of Music, Durham University, United Kingdom
2Department of Electrical Engineering, Indian Institute of Technology Bombay, India

Multimodal Classification Architecture

Motivation

A
Video Pooling Layer

Left wrist x

Video Softmax

Left wrist y

Filtering +Amp norm

Right wrist x
Right wrist yVideo Conv Layer

Source / Latent
Fusion Prediction(C/D)

Openpose keypoints
SourceFusionF0Voicing

A singer’s gesture while performing a raga

Tonic norm +Amp normC• Gesture styles are idiosyncratic (varies between
teacher-student or siblings)

Source Fusion Conv Layer

Multimodal Classification Objective
Train a deep learning classifier using• Unimodal features from audio and video• Try different multimodal classification methods
and compare performance

Dataset Description and RagaInformation
We use an OSF dataset comprising alap
(2 takes/(raga, singer) x 3 mins/take) andcharacteristicphrases(pakad,4takes/raga x 20s/take) in 9 ragas, sungby 3 professional artists (~3.5 hours total)

Latent fusion
1*1 convolutionFusion Pooling Layer Fusion Softmax

E

Audio Conv Layer(s)B

Audio Pooling Layer

Audio Softmax

Multimodal raga classification architecture

Extracting Video Features

Multimodal Experiments• Source Fusion – Combine audio and videofeatures (model C)• Nonetheless,gestures
often
seem
toaccompany (and illustrate) aspects of melody• “Can ragas be classified using movement
information alone, or can movement information
help to disambiguate raga identity?”

DInception block

Audio signal• Musical gesture studies [1] build on those ofspeech-accompanying gesture• Indian vocal music has featured prominently inthis field due to its rich use of manual gestures

Latentfusion

Late Fusion
Prediction(E)• Latent Fusion – Use frozen weights of the bestmodel from (A) and (B) and train inception andfinal layers (model D)
Sample OpenPose output & Sample video features fornormalization box
one 12s clip

• We use OpenPose 2D pose estimation [2] totrack a set of upper-body keypoints• Normalize keypoints based on squarebounding box around singer to a range of [0,1]• Late fusion – Train a classifier (RF etc.) on topof predicted softmax of unimodal models A andB (model E)

Model Results• Use only the positions of right and left wrists• Any missing data is interpolated and lowpass
filtering applied for smoothing

Train/Test Splits

Unimodal validation accuracy

Multimodal validation accuracy

Fig. 7: Seen and Unseen Singer splits

The notes of each raga; S is the tonic note

Extracting Audio Features

Seen and unseen split sizes by 12s segments• Recordings are split into overlapping 12ssegments (order of phrase duration)• Seen singer split - 1 alap take of one singer invalidation, rest in train

Sample audio pitch andvoicing mask

• Unseen singer split - All recordings of one
singer in the validation, rest in train

Architecture Components• Convolutional layers for feature extraction

Audio preprocessing• We use source separation of vocal frombackground tanpura.
• We extract vocal pitch and binary voicing at 10
ms intervals. Interpolate across short silence
segments (<250 ms)• We apply tonic normalization to obtain pitch incents with reference to the singer’s tonic.

• Inception layer to process multiscaleinformation• Layers separately hyperparameter tuned forindividual experiments

Unimodal Experiments• Unimodal Video- Use video features (model A)

Comparison of unimodal and latent fusion models.
3-bit code indicating if audio, video and latent fusion
models are correct (1) or wrong (0) respectivelyConclusions• Unimodal audio results much better than video• There is complementary information in video toimprove multimodal performance over audiowith latent fusion

References[1] Godøy, R. Inge, M. Leman, eds. Musical gestures:Sound, movement, and meaning. Routledge, 2010.
[2] Cao, Zhe, et al. "Realtime multi-person 2d pose
estimation using part affinity fields." Proceedings of theIEEE conference on computer vision and patternrecognition. 2017.

• Unimodal Audio- Use audio features (model B).
Scan accompanying QR code forsupplementary material.