Music Scene Description Project

Toward Audio-based Real-time Music Understanding

Overview
Introduction

Previous Worko Our goal
• Build a real-time system that can understandreal-world music signals (CD recordings)
in a human-like fashion- hum the melody- notice a phrase being repeated- find chorus sections• Useful in various applications
MIR, music production/editing, entertainment, etc.
Brain mechanisms have not been understoodDifficult to implement these abilities

o Two popular approaches• Sound source separation• Automatic transcription
Neither separation nor transcription
is necessary or sufficientfor understanding music• Human auditory system does not extract
each individual audio signal
Even if a mixture cannot be separated,that the mixture includes certain
components can be understood• Untrained listeners understand music without
mentally representing audio signals as scores

Music Scene Descriptiono Real-time music-scene-description system• Obtain descriptions intuitively meaningful tountrained listeners
from real-world audio signals containing simultaneous
sounds of various instruments (w/ or w/o drum-sounds)• Five descriptions

Even if we could derive separated signals andmusical notes, it is still difficult to obtainhigh-level descriptions like melody and chorus

Beat Structure
o Audio-based real-time beat-tracking method• Recognize a hierarchical beat structure
Quarter-note and measure levels

Consider what is to be achieved to understand music
Hierarchicalbeat structure
Compact disc (CD)• Advantage
Track beats above the quarter-note levelby using three kinds of musical knowledge- onset times- chord changes- drum patterns

Melody line
Bass line

time

Musicalaudio signals

Local descriptions

Repeated
sections
Chorussections ...

samesame• Multiple agent architecture• Overview

chorus ...
chorus chorus

Global descriptions
Compact disco Real-time methods

Manager• Audio-based real-time beat-tracking method[Goto and Muraoka, 1999][Goto, 2001a]• Predominant-F0 estimation method for
detecting melody and bass lines (PreFEst)[Goto, 2001b][Goto, 2003b]• Chorus-section detection method (RefraiD)[Goto, 2003a][Goto, 2003c]

Audio signals

Beat structure

12 agentso Beat-tracking application• Beat-driven real-time computer graphics
Various movements and lighting properties
can be changed with musical beatsMasataka Goto
121,2

"Information and Human Activity," PRESTO, JST
National Institute of Advanced Industrial Sci. and Tech. (AIST)

Methods
timeStrong beatWeak beatGreen: the beginning of measure

Beat Structure

Beat-driven Dancers

Melody and Bass Lines

Chorus/Repeated Sections

Melody and Bass Lines

Chorus/Repeated Sectionso Predominant-F0 estimation method foro Chorus-section detection method (RefraiD)

detecting melody and bass lines (PreFEst)• Estimate the fundamental frequency (F0)of melody and bass lines• Detect all the chorus sections in a songand several repeated sections
time

Melody line
Bass line• Advantage• Advantage
Deal with monaural complex music signals- Not assume the number of sound sources- Not locally trace frequency components- Not rely on F0's frequency component- Obtain a list of the start and end points ofevery chorus section in CD recordings- Detect modulated chorus sections (with key change)• Overview
Audio signals

Audio signals• Overview

Extracting 12-dimensional chroma vectors

Instantaneous frequency calculation
Extracting frequency components

Calculating similarity between chroma vectors

Limiting frequency regions
BPF for melody line

Listing repeated sections

BPF for bass line

Integrating repeated sections
Frequency components
Forming F0's probability density function

Integrating repeated sections with modulation

F0's probability density function

Selecting chorus sections
List of chorus sections

Agent
Agent

Salience
detector

Agent• Regard the most repeated sections

Most predominant F0 trajectory
Melody line

as the chorus sections in popular music

Bass line

Detect without using prior information about
spectral characteristics of chorus sections• MAP estimation using the EM algorithm(maximum a posteriori probability)(expectation-maximization)

Introduce original mixture density model• Extract 12-dimensional chroma vectors

contain every possible harmonic structurewith different weights (model parameter)(Sum of power at frequencies of each pitch class)Detect modulated repetitionby shifting chroma vectors

STFT power spectrum
log-scale frequency+Observed PDF

List of repeated sections+ ++130 Hz+Mixture density model

F0's PDF

Chroma vector

8 kHz

Summation

Estimate the F0's PDF (probability density function)(relative dominance of every possible F0)

C C# D D# E F F# G G# A A# B

before modulation
C C# D D# E F F# G G# A A# B

badafter modulation

C C# D D# E F F# G G# A A# B

good!

C C# D D# E F F# G G# A A# B

C# D D# E F F# G G# A A# B C

modulated by 1 semitone upward

modulation of 1 semitone: 1 shift

URL of the Masataka Goto's Home Page:
http://staff.aist.go.jp/m.goto/Metadata
Metadata Editoro Multipurpose music-scene labeling editor• Hand-label musical pieces with metadata(correct descriptions)• Evaluate music-scene-description methods

o Functions• Deal with both audio files and SMFs
• Support interactive audio/MIDI playback• Show subwindows in which any selected
descriptions can be displayed and edited• Support practical editing aids
- magnifying-glass function- region-based cut-and-paste operation- cursor movement between
context-dependent grid points (e.g., beats)o RWC Music Database: Popular Music• Hand-labeled chorus sections of all 100 songs(RWC-MDB-P-2001 No.1 - 100)

• Evaluated the RefraiD
Compare the output with correct sections
Correctly detected in 80 of the 100 songs

Conclusiono Music Scene Description Project• Build a music-scene-description system
Understand real-world audio signals w/oderiving musical scores or separating signals• Develop a metadata editor
Enable a user to hand-label audio files/SMFs
with descriptions of the music in those files• Hand-labeled 100 songs of RWC-MDB-P-2001

with their chorus sections
2003/10/28 ISMIR 2003 poster