Journal of the DBSJ Vol.7, No.1

A New Feature forMusical Genre Classification of
MPEG-4 TwinVQ Audio Data

to have these functions.
To retrieve a piece of music, conventional music retrieval
systems usually use keywords such as the title, artist, musical genre, etc. It is difficult to describe musical genre of apiece of music. Several researchers proposed a content-based
musical genre classification from audio signal [4, 9, 13, 19].
♦ We propose a new music feature for content-based musicalMichihiro
Takaya MORITA
genre classification in the compressed domain of MPEG-4
KOBAYAKAWATwinVQ audio [11].
♥The rest of this paper is organized as follows. Section 2♠♣ shows the related works of musical genre classification andMamoru HOSHI
Tadashi OHMORI music information retrieval in the compressed domain. Section 3 briefly introduces MPEG-4 TwinVQ audio compression. Section 4 proposes a musical feature using LSP paThis paper proposes a new musical feature to
rameter directly extracted from a bitstream of the TwinVQ
classify MPEG-4 TwinVQ compressed data into muaudio
data and the Discrete Wavelet Transform (DWT).
sical genre without decoding to audio signals. To
Section 5 describes experiments on 2, 196 compressed data
extract the musical feature, we use the LSP (Lineand discusses the performance of the musical feature for muSpectrum Pair) parameters directly extracted fromsical genre classification. Finally, Section 6 concludes thisa bitstream without any computation, and the Dispaper.
crete Wavelet Transform (DWT). We experimented
on 2, 196 compressed music data collected from 10
2. Related Workmusical genres and evaluated the performance of themusical feature for musical genre classification. The
We briefly describe previous works on signal-based musimaximum of average correct ratio for musical genre
cal genre classification and on music information retrieval
classification was 81.7%. Experiment showed thatin the compressed domain MPEG audio standard.
the musical feature had very good performance for
The following basic features are often used for analyzing
musical genre classification in the compressed doaudio contents:
main of MPEG-4 TwinVQ audio compression.
• energy [7, 12, 20],• normalized subband energy [12],• (Fs , Ft )-subband energy between Fs -th subband and Ft 1. Introduction
th subband,
Audio compression techniques are widely used. We can• spectral centroid [12, 20],
obtain compressed music data through music distribution• root mean squared subband vector, mean subband
systems that contain hundreds of thousands of the comvalue, rolloff point, low energy, spectral flux [20],pressed music data. As compressed music data is generated,• cepstrum [15],stored and distributed continuously, methods for effectively• Mel Frequency Cepstrum Coefficients (MFCC) [18].
managing compressed music data are required.
They are based on the subband values or calculated in the
In addition to a compression function, functions for depolyphase filterbank analysis.
scribing the contents of compressed music data are required
First, we show several works on musical genre classificain order to effectively use the compressed data. Comprestionin the signal domain.
sion techniques, such as MPEG-7 or MPEG-21, are required
Tzanetakis et.al extracted nine features as the “musical♥Non-member Graduate School of Information Syssurface features”and rhythm features. The musical surface
tems,
the University of Electro-Communications
features are based on the centroid, rolloff, flux, zerocrossing
kobayakawa@computer.org, kobayakawa@acm.org♦
and low energy, and the rhythm features are based on the
Non-member Graduate School of Information Systems,
the University of Electro-Communications
Discrete Wavelet Transform [18].
morita@hol.is.uec.ac.jp
Li et.al proposed the Daubechie Wavelet Coefficient His♠
Non-member Graduate School of Information Systograms for musical genre classification and evaluated thetems,
the University of Electro-Communications
performance using several classifiers [4].
hoshi@is.uec.ac.jp♣
Tzanetakis [17], Mandel et.al [8], Lidy et.al [6], Guaus
Member Graduate School of Information Syset.al [1] joined in MIREX2007 Evaluation Task for audio
tems,
the University of Electro-Communications
omori@is.uec.ac.jp1    Vol.7, No.1



Journal of the DBSJ Vol.7, No.1
/

MDCT

/

/

Bark scaleenvelope
prediction VQ

power
analysisinterleave-

LPC
analysis

LPC
spectrumpitch
analysisweighted
VQ

LSP
VQ
perceptual
model

bitstream

output signal

IMDCT

LSP

pitch

LPC spectrum
decoding

pitch decoding*+

Bark scaleenvelope

power

Bark scale envelope power decoding
decoding**

flattened
MDCT
coefficient

decodeinverce interleave

Figure. 1 Architecture of MPEG-4 TwinVQ audio compression.

genre classification [9].
Table. 1 Genre and the number of compressed data.
Tzanetakis used the spectral centroid, rolloff, flux, MFCC
genre
# of compressed dataas the musical features, and used a linear support vector mabaroque223
chine as a classifier. Mandel et.al used the spectral features
bossanova
119based on MFCCs and used support vector machines. Lidy
dance
176
et.al extracted rhythm pattern and symbolic feature fromhiphop196
audio signals and used the support vector machines. Guaus
jazz
431
et.al used a set of timbre descriptors and rhythm descriptors
march91, and used support vector machines.
oldies
435
The summary results of MIREX 2007 Evaluation Task forrock200
audio genre classification are shown at “http://www.musictango
220
ir.org/mirex2007/index.php/Audio Genre Classification Results”.
waltz
99
Next, we show works on music information retrieval in the
total
2, 196compressed domain.
Liu et al. proposed a method for classifying the encoded
data to identify a singer. They used the energy of a frame
for classification [7].
Pye proposed a method for managing music data [15].
3. TwinVQ audio compression
They classified the MP3 music data into several genres andThis section briefly describes the architecture of MPEGlabeled them artist name using cepstrum.
4 TwinVQ audio compression. The details of TwinVQ
Nakajima et al. proposed a method for classifying theaudio compression are referred to [11] or the URL
encoded audio data [12]. They discriminated between short
http://www.twinvq.org/. A sequence of original signals of apieces of silence and pieces of non-silence by using energy,
piece of music is divided into frames. The MDCT (Modified
and then classified pieces of non-silence into music, speech,
Discrete Cosine Transform) is then applied to the signals ofand applause by using the normalized subband energy andeach frame. For the MDCT coefficients of each frame, fivecenter frequency of subband energy on 1 second of MPEG
analyses, namely, LPC analysis, Pitch analysis, Barkscaleaudio data.
envelope analysis, Power analysis and Weighted VQ analyTzanetakis et al. proposed a method for segmenting thesis, are carried out in this order (Figure 1). In LPC analysis,
encoded data into short pieces and classifying them into
the LSP (Line Spectrum Pair) parameters are stored in themusic and speech [20].
bitstream. They are used for describing the power envelopeof MDCT coefficients of each frame.

2    Vol.7, No.1



Journal of the DBSJ Vol.7, No.1
Table. 2 Average correct ratio of the (D, L) musical feature vector f D,L for the test data (%).
L

4.

D

1

2

3

4

5

6

7

8

9

10

1

33.6

48.6

50.7

52.3

53.9

54.4

54.5

58.6

58.8

59.1

2

55.4

65.0

67.5

67.8

69.0

68.5

69.2

71.9

72.2

71.7

3

59.2

67.8

69.7

70.4

71.7

71.1

71.6

75.4

75.1

75.1

4

62.4

70.2

71.8

72.6

73.9

74.0

74.9

77.5

77.4

77.7

5

63.7

70.4

73.2

73.6

74.6

74.7

76.1

78.4

78.2

78.0

6

65.8

71.9

74.4

74.9

76.4

77.3

77.9

80.9

80.4

80.5

7

67.1

73.6

76.3

76.0

77.9

78.0

78.6

80.7

81.1

81.5

8

67.8

74.0

76.4

76.7

78.1

78.6

78.9

81.2

81.6

81.0

9

67.9

75.2

77.3

77.7

78.7

78.7

79.5

81.7

81.5

81.5

10

69.3

75.5

77.4

77.6

78.7

78.6

79.5

81.5

81.6

81.5

11

69.4

76.0

77.7

77.7

78.5

78.5

79.0

81.0

81.2

80.8

12

69.7

76.6

78.0

77.7

78.3

78.4

78.8

80.7

80.8

80.5

13

70.4

76.5

77.8

78.1

78.4

79.3

79.3

80.9

80.9

81.2

14

70.9

76.8

78.2

78.3

79.0

79.4

79.0

80.6

80.5

80.0

15

71.1

77.2

78.2

78.2

79.1

79.2

79.0

80.1

80.4

79.6

16

71.2

77.0

78.7

78.3

79.3

79.0

78.3

80.2

80.1

78.9

17

71.6

77.2

79.0

78.5

79.4

78.6

78.1

80.0

79.8

78.2

18

71.9

77.7

79.2

78.8

79.5

78.8

78.5

80.5

79.7

78.5

19

72.4

78.4

79.4

79.3

79.4

79.7

78.5

80.2

79.7

78.8

20

72.1

78.4

79.4

79.0

79.4

79.1

78.2

80.2

79.4

78.4

Musical Feature

wavelet coefficients cd,l,t of level l,This section describes a musical feature for musical genre
classification of MPEG-4 TwinVQ audio data. The musical feature is based on the sequence of LSP parameters obtained from the bitstream and on multi-resolution analysis
by DWT.
We define the LSP parameter vector of the m-th frame by
LSP
LSP
LSP T
ω LSP
= (ωm,1, . . . , ωm,d
, . . . , ωm,20
) ,
m

(1)

LSP
where ωm,dis the d-th LSP parameter of the m-th frame,
and T denotes the transposition. The sequence ΩLSP consisting of N LSP parameter vectors is represented by 20×N
matrixΩLSP = (ω LSP, ω LSP, . . . , ω LSP
),
(2)
1
2
N

where N is the number of frames of TwinVQ audio data.
The d-th row of the matrix ΩLSP is expressed as
LSP
LSP
,
, . . . , ωN,d= ω1,dω LSP
dP{cd,l,t }2N2l,

and the standard deviation σd,l

v
u
N
u
2l
u1 X
σd,l = t(c2N2l i=1d,l,i − pd,l )2.

We then define the spectrum vector pd,L by the spectrum
up to level L as
pd,L ≡ (pd,1 , pd,2 , . . . , pd,L ),d = 1, . . . , 20,

and the standard deviation vector up to level L as
σ d,L ≡ (σd,1 , σd,2 , . . . , σd,L ).

(3)

and is called “the d-th sequence of length N ”.
Then, by using DWT with the base Haar, we decompose
the d-th sequence ω d and then obtain the detailed wavelet
coefficients cd,l,t (l = 1, . . . , L; t = 1, . . . , b 2Nl c ) up to levelL.
We calculate the mean spectrum pd,l of level l (l =1, . . . , L) defined by the average of the power of the detailed

3

pd,l =We define the vector f D,L of the compressed music data
up to order D and wavelet decomposition level L as
f D,L ≡ (p1,L , , . . . , pD,L , σ1,L , . . . , σD,L ),1 ≤ D ≤ 20,

1 ≤ L ≤ 10.

and call it “the (D, L) musical feature vector”.

    Vol.7, No.1


5.

Journal of the DBSJ Vol.7, No.1

Experiments

This section evaluates the performance of the musical feature for genre classification.
We can directly obtain the LSP parameter vector (20dimensional) of each frame. The time of extracting the
LSP parameter vector was only 0.9µsec on average (variance 5.4 × 10−23 , minimum 0.1µsec, maximum 1.4µsec )(OS: Linux, CPU: AMD Athron 64 X2 Dual Core Processor
4600+, Memory: 3GB). For example, the computation timeof length of 10, 000 frames (play time is 232.2 sec) is only
90msec on average.

5.1

Data set and Classifier

Data set
We experimented on 2, 196 compressed music data of 10
musical genres such as “baroque”, “bossanova”, “dance”,“hiphop”, “jazz”, “march”, “oldies”, “rock”, “tango” and“waltz”.
We collected the pieces of music from the collected worksof “baroque”, “bossanova”, “dance”, “oldies” and “tango”,
and from some albums of march and waltz. In addition, we
selected the pieces of music of hiphop and rock from ourlibrary consisting of 20, 000 pieces of music. Table 1 shows
the number of pieces of music of each musical genre.

Classifier
We used the Discriminant Analysis (DA) as a classifier(Figure 2) to evaluate the performance of the (D, L) musicalfeature vector. The DA is a simple and widely used standard
statical classifier.

5.2

Musical Genre Classification

This section evaluates the classification performance ofthe proposed musical feature.
We applied the MPEG-4 TwinVQ encoder to each signalof piece of music and obtained 2, 196 compressed music data,then compute the musical feature vector for each compressedmusic data.
The musical genre classification was done as follows.
Firstly, as training data we randomly selected 50 percentof the compressed music data of the same genre and used
the rest as the test data (50%).
The system executed discriminant analysis to the set of
the training data, and obtained the discriminant space andthe centroid vector for each musical genre.
To classify the test data, the system maps the musicalfeature vector of the test data into the discriminant space
and then compute the distance between the test musicalfeature vector and centroid points, and returns the genre of
the nearest centroid as the musical genre of the test data.
For each set of the (D, L) musical features, the process

4

above was repeated three times and the average accuracyof classification was obtained. We show the performanceof the (D, L) musical feature in the form of a confusion
matrix on percentage (e.g., Table 3 for the (9, 8) musicalfeature vector). The rows correspond to the actual genre
and the columns to classified genre. For example, the cell of
row “dance”, column “hiphop” with value 2.27 means that
2.27% of the dance music was wrongly classified as hiphop
music.
The average of diagonal elements of the confusion matrix
for the (D, L) musical feature shows the total performanceof the (D, L) musical feature for musical genre classification.
The average is called the “average correct ratio”.
We examine how the number D of LSP parameters andthe wavelet decomposition level L affect on the performanceof musical genre classification.
Table 2 shows the average correct ratio of f D,L for D = 1∼ 20, L = 1 ∼ 10.
The average correct ratios for 6 ≤ D ≤ 14 and 8 ≤ L ≤ 10
were greater than or equal to 80.0%. The maximum ratio
was 81.7% (for f 9,8 ).
Hereafter, we examine the performance of musical genre
classification using the (9, 8) musical feature vector f 9,8 .
The results of musical genre classification using the vectors
f 9,8 are shown in Table 3 (confusion matrix). All diagonal elements in Table 3 were greater than or equal to 70.0,
except waltz (67.35%). The correct ratios for five genres:baroque, hiphpo, jazz, march and rock were greater than orequal to 86.36. Waltz music had the minimum (worst) correct ratio 67.35. Waltz music was misclassified as baroque(12.24%), and jazz, march and tango (6.12% each). The
misclassifications are similar to what a human would do.
The results of experiments showed that the musical genre
classification by using the (9, 8) musical feature vector of
LSP parameter had very good performance.

6.

Conclusion

This paper proposed a new musical feature for musicalgenre classification of MPEG-4 TwinVQ compressed music
data. The key idea was to use the LSP parameters directly
extracted from bitstream in the MPEG-4 TwinVQ audio
data without any computation. We applied the Discrete
Wavelet Transform to a sequence of the LSP parameter andobtained the (D, L) musical feature vector. To evaluate theperformance of the musical feature, we experimented musical genre classification on 2, 196 compressed music data of 10
musical genres. We got the maximum average correct ratio
81.7% for the (9, 8) music feature vector based on the LSP
parameter. Although the used classifier (DA) was very simple, the proposed music feature had very good performancefor musical genre classification.

    Vol.7, No.1



Journal of the DBSJ Vol.7, No.1
Classification space generator

Sets of traning data

Feature Extraction

baroque

Genre Assigment

User interface
Compressed music data

Genre

Feature Extraction

Top 1
Nearest Neighbor Searchbossanova

Discriminant Analysis

Presentative Points
waltz

Query generator
Figure. 2 Musical genre classification using the discriminant analysis.

Table. 3 The accuracy of genre classification using the (9, 8) musical feature vector f 9,8 for the test data (%).
classified as
genre

baroque

bossa

dance

hiphop

jazz

march

oldies

rock

tango

baroque

86.36

0.00

0.00

0.00

0.00

6.36

1.82

0.00

1.82

3.64

bossa

0.00

71.19

1.69

1.69

11.86

1.69

10.17

0.00

1.69

0.00

dance

4.55

4.55

79.55

2.27

3.41

0.00

3.41

2.27

0.00

0.00

hiphop

0.00

2.00

5.00

91.00

1.00

0.00

1.00

0.00

0.00

0.00

jazz

0.93

3.72

0.47

0.00

85.58

0.93

2.79

0.47

4.65

0.47

march

2.22

0.00

0.00

0.00

0.00

88.89

0.00

0.00

2.22

6.67

oldies

1.32

1.32

3.08

0.44

7.05

0.88

78.41

0.88

5.73

0.88

rock

2.00

2.00

4.00

0.00

0.00

0.00

2.00

90.00

0.00

0.00

tango

3.00

0.00

0.00

0.00

10.00

1.00

8.00

0.00

78.00

0.00

waltz

12.24

0.00

0.00

0.00

6.12

6.12

2.04

0.00

6.12

67.35

We can obtain the approximate values of the LPC coefficients and the LPC cepstrum from the LSP parameterswith some computations, and compute the musical feature
vectors using the approximate the LPC coefficients and the
LPC cepstrum in the same way as described in section 4.
The performance of them are found in [10].
In this paper, we focused on the musical feature using
the LSP parameters, because they can be directly extracted
from the bitstream without any computation and are fundamental information of the MPEG-4 TwinVQ compresseddata.
The contribution of this paper is to present a new musicalfeature to classify MPEG-4 TwinVQ compressed data intomusical genre without decoding to audio signals.

[References][1] E. Guaus and P. Herrera.
A Basic System
For Music Genre Classification. http://www.musicir.org/mirex/2007/abs/GC guaus.pdf, 2007.
[2] J.-S. R. Jang and H.-R. Lee. Hierarchical Filtering
Method for Content-based Music Retrieval via Acoustic
Input. In Proceedings of the Ninth ACM International

5

waltz

Conference on Multimedia (MM2001), pages 401–410,
2001.
[3] N. Kosugi, Y. Nishihara, T. Saketa, M. Yamamuro,and K. Kushima. A Practical Query-By-Humming System for a Large Music Database. In Proceedings of theEighth ACM International Conference on Multimedia(MM2000), pages 333–342, 2000.
[4] T. Li, M. Ogihara and Q. Li. A comparative study
on content-based music genre classification. In Proceedings of the International ACM SIGIR Conference
on Research and Development in Information Retrieval(SIGIR2003), pages 282–289, 2003.
[5] T. Lidy and A. Runber. Evaluation of Feature Extractions and Psycho-Acoustic Transformations for Music Genre Classification. In Proceedings of International Symposium on Music Information Retrieval (ISMIR2005), pages 34–41, 2005.
[6] T. Lidy, A. Runber, A. Pertusa, and J. M. Inesta.
Combining Audio and Symbolic Descriptors forMusic Classification from Audio. http://www.musicir.org/mirex/2007/abs/AI CC GC MC AS lidy.pdf,
2007.

    Vol.7, No.1

[7] C.-C. Liu and P.-H. Tsai. Content-based Retrieval of
MP3 Music Objects. In Proceedings of the Eleventh International Conference on Information and Knowledge
Management (CIKM2002), pages 506–511, 2002.
[8] M. Mandel and D. P. W. Ellis.
LABROSA’S
Audio Music Similarity and Classification Submissions.
http://www.music-ir.org/mirex/2007/abs/
AI CC GC MC AS mandel.pdf, 2007.
[9] MIREX2007.
http://www.music-ir.org/mirex/
2007/index.php/Audio Genre Classification.
[10] T. Morita, M. Kobayakawa, M. Hoshi, and T.
Ohmori. Automatic Musical Genre Classification using TwinVQ. In Proceedings of the Nineteenth Data
Engineering WorkShop 2008 (DEWS2008), E8-1, 2008
(in Japanese).
[11] T. Moriya, N. Iwakami, K. Ikeda, and S. Miki. Extension and Complexity of TwinVQ Audio Coder. In
Proceedings of International Conference on Acoustics,
Speech and Signal Processing (ICASSP1996), pages
1029–1032, 1996.
[12] Y. Nakajima, Y. Lu, M. Suguro, A. Yoneyama, H.
Yanagihara, and A. Kurematsu. A Fast Audio Classification From MPEG Coded Data. In Proceedings ofIEEE International Conference on Acoustics, Speechand Signal Processing (ICASSP1999), pages 3005–
3008, 1999.
[13] N. M. Norowi , S. Doraisamy and R. Wirza. Factors
Affecting Automatic Genre classification: An investigation Incorporating Non-Western Musical Forms. In
Proceedings of International Symposium on Music Information Retrieval (ISMIR2005), pages 13–20, 2005.
[14] S. Pfeiffer, S. Fischer, and W. Effelsberg. Automatic
Audio Content Analysis. In Proceedings of the Fourth
ACM International Conference on Multimedia, pages
21–30, 1996.
[15] D. Pye. Content-based Methods for Management of
Digital Music. In Proceedings of IEEE International
Conference Acoustics, Speech, and Signal Processing(ICASSP2000), pages 2437–2440, 2000.
[16] J. Shifrin, B. Pardo, C. Meek, and W. Birmingham.
HMM-Based Musical Query Retrieval. In Proceedingsof Joint Conference on Digital Libraries (JCDL2002),
pages 295–300, 2002.
[17] G. Tzanetakis. MARSYS Submissions to MIREX207.
http://www.music-ir.org/mirex/2007/abs/
AI CC GC MC AS tzanetakis, 2007.
[18] G. Tzanetakis, G. Essl and P. Cook.
Automatic Musical Genre Classification of Audio Signals.
In Proceedings of International Symposium on Music Information Retrieval (ISMIR2001),
http://ismir2001.ismir.net/pdf/tzanetakis.pdf, 2001.
[19] G. Tzanetakis and P. Cook. Music genre classification

6

Journal of the DBSJ Vol.7, No.1
of audio signals. In IEEE Transaction on Speech andAudio Processing, Vol. 10, No. 5, pages 293–302, 2002.
[20] G. Tzanetakis and P. Cook. Sound Analysis using
MPEG Compressed Audio. In Proceedings of IEEE
International Conference Acoustic, Speech, and Signal
Processing (ICASSP2000), pages 761–764, 2000.

Michihiro KOBAYAKAWA
He received Dr. E. degree from the University of ElectroCommunications in 2001. From 2001 to 2008, he was on
Graduate School of Information Systems, The University ofElectro-Communications. He is an Associate Professor of
Okinawa National College of Technology, Okinawa, Japan.
His interests include a content-based multimedia retrieval,and algorithms and data structures for multimedia data retrieval. He is a member of ACM, IEEE CS and IEICE.

Takaya MORITAHe received the degree of Master of Engineering from the
University of Electro-Communications in 2008. He works
for Hitachi Corporation Ltd. His research interests include content-based music information retrieval and music
retrieval.

Mamoru HOSHIHe received Dr. E degree in mathematical engineeringfrom the University of Tokyo in 1985. He is a Professorof Graduate School of Information Systems, The University
of Electro-Communications. His research interests include
algorithms and data structures for searching, multimedia retrieval, and random number generation. He is a member of
ACM, IEEE CS, IEEE IT, IEICE, IPSJ, and SITA.

Tadashi OHMORIHe is an Associate Professor in Graduate School of Information Systems, The University of Electro-Communications.
He received his Dr. Eng. degree from The University ofTokyo, 1990. His research interests include parallel databasemachines, transaction processing, and Web mining.

    Vol.7, No.1