AUTOMATIC AUDIO TAG CLASSIFICATIONVIA SEMI-SUPERVISED CANONICAL DENSITY ESTIMATION
Jun Takagi† , Yasunori Ohishi‡ , Akisato Kimura‡ , Masashi Sugiyama† , Makoto Yamada† , Hirokazu Kameoka‡†Graduate School of Information Science and Engineering, Tokyo Institute of Technology‡NTT Communication Science Laboratories, NTT Corporation
ABSTRACT
We propose a novel semi-supervised method for building
a statistical model that represents the relationship between
sounds and text labels (“tags”). The proposed method, namedsemi-supervised canonical density estimation, makes use ofunlabeled sound data in two ways: 1) a low-dimensional
latent space representing topics of sounds is extracted bya semi-supervised variant of canonical correlation analysis,and 2) topic models are learned by multi-class extension ofsemi-supervised kernel density estimation in the topic space.
Real-world audio tagging experiments indicate that our proposed method improves the accuracy even when only a small
number of labeled sounds are available.
Index Terms— Audio tag classiſcation, semi-supervised
learning, topic model, canonical correlation analysis, kerneldensity estimation
1. INTRODUCTION
A central goal of music information retrieval is to develop asystem that can efſciently store and retrieve sounds from alarge database of musical contents. The most common way
is to use metadata such as the name of composers and artists,
the title of songs, and the release date of albums; furthermore,
one may also use various additional information such as genres, instruments, song lyrics, and reviews. These metadata
can be used as input to query-by-keyword systems and collaborative recommender systems [1, 2, 3]. However, all thesesystems have a common drawback called the new item problem—when a new song is added to the database, it needs to beannotated manually. This requires a large amount of humanlabor, and thus keeping the systems updated is highly costly.
To cope with this problem, most previous works have
tried to automatically associate sounds with words for queryby-text retrieval or music annotation [4, 5, 6, 7, 8, 9, 10, 11].
Recently, inference techniques based on topic models, suchas probabilistic latent semantic analysis (pLSA) and latent
Dirichlet allocation (LDA), have been exploited for automatic image annotation and retrieval [12, 13]. Since canonical correlation analysis (CCA) [14] can be interpreted as
an approximation to Gaussian pLSA and also be regarded as
an extension of Fisher linear discriminant analysis (FDA) tomulti-label classiſcation [15], learning topic models through
CCA is not only computationally efſcient, but also promising
for multi-label audio annotation and retrieval.
All of the methods explained above considered a supervised learning setup, where annotated sounds are used as

978-1-4577-0539-7/11/$26.00 ©2011 IEEE

Fig. 1. Topic model for audio tag classiſcation
training data. Thus, in order to further improve the annotation and retrieval accuracy, high-quality semantic information
about sounds would be necessary. However, gathering suchhigh-quality semantic information is expensive, and thus a
big raise in the number of annotated sounds cannot be expected in reality. On the other hand, a large number ofunannotated sounds which have not yet been registered tothe system can be readily collected. Thus, a semi-supervised
learning approach which utilizes a small number of annotatedsounds and a large number of unannotated sounds would bepromising.
In this paper, we propose a semi-supervised learning
method for generic topic models named semi-supervised
canonical density estimation (SSCDE). SSCDE fully makes
use of unannotated sounds for both feature extraction andmodel estimation. More speciſcally, we estimate a lowdimensional latent space representing topics of music byapplying a semi-supervised variant of CCA called SemiCCA[16], which extends the ordinary CCA to be able to utilize
both paired and unpaired samples. Then, in the estimatedlatent space, topic models are learned by multi-class extension of semi-supervised non-parametric density estimation
called semi-supervised kernel density estimation (SSKDE)[17]. Through real-world audio tagging experiments, wedemonstrate the effectiveness of the proposed approach.
2. FRAMEWORKFigs. 2 and 3 show the framework of the proposed method+Nxbrieƀy. Let X = {xn }Nand Y = {y n }N
n=1 be a set
n=1of audio and semantic features with Dx and Dy dimensions,where N and Nx are the number of labeled and unlabeledsounds, respectively. A topic model is estimated from feature
vectors (X, Y ), which consists of two steps.

2232

ICASSP 2011

Fig. 2. Framework of the proposed method (1st part)Fig. 3. Framework of the proposed method (2nd part)The ſrst step is to generate a latent variable Z =+Nxof Dz dimensions with SemiCCA. More specif{z n }Nn=1
ically, a pair (fx , fxy ) of functions fx (x) and fxy (x, y) isderived from (X, Y ) as training samples via SemiCCA, andlatent variables Z are generated from (X, Y ) with (fx , fxy ).
We describe the detail of the ſrst step in Section 4.1.
The second step is to set up a topic model with the help ofkernel density estimation (KDE) on the latent space:
N +N
p(x, y) = (N + Nx )−1 n=1 x p(x|z n )p(y|z n ).
(1)
The main procedure in this step is to estimate the conditional
densities p(x|z n ) and p(y|z n ) of features (x, y) for every
latent variable z n . Note that a conditional density p(y|z n )(n = N + 1, N + 2, . . . , N + Nx ) can be derived even though
the corresponding text label y n does not exist. We show thedetailed procedure in Section 4.2.
Once the model estimation has been ſnished, we are readyto annotate an unseen sound s through maximum a posteriori
 can(MAP) estimation. The most probable semantic feature y
be derived by using an audio feature x(s) extracted from agiven sound s as = argmax p(y|x(s) )y(2)N +Nx(3)y∈[0,1]Dy= argmax
y∈[0,1]Dyn=1p(x(s) |z n )p(y|z n ).

 exceeds a pre-deſned threshold
When the d-th element of yθ, the text word of index d is provided to the given sound s.
3. FEATURE REPRESENTATIONS

signal. Speciſcally, we represent the audio with a time series of Mel-frequency cepstral coefſcients (MFCCs) featurevectors which are popular features for speech recognition andmusic classiſcation [9]. A time series of MFCC vectors is extracted by sliding a half-overlapping, short-time window (23ms) over the audio ſle. The dynamic features (MFCC-Delta)
are derived by calculating the ſrst and second instantaneous
derivatives of each element over consecutive vectors and areappended them to the vector of MFCCs. We use the ſrst 13
MFCCs resulting in about 5,200 39-dimensional feature vectors per minute of audio contents. We create a vector quantization (VQ) codebook of size Dx using about 1,000,000 feature vectors which are sampled randomly so that each soundis represented by 500 feature vectors. Then, normalized code
histograms of VQ results are used as Dx -dimensional audio
N +Nx
feature vectors X = {{xn }N
n=1 , {xn }n=N +1 } (Dx =1,024),
representing the acoustic characteristics of sounds, where
N +Nx{xn }N
n=1 are labeled sounds, while {xn }n=N +1 are unlabeled sounds.
On the other hand, we also extract Dy -dimensional binary
annotation vectors Y = {y n }N
n=1 (Dy = 230) from the 2,012
sounds. Each element of y is set to 1 if the corresponding
word is annotated and 0 otherwise.
4. ASSOCIATING SOUNDS WITH WORDS
We describe a semi-supervised learning method for generic
topic models called SSCDE to associate souonds with words.
4.1. Semi-supervised CCA

We describe an annotated sound corpus “Freesound” used inour experiments and introduce audio and semantic feature
representations.
The Freesound Project is a collaborative database of Creative Commons licensed sounds [18]. Each sound sample has
been annotated using a vocabulary (e.g., genre, instrumentation, emotion, style, rhythm). We extract 2,012 WAV ſles
with a 44.1 kHz sampling rate, a 16 bit depth, and mono orstereo. The stereo audio ſles are converted to the mono ones
by taking the average of the left and right channels. Then, weobtain a vocabulary of 230 words used for annotating thesesounds.
Each sound is represented as a bag-of-feature-vectors
calculated by analyzing a short-time segment of the audio2233

We have proposed SemiCCA [16] that combines CCA withprincipal component analysis (PCA) for utilizing unlabeled
samples. Let us explain the idea of SemiCCA using an illustrative two-dimensional data set depicted in Fig. 4, wherelabeled (resp. unlabeled) samples are plotted with white(resp. red and blue). When only the labeled samples are used,poor projection directions may be obtained by CCA due tooverſttng. In contrast, unlabeled samples may be used for revealing the global structure in each domain. Note that once abasis in one sample space is rectiſed, the corresponding basis
in the other sample space is also rectiſed so that correlations
between two bases are maximized.
Motivated by the above illustration, we smoothly combine the eigenvalue problems of CCA and PCA. More specif-Fig. 5. SSKDE procedure

Fig. 4. Effects of unlabeled samples in SemiCCA
ically, the solution of SemiCCA is given by the leading generalized eigenvectors of the following generalized eigenvalue
problem:
Bw = λCw, w = (wx , wy )

0S (L)
0S xxxy,
B=β+ (1 − β)0S yyS (L)
0yx
 (L)



S xx
00I DxC=β+(1−β)
0I Dy ,0S (L)
yy(4)
(5)
(6)

where S xx is a scatter matrix of X, S (L)xy is a scatter matrix obtained from a pair of labeled samples in (X, Y ), (allthe other scatter matrices can be deſned similarly), and β(0 ≤ β ≤ 1) is a constant named a trade-off parameter. Thetrade-off parameter controls the trade-off between CCA andPCA. Namely, when β = 1, the problem is reduced to theCCA eigenvalue problem, while when β = 0 the problem isreduced to the PCA eigenvalue problem, under the assumption that X and Y are uncorrelated. In general, SemiCCA
with a trade-off parameter 0 < β < 1 inherits the properties of both CCA and PCA so that the global structure in eachdomain and the co-occurrence information of paired samples
are smoothly controlled. Picking up the top Dz generalized
eigenvectors as row vectors, we can obtain Dz -dimensional
mappings W x and W y .
By introducing a stochastic interpretation of CCA [15],we can derive two types of functions fxy and fx to obtain alatent variable z as follows:
fxy (x, y) = Λ1/2 (I Dz + Λ)−1 (W x x + W y y),fx (x) = Λ1/2 W x x,(7)(8)

where Λ is the diagonal matrix with the d-th diagonalcomponent being the d-th largest singular value λd (d =1, 2, . . . , Dz ).

dio feature x and a semantic feature y as
N +N
p(x, y) = (N + Nx )−1 n=1 x p(x|z n )p(y|z n ), (9)p(x|z n ) = κ(fx (x) − z n ),(10)Dy(11)p(y|z n ) = d=1 p(yd |z n ),p(yd |z n ) = μδ(yd − yn,d ) + (1 − μ)Nd /N,(12)where each latent variable z n is calculated aszn=fxy (xn , y n ), n = 1, . . . , N
fx (xn ),n = N + 1, . . . , Nx(13)κ(·) is a Gaussian kernel with a paramter γ, δ(·) is the Diracdelta, yn,d is the d-th element of y n , Nd is the number of theaudio signals containing the d-th word in labeled sounds, andμ (0 < μ < 1) is a parameter representing how reliable a givenlabel is.
Note that this strategy with the traditional KDE can utilize
only labeled samples to estimate a topic model p(x, y) due tolack of labels (cf. Eq. (12)). This indicates that the accuracyof density estimation heavily relies on the number of labeled
samples. Thus, we introduce an idea of semi-supervised KDE[17] used for discrimination tasks, and extend it to multi-label
classiſcation.
From Eq. (11), each conditional density p(yd |z n ) can beviewed as a posterior of the d-th “class” given a “feature”
z n . Then, we can apply the idea of SSKDE to the estimationof this conditional density. For simplicity, we introduce thefollowing matrix forms+Nx
P = {Pn,m }Nn,m=1 ,+Nx y
F = {Fn,d }Nn=1 d=1 , (14)

κ(z n − z m )Pn,m = N +Nx,m =1 κ(z n − z m )Fn,d = p(yd |z n ).

D(15)
(16)

What we want to derive is the matrix F of conditional densities, which can be obtained through an iterative procedure(see [17] for detail). Fig. 5 brieƀy depicts the procedure ofSSKDE.
5. EXPERIMENTS

4.2. Semi-supervised KDE
Based on the idea of KDE, we introduce the following topic
model p(x, y) [19] describing the relationship between an au-2234

This section describes the results for the automatic audio annotation task. We used 2,012 sounds in the Freesound dataset,Table 1. Precision, Recall, and F-value
Precision Recall
CCA (N =1,000)
0.355
0.240
SemiCCA (N =1,000, Nx =912)
0.355
0.274
SSCDE (N =1,000, Nx =912)
0.359
0.333
CCA (N =1,912)
0.462
0.356

F-value
0.287
0.310
0.345
0.402

ysis for audio-video synchronization and audio-visual speech
recognition.
7. ACKNOWLEDGMENT

Fig. 6. Results for the automatic annotation task
and separated them into 1,912 sounds (N + Nx = 1, 912) for
training and 100 sounds for evaluation. Parameters Dz , β, γ,
μ were set to 100, 0.99, 0.8, 0.99, respectively. We determined these parameters experimentally. As the evaluation
measures, we calculated the precision, recall and F-value
based on evaluation dataset.
Fig. 6 shows the experimental results for the automatic
annotation task, where the threshold θ (see the last sentencein Section 2) is varied from 0 to 5.0. In Tab. 1, we show recall
and precision when the F-value reach a maximum value. We
separated 1912 sounds for training into 1000 labeled sounds(N = 1, 000) and 912 unlabeled sounds (Nx = 912), andlearned the proposed model using these sounds. We compared the proposed model with the CCA-based model using all of training data as labeled sounds (N = 1, 912), theCCA-based model using only X (L) (N = 1, 000), and theSemiCCA-based model using X (L) and X (U ) , namely multiclass SSKDE was removed from the proposed method. As thetable shows, the topic model built with the help of SSCDE
outperformed that of SemiCCA. This is because the proposed
method incorporate semi-supervised latent space estimation
with SemiCCA and semi-supervised nonparametric model estimation with SSKDE into the existing learning method oftopic models.
6. CONCLUDING REMARKS
We have developed a new and efſcient method to learn
topic models in a semi-supervised manner, named semisupervised canonical density estimation (SSCDE), and presented a way to integrate it to audio annotation/retrieval.
The proposed method contained two novel contributions: (a)semi-supervised latent space estimation with SemiCCA and(b) semi-supervised non-parametric model estimation withSSKDE. Experiments with thousands of audio signals havedemonstrated that the proposed method is promising.
In the future, we plan to evaluate the proposed method
in detail on a larger database. We also want to compare theperformance of the proposed method with that of the conventional methods [9, 11] and apply to various challenging
real-world problems e.g., multi-modal event correlation anal-The authors are grateful to Dr. Gordon Wichern of Arizona State
University for assistance with experiments. The authors would liketo thank Mr. Takuho Nakano of the University of Tokyo, Dr. Hitoshi Sakano, Dr. Eisaku Maeda, Dr. Katsuhiko Ishiguro, Dr. Kunio
Kashino and Dr. Hidehisa Nagano of NTT Communication Science
Laboratories for their valuable discussions.
8. REFERENCES[1] W. W. Cohen and W. Fan, “Web-collaborative ſltering: Recommending music by crawling the Web,” in Proc. WWW9, 2000.
[2] A. Uitdenbogerd and R. van Schyndel, “A review of factors affecting
music recommender success,” in Proc. ISMIR 2002.
[3] P. Knees et al., “Artist classiſcation with web-based data,” in Proc.
ISMIR 2004.
[4] M. Slaney, “Semantic-audio retrieval,” in Proc. ICASSP 2002.
[5] B. Whitman et al., “Automatic record reviews,” in Proc. ISMIR 2004.
[6] P. Knees et al., “A music search engine built upon audio-based andweb-based similarity measures,” in Proc. SIGIR 2007.
[7] D. Torres et al., “Identifying words that are musically meaningful,” inProc. ISMIR 2007.
[8] L. Barrington et al., “Combining feature kernels for semantic musicretrieval,” in Proc. ISMIR 2008.
[9] D. Turnbull et al., “Semantic annotation and retrieval of music andsound effects,” IEEE TASLP, vol. 16, no. 2, pp. 467–476, 2008.
[10] R. Takahashi et al., “Building and combining document and music spaces for music Query-By-Webpage system,” in Proc. INTERSPEECH 2008.
[11] D. Turnbull et al., “Combining audio content and social context for
semantic music discovery,” in Proc. SIGIR 2009.
[12] K. Barnard et al., “Matching words and pictures,” J. Mach. Learn. Res.,
vol. 3, pp. 1107–1135, 2003.
[13] Li Fei-Fei and P. Pietro, “A Bayesian hierarchical model for learning
natural scene categories,” in Proc. CVPR 2005.
[14] H. Yanai and S. Puntanen, “Partial canonical correlation associatedwith the inverse and some generalized inverse of a partitioned dispersion matrix,” in Proc. the third Paciſc Area Statistical Conference on
Statistical Sciences and Data Analysis, 1993, pp. 253–264.
[15] F. R. Bach and M. Jordan, “A probabilistic interpretation of canonicalcorrelation analysis,” Tech. Rep. 688, Department of Statistics, University of California, Berkeley, 2005.
[16] A. Kimura et al., “SemiCCA: Efſcient semi-supervised learning ofcanonical correlations,” in Proc. ICPR 2010.
[17] M. Wang et al., “Semi-supervised kernel density estimation for video
annotation,” Computer Vision and Image Understanding, vol. 113, no.
3, pp. 384–396, 2009.
[18] The Freesound Project, “http://www.freesound.org/,” .
[19] T. Harada et al., “Image annotation and retrieval based on efſcientlearning of contextual latent space,” in Proc. ICME 2009.

2235