IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 16, NO. 2, FEBRUARY 2008

435

An Efficient Hybrid Music Recommender
System Using an Incrementally Trainable
Probabilistic Generative Model
Kazuyoshi Yoshii, Student Member, IEEE, Masataka Goto, Kazunori Komatani, Tetsuya Ogata, Member, IEEE,and Hiroshi G. Okuno, Senior Member, IEEE

Abstract—This paper presents a hybrid music recommendersystem that ranks musical pieces while efficiently maintaining
collaborative and content-based data, i.e., rating scores given byusers and acoustic features of audio signals. This hybrid approachovercomes the conventional tradeoff between recommendation
accuracy and variety of recommended artists. Collaborative
filtering, which is used on e-commerce sites, cannot recommend
nonbrated pieces and provides a narrow variety of artists. Content-based filtering does not have satisfactory accuracy because itis based on the heuristics that the user’s favorite pieces will have
similar musical content despite there being exceptions. To attaina higher recommendation accuracy along with a wider variety ofartists, we use a probabilistic generative model that unifies thecollaborative and content-based data in a principled way. Thismodel can explain the generative mechanism of the observed
data in the probability theory. The probability distribution overusers, pieces, and features is decomposed into three conditionallyindependent ones by introducing latent variables. This decomposition enables us to efficiently and incrementally adapt the model
for increasing numbers of users and rating scores. We evaluatedour system by using audio signals of commercial CDs and theircorresponding rating scores obtained from an e-commerce site.
The results revealed that our system accurately recommended
pieces including nonrated ones from a wide variety of artists andmaintained a high degree of accuracy even when new users andrating scores were added.
Index Terms—Aspect model, hybrid collaborative and contentbased recommendation, incremental training, music recommendersystem, probabilistic generative model.

I. INTRODUCTION

THE importance of music recommender systems is increasing because many online services that manage large
music collections do not provide users with fully satisfactory
access to their collections [1], [2]. Standard retrieval systems
Manuscript received July 4, 2007; revised September 21, 2007. This work
was supported in part by the Ministry of Education, Culture, Sports, Science,
and Technology (MEXT), CrestMuse Project, Japan Science and Technology
Agency (JST), and a Grant-in-Aid for Scientific Research from the JSPS Research Fellowship. The associate editor coordinating the review of this manuscript and approving it for publication was Dr. Gaël Richard.
K. Yoshii, K. Komatani, T. Ogata, and H. G. Okuno are with the Department of Intelligence Science and Technology, Graduate School of Informatics,
Kyoto University, Kyoto 606-8501, Japan (e-mail: yoshii@kuis.kyoto-u.ac.jp;
komatani@i.kyoto-u.ac.jp; ogata@i.kyoto-u.ac.jp; okuno@i.kyoto-u.ac.jp).
M. Goto is with the National Institute of Advanced Industrial Science andTechnology (AIST), Tsukuba 305-8568, Japan (e-mail: m.goto@aist.go.jp).
Color versions of one or more of the figures in this paper are available online
at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TASL.2007.911503

force users to discover their favorite musical pieces by using
queries including the songs’ titles or artist names. To overcome
this limitation, many researchers focus on music information
retrieval (MIR), which enables users to discover their favorite
pieces on the basis of musical content such as genre, rhythm,
and melody [3]. However, many users have difficulty in stating
their musical preferences as queries. In fact, most users ofmusic streaming services that let one freely listen to numerous
songs for a flat fee, want to access their favorites one after
another without querying. Recommender systems should thus
be able to select musical pieces that will likely be preferred byestimating user preferences. So far, two major techniques havebeen proposed: collaborative and content-based filtering, andthey have complementary properties.
Collaborative methods [4]–[8] recommend musical pieces
to the user by considering how someone else rated them. For
example, suppose that there is a target user who likes piece A.
If many others like A and B, B will be recommended to theuser. This technique is widely utilized in practical e-commerce
services (e.g., Amazon.com and the iTunes music store) andhas been demonstrated to be rather effective. However, there
are two problems. The first problem is that pieces that havenot been rated (e.g., newly released CDs and less well-known
songs) cannot be recommended. This is known as the new-item
problem or the cold-start problem. Therefore, the chances ofencountering unexpected favorites are limited. The second
problem is that the artists of the recommended pieces tend
to be the same and are often well known because most users
tend to rate highly musical pieces by the same artists. Such
recommendations are unsatisfactory or meaningless.
Content-based methods [9]–[12] recommend musical pieces
similar to the users’ favorites in terms of musical properties.
This results in a large variety of artists; i.e., various pieces arerecommended even when they have not been rated. However,these methods have essential problems concerning accuracyof recommendations because similarity in content is only oneof many factors characterizing user preferences. In addition, it
is difficult to associate user preferences with musical content
by using a real database where most users provide few ratingscores. Unfortunately, reliable methods of doing this have notbeen established. For example, although Hoashi et al. [9] tried
to model user preferences, their method was only verified using
an artificial database where 12 subjects were asked to give
rating scores. Logan [10] did not use real rating scores andinstead took a set of songs in a CD album as a particular user’s

1558-7916/$25.00 © 2008 IEEE

436

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 16, NO. 2, FEBRUARY 2008

set of favorites. Celma et al. [11] developed a recommendersystem that models user preferences by collecting profiles ofXML formats from the web. However, their system was notevaluated. Some studies used relevance feedback from users to
improve the accuracy of recommendations [9], [12].
To solve the problems of the above techniques, we developed a hybrid recommender system that utilizes both the user’s
rating and musical content. Our goal was to get more accurate
recommendations referring to a large variety of artists. Here,a fundamental problem is that the observed rating scores andacoustic features incompletely represent user preferences. To
overcome this problem, we used a Bayesian network model
called a three-way aspect model that integrates both of the observed data [13]. This model can directly represent unobservable user preferences as part of a generative mechanism for theobserved data. To our knowledge, our system is the first to apply
the aspect model to content-based data extracted directly frommedia files (audio signals) as opposite to text-based annotationsof media files.
Nevertheless, a critical problem concerning the computational cost emerges in practical situations. Most academic
recommender systems are based on offline methods that require
costly recalculations over all the observed data when nonregistered users or new rating scores are added. To solve thisproblem, we developed an incremental online training methodthat partially updates the parameters of the three-way aspectmodel at low computational cost.
The rest of this paper is organized as follows. Section II specifies the requirements for the recommendations we focus on
and the recommendation task. Section III reviews the conventional methods. Section IV explains our hybrid recommendersystem with the incremental training method. Section V reports
on our experiments that used real rating scores collected froma web-shopping site, Amazon.co.jp. Section VI summarizes the
key points of this study.
II. SPECIFICATIONS FOR MUSIC RECOMMENDATION
We first describe four requirements for designing recommender systems and define the recommendation task. Note that
our target users are those who want to listen to many of theirfavorite songs by using music streaming services that have aflat price (e.g., Last.fm and Pandora) rather than customers of
web-shopping sites (e.g., Amazon.com).
A. Our Goal
We aimed at developing a recommender system that satisfies
the following requirements.
1) High recommendation accuracy
Given a target user, a better recommender system should
select more favorite pieces and fewer disliked ones from areal database in which the number of rating scores given
by users is not sufficient.
2) Rich variety of artists
If the recommended pieces were performed by various
artists unfamiliar to the target user, his or her chances of
discovering new artists who play music that matches his or
her preference would increase.
3) Capability of recommending nonrated pieces

This capability enables users to find appropriate pieces thathave been given a few or no rating scores. In addition, it
increases the variety of artists for the recommended pieces.
4) Prompt responses
If the target user adds rating scores, the recommended
pieces should be reselected in real time. The recommendersystem should be able to deal with the increase in observed
data at low computational cost.
Collaborative methods and content-based ones, which havecomplementary properties, cannot simultaneously satisfy the
first three requirements, as discussed in Section I. We believethat advantages of these methods can be combined by using
both collaborative data (rating scores) and content-based data(acoustic features). In addition, we should pay special attention
to the fourth requirement.
B. Recommendation Task
An objective of music recommendation is to rank musicalpieces that have not been rated by the target user. We let
be users andbe pieces,is the number of the users andis that of the
where
and
denote variables, which
pieces. Here, let
are treated as probabilistic ones in the probability theory. We
assumed that and were registered in the system in advance.
Additional metadata such as titles, artist names, and genre labels
are not used to make recommendations. Rating data should also
be reserved in the system. In this paper, we focus on scores on
be a rating score given
a 0-to-4 scale as rating data. We let
is an integer between 0 and 4
to piece by user , where(4 being the best). By collecting all the rating scores, the rating
matrix, , is obtained as(1)
When user has not rated piece , is substituted forasa symbol, representing an “empty” score for convenience. Notethat most scores in are empty because each user will have
rated a few pieces in . Collaborative methods only use forthe recommendation.
Content-based data is required to use content-based methods.
We assumed that audio signals of the pieces represented bywould be available. The content of each piece is represented asa single vector of several musical elements extracted from thebe these elcorresponding audio signal. Let
is the number of them (dimension of content
ements, where
also be a variable. We letbe thevectors). Here, letvalue of element in piece . By collecting all the content vectors, the content matrix is obtained by(2)
Given target user
, content-based methods use and not
but
for the recommendation. That is, they do
not use scores given by other users in .
III. CONVENTIONAL RECOMMENDATION METHODSWe describe the conventional methods that were used forthe comparative experiments discussed in Section V. As mentioned in Section I, the conventional methods can be categorized

YOSHII et al.: EFFICIENT HYBRID MUSIC RECOMMENDER SYSTEM

437

Fig. 1. Memory-based collaborative filtering method. Calculating similarities
in rating-score vectors between target user and others.

into collaborative and content-based filtering. They can furthermore be categorized from the viewpoint of methodology intomemory-based and model-based methods.
To make recommendations, memory-based methods operate
ifover the entire rating matrix, , (and content matrix
needed). Model-based methods, in contrast, use these databases
to train models that estimate user preferences, which are thenused to make recommendations. In general, the latter can makeprompter recommendations once the models are constructed.
However, the computational cost involved in training thesemodels tends to be high. Some studies on text-based recommendation have reported that model-based methods outperform
memory-based ones in terms of recommendation accuracy [6].
A. Collaborative Filtering
We review major methods of collaborative filtering.
1) Memory-Based Methods: Typical memory-based
methods try to predict the unknown rating scores of musical pieces that have not been rated by a target user, byconsidering someone else’s scores for those pieces, as outlinedin Fig. 1. That is, these methods are fundamentally based onheuristics. Given a target user , letbe his or her predicted
rating score for piece , which is given by
(3)
where
is the average rating score of user andis thatis a weight that reflects the preference simiof user .
larity between users and , and is a normalizing factor sothat the absolute values of the weights add up to unity. That is,. After the score is predicted, pieces are ranked.
according to
Several measures are used to calculate similarity. The mostpopular is the Pearson correlation coefficient [5] with whichsimilarity is defined as(4)

where summations overare for pieces rated by both and. However, there are usually very few of those pieces when
the rating matrix
is sparse. Therefore, this basic similarity
calculation often fails.
To solve this problem, empty scores in are replaced with
to 2.5,a default score . We empirically set the value forFig. 2. Model-based collaborative filtering method. Ranking musical pieces byusing an aspect model with latent variables.

which is biased (cf., a neutral score is 2 on the 0-to-4 scale),because most users tend to give high scores (3 and 4) more often
than low ones (0 and 1).
2) Model-Based Methods: From the viewpoint of the probability theory, the unknown rating scores of a target user can bepredicted by calculating their expected values(5)
is a probabilistic variable, andwhereis the probability of score when user and pieceareobserved. This probability is estimated by using actual ratingby assuming probabilistic models [6]. Note thatscores inthese methods are based not on some ad hoc heuristic rules but
on a statistical model learned from the underlying data using
machine learning techniques.
An alternative approach is to rank musical pieces for given, which is the conditional probauser according towhen user is observed. Here, we assume
bility of piecethat the co-occurrence of piece and user (e.g., events thatuser listens to piece ) is likely to be observed if user
prefers piece , i.e., the joint probability,, increases asorbecomes large. Therefore, the probability
score
represents how likely user is to prefer, Hofmann and Puzicha [14] used
piece . To estimate
a probabilistic generative model called an aspect model. A data
mining technique based on the aspect model is known as theprobabilistic latent semantic analysis (pLSA) [15].
The aspect model introduces multiple latent variables, which
represent conceptual genres, as outlined in Fig. 2. Let
be these variables, whereis the number ofthem. An interpretation of this model is that user stochasticallyselects conceptual genre according to his or her preference, and then stochastically generates piece according. The conceptual genres, which do not
to the probability
correspond to “genres” in the general sense, are not given inadvance. They are automatically determined so that the model
provides the best explanation of the generative mechanism forthe observed rating scores. To put this more concretely,andare statistically estimated by using the EM algorithm[16] (the method of estimating parameters is a simplified version of ours described in Section IV). After the parameters are
estimated, musical pieces are ranked for each user according
.
to

438

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 16, NO. 2, FEBRUARY 2008

Fig. 3. Memory-based content-based filtering method. Calculating similarities
in musical content between the user’s favorites and other pieces.
Fig. 4. Model-based content-based filtering method. Estimating the preferences of the target user by using his or her rating scores.

B. Content-Based Filtering
We review major methods of content-based filtering.
1) Memory-Based Methods: Typical memory-based
methods try to rank musical pieces on the basis of the similarity of musical content by representing user preferences in adenote amusical-content space. Let
be acontent vector of piece . Let
set of pieces that were given positive scores (3 or 4) by user .
be a set of pieces
In the same way, letthat were given negative scores (0 or 1). Given target user ,the algorithm is
is not empty, the set of content vectors
1) If
represents the musical taste of . If
is not empty, the set of vectors,
represents the musical dislikes of .
and the2) The similarities between content vectors incan be calculated with a simcontent vector
be the maxilarity measure, as outlined in Fig. 3. Let
imum similarity, which indicates how likely will prefer. Then,is calculated for each piece . Ifisempty,is set to zero.
be the maximum similarity between content vec3) Let
and content vector, which istors iniscalculated in the same way in the previous step. If
empty,is set to zero.
that have not been rated
4) The musical pieces, which isby are ranked according to the total value(6)
The cosine measure is often used to calculate the similarities
between two vectors [9]. Note that if only provides neutralscores (2), random pieces are recommended.
2) Model-Based Methods: Recommending musical pieces
can be viewed as categorizing them into two classes: favorites
and disliked. This is a standard machine-learning problem.
Naïve Bayes models have often been used to categorize
text documents (e.g., spam-mail filtering) [17]. However, one
problem is that we should define and calculate musical elements
which correspond to words in text documents. We propose anovel method of solving this problem in Section IV-B. Oncethe content matrix is obtained with our method, the binary
categorization models are given by(7)(8)

indicates how likely user is to like
where
indicates how likely is(dislike) piece .
to like (dislike) element , which is trained as outlined in Fig. 4(9)Finally, musical pieces are ranked for eachtotal value
which is obtained byaccording to the(10)is empty,
Ifconvenience.

is set to one for

IV. HYBRID RECOMMENDER SYSTEM
We propose a hybrid music recommender system that theoretically integrates collaborative data (rating scores of users) andcontent-based data (acoustic features of audio signals) to meetthe four requirements described in Section II-A. First, we will
discuss the problems with integrating both data. Next, we will
explain our elegant approach based on a probabilistic generativemodel that can be incrementally trained.
A. Problems
We need to solve two problems to design a hybrid recommender system. Each problem is discussed below.
1) Reliable Integration: The first problem is to reflect the collaborative and content-based data when making recommendations. An easy solution is to use collaborative and content-based
methods in parallel [18]–[20] or in cascade [21]–[25]. However,such an approach has drawbacks. Although meta recommendersystems have been proposed to select a recommender systemamong conventional ones on the basis of certain quality measures [18], [19], the disadvantages of the selected system are
inherited. Moreover, the heuristics-based integration dealt within other studies lacks a principled justification. For example,
Claypool et al. [20] proposed a linear combination of ratingscores predicted by several conventional systems. Typical cascade systems [21]–[23] first represent user preferences by using
content-based data and then make recommendations in a collaborative way that calculates the similarities of the contentbased user preferences. Melville et al. [24] used a collaborative
method after predicting unknown rating scores by using content-

YOSHII et al.: EFFICIENT HYBRID MUSIC RECOMMENDER SYSTEM

based data. A similar content-boosted approach was proposed
by Hayes [25]. A principled way of integration is to take amodel-based approach, which statistically estimates user preferences on the basis of a unified model. Here, note that the observed data (rating scores and acoustic features) are incomplete;i.e., these data partially represent the latent user preferences.
2) Efficient Calculation: The second problem, which has
been scarcely dealt with, is to efficiently adapt a recommendersystem according to the increase in rating scores and users. An
easy solution is to take a memory-based approach, which is originally free from this problem because the whole data is always
used to make recommendations. However, this results in the late
responses. Yu et al. [26] tried to overcome this disadvantage
by using a probabilistic method in a pure collaborative filteringcontext. On the other hand, Zhang et al. [27] proposed an efficient method that incrementally trains an aspect model used
for model-based collaborative filtering. To our knowledge, there
are no studies on incremental adaptation of hybrid recommendersystems. We need to carefully design a hybrid architecture whileconsidering whether the previous prominent methods can be applied or not.
B. Our Approach
To solve the problems, we take the following strategies:
1) Model-Based Integration: We use a probabilistic generative model, called a three-way aspect model, proposed byPopescul et al. [13]. This model is an extended version of
the aspect model (pLSA) proposed by Hofmann [15] (seeSection III-AII), and it explains the probabilistic generative
mechanism for the observed data (rating scores and acoustic
features) by introducing a set of latent variables. As part of
the generative mechanism, the model directly represents user
preferences (latent favorite genres), which are statisticallyestimated with a theoretical proof. This estimation makes therecommendations more reliable.
2) Incremental Training: We propose a method that incrementally adapts the three-way aspect model on the basis of an
extended version of Zhang’s method [27].
C. Model-Based Integration Method
We will discuss how the three-way aspect model is applied tomusic recommendation. After that, we explain its implementation.
1) Analogy to Document Recommendation: Popescul’s hybrid model cannot be directly applied to our system because it
was designed for recommending text documents. The document
content is represented on the basis of the “bag-of-words” model
originally proposed in the field of language processing, i.e., thecontent of a document is represented as a set of frequencies of
informative words.
To apply the three-way aspect model to music recommendation, the content of each piece should be represented as a single
vector in which all dimensions are semantically equivalent. For
example, each dimension always represents a word frequency
with Popescul’s method. In addition, all dimensions of each
vector should add up to unity.
2) Application to Music Recommendation: We proposea “bag-of-timbres” model in analogy with the bag-of-words

439

Fig. 5. Asymmetric representation of our aspect model using polyphonictimbre weights as music content.

model to meet the above-mentioned conditions. The content
of each piece is represented as a bag of multiple timbres, i.e.,a set of weights of polyphonic timbres. Aucouturier et al. [28]proposed the original concept of polyphonic timbres, which do
not represent the perceptual “sounds” of individual instruments
but of their combinations (mixed sounds). These features areimportant factors that characterize the textures of musicalpieces. In addition, polyphonic timbres can easily be extracted
from various audio signals because the instrument parts do
not need to be separated (this separation is quite difficult).
However, Aucouturier et al. pointed out that this approach hasa performance limitation on timbral-similarity-based audio
clustering. In contrast, we expect that this approach works wellby integrating it with the “wisdom-of-crowds” approach basedon collaborative data.
3) Three-Way Aspect Model Based on Bags of Timbres: The
observed data in the three-way aspect model are associatedwith latent variables, whereis the total
number, as outlined in Fig. 5. Each latent variable corresponds
to a conceptual genre. Given a target user , a set of conditional
probabilities
reflects the musical taste of .
One possible interpretation is that user stochastically selects,
conceptual genre according to his or her preference,and polyphonicand then stochastically generates piecetimbre according to their probabilities
and. We
assumed the conditional independence of users, pieces, andtimbres through the latent genres. In this model, all users andall musical pieces can be observed for selection of any genre,
whereas most clustering methods assign each user and eachpiece to a single genre class. To deal with the increase in ratingscores and users, we only need to update user preferences.
4) Calculation of Bags of Timbres: To calculate bags of timbres from audio signals, we used Mel-frequency cepstral coefficients (MFCCs), which have been used in many studies on genre
classification [29]. Their method was used to apply a Gaussian
mixture model (GMM) to the MFCCs extracted from each musical piece. The similarity between two pieces was measured
as the reciprocal of the distance between corresponding GMMs
that was obtained by sampling.
We also built a GMM for MFCCs extracted from each piece.
GMMs in total. We assumed that eachThat is, we obtained
Gaussian in a GMM would represent the MFCC distributionof a particular polyphonic timbre, i.e., the mixture weights of

440

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 16, NO. 2, FEBRUARY 2008

Gaussians would correspond to the weights of timbres. Unfortunately, we could not use Aucouturier’s method because theGaussians in one GMM were different from those in the other
GMMs. Therefore, each GMM represented a different combination of polyphonic timbres.
Our unique idea to solve this problem was to let the bags oftimbres of all the pieces share the same combination of Gaussians. The means and covariances of the Gaussians were estimated by using numerous MFCCs extracted not individuallyfrom each piece but from all the pieces, and the mixture weightsof the Gaussians were discarded in this estimation. The weightsof the polyphonic timbres in each piece were obtained as the
mixture weights of the fixed Gaussians in that piece; only the
mixture weights were reestimated by using the MFCCs of thesingle piece.
First, 13-dimensional MFCCs were extracted from audio
signals sampled at 16.0 kHz by applying short-time Fouriertransformation (STFT) with a Hanning window of 200 ms.
The shifting interval was 100 ms. Then, 28-dimensional feature vectors were obtained (MFCCs, energy, and their delta
be feature vectors
components). We letis the total number. Next, theextracted from piece , whereparameters of the Gaussians were estimated for all the pieces
by using the expectation-maximization (EM) algorithm [16],.
where the number of mixtures was set to 64. That is,
The number of mixtures , which corresponds to the GMM
complexity, influenced the recommendation accuracy defined inwas sufficient
Section V-B. We empirically found that
because the recommendation accuracy did not almost improve
to be any higher.
much if we took
be the th Gaussian, which is given byHere, let

(15)
(16)
is the prior probability of user . Marginalizing ,where
we obtain the joint probability distribution
over ,
, and
(17)

(12)

where
is the prior probability for genre , andis theprobability that genre will generate user .
,The unknown model parameters are,, and, which should be estimated by using therating matrix and content matrix . After they are estimated,the musical pieces are ranked for a given user according to.
6) Estimation of Model Parameters: We explain the methodof estimating the model parameters using the EM algorithm[16]. Here, we assume that each event occurs independently.
The likelihood of the parameters for the observed data is givenby, which is the weight of timbre in piece

(18)

(11)

where is the mean vector and is the covariance matrix.
is the squared Mahalanobis distance given by

The content value,, is obtained byandRecall that in the definitions of•is the rating score of user for piece . A default
rating score (2.5) was substituted for the empty scores inour method, as described in Section III-A.
is the weight of polyphonic timbre in piece .
•co-ocThis assumption is based on the general fact that
more or thecurs more frequently if user prefers piece
weight of timbre in piece is higher.
is defined as the co-occurIn the same way, a tuple
,
,
, and
. Eachrence of four variables, corresponds to unobservable events, wheretuple,
user selects genre , and then genre simultaneously generbe the probability
ates timbre and piece . Let. The assumed conditional inof the co-occurrence
dependence over , , and through leads to a symmetricform ofas follows:

(19)
(13)
is a normalizing factor so that.
where
5) Formulation of Three-Way Aspect Model: We explain themathematical formulation for the three-way aspect model. Let, be the co-occurrence of three probabilistic
a tuple,,, and. Each tuple,,
variables
corresponds to observations where user listens to timbre inbe the number of these observations.
piece . Let
In this paper, we assume thatis proportional to theandproduct ofGiven the observed data (rating matrix), the log-likelihood, , is obtained as

and content matrix(20)
We use the EM algorithm [16] to estimate the parameters so that
log-likelihood reaches a local maximum as follows:
E step(21)(14)

YOSHII et al.: EFFICIENT HYBRID MUSIC RECOMMENDER SYSTEM

441

M step
(22)
(23)
(24)
(25)
where we have introduced a parameter that is set to one in thebasic EM algorithm. These steps are iterated alternately untilconverges to the local maximum. Note that the local maximum
problem cannot be ignored in practice because the matrices
and are sparse.
To solve this sparseness problem, we use the deterministic annealing EM algorithm (DAEM) [30], which is a variant of the
EM algorithm. The DAEM algorithm is equivalent to the tempered EM algorithm proposed by Hofmann [15] and was used
for training Popescul’s three-way aspect model [13]. works
as an inverse computational temperature, which is gradually inand increased. The DAEM algorithm starts withwhen log-likelihood
creases with ratio , using
converges. Finally, is set to one. It is sufficient to set
to
0.1 and to 1.2 in practice.
The computational complexity of training the model with this, considering thatmethod isandremain constant and the number ofusers and number of musical pieces are not fixed. This meansthat both computational time and memory use increase rapidly. One possible solution is to categorize
according tousers and musical pieces into fewer groups. This will be donein a future work.

1) Updating Profiles of Registered Users: Given a specific,user , the conditional probability distribution
which is called a user profile, captures his or her musical preferrepresents how likely user is to selectences. Recall thatconceptual genre according to his or her musical preferences.
The three-way aspect model assumes that the profiles of all users
are independent, as outlined in Fig. 5. Therefore, when a user
gives new rating scores, we only need to update his or her profile
without affecting the profiles of others to keep the log-likelihood
maximized. This results in lower computational cost.
,
We aim at updating a profile of user :where
is a registered user who gives new rating scores.
We assume that model parameters other than the profile of user
are constant. Therefore, maximizing the log-likelihood isequivalent to maximizing the sum of terms including user
in . We letbe the log-likelihood for the observed data
concerning user , as follows:
(26)
(27)
where we have introduced a new operator
an arbitrary value), which represents
Jensen’s inequality, we can rewrite (27) as

for

( is
. Using
(28)
(29)

(30)

D. Incremental Training Method
To achieve prompt responses, we propose an efficient incremental training method that partially updates the parametersof the three-way aspect model. Ours is an extended version of
Zhang’s method [27], which was designed for incrementallytraining a topologically different aspect model. After this, wewill call the three-way aspect model that was initially obtainedusing EM-based training a base model. We will call a modelthat was obtained by incrementally training the base model an
updated model.
Our method individually addresses the following two cases
to obtain the updated model.
who1) Recommendation given to a registered user
provides new rating scores.
who2) Recommendation given to a nonregistered user
provides some rating scores.
While the size of the model (the number of parameters) remains
unchanged in the first case, it increases in the second because
nonregistered users are added. Our method differs from typicalincremental training methods based on a fixed model size inthis regard. Next, we will explain the incremental training algorithms for both bases.

whereis a supplementary function given by(31)

Becauseandare almost constant, maximizing
is approximately equivalent to maximizing the first termof (30). Therefore, we can obtain the following maximization
problem:maximize(32)(33)
is a constrained objective function and (33) is a conwhere
straint function. Here, we use the Lagrange multiplier method[31] to solve this problem. Introducing an unknown multiplier, we define
as
(34)

442

IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 16, NO. 2, FEBRUARY 2008

The partial derivative ofwith respect tois given by

TABLE ICOMPOSITION OF RATING SCORES(35)
To maximize, (35) should be zero. Therefore, we obtain(36)

Substituting (36) into (33) gives
(37)

(38)
(39)
Substituting (39) into (36), we finally obtain the updating formula(40)

The computational complexity of updating the profile of useris, whereis the number of pieces that were
newly rated by user . To update the profile, we only need to reterms concerning these pieces in each sumcalculate themation of the updating formula (40).
2) Creating Profiles of Nonregistered Users: We aim at cre, whereis aating a profile of user :.
nonregistered user who has some rating scores
Note that these scores were not used for training the base model.
We can use the updating formula (40) in this case to create theand, which were estimated forprofile by using
the rating scores of registered users .
V. EVALUATION
We will report on several experiments that were conducted
to determine whether our hybrid recommender system satisfies
the four requirements described in Section II-A. First, we compared our method based on the three-way aspect model with thefour conventional methods described in Section III in terms ofrecommendation accuracy. Next, we evaluated our method interms of variety of recommended artists and capability of recommending nonrated pieces. Finally, we evaluated our incremental training method in terms of recommendation accuracy.
A. Experimental Conditions
It is ideal to use large-scale rating data in which the number ofrating scores given by users is sufficient to conduct reliable comparative experiments, However, collecting rating scores basedon questionnaires is extremely time consuming. In addition, theratio of negative scores tends to be much higher in artificial

rating data (e.g., the rating data in the previous study [9]) than inreal rating data. Users tend to voluntarily give much more positive scores than negative scores.
To deal with this problem, we collected real rating scoresfrom web sites [32]. Amazon.co.jp provides application programming interfaces (APIs) that allow us to download almost
all information on a web site.1
The musical pieces we used were Japanese songs on single
CDs that were ranked in weekly top-20 sales rankings from
April 2000 to December 2005. The corresponding scores withuser IDs were collected from Amazon.co.jp. If a user has rated
multiple pieces, we can identify the scores given by the sameuser. However, there were many unreliable users and pieces that
had few/no scores. To deal with this problem, we selected users
and pieces so that the number of scores given by a user and thenumber of scores given to a piece were always more than 4. As
was 316 andwas 358. Table I lists the coma result,position of actual scores in the rating matrix . The density of
was 2.19%, which was almost equal to the density in the previous study [6]. This means contains practical data.
By using the prepared rating data, we compared our hybrid
method based on the three-way aspect model with the four conventional methods described in Section III:1) Memory-based collaborative method using the Pearson
correlation coefficient (called memory-CF).
2) Model-based collaborative method using Hofmann’s aspect model (called model-CF).
3) Memory-based content-based method using the cosine distance measure (called memory-CB).
4) Model-based content-based method using the naïve Bayesmodels (called model-CB).
Recall that(the number of polyphonic timbres) was 64.
(the number of latent variables) was set to 10.
B. Evaluation Measure
The experiments were conducted with tenfold cross validation; i.e., a training matrix
and an evaluation matrix
werecreated from the rating matrix by randomly masking 10% of
the actual scores in , as outlined in Fig. 6. The five methods,
including ours, were used to rank musical pieces for each user
and the content matrix , if needed.
by using
We devised an evaluation measure to calculate the accuracyof recommendation that focuses on the ratio of favorite pieces
to recommended pieces whose scores are masked. We examined. Fig. 7 showsthe entire top- rankings of all users
an example for the case of. Note that we could not eval)uate all the recommended pieces (the total number was
because most of them had not actually been rated by users (thebe the total
corresponding scores were in ). Here, we let
number of recommended pieces whose scores were masked but
1[Online]. Available: http://www.amazon.com/gp/aws/landing.html.

YOSHII et al.: EFFICIENT HYBRID MUSIC RECOMMENDER SYSTEM

443

TABLE IIRECOMMENDATION ACCURACY: A

Fig. 6. Preparation of tenfold cross validation. Creating training matrix R andevaluation matrix R by randomly masking 10% of the actual scores in rating
matrix R.

TABLE III
RECOMMENDATION INACCURACY: A

C. Evaluation of Recommendation Accuracy

Fig. 7. Evaluation of recommendation accuracy. Calculating ratio of recommended pieces that were given highest actual scores.

were actuallyand let be. We letously, was much less thanto , i.e.,. Obvibe the ratio of(41)

and a lower value forindicate better
A higher value foris not equal to 20% even when recperformance. Note thatommending random pieces. In this case,are 57.9%,
19.1%, 8.57%, 4.85%, and 9.54%, respectively. They are equal
to the ratios listed in Table I.
The precision and recall rates, which are commonly used toevaluate document recommender systems, were not suitable forbe the total number of pieces whose
our task. Instead, we letscores were masked but were actually . In the case of Fig. 7,
are 2, 1, 1, 1, and 1, respectively. The precision andrecall rates,
and , are given by(42)
would inIf we used these measures, higher values for anddicate better performance. Obviously, and are proportional
because the denominators are constant. Therefore, a way
toof improving performance is to increase , i.e., to recommend
more pieces whose scores were masked but were actually rated.
However, such recommendations are meaningless. In addition,andare not related to and . This means the disliked
pieces in the recommended ones are not taken into account.n

The results of the comparative experiments revealed that our
hybrid method outperformed the four conventional methods in(Table II). For,
terms of recommendation accuracy
the accuracies obtained with our method were much higher than, although the accuracy
those with the other methods. Forof our method (93.5%) was slightly less than the best accuracy
obtained by model-CF (95.2%), the difference was small.
Note that our experiments with rating scores collected from
Amazon.co.jp were advantageous to collaborative methods.
Users tend to give positive scores (3 and 4) to most musicalpieces performed by their favorite artists. It is more effective
to focus on positive scores than on acoustic features to deal
with this tendency. When we increased , the accuracy of our
method gradually deteriorated
while the accuracy of model-CF more rapidly deteriorated. This indicates that acoustic
features are important factors that characterize user preferences, although artist names are the most dominant factors at
Amazon.co.jp. In the future, we plan to check whether musicstreaming services that have flat pricing have this feature.
Our method yielded the lowest recommendation inaccuracy(Table III). Our method recommended the fewest pieces thatwere actually hated by users. This is an important aspect for
ensuring stress-free environment.
We confirmed that our bag-of-timbres model worked well for
representing musical content derived from audio signals. The
accuracies achieved by memory-CB were slightly better thanthose by memory-CF. These results also proved the importanceof acoustic features in modeling user preferences. However, as
we pointed out in Sections I and IV-C2, only focusing on thecontent-based aspect of user preferences limits the recommendation accuracy improvement.
D. Evaluation of Artist Variety
We propose two measures to calculate the variety of artists.
Artists who perform musical pieces correspond to the authorsof documents and directors of movies in other recommendation
tasks. However, variety has not been investigated in the field oftext-based recommendation. Our study is the first to examine thevariety of artists. Given a target user , we letbe the number.
of artists in pieces that were recommended to user
We then let
be the number of recommended pieces by new444IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 16, NO. 2, FEBRUARY 2008

TABLE IVRATIO OF NUMBER OF ARTISTS: V =x

TABLE VICAPABILITY OF RECOMMENDING NONRATED PIECESTABLE VPERCENTAGE OF PIECES BY NEW ARTISTS: V =xartists whose pieces had not been rated by user.
be the average forandbe that forover allWe letandwere less than . Higher values
users. Obviously,
andindicate a larger variety of artists.
of
The results showed that our method recommended a sufficiently diverse variety of artists (Tables IV and V). Note thatthe values listed in the tables are normalized by the number ofrecommended pieces in order to compare all results directly.
gotten with our method were the second best
The values offor. The values ofwere the third best. However,we think that the differences in measures between ours and thebest methods were small. More important is the simultaneousachievement of high recommendation accuracy and rich variety.
We proved that memory-CF, which is used in many e-commerce services, provides only a limited variety of artists.
andfor memory-CF were the lowest for.
was especially small (0.629) in the case of. This means
the probability that a user will discover an unknown artist with a
top-1 ranked piece is at most 62.9%, which is much lower thanthe probabilities obtained with the other methods (about 90%).
andwere generally good for the conOn the other hand,tent-based methods. However, many disliked pieces performedby various artists contributed to enriching variety. Indeed, thesemethods were inferior to the others in terms of recommendation
accuracy, as listed in Tables II and III.
E. Evaluation of Capability of Recommending Nonrated
Musical Pieces
To evaluate the recommendation accuracy for nonrated musical pieces, we did another tenfold cross validation by masking
actual scores of 10% of ; i.e., the training matrix
includednonrated pieces and the evaluation matrix
included the actual scores for these pieces. The results revealed
that our method could make reasonable recommendations fornonrated pieces (Table VI). It competed with the content-based
methods in terms of accuracy of recommending favorite pieces), although
was not high. Its probabil(the values forities of recommending disliked pieces were similar to those of
the conventional methods. Note that the recommended nonrated
pieces amounted to less than 5% of that of all recommended
pieces.

Fig. 8. Recommendation accuracy A versus the rating scores for incremental
training.

F. Evaluation of Incremental Training Method
We evaluated the accuracy of incremental training for the twocases described in Section IV-D.
1) Recommendations for Registered Users: The objective of
this experiment was to assess the decrease in recommendation
accuracy caused by reducing the number of rating scores thatwere used to construct the base model. In addition, we tried toclarify the differences in recommendation accuracy between thebase model and updated models.
Let us first explain the experimental procedure. We used theto prepare a base model and a total of ten uprating matrix
dated models. The base model was then constructed by using
as training data. The updated models was obtained as follows.
was prepared by randomly
1) A temporary rating matrixof actual scoresmasking thein . If is zero,is equal to .
as training
2) A temporary base model was built by using
data.
3) An updated model was obtained by adding the masked
scores, i.e., by using .
Each model was used to rank the musical pieces. To calculate
the recommendation accuracies, we used the evaluation matrix
for all the settings. We iterated these procedures ten times
while switching the rating matrices that were prepared for tenfold cross validation described in Section V-B.
Fig. 8 plots the results. Our method can promptly and appropriately adapt recommendations according to the increase in
rating scores. We found that the accuracy barely deteriorated
even when the number of rating scores used to update the base.
model was increased to the number for building it
An interesting fact is that the difference in accuracy between

YOSHII et al.: EFFICIENT HYBRID MUSIC RECOMMENDER SYSTEM

445

,

,
,
, and
,
in the cases of). This is a reasonable
observation because rating scores given by nonregistered users
were not used to train the base models. Therefore, the recommendation accuracy for nonregistered users tended to be unstable. However, by doing the t-test, we found that there were nodifferences in average accuracy in any of the cases (
,
,
,
, and
,
in the cases of).
G. Discussion
Fig. 9. Recommendation accuracy for registered and nonregistered users.

the base and updated models narrowed as the number of recomincreased. The largest difference was about
mended pieces. However, the rec5% in evaluating the top-1 rankings
ommendation accuracy was sufficiently high even in this caseand was higher than the other “nonincremental” methods listed
in Table II.
2) Recommendations for Nonregistered Users: The objective of this experiment was to compare the accuracies of recommendations given to registered users with those given to nonregistered users. Smaller differences in recommendation accuracy
indicate better performance.
is
The experimental procedure using the training matrix
as follows.
were randomly selected from . They
1) 10% of users
be thewere regarded as nonregistered users. We letremaining users (registered users).
was obtained by removing2) A reduced training matrix
from . Therefore, the size ofwas 90% of thatof .
3) A temporary base model was constructed by usingastraining data.
, we did4) To calculate the recommendation accuracy forthe following:in the base model were updated bya) Profiles ofagain.
usingb) Recommendations based on the updated profiles werein evaluevaluated by using the rating scores ofation matrix .
, weTo calculate the recommendation accuracy for
did the following:
were created by using the ratinga) Profiles ofscores ofthat were removed in Step 2).
b) Recommendations based on the created profiles werein .
evaluated by using the rating scores of
We iterated these procedures ten times while switching therating matrices that were prepared for the tenfold cross validation described in Section V-B. We repeated the experiment
four times and computed the average and variance of therecommendation accuracies.
Fig. 9 clearly shows that our method can make accurate recommendations to nonregistered users as well as to registeredusers. We found differences in the variance of accuracy in allcases through the F-test at a significant level of 5% (We would like to discuss the computational time required toobtain the base and updated models using a standard computerwith a 3-GHz Pentium-4 processor. The system required about
10 min to obtain the base model with the EM-based training
method. In contrast, the system required only 5 s to update thebase model, i.e., to obtain an updated model, with the incremental training method.
There are three remaining issues.
1) The current system cannot incrementally register nonregistered pieces (e.g., new releases) to the three-way aspectmodel because the current system only takes into accountthe addition of new scores or nonregistered users. We
expect that this problem can be solved by introducing amodel-updating formula in the same way as described in
Section IV-C.
2) It is necessary to determine until when the model can beincrementally updated, i.e., the point at which the decrease
in recommendation accuracy exceeds the user’s tolerance.
To determine an optimized timing for retraining the wholemodel, we plan to conduct a user study by deploying oursystem on realistic large databases.
3) It is important to examine whether semantic properties oflatent variables are similar to those of existing genres. To
do this, it would be better to use realistic large databases
having large variety of genres and moods.
VI. CONCLUSION
This paper presented a hybrid music recommender systemthat ranks musical pieces by comprehensively consideringcollaborative and content-based data, i.e., rating scores derived
from users and acoustic features derived from audio signals.
To create our system, we used a probabilistic generative model
called a three-way aspect model. This model can theoretically explain the generative mechanism for both kinds of theobserved data by introducing a set of latent variables, which
conceptually correspond to genres. One possible interpretationof the generative mechanism is that a user stochastically selects
a genre according to his or her preferences and then the genre
stochastically generates a musical piece and an acoustic feature.
That is, the joint probability distribution over users, pieces, andfeatures is decomposed into three independent distributions,which are respectively conditioned by genres. These distributions are statistically estimated so that the probability ofgenerating the observed data is maximized. This allows us toincrementally train the aspect model according to the increasein users and rating scores at low computational cost, i.e., weonly need to partially update the parameters.

446IEEE TRANSACTIONS ON AUDIO, SPEECH, AND LANGUAGE PROCESSING, VOL. 16, NO. 2, FEBRUARY 2008

The main contributions can be summarized as follows.
1) We proposed a hybrid recommender system that has threefundamental capabilities: a high degree of recommendation accuracy, a large variety of artists, and a capability forrecommending nonrated pieces.
2) We proposed an incremental training method that satisfies
an important requirement: a prompt response without deteriorating accuracy.
3) We proposed a bag-of-timbres model that represented timeseries of MFCCs as a single vector. Our flexible method
can be applied to various audio signals or to the time seriesof various musical features.
4) We demonstrated these capabilities and proved the effectiveness of our bag-of-timbres model by conducting experiments that used real rating scores.
In the future, we plan to use various audio-based features suchas tempi, pitches, and rhythmic patterns for improving the representation of the musical content. We may try several toolkits
such as MARSYAS [33] and CLAM [34] for automatic feature
extraction. In addition, we plan to apply our method to a social networking service (SNS) in which users are introduced toothers with similar musical preferences.
ACKNOWLEDGMENT
The authors would like to thank Dr. H. Asoh (AIST, Japan)
for his insightful comments.
REFERENCES[1] A. Uitdenbogerd and R. van Schyndel, “A review of factors affecting
music recommender success,” in Proc. Int. Conf. Music Inf. Retrieval(ISMIR), 2002, pp. 204–208.
[2] G. Adomavicius and A. Tuzhilin, “Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions,” IEEE Trans. Knowl. Data Eng., vol. 17, no. 6, pp. 734–749,
Jun. 2005.
[3] R. Typke, F. Wiering, and R. Veltkamp, “A survey of music information retrieval systems,” in Proc. Int. Conf. Music Inf. Retrieval (ISMIR),
2005, pp. 153–160.
[4] P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl, “GroupLens: An open architecture for collaborative filtering of netnews,” inProc. ACM Conf. Computer-Supported Cooperative Work, 1994, pp.
175–186.
[5] U. Shardanand and P. Maes, “Social information filtering: Algorithms
for automating word of mouth,” in Proc. ACM CHI’95 Conf. Human
Factors Comput. Syst., 1995, pp. 210–217.
[6] J. Breese, D. Heckerman, and C. Kadie, “Empirical analysis of predictive algorithms for collaborative filtering,” in Proc. 14th Conf. Uncertainty Artif. Intell. (UAI), 1998, pp. 43–52.
[7] W. Cohen and W. Fan, “Web-collaborative filtering: Recommending
music by crawling the web,” Comput. Netw., vol. 33, no. 1–6, pp.
685–698, 2000.
[8] G. Linden and B. Smith, “Amazon.com recommendations: Item-toitem collaborative filtering,” IEEE Internet Comput., vol. 7, no. 1, pp.
76–80, Jan./Feb. 2003.
[9] K. Hoashi, K. Matsumoto, and N. Inoue, “Personalization of user profiles for content-based music retrieval based on relevance feedback,”
Proc. ACM Multimedia, pp. 110–119, 2003.
[10] B. Logan, “Music recommendation from song sets,” in Proc. Int. Conf.
Music Inf. Retrieval (ISMIR), 2004, pp. 425–428.
[11] O. Celma, M. Ramirez, and P. Herrera, in Proc. Int. Conf. Music Inf.
Retrieval (ISMIR), 2005, pp. 464–467.
[12] B. Logan, “Content-based playlist generation: Exploratory experiments,” in Proc. Int. Conf. Music Inf. Retrieval (ISMIR), 2002, pp.
295–296.
[13] A. Popescul, L. Ungar, D. Pennock, and S. Lawrence, “Probabilistic
models for unified collaborative and content-based recommendation insparse-data environments,” in Proc. 17th Conf. Uncertainty Artif. Intell.
(UAI), 2001, pp. 437–444.

[14] T. Hofmann and J. Puzicha, “Latent class models for collaborative
filtering,” in Proc. Int. Joint Conf. Artif. Intell. (IJCAI), 1999, pp.
688–693.
[15] T. Hofmann, “Probabilistic latent semantic analysis,” in Proc. 15th
Conf. Uncertainty Artif. Intell. (UAI), 1999, pp. 289–296.
[16] A. Dempster, N. Laird, and D. Rubin, “Maximum likelihood from incomplete data via the EM algorithm,” R. Statist. Soc. B, vol. 39, pp.
1–38, 1977.
[17] R. Mooney and L. Roy, “Content-based book recommending using
learning for text categorization,” in Proc. ACM Conf. Digital Libraries,
2000, pp. 195–204.
[18] D. Billsus and M. Pazzani, “User modeling for adaptive news access,”
User Modeling and User-Adapted Interaction, vol. 10, no. 2–3, pp.
147–180, 2000.
[19] T. Tran and R. Cohen, “Hybrid recommender systems for electroniccommerce,” in Proc. Knowledge-Based Electronic Markets, Papers
From the AAAI Workshop, 2000, Tech. Rep. WS-00-04, AAAI Press.
[20] M. Claypool, A. Gokhale, T. Miranda, P. Murnikov, D. Netes, and M.
Sartin, “Combining content-based and collaborative filters in an online
newspaper,” in Proc. ACM SIGIR Workshop Recommender Syst., 1999.
[21] M. Balabanovic and Y. Shoham, “Fab: Content-based, collaborative
recommendation,” Commun. ACM, vol. 40, no. 3, pp. 66–72, 1997.
[22] M. Pazzani, “A framework for collaborative, content-based, and demographic filtering,” Artif. Intell. Rev., vol. 13, no. 5–6, pp. 393–408,
1999.
[23] R. Stenzel and T. Kamps, “Improving content-based similarity measures by training a collaborative model,” in Proc. Int. Conf. Music Inf.
Retrieval (ISMIR), 2005, pp. 264–271.
[24] P. Melville, R. Mooney, and R. Nagarajan, “Content-boosted collaborative filtering for improved recommendations,” in Proc. Nat. Conf.
Artif. Intell. (AAAI), 2002, pp. 187–192.
[25] C. Hayes, “Smart radio: Building community-based Internet music
radio,” Ph.D. dissertation, Trinity College Dublin, Dublin, U.K., 2003.
[26] K. Yu, A. Schwaighofer, V. Tresp, X. Xu, and H.-P. Kriegel, “Probabilistic memory-based collaborative filtering,” IEEE Trans. Knowl.
Data Eng., vol. 16, no. 1, pp. 56–69, Jan. 2004.
[27] L. Zhang, C. Li, Y. Xu, and B. Shi, “An efficient solution to factor
drifting problem in the pLSA model,” in Proc. Int. Conf. Comput. Inf.
Technol. (CIT), 2005, pp. 175–181.
[28] J.-J. Aucouturier, F. Pachet, and M. Sandler, “‘The way it sounds’:
Timbre models for analysis and retrieval of music signals,” IEEE Trans.
Multimedia, vol. 7, no. 6, pp. 1028–1035, Dec. 2005.
[29] J.-J. Aucouturier and F. Pachet, “Musical genre: A survey,” New Music
Res., vol. 32, no. 1, pp. 83–93, 2003.
[30] N. Ueda and R. Nakano, “Deterministic annealing EM algorithm,”
Neural Netw., vol. 11, no. 2, pp. 271–282, 1998.
[31] G. Arfken, “Lagrange multipliers,” in Mathematical Methods for
Physicists, 3rd ed. New York: Academic, 1985, pp. 945–950.
[32] M. Zadel and I. Fujinaga, “Web services for music information retrieval,” in Proc. Int. Conf. Music Inf. Retrieval (ISMIR), 2004, pp.
478–483.
[33] G. Tzanetakis and P. Cook, “MARSYAS: A framework for audio
analysis,” in Organized Sound. Cambridge, U.K.: Cambridge Univ.
Press, 2000.
[34] X. Amatriain, J. Massaguer, D. Garcia, and I. Mosquera, “The CLAM
annotator: A cross-platform audio descriptors editing tool,” in Proc.
Int. Conf. Music Inf. Retrieval (ISMIR), 2005, pp. 426–427.

Kazuyoshi Yoshii (S’05) received the B.E. degree
and the M.S. degree in Informatics from Kyoto University, Kyoto, Japan, in 2003 and 2005, respectively.
He is currently pursuing the Ph.D. degree in the
Department of Intelligence Science and Technology,
Graduate School of Informatics, Kyoto University.
He is supported by the JSPS Research Fellowships
for Young Scientists (DC1).
His research interests include music scene analysis, music recommendation, and human–machineinteraction.
Mr. Yoshii received nine awards including the IPSJ Yamashita SIG ResearchAward, the FIT 2004 Paper Award, the Interaction 2006 Best Presentation
Award, and the Best-in-Class Award of MIREX 2005. He is a member of theInformation Processing Society of Japan (IPSJ) and the Institute of Electronics,
Information, and Communication Engineers (IEICE).

YOSHII et al.: EFFICIENT HYBRID MUSIC RECOMMENDER SYSTEM

Masataka Goto received the Doctor of Engineering
degree from Waseda University, Tokyo, Japan, in
1998.
He then joined the Electrotechnical Laboratory(ETL), which was reorganized as the National Institute of Advanced Industrial Science and Technology(AIST), Tsukuba, Japan, in 2001, where he has beena Senior Research Scientist since 2005. He served
concurrently as a Researcher in Precursory Research
for Embryonic Science and Technology (PRESTO),
Japan Science and Technology Corporation (JST),from 2000 to 2003, and has been an Associate Professor in the Department
of Intelligent Interaction Technologies, Graduate School of Systems andInformation Engineering, University of Tsukuba, since 2005.
Dr. Goto received 21 awards, including the IPSJ Best Paper Award, IPSJ Yamashita SIG Research Awards, and Interaction 2003 Best Paper Award. He is amember of the IPSJ, ASJ, JSMPC, IEICE, and ISCA.

Kazunori Komatani received the B.E. degree, M.S.
degree in Informatics, and the Ph.D. degree, all from
Kyoto University, Kyoto, Japan, in 1998, 2000, and2002, respectively.
He is currently an Assistant Professor in the
Department of Intelligence Science and Technology,
Graduate School of Informatics, Kyoto University.
His research interests center on spoken language
processing, especially on spoken dialogue systems.
Dr. Komatani has received several awards including the 2002 FIT Young Researcher Award and2004 IPSJ Yamashita SIG Research Award from the Information Processing
Society of Japan (IPSJ), and the RSJ/SICE Award for IROS-2006 Best Paper
Nomination Finalist. He is a member of the IPSJ, Institute of Electronics,
Information and Communication Engineers (IEICE), Association for Natural
Language Processing (NLP), Japanese Society for Artificial Intelligence(JSAI), Association for Computational Linguistics (ACL), and International
Speech Communication Association (ISCA).

447

Tetsuya Ogata (M’00) received the B.S., M.S., andD.E. degrees in mechanical engineering from Waseda
University, Tokyo, Japan, in 1993, 1995, and 2000,
respectively.
From 1999 to 2001, he was a Research Associatewith Waseda University. From 2001 to 2003, he wasa Research Scientist in the Brain Science Institute,
RIKEN. Since 2003, he has been a Faculty Memberin the Department of Intelligence Science and Technology, Graduate School of Informatics, Kyoto University, Kyoto, Japan, where he is currently an Associate Professor. Since 2005, he has been a Visiting Associate Professor in the
Humanoid Robotics Institute, Waseda University. His research interests include
human–robot vocal-sound interaction, dynamics of human–robot mutual adaptation, and active sensing with robot systems.
Dr. Ogata received the 2000 JSME Outstanding Paper Medal from the Japan
Society of Mechanical Engineers, the Best Paper Award of IEA/AIE-2005, andthe RSJ/SICE Award for IROS-2006 Best Paper Nomination Finalist. He is amember of the IPSJ, JSAI, RSJ, JSME, HIS, and SICE.

Hiroshi G. Okuno (SM’06) received the B.A. and
Ph.D. degrees from the University of Tokyo, Tokyo,
Japan, in 1972 and 1996, respectively.
He worked for Nippon Telegraph and Telephone,
Kitano Symbiotic Systems Project, and Tokyo University of Science. He is currently a Professor in the
Department of Intelligence Science and Technology,
Graduate School of Informatics, Kyoto University.
He was a Visiting Scholar at Stanford University,
Stanford, CA, and a Visiting Associate Professor at
the University of Tokyo. He has done research inprogramming languages, parallel processing, and reasoning mechanisms in AI,
and he is currently engaged in computational auditory scene analysis, musicscene analysis, and robot audition. He edited with D. Rosenthal Computational
Auditory Scene Analysis (Lawrence Erlbaum Associates, 1998), with T. Yuasa
Advanced Lisp Technology (Taylor and Francis, 2002), and with M. Ali New
Trends in Applied Artifical Intelligence (Springer-Verlag, 2007).
Dr. Okuno has received various awards including the 1990 Best Paper Awardof JSAI, the Best Paper Awards of IEA/AIE-2001 and 2005, IEEE/RSJ Nakamura Award for IROS-2001 Best Paper Nomination Finalist, and the RSJ/SICE
Award for IROS-2006 Best Paper Nomination Finalist. He was also awarded
the 2003 Funai Information Science Achievement Award. He is a member of
the IPSJ, JSAI, JSSST, JSCS, RSJ, ACM, AAAI, ASA, and ISCA.