MUSIC EMOTION RECOGNITION WITH ADAPTIVE AGGREGATIONOF GAUSSIAN PROCESS REGRESSORS
Satoru Fukayama and Masataka Goto
National Institute of Advanced Industrial Science and Technology (AIST), Japan
ABSTRACTThis paper describes a novel method for estimating the emotions elicited by a piece of music from its acoustic signals.
Previous research in this field has centered on finding effective acoustic features and regression methods to relate features to emotions. The state-of-the-art method is based on amulti-stage regression, which aggregates the results from different regressors trained with training data. However, after
training, the aggregation happens in a fixed way and cannot
be adapted to acoustic signals with different musical properties. We propose a method that adapts the aggregation bytaking into account new acoustic signal inputs. Since we cannot know the emotions elicited by new inputs beforehand, weneed a way of adapting the aggregation weights. We do so
by exploiting the deviation observed in the training data using Gaussian process regressions. We confirmed with an experiment comparing different aggregation approaches that ouradaptive aggregation is effective in improving recognition accuracy.
Index Terms— Music emotion recognition, Gaussian
process regression, Adaptive aggregation, Multi-level regression
1. INTRODUCTION

feature extraction

x1 x2 x3
proposed method
existing method
x1 x2 x3 x1 x2 x3
regressors

Gaussian process
regressors

estimation of points

estimation of distribution

aggregation

adaptive aggregation

valencegroundtruth

ACCURACY valence
IMPROVED
estimation

arousalMusic emotion recognition is a task to estimate how a pieceof music affects the emotions of a listener. It works by extracting features from music audio and applying a regression
model or classification model to map those features into an
emotion representation. Psychological studies have proposed
two-dimensional values in the Arousal-Valence (AV) plane
for representing emotion [1]. The task we focus on in this research is to estimate the AV value from the musical audio input. Specifically, the AV values are estimated for segments ofmusic lasting 30 seconds. This problem setting is the same asthat of the Emotion in Music Task at MediaEval Workshops,
which is the leading evaluation campaign on music emotion
recognition [2, 3].
Previous methods have used carefully selected features toestimate emotions. Linear regression methods such as multivariate regression analysis have been used to map those features to emotions [4, 5]. Feature selection algorithms are

978-1-4799-9988-0/16/$31.00 ©2016 IEEE

audio

71

ground
un
nd
truth
utth

estimation

arousalFig. 1. Fixed aggregation vs. proposed adaptive aggregationof Gaussian process regressors for music emotion recognition.
also applied to find important features to improve the estimation [6]. Recently, emotion recognition has been improvedby automatically tuning the importance of each feature instead of carefully choosing useful ones. One approach ismulti-level regression which creates individual regressors using each feature and then aggregates the results from those regressors with another regressor [7, 8]. Another approach is to
apply non-linear regression methods and make use of the nonlinear dimensionality reduction. Powerful regressors such asneural networks [9, 10, 11], support vector regressions [12],and Gaussian process regressions [13] have been used.
Although state-of-the-art methods have significantly im-

ICASSP 2016

proved performance, they cannot change the importance ofeach feature once it has been trained using training data. It
is not clear whether all kinds of emotions on the AV plane
can be estimated from a fixed importance for each feature. As
we show in our research, the importance of features differs
when estimating low-arousal and high-arousal areas. However, there is as yet no method that can adaptively change theimportance of each feature depending on what emotion wewant to estimate.
We propose a novel method of emotion estimation that
can adjust the importance of each feature depending on theaudio input. The technical difficulty here is the estimationof the importance before knowing the emotion of the audio
input. If we know the emotion of the audio beforehand, one
strategy is to adjust the importance by optimizing it to matchthe result to that emotion. However, in our case, the emotion
can be known only once the estimation has been performed;
hence, this strategy cannot be taken.
We overcome this difficulty by using Gaussian process(GP) regression. The overview and novelty of our method areshown in Figure 1. GP regression can predict the mean andvariance of an estimated AV value from a new audio input.
By preparing multiple GP regressors wherein each is trainedwith a different feature, we obtain the means and variances ofthe estimated AV values considering each feature. The feature is not important when the feature values in the training
data take various values for similar AV values, and this leads
to a large variance when an AV value is estimated with GP regression from a new piece of audio. Thus, we can obtain theimportance of each feature by calculating the variance of AV
values with individual GP regressors. Since the importanceof each feature varies depending on the audio input, we aggregate the results from the individual regressors adaptively
to estimate the emotion. The mathematical formulation forthis procedure is described in Section 3, and in Section 4 weshow that it can improve the performance of music emotion
recognition.
2. RELATED WORK
Our research can be related to previous music emotion recognition methods from two viewpoints: selecting features andregression methods. In this section, we summarize some important insights from the review papers [14, 15, 16].
2.1. Features for emotion recognition
It is usually argued that AV values are related to the musical content, such as tempo, pitch, loudness, timbre forarousal, and mode and harmony for valence [17]. In particular, audio features, such as mel-frequency cepstral coefficients (MFCCs), chroma (i.e., pitch class profile), spectral
descriptors (e.g., spectral contrast, centroid, flux, rolloff, andflatness) are used [6]. Text information is often combined72with audio features [14]. For example, lyric and metadata
such as genre tags and social media tags can be used [16, 18].
2.2. Regression methods
Emotion recognition is usually seen as a problem of regression between features and AV values. The regression canbe linear or non-linear. Multivariate regression is a powerful linear method used in emotion recognition [4]. Non-linear
regressors include support vector regression [12], deep recurrent neural networks [19, 10], continuous conditional random
fields [11], conditional random field [20], k-nearest neighbors [21] and Gaussian process [13].
Multi-level regression is used to gather information fromindividual regressors which are trained with individual feature. Schmidt et al. proposed fusing results from individual
regressors to improve performance [7]. They attempted two
different fusion topologies: in one case, the secondary regressors receive only AV estimates respectively; in the other, thesecondary regressors receive both arousal and valence values
from the first-level regressors.
3. ADAPTIVE AGGREGATION OF
GAUSSIAN PROCESS REGRESSORS
Given an excerpt of music audio with a fixed length, we wantto estimate the AV values by aggregating the predictions of
N different features x1 , . . . , xN . Since the same discussion
applies to estimating the arousal and the valence, we will only
discuss the case of estimating one of them, and will represent
its value as y. The prediction from the nth feature is yn , andour goal is to find an appropriate weight wn for this feature.
3.1. Quantifying the importance of features
Let us think of a transform which maps the nth feature xn
into an emotion yn . If we know that the ground truth of emotion for the given audio is y ∗ , the estimation error n for thisfeature can be defined as n = y ∗ − yn .
Suppose we have two different estimates for emotion, yn
and ym , from the nth and mth features, respectively. If n ismore likely to take a value closer to 0 compared with m , wecan say the nth feature is more important than the mth featurein estimating the true value y ∗ . To quantify the importance offeatures, we assume
 every n follows a Gaussian distri thatbution: n ∼ N 0, σn2 . The nth feature is more important2
in estimation than the mth feature when σn2 < σm.
3.2. Adaptive aggregation
Now we will show how to aggregate the estimation results
obtained from N different features. From the assumption thateach estimation error n , (n = 1, . . . , N ) follows a Gaussiandistribution, we can derive N probability distributions:
Pn (y) = N yn , σn2 , n = 1, . . . , N(1)

The maximum likelihood estimation of y can be understood
to be a value that maximizes the joint probability PJ , whichis calculated by multiplying Pn (y) for all n = 1, . . . , N [22].
We getN11
PJ (y) =(2)
Pn (y) ∝exp − ξ 2
σ1 · · · σ N
2
n=1
where
ξ2 =

N
2

(y − yn )
n=1

σn2

.

(3)

To maximize PJ (y) with respect to y, we need to mini2
mize ξ 2 . By solving dξdy = 0, we obtain the maximum likelihood estimation of y:N1 
wn yn ,
W n=1

1
,
σn2

4. EXPERIMENT ON MUSIC EMOTIONRECOGNITION
We conducted an experiment to verify the effectiveness of our
method. We estimated the AV values from fixed lengths (30seconds) of music audio. We compared the emotion recognition accuracies obtained with different methods.
4.1. Compared methods
There were two points of comparison. First, we compared theperformance of our proposed adaptive aggregation approachto that of an existing multi-level regression approach [7, 8].
Second, we compared both of these to a baseline which does
not use aggregation. All three methods tested were based on
GP regression. They are:
(1) Adaptive aggregation of results from GP regressors (theproposed method)(4)(2) Fixed aggregation of results from GP regressors using
multivariate regression, andNwhere W = n=1 wn . We have thus estimated the emotion y
as a weighted average wherein the weights are the normalized
inversed squares of the variances.

(3) GP regression using all features as a single feature (noaggregation).

y=

wn =

4.2. Experiment procedure
3.3. Gaussian process regressors
The discussion so far indicates that if we can get Gaussian
distributions Pn (y) , n = 1, . . . , N as equation (1), we cancalculate the optimal estimation of y. This final subsection
describes how we can obtain those distributions from training
data by using Gaussian process regression.
(1)(K) (1)(K)
Let {xn . . . xn , yn . . . yn } be the training data having K data points for the nth feature. Moreover, let yn be a
(1)

(K)

T

vector containing the training data, yn = yn . . . yn,
where T denotes the transpose of a vector. According to thederivation of GP regression
 [13], we get a Gaussian distribution Pn (y) = N yn , σn2 where the mean yn and the variance σn2 are
−1

2
yn ,
(5)
yn = kT∗ K + σobs I
−1

2
2
T
2
σn = k (xn , xn ) + σobs − k∗ K + σobs I
k∗ . (6)Here, k (·, ·) is a squared-exponential covariance function:


2
1
(i)
(j)
2
(i)
(j)
,
(7)
k xn , xn = σf exp − 2 xn − xn2lwhere I denotes the identity matrix,(1)(K)
and k∗ = k xn , xn . . . k xn , xn

, K = Kij =(i)(j)
xn , x nk. The parameters σobs , σf and l are optimized
through training to maximize the log-likelihood of the training data.

73

We followed three steps in our experiment. First, low-level
audio descriptors related to emotion were extracted from theaudio. Specifically, we used a set of descriptors from thestate-of-the-art method that came in first place at the MediaEval Emotion in Music task in 2014 [10]. This is the official
set of descriptors used in the 2013 INTERSPEECH Computational Paralinguistics Challenge (ComPareE) [23]. It contains
65 types of descriptors, their first-order derivatives, and statistics (mean, standard deviation, skewness, max., min.) calculated from an audio clip. As a result, we obtained a 6373dimensional vector describing the content of the audio.
Second, we generated features and constructed GP regressors. We created 3 different feature sets by dividing the original 6373-dimensional vector into three, collecting those descriptors related to the spectral descriptors, MFCCs, and others, respectively. To construct the GP regressors, we used theEmotion in Music Database as training data, which consists of
744 audio clips and annotations of AV values on a scale from
1 to 9 [24], and we normalized them to a scale from −1.0 to
1.0. We used 619 randomly chosen clips as training data andthe rest (125 clips) as evaluation data. The Constrained Optimization BY Linear Approximation (COBYLA) method [25]was used to optimize the GP parameters.
Finally, the results from the regressors were aggregated toobtain the AV value estimate. We calculated the AV value as aweighted average of the means of the estimated Gaussian distributions from the GP regressors, wherein the weights were
set to the normalized inversed square of the variances. Whenmethod(1) adaptive aggregation of GP regressors(2) fixed aggregation of GP regressors(3) GP regression

arousalR2
RMSE
0.636 ± 0.040 0.206 ± 0.014
0.619 ± 0.042 0.211 ± 0.014
0.488 ± 0.027 0.238 ± 0.053

valence
R2
RMSE
0.413 ± 0.043 0.230 ± 0.011
0.397 ± 0.064 0.234 ± 0.013
0.399 ± 0.073 0.233 ± 0.012

Table 1. Comparison of methods of estimating emotion from audio in the MediaEval 2013 Emotion in Music development set
and 10-fold cross-validation. The proposed adaptive method, evaluated with the R-squared value (official score in MediaEvaltask), outperformed the fixed aggregation method by 2.7% for estimating arousal and 4.0% for valence.
conducting the fixed aggregation, we split the training data
(619 clips) into 309 clips and 310 clips and used the former
clips to construct the GP regressors and the latter ones forthe multivariate regression between the results from the GP
regressors and ground-truth.
4.3. Evaluation metrics
We calculated the R-squared value (R2 ) and the root meansquared error (RMSE) by following the official evaluation
scheme used in the MediaEval Emotion in Music task. Since
R2 has several definitions, we chose the one which was usedin the previous music emotion recognition research [13]:2(y∗(k) −y(k) )R2 = 1 − k ∗(k) ¯∗ 2 , where y ∗(k) is the ground truth
−y )
k (y
for the k th audio clip, and y¯∗ is the mean of the ground truth
values. The estimated values are close to the ground truth
data when R2 is close to 1.0, and RMSE is close to 0.0. Boththese measures were calculated with 10 different combinations of training data and evaluation data, and the means andstandard deviations of the values were used to measure theperformance (10-fold cross-validation).
4.4. Results
The results shown in Table 1 indicate that adaptive aggregation of regressors improved accuracy. For both arousal andvalence estimations, adaptive aggregation of GP regressors
performed the best out of the three methods. Comparing
adaptive aggregation with fixed aggregation, we can see theR-squared value for adaptive aggregation had a 2.7% improvement for arousal and a 4.0% improvement for valence.
For arousal, both aggregation approaches performed better
than the GP regression using all features as a single feature, but for valence, only the proposed adaptive aggregation
method gave an improvement.
5. DISCUSSION
We confirmed that adaptive aggregation of regressors can improve the performance of music emotion recognition. GP regression was suitable for our aggregation methodology since
it can predict not only the estimation values but also the variances.

74

The results imply that the importance of each feature set
can differ depending on the emotion that we are estimating.
We actually found that the feature set most important for
estimating arousal value around 0.425 was the MFCC-related
feature set (weight 0.52), whereas for estimating arousal
value around −0.5, the spectral-related feature set had thebiggest importance (weight 0.60). Although we could not
find a simple explanation for which is the most importantfeature set in which part of the AV plane, we confirmedthat using fixed weights for aggregating the regressors is toocoarse an approximation to reflect the relative importance ofeach feature set at all points in the plane.
Our method matched the performance of one state-of-theart approach, but did not outperform it. The best performanceusing the same dataset reported so far is R2 = 0.704 for estimating arousal by using deep recurrent neural networks [19].
However, considering that the highest performance using asingle GP regressor nearly matches that one (R2 = 0.695 ±
0.046) with a novel selection of features and kernel functions [13], trying various features and kernel functions may
eventually yield even higher estimation performance.
The way of determining the optimal number of featuresets for aggregation remains to be investigated. Althoughadding new feature sets can increase the adaptiveness of our
method, adding too many feature sets by dividing a feature set
into too many sets can decrease the benefit of the non-linearity
of GP regression.
6. CONCLUSION
We proposed a music emotion recognition method based ona novel aggregation of Gaussian process regressors. The
method exploits the variance of the training data and adapts
the weights for aggregating the results depending on the input
audio. Experimental results showed that our method could
actually improve the estimation performance. Our future
work includes investigating the effect of varying the numberof feature sets.
Acknowledgements
The authors thank Jordan B. L. Smith for helping to polishthis paper. This work was supported in part by OngaCREST,JST.

7. REFERENCES[1] J. A. Russel, “A circumplex model of affect,” Journalof Personal Social Psychology, vol. 39, pp. 1161–1178,
June 1980.
[2] M. Soleymani, M. N. Caro, E. M. Schmidt, and Y.-H.
Yang, “The MediaEval 2013 Brave new Task: Emotionin Music,” in Proceedings of MediaEval 2013 Workshop, 2013.
[3] A. Aljanaki, Y.-H. Yang, and M. Soleymani, “Emotionin Music task at MediaEval 2014,” in Proceedings ofMediaEval 2014 Workshop, 2014.
[4] T. Eerola, O. Lartillot, and P. Toiviainen, “Prediction ofmultidimensional emotional ratings in music from audio
using multivariate regression models,” in Proceedings of
ISMIR 2009, 2009, pp. 621–626.
[5] A. Aljanaki, F. Wiering, and R. C. Veltkamp,“MIRUtecht participation in MediaEval 2013: emotion
in music task,” in Proceedings of MediaEval 2013 Workshop, 2013.
[6] E. M. Schmidt, D. Turnbull, and Y. E. Kim, “Feature
selection for content-based, time-varying musical emotion regression,” in Proceedings of MIR 2010, 2010, pp.
267–273.
[7] E. M. Schmidt, F. Eyben, and Y. E. Kim, “Prediction oftime-varying musical mood distributions from audio,” inProceedings of ISMIR 2010, 2010, pp. 465–470.
[8] Y. Fan and M. Xu, “MediaEval 2014: THU-HCSIL approach to emotion in music task using multi-level regression,” in Proceedings of MediaEval 2014 Workshop,
2014.
[9] F. Weninger, F. Eyben, and B. Schuller, “The TUM
approach to the MediaEval music emotion task using
generic affective audio features,” in Proceedings of MediaEval 2013 Workshop, 2013.
[10] E. Coutinho, F. Weninger, B. Schuller, and K. R.
Scherer, “The Munich LSTM-RNN approach to the MediaEval 2014 ´’Emotion in Music´’ task,” in Proceedingsof MediaEval 2014 Workshop, 2014.
[11] V. Imbrasaite and P. Robinson, “Music emotion tracking with continuous conditional neural fields and relative representation,” in Proceedings of MediaEval 2014
Workshop, 2014.
[12] B. Han, S. Rho, R. B. Dannenberg, and E. Hwang,“SMERS: music emotion recognition using support vector regression,” in Proceedings of ISMIR 2009, 2009, pp.
651–656.

75[13] K. Markov and T. Matsui, “Music genre and emotion
recognition using Gaussian process,” IEEE Access, vol.
2, pp. 688–697, June 2014.
[14] Y. E. Kim, E. M Schmidt, R. Migneco, B. G. Morton,
P. Richardson, J. Scott, J. A. Speck, and D. Turnbull,“Music emotion recognition: a state of the art review,”
in Proceedings of ISMIR 2010, 2010, pp. 255–266.
[15] Y.-H. Yang and H. H. Chen, “Machine recognition ofmusic emotion: a review,” ACM Transaction on intelligent systems and technology, vol. 3, pp. 40:1–40:30,
2012.
[16] M. Barthet, G. Fazekas, and M. Sandler, “Multidisciplinary perspectives on music emotion recognition: implications for content and context-based models,” inProceedings of CMMR 2012 (Lecture Notes in Computer Science 7900), 2012, pp. 228–252.
[17] A. Gabrielsson, “The influence of musical structure on
emotional expression,” Music and emotion: Theory andresearch, pp. 223–248, 2001.
[18] C. Laurier, J. Grivolla, and P. Herrera, “Multimodal music mood classification using audio and lyrics,” in Proceedings of ICMLA 2008, 2008, pp. 688–693.
[19] F. Weninger, F. Eyben, and B. Schuller, “On-line
continuous-time music mood regression with deep recurrent neural networks,” in Proceedings of ICASSP
2014. IEEE, 2014, pp. 5449–5453.
[20] W. Yang, K. Cai, B. Wu, Y. Wang, X. Chen, D. Yang,and A. Horner, “Beatsens’ solution for MediaEval 2014
emotion in music task,” in Proceedings of MediaEval
2014 Workshop, 2014.
[21] A. Wieczorkowska, “Towards extracting emotions frommusic,” in Proceedings of the International Workshop
on Intelligent Media Technology for Communicative Intelligence, 2004, pp. 228–238.
[22] J. Hartung, G. Knapp, and B. K. Sinha, “Statisticalmeta-analysis with applications,” Wiley Series in Probability and Statistics, 2008.
[23] F. Weninger, F. Eyben, B. W. Schuller, M. Mortillaro,and K. R. Scherer, “On the acoustics of emotion in audio: what speech, music, and sound have in common,”Frontiers in Psychology, vol. 4, pp. 1–12, May 2013.
[24] M. Soleymani, M. N. Caro, and E. M. Schmidt, “1000songs for emotional analysis of music,” in Proceedingsof CroudMM 2013. ACM, 2013.
[25] A. R. Conn, K. Scheinberg, and P. L. Toint, “On the convergence of derivative-free methods for unconstrained
optimization,” Approximation theory and optimization,
pp. 83–108, 1997.