Recognition and Analysis of Emotion in IndonesianConversational Speech
Nurul Lubis1,2 , Sakriani Sakti1 , Graham Neubig1 , Tomoki Toda1 ,Dessi Lestari2 , Ayu Purwarianti2 , and Satoshi Nakamura1
1Nara Institute of Science and Technology
2Institut Teknologi Bandung

Abstract: The importance of incorporating emotional aspect in human computer interaction
continues to arise. Unfortunately, exploration of the subject in Indonesian is still very lacking.
This paper presents the first study of emotion recognition in Indonesian on conversational speech.
We construct our corpus, IDESC, by making use of television talk show recordings in various topics
of discussion, yielding colorful emotional utterances. Using the corpus, we then build a support
vector machine (SVM) that classifies Indonesian speech in terms of emotion based on its acousticfeatures. We perform feature selection and parameter optimization while building the classifier tooptimize the recognition performance, resulting in absolute 11.9% increase of accuracy. Lastly, weperform analyses on our corpus and evaluation result to gain better insight of emotion occurencein Indonesian speech.

1Introduction

Human computer interaction technologies aim at
the most natural form of interaction possible by matching that of human and human. In that sense, theinteraction should not only focus on completion ofcertain tasks, but also engagement with user on anemotional level. This requires a set of capabilities ina machine; recognizing, interpreting, processing, andsimulating human aﬀects.
To examine diﬀerent sides and angles of this problem a number of emotional challenges have been heldfrom year to year, e.g INTERSPEECH [1] [2] andAVEC [3] [4]. Along with research and studies in thisfield, interaction between human and computer haveadvanced to better facilitate the emotional aspect ofthe interaction.
For Asian languages, there exist a number of studies and findings related to emotion in computing. ForChinese, researchers have studied the eﬀect of switching the stimulus in user, involving aﬀective systems[5]. In Tagalog, an automated narrative storyteller
was constructed with an average precision of 86.75%
in expressing a particular emotion [6]. Unfortunately,in Indonesian, research on emotion recognition is nonexistent—even the resource to conduct studies andresearch on is still very lacking.

This paper presents the first study of emotion recognition in Indonesian. We construct a speech corpus
from television talk show recordings in various topics
of discussion, yielding colorful emotional utterances.
Utilizing the corpus, we then train and evaluate ouremotion classifier. We observe and analyze the training and evaluation process to have better insight ofemotion in Indonesian speech.

2Previous Works

One of the early studies on speech based emotion
recognition is performed on acted utterances in theEnglish language [7]. The study reports a novel approach for classifying speech based on its emotion
content and the promising acoustic features for thetask. More recently, real-time recognition have been
constructed using acted emotion corpus intended forteaching autistic children about simple and complex
emotions [8].
Emotion recognition has been applied in spoken dialogue systems to deliver a more natural experience touser. This includes studies on emotion triggers on human spoken dialogue [9] and generation of emotionallycoloured conversation [10]. In this context, spontaneous, naturalistic, or induced emotional speech datais preferable as they better mimic interaction between
human and computer.
Despite the number of research in emotion recognition, only a few utilize human-human conversationthat really mimics natural interaction—most makeuse of speech data recorded in specific situation, guidelines, or scenario, thus compromising the nature andemotion range of the resulting data. In this study, we
utilize real human conversations to ensure applicability to actual human-computer interaction.

related subjects. The second show is ”Kick Andy”,
with topics in the area of humanity. The third show
is ”Just Alvin”, with focus in celebrities, entertainment, and lifestyle. The diﬀerent topics are expected
to provide more varied emotion content in the collected data.
In total, the three talk show episodes are 2 hours,
25 minutes, and 39 seconds in length. Audio is available at 16 kHz and 16 bits per sample. There are 18speakers in total; 12 male speakers and 6 female.

33.2

Corpus Construction

Construction of our corpus, the Indonesian Emotional Speech Corpus (IDESC), comprises three steps.
The first is data collection, during which contents forthe corpus are gathered. After collection, the content
is then segmented into speech utterances. Each segment is then annotated or labeled based on its emotion content. The overall construction process of thecorpus is demonstrated in Fig. 1. Each of these steps
will be explained in detail in this section.

The collected data is segmented into speech utterances. We segment the speech manually to ensure
quality, as segmentation using an existing automatic
speech recognizer may introduce errors to the result.
During the process, we make sure the emotion content
is consistent for each segment. This is done so thatthe resulting segments are relevant in emotion recognition. However, this doesn’t limit other approach ofsegmentation in the corpus to suit other task in advanced human-computer interaction.
Segmentation is done using speech processing tool
Audacity.1 . In total we obtained 2179 speech segments worth 1 hour, 34 minutes, and 49.7 seconds inlength.

3.3

図 1: The steps of corpus construction

3.1

Data Collection

We collect the speech data in Indonesian from various television talk shows. Television talk shows provide clean speech recordings with distinguishable dialogue turns, resulting in good quality speech data.
The dialogue format gives us natural speech utterances. Furthermore, with diﬀerent guests in eachepisode, speech data from a number of speakers canbe obtained.
We select three episodes from diﬀerent talk shows
to cover broader range of emotions. The first show
is ”Mata Najwa”, with discussions focusing on politic

Segmentation

Labeling

We label the segments manually based on humanrecognition. 5 human labelers are employed, 3 females and 2 males. Before labeling, the labelers arebriefed regarding the corpus and the objective of thetask. We define 5 emotion labels derived from thewidely-known valence-arousal scale: neutral, happiness, anger, sadness, and contentment. These classes
are general, yet it covers all emotions in daily humaninteractions. This set of emotion labels gives a goodfoundation in further development of IDESC, wheremore specific emotion terms can be defined.
A segment is labeled neutral if not enough aﬀect
is detected in the speech. If a speech segment shows
active expression of a positive emotion, it is labeledhappiness; if it’s of negative emotion, it’s labeled sadness. Passive expression of positive emotion yields
the contentment label, and on negative emotion, sadness. This labeling rule is simple and straightforward,
1 http://audacity.sourceforge.net/

as we want to start with a less complicated task andprogress as we develop IDESC. After all segments arelabeled, we obtain the finished emotional corpus inIndonesian.

4

Experiment Set Up

In training our SVM, we employ the widely usedlibrary for support vector machines, libSVM [11] using the Radial Basis Function (RBF) kernel. Fig. 2gives an overview of the emotion recognition model
construction, followed with evaluation. The process
comprises scaling, feature selection, and parameter
optimization to obtain a model that is as good aspossible. These steps are recommended in [12] andhave been applied in similar tasks by [8].

Cepstral features (13)
MFCC 0-12
Spectral features (35)
Mel-Spectrum bins 0–25, zero crossing rate,
25%, 50%, 75%, and 90% spectral roll-oﬀ
points, spectral flux, centroid, relative position of spectral maximum and minimum
Energy features (6)logarithmic energy, energy in bands from
0–250 Hz, 0–650 Hz, 250–650 Hz, 1–4 kHz,
3010–9123 Hz
Voicing-related features(3)F0 (subharmonic summation (SHS) followed
by Viterbi smoothing), F0 envelope, probability of voicing
表 1: Low level descriptors of the feature set

図 2: Experiment and evaluation flow

In the experiment, we exclude the segments with aneutral label to keep our focus in recognizing presentemotion. We part our IDESC into training set andtest set with a 85:15 ratio. We use the training set
to construct our emotion recognizer and the test set
to subsequently evaluate it. We obtain 1155 segmentsin the training set with a distribution as follows: 204
segments labeled as happiness, 467 as anger, 228 assadness, and 459 as contentment.
On the training set, we extract acoustic features as
the basis of emotion recognition. For reproducibility,
we employ an open-source feature extraction toolkit
openSMILE [13]. The employed feature set is the official openSMILE emo large.conf feature set. We
choose a large feature set to extract as much relevant
information as possible from speech to be filtered byfeature selection in the next step.
The feature set includes detailed statistical description of the basic speech features with many spectral

and further descriptors. The low level descriptors(LLD) are listed in Table 1, categorized into cepstral, spectral, energy, and voicing-related features.
For each LLD as well as its delta and accelerationcoeﬃcients, 39 statistical functionals are computed.
These functionals include values such as mean, standard deviation, percentiles and quartiles, linear regression functionals, or local minima/maxima relatedfunctionals.
For each segment, we scale the value of the extracted features to a [-1, +1] range to avoid overpowering of features of bigger value and numerical difficulties in further SVM computation. After scaling
the features, we perform feature selection based on Fscore to eliminate features that are possibly irrelevant
or causing noise in the training data.
First, we calculate the F-score of each extracted feature and sort them in descending order and exclude
features with F-score of 0. By continually dividing thenumber of features by 2, we loop to experiment with
diﬀerent numbers of top scoring features. On eachloop, we obtain the average recognition rate by performing 5-fold cross validation. The subset with thebest validation rate is chosen as the optimal featureset.
During feature selection, we also perform parameter optimization using the grid search algorithm whileevaluating each feature subset. This is done in abrute-force manner by testing pairs of learning parameters cost and gamma (C, γ)|C ∈ {2−5 , 2−2 , ..., 213 ,215 }, γ ∈ {2−15 , 2−13 , ..., 2, 23 } and choose the pairwith the best 5-fold-cross-validation rate. Based on
the experiment, we build our emotion classifier accordingly.

5Experiment Result and Analysis

In this section we lay out detailed analysis on ourcorpus construction and experiment described in previous sections. First, we analyze the emotion content
on IDESC to get better insight of the corpus. Second,
we evaluate our model by performing emotion recognition on our test set and subsequently analyze theresult. Third, we try to further investigate the eﬀectof learning optimization in our experiments.

5.1

図 3: Agreement level between annotators measured
with Fleiss’ Kappaof audience. However, at certain parts, diﬀerent emotions do naturally occur as the result of the topic discussed. This correlation between topic of discussion
and emotion occurrence will be beneficial in further
data collection of certain emotion content.

Data Analysis

We analyze the emotion content of our constructed
speech corpus, IDESC. Firstly, we analyze the agreement level between human annotators using Fleiss’Kappa. These numbers are presented in Fig. 3. Thestatistic shows that the annotators have the highest agreement on labeling segments with anger emotion, with moderate agreement at 0.55, while lowest
on neutral emotion, with slight agreement at 0.28.
Anger ranked first on annotators’ agreement level, followed by happiness, contentment, sadness, and neutral. This tells us that active emotions are more uniformly recognizable than passive ones, or the absenceof emotion. The overall agreement level of the annotators on the entire corpus reaches 0.23, which can beinterpreted as slight agreement.
Secondly, from each talk show, as well as IDESC inoverall, we visualize its distribution of emotion labels.
We detect diﬀerent tendency of emotion occurrencesin each talk show. We then try to draw correlation ofthis tendency to the topic of the discussion. The distribution of the emotion class in IDESC is visualized
in Fig. 4.
In general, neutral and passive-positive emotion seemto be the most common occurrence in the collecteddialogues. This is expected, as talk shows are rather
formal in format and broadcasted to a large amount

図 4: Distribution of emotion labels in the corpus

We analyze the emotion content (aside from neutral) of each talk show and correlate it to its main
topic. ”Mata Najwa”, which provides a discussion
centering in politics, is the talk show with most positivenegative emotion occurrences. On the other hand,the story telling of enriching life experiences in ”KickAndy” gives us the many passive-positive occurrences.
Meanwhile, ”Just Alvin” with its focus in entertainment and celebrities, seem to have balanced occurrences on the four labels of emotion. Overall, we obtain emotion label distribution as shown in the firstbar of Fig. 4. The composition is slightly imbalanced, with happiness and sadness as the least occurring emotions.

5.2

Feature Selection and Parameter
Optimization

We evaluate our learning optimization algorithm
explained in Sec. 4. We build 4 models with different learning optimizations and perform recognitionof our test data on them. The test set consists of 202
speech segments in Indonesian with emotion distribution as follows: 32 segments labeled as happiness, 71
as anger, 33 as sadness, and 66 as contentment.
The first model is build with no learning optimization. On the second model, we apply only feature
selection and on the third, parameter optimization.
The fourth model is built with both feature selectionand parameter optimization. Based on the recognition, we calculate 4 performance measures as comparison of the models: accuracy, precision, recall, andF-score. The unoptimized model serves as the baseline. These numbers are presented in Fig.5.

表 2: Confusion matrix for prediction of test set

表 3: One-against-one and one-against-all emotion
recognition accuracy

mized model with the best scores on almost all performance measures compared to the rest of the models.

5.3

図 5: Comparison of model performance with diﬀerent
learning optimization

Our figure shows that with feature selection only,
we’re able to obtain a model with higher F-score thanunoptimized model, but with lower accuracy. On
the other hand, parameter optimization alone is able
to improve recognition performance on all measurements. This tells us that while selecting the properfeatures is important in building a classifier, to fit themodel to the problem is just as essential if not more.
Combining the two techniques gives us the fully opti-Emotion Recognition

To analyse emotion in Indonesian speech further,
we take a closer look on recognition by our fully optimized model, the proposed model in our study. Confusion matrix of the recognition by the model is presented in Table 2, with correct prediction marked withboldface.
Most incorrect predictions happen when a segment
is or should be predicted as contentment. On theother hand, anger is the best recognized emotion. This
demonstrates prominence of acoustic characteristicsof the emotions; with contentment being the least
prominent and anger being the most. It’s worth noting that both emotions are of diﬀerent valence andarousal range.
Furthermore, Table 3 presents recognition accuracy
for the one-against-one and one-against-all scenarios,with the highest performance boldfaced and the lowest underlined. Classification between happiness andanger emotion yields the best accuracy for one-againstone classification, while sadness and contentment yields
the lowest. On one-against-all, the best accuracy isobtained while classifying anger emotion from the rest,
and the least, contentment.

6Conclusion

We present a study on emotion recognition in Indonesian. We build an emotion recognizer from anemotionally colorful speech corpus in Indonesian, IDESC.
In constructing our emotion recognizer, we attempt toobtain the best resulting model possible by optimizing the learning process with feature selection andparameter optimization. As a result, we achieve multiclass classification accuracy of 68.31% for 4 emotion
classes. We perform analysis on our data as well as
the experiment process and result to get more insightof emotion in Indonesian specch. This information ishighly beneficial for further development in the language.
The overall result of this study is widely open forimprovements. More data in Indonesian can be obtained in answering the scarcity of resource. Furthermore, a more complex emotion structure can be defined to obtain a model capable of recognizing more
specific emotions. We look forward to get meaningful comparison to this work by testing diﬀerent techniques and approaches to our corpus.

[5] Liusheng Wang and Jing Li, “The switching effect involving the aﬀective system in Chinese affective concept processing,” Universal Journal ofPsychology, vol. 2, no. 5, pp. 151–157, 2014.
[6] John Christopher P. Gonzaga, Jemimah A.
Seguerra, Jhonnel A Turingan, Mel Patrick A.
Ulit, and Ria A. Sagum, “Emotional techybasyang: An automated filipino narrative storyteller,” International Journal of Future Computer and Communication, vol. 3, pp. 271–274,
August 2014.
[7] Frank Dellaert, Thomas Polzin, and Alex Waibel,“Recognizing emotion in speech,” in Spoken Language, 1996. ICSLP 96. Proceedings., Fourth International Conference on. IEEE, 1996, vol. 3,pp. 1970–1973.
[8] Tomas Pfister and Peter Robinson, “Real-time
recognition of aﬀective states from nonverbal features of speech and its application for public
speaking skill analysis,” Aﬀective Computing,IEEE Transactions on, vol. 2, no. 2, pp. 66–78,
2011.

[1] Bjoern Schuller, Stefan Steidl, and Anton Batliner, “The INTERSPEECH 2009 emotion challenge.,” in INTERSPEECH. Citeseer, 2009, vol.
2009, pp. 312–315.

[9] Nurul Lubis, Sakriani Sakti, Graham Neubig,
Tomoki Toda, Ayu Purwarianti, and Satoshi
Nakamura, “Emotion and its triggers in humanspoken dialogue: Recognition and analysis,” inProceedings of International Workshop on Spoken Dialogue Systems, 2014, pp. 224–229.

[2] Bjoern Schuller, Stefan Steidl, Anton Batliner,
Felix Burkhardt, Laurence Devillers, Christian A
Mueller, and Shrikanth S Narayanan, “The INTERSPEECH 2010 paralinguistic challenge.,” inINTERSPEECH, 2010, pp. 2794–2797.

[10] Ellen Douglas-Cowie, Roddy Cowie, Cate Cox,
Noam Amier, and DKJ Heylen, “The sensitive artificial listner: an induction technique forgenerating emotionally coloured conversation,”
2008.

[3] Bjoern Schuller, Michel Valstar, Florian Eyben,
Gary McKeown, Roddy Cowie, and Maja Pantic,“Avec 2011–the first international audio/visualemotion challenge,” in Aﬀective Computing andIntelligent Interaction, pp. 415–424. Springer,
2011.

[11] Chih-Chung Chang and Chih-Jen Lin, “LibSVM:a library for support vector machines,” ACM
Transactions on Intelligent Systems and Technology (TIST), vol. 2, no. 3, pp. 27, 2011.

参考文献[4] Bjoern Schuller, Michel Valster, Florian Eyben,
Roddy Cowie, and Maja Pantic, “Avec 2012: thecontinuous audio/visual emotion challenge,” inProceedings of the 14th ACM international conference on Multimodal interaction. ACM, 2012,
pp. 449–456.

[12] Chih-Wei Hsu, Chih-Chung Chang, Chih-Jen
Lin, et al., “A practical guide to support vector classification,” 2003.
[13] Florian Eyben, Martin Woellmer, and Bjoern
Schuller, “Opensmile: the munich versatile andfast open-source audio feature extractor,” in Proceedings of the international conference on Multimedia. ACM, 2010, pp. 1459–1462.