IEICE TRANS. INF. & SYST., VOL.E101–D, NO.8 AUGUST 2018

2092

PAPER

Construction of Spontaneous Emotion Corpus from Indonesian TV
Talk Shows and Its Application on Multimodal Emotion
Recognition
Nurul LUBIS†a) , Dessi LESTARI†† , Nonmembers, Sakriani SAKTI†,††† , Member,
Ayu PURWARIANTI†† , Nonmember, and Satoshi NAKAMURA†,††† , Member

SUMMARY As interaction between human and computer continues todevelop to the most natural form possible, it becomes increasingly urgent
to incorporate emotion in the equation. This paper describes a step toward extending the research on emotion recognition to Indonesian. The
field continues to develop, yet exploration of the subject in Indonesian isstill lacking. In particular, this paper highlights two contributions: (1) the
construction of the first emotional audio-visual database in Indonesian, and(2) the first multimodal emotion recognizer in Indonesian, built from theaforementioned corpus. In constructing the corpus, we aim at natural emotions that are corresponding to real life occurrences. However, the collection of emotional corpora is notably labor intensive and expensive. To
diminish the cost, we collect the emotional data from television programs
recordings, eliminating the need of an elaborate recording set up and experienced participants. In particular, we choose television talk shows due
to its natural conversational content, yielding spontaneous emotion occurrences. To cover a broad range of emotions, we collected three episodes in
diﬀerent genres: politics, humanity, and entertainment. In this paper, we
report points of analysis of the data and annotations. The acquisition of
the emotion corpus serves as a foundation in further research on emotion.
Subsequently, in the experiment, we employ the support vector machine(SVM) algorithm to model the emotions in the collected data. We perform
multimodal emotion recognition utilizing the predictions of three modalities: acoustic, semantic, and visual. When compared to the unimodal result,in the multimodal feature combination, we attain identical accuracy for the
arousal at 92.6%, and a significant improvement for the valence classification task at 93.8%. We hope to continue this work and move towards afiner-grain, more precise quantification of emotion.
key words: Indonesian, audio-visual corpus, multimodal emotion recognition, spoken language processing

1.

Introduction

Emotion is an aspect yet to be fully replicated that is able
to provide a richer and more natural human-computer interaction. Particularly in recent decades, the field has witnessed substantial advancements of studies and research on
emotion. Further, researchers and scholars are putting theeﬀort into real life applications through the construction
Manuscript received November 6, 2017.
Manuscript revised March 26, 2018.
Manuscript publicized May 10, 2018.
†
The authors are with the Augmented Human Communication
Lab, Nara Institute of Science and Technology, Ikoma-shi, 630–
0192 Japan.
††
The authors are with the School of Informatics and ElectricalEngineering, Institut Teknologi Bandung, Indonesia.
†††
The authors are with the RIKEN, Center for Advanced Intelligence Project AIP, Ikoma-shi, 630–0192 Japan.
a) E-mail: nurul.lubis.na4@is.naist.jp
DOI: 10.1587/transinf.2017EDP7362

of complex and emotionally advanced system, in particular spoken dialogue systems, e.g. Sensitive Artificial Listener [1], personable in-car assistant [2], and an embodied
conversational companion [3].
This increasing interest in the topic is partly owing to
the challenges held globally. In 2009 and 2010, INTERSPEECH spotlighted research on emotion in their annual
challenges [4], [5]. With stricter focus on emotion recognition, the Audio Visual Emotion Challenge (AVEC) is held
annually to address issues and unify solutions in the field. It
is held for the first time in 2011, aiming at multimedia processing and machine learning methods for automatic emotion analysis [6]. The challenge developed from discreteemotional states into real time dynamic values in 2012 [7],up until the inclusion of physiological signals in 2015 and2016 [8], [9].
However, the result of these advancements, challenges,
and research on emotion are often language dependent. Unfortunately, adaptation of these works into a new language
is not straightforward; a study of English-German multilingual emotion recognition using interactive voice response(IVR) recordings argues that for diﬀerent languages, different sets of features are needed to obtain optimal emotion recognition performance [10]. Even though eﬀorts oncross-lingual recognition is underway, uniformly high performance across the languages is still diﬃcult to obtain [11].
This means, language specific research is necessary for application in a certain language.
In Asian languages, findings in aﬀective computing continue to emerge. In Chinese, speech based emotion recognition for three emotion classes has been researched [12]. In Tagalog, an automated narrative storyteller
was constructed with avarage precision of 86.75% in expressing a particular emotion [13]. On the other hand, in Indonesian, even though spoken language technology starts toactively develop since the past few years, research on emotion recognition is still non-existent. Currently, there exist
a number of studies on automatic speech recognition [14],machine translation [15], and speech synthesis [16]. Unfortunately, for emotion recognition, even the resource to conduct studies and research on is still very lacking.
This is the reason we initiate the research on emotion
recognition in Indonesian. In this eﬀort, emotional corpora
act as the starting point. As most approaches are supervised,c 2018 The Institute of Electronics, Information and Communication Engineers
Copyright ⃝LUBIS et al.: CONSTRUCTION OF SPONTANEOUS EMOTION CORPUS FROM INDONESIAN TV TALK SHOWS AND ITS APPLICATION

2093

the necessity of training data for experiments and modelingis non-negotiable. Moreover, the nature of the data will determine the quality of the system. This means, it is important to collect natural data that has as small a gap as possible
with real life emotion occurrences.
However, the collection of such emotional data for research purposes is a sensitive matter, often raising moral
and ethical issues [17]. Spontaneous emotion occurrence islikely to be kept private, as it is inherently a personal humanexperience. On the other hand fabrication of emotion is nontrivial and it is extremely diﬃcult to match real life emotion.
To tackle this, we construct the corpus in Indonesian fromTV programs, particularly talk shows, containing real conversations and spontaneous emotions. The format of TV
talk shows represents a typical social conversation between
a small group of people, where various social topics thatelicit story-telling, opinions, and emotions are discussed.
In the experiment, we utilize the collected data and employ the support vector machine (SVM) algorithm to model
the emotions contained within. We perform multimodal
emotion recognition utilizing the predictions of three modalities: acoustic, semantic, and visual. In this paper, we report
the result of our corpus construction and emotion recognition experiment, followed with detailed analyses.
2.

Related Work

Emotions in Indonesian language have been previously
studied from the point of view of psychology [18], [19]. In
aﬀective computing, a number of works in Indonesian have
not yet emerged until recently. One of the latest works on
Indonesian Twitter data proposed an emotion lexicon argued to boost sentiment analysis performance [20]. Similarly, [21] attempted to recognize emotion from Indonesian
tweets using linguistic, semantic, and orthographic features.
On the other hand, [22] proposed a gamelan (Indonesian traditional instrumental orchestra) music emotion recognition
for a robot puppeteer, able to distinguish delightful song,fearful song, and noise.
However, to our knowledge, the existing works in Indonesian are limited to unimodal recognition on textual andmusical data. As studies on emotion recognition ultimately
aims at enhancing the quality of human-computer interaction (HCI), recognition on fundamental communication
channels such as acoustic and visual is essential to accommodate natural spoken language interaction between a user
and a system [23], [24].
Toward this goal, we extend the works in Indonesian
for spoken language by utlizing multimodal features. In thiswork, we first construct an Indonesian multimodal corpus
suitable for real life application. We adapt the approachin [25] and utilize television program recordings, to construct the first audio-visual emotional corpus in Indonesian.
We subsequently conduct experiments with the constructed
corpus using SVM by utilizing three diﬀerent facets of information, potentially containing emotional clues: acoustic,
semantic, and visual. The combination of these modalities

Fig. 1

Emotion dimensions and common terms.

is hoped to provide a fuller observation of emotion occurrences.
3. Emotion Definition
Defining and structuring emotion is essential in observing
and analyzing its occurrence. In this work, we define theemotion scope based on the circumplex model of aﬀect [26].
This model highlights the emotion felt as opposed to underlying process of it, and is able to represent both primary andsecondary emotion. An advantage of this model is that it
is intuitive and also easily adaptable and extendable to either discrete or dimensional emotion definitions. The longestablished dimension are core to many works in aﬀective
computing and potentially provides useful information evenat an early stage of research. Furthermore, this allows usefulcomparison to a wide range of related works.
Two dimensions of emotion are defined: valence andarousal. Valence measures positivity or negativity of emotion; e.g. joy is indicated by positive valence while fear isnegative. On the other hand, arousal measures the activityof emotion; e.g. depression is low in arousal (passive), whilerage is high (active). Figure 1 illustrates the valence-arousal
dimension in respect to a number of common emotion terms.
4. Construction of Indonesian Spontaneous Emotional
Audio-Visual Corpus
Although there has been an increase of interest in constructing corpora containing social interactions [27], [28], there is
still a lack of spontaneous and emotionally rich corpora especially in low-resource languages. To bridge this gap, weconstruct a corpus of spontaneous social-aﬀective interaction in the wild. We utilize various television talk shows
containing natural conversations and real emotion occurrences. The steps of corpus construction are visualized inFig. 2, and each will be described in detail in this section.
4.1

Spontaneous Emotional Data Collection

Many of emotion-related works rely on acted emotional
data. While this provides rich emotional content, the portrayed emotions diﬀer considerably from real occurrences in

IEICE TRANS. INF. & SYST., VOL.E101–D, NO.8 AUGUST 2018

2094

Fig. 3Fig. 2The steps of corpus construction

everyday life, causing a potential mismatch when the technology is applied to real human-computer interaction. In
this paper, we utilize emotion occurrences that is closer tothe natural ones by utilizing television talk show recordings.
We collect the spontaneous emotional data in Indonesian from various television programs, in particular talkshows. This approach is previously utilized on the “Vera am
Mittag” corpus [25]. Interactions in talk show setting well
represent typical social conversation, where a small numberof speakers are involved and various emotion-inducing topics are discussed. Furthermore, television talk shows provide clean recordings with distinguishable dialogue turns,
resulting in good quality conversational data. As interactions are not scripted, the format gives us natural utterances with spontaneous emotion occurrences. Television
talk shows also provide speaker variance. With diﬀerent
guests in each episode, we are able to gather data from avariety of speakers.
We select three episodes from diﬀerent kinds of talkshows to cover a broader range of emotion. The selected
talk shows are well-known in the country with high viewership, containing discussions on engaging and interesting
topics that trigger various emotions from the speakers. The
first show is “Mata Najwa,” with discussion focusing on
politic related subjects. The second show is “Kick Andy,”
with topics in the area of humanity. The third show is “Just
Alvin,” with a lighter focus in celebrities, their career andlifestyle. The diﬀerent topics are expected to provide morevaried emotional content in the collected data.
The three talk show episodes are 2 hours, 25 minutes,
and 39 seconds in length. There are 18 speakers in total; 12
male and 6 female. Even though the speech-speaker ratio
is not uniform due to the diﬀerent speaker roles in the talkshow, this nonetheless oﬀers speaker variance in the corpus.
4.2 Annotation
4.2.1

Labels

We perform the annotation on the sentence level in terms ofemotional states. We define a sentence as a continuous utterance of a speaker until either one of the following conditions
is met: 1) a full sentence is produced, or 2) it is proceeded by
a diﬀerent speaker. Segmentation with these criteria resultsin 847 sentences, or speech segments, in the corpus.
As elaborated in Sect. 3, in this paper we follow thecircumplex model of aﬀect as the computational model of
emotion [26]. We define two emotion dimensions as the descriptor of felt emotion; valence and arousal. Valence mea-

Overview of annotation proceduresures the positivity or negativity of emotion while arousal
measures the activity. Following this model, the emotion annotation of the corpus consists of the level of valence (val)and (aro). The value of each dimension can be as low as −3
and as high as 3 with a discrete step of 1. This provides a
granularity that balances between details of information andcognition load for the annotators.
4.2.2

Procedure

In annotating the corpus, we bear in mind that language andculture aﬀect how emotion is perceived and expressed in aninteraction. We carefully select 3 human annotators. Everyannotator is required to be (1) a native Indonesian speaker,
and (2) knowledgeable of the Indonesian culture of communication. With these requirements, we try to ensure that the
annotators can observe emotion dynamics of the interaction
to the furthest extent in addition to recognizing the emotion
appropriately. To ensure consistency, we have each annotator annotate the full corpus.
Figure 3 gives an overview of the annotation procedure.
Before annotating the corpus, the annotators are briefed andgiven a document of guidelines to get a clearer picture of
the task and its goal. The document provides theoreticalbackground of emotion in discourse as well as a number ofexamples.
After the annotators are briefed, firstly, we ask them to
do preliminary annotation by working on a small subset ofthe corpus. This step is done to let them get familiar with thetask. Furthermore, with the preliminary result, we are able
to confirm whether the annotators have fully understood theguidelines, and verify the quality and consistency of theirannotations.
We manually screen the preliminary annotation resultand give feedback to the annotators accordingly. They are
asked to revise inconsistencies with the guidelines if there
are any. This process is repeated until the quality of thepreliminary annotation is suﬃcient. Once their results areverified, the annotators are authorized to work on the rest of
the corpus. We perform the same screen-and-revise process
on the full corpus annotation to achieve a tenable result.
5. Corpus Analysis
We inspect properties of the corpus to gain better insightof the data contained within. We look over the conversational aspect of the collected data through the distributionof segments length in the corpus. Furthermore, we examine
the quality of emotion annotations by looking at the interannotator agreement.

LUBIS et al.: CONSTRUCTION OF SPONTANEOUS EMOTION CORPUS FROM INDONESIAN TV TALK SHOWS AND ITS APPLICATION

2095

Fig. 4

Number of segments in respect to the duration

5.1 Length of Segments
We plot the distribution of number of segments according totheir duration on Fig. 4. In the figure, the y-axis shows thenumber of segments with respect to the x-axis, which shows
the duration. The segments have high variance of duration
with a peak at around 1 second. The average duration ofsegments is 7.57 seconds.
5.2 Annotators Agreement
To analyze the consistency of annotation, we investigate theannotators agreement for both emotion dimension annotations. We calculate mean Pearson’s correlation coeﬃcientsr of the three annotators for each emotion dimension. Pearson’s r measures the strength and direction of linear relationship between two variables. An absolute value of r between
0.0 and 0.3 is interpreted as weak correlation, and greaterthan 0.3 up to 0.5 as moderate correlation. For val and aro,
we achieve moderate correlations of 0.33 and 0.37, respectively. This shows similar levels of agreement between the
annotators in perceiving both valence and arousal, suggesting uniform capabilities in perceiving both dimension in thesentence segments.
5.3 Emotion Label Distribution
We average the annotation of the 3 annotators to obtain the
average valence and arousal of a segment. The majority of
the collected segments lies in the [0, 1] (43.80% for valence
and 46.22% for arousal) and [1, 2] (31.28% for valence and45.33% for arousal) value ranges. There is little coverageof the extreme values (−3 and 3) and the negative spectrumof either dimensions. This observation shows some bias towards the positive regions for both valence and arousal.
A possible explanation for this is the nature of television talk shows in the first place. Discussions in the television talk shows are normally designed to be engaging andenjoyable for the viewers, as is the case with the collectedepisodes. In such situations, the occurrence of deactivated(negative arousal) and negative (negative valence) emotions

Fig. 5

Combination scheme of multiple modalities

appear to be rare. This points to the limitation of the conversational scope of television talk shows, which should be
addressed upon corpus expansion in the future.
6. Multimodal Emotion Recognition
We attempt to train a high-accuracy emotion recognizer byperforming a feature combination of diﬀerent modalities. In
the following experiments, we train unimodal classifiers using acoustic, semantic, and visual features, and fuse theminto a multimodal one by combining their predictions.
Figure 5 visualizes the combination scheme proposedin this work. The model construction starts with the training of several classifiers using various unimodal features.
We then utilize these unimodal models to make prediction
on subsets of the corpus. We utilize their predictions, inthe form of probability distributions, as classification features for the multimodal model. By including features ofdiﬀerent modalities, we hope to be able to observe a more
complete picture of emotion occurrences. To the best of
our knowledge, this constitutes the first multimodal emotion
recognition eﬀort in Indonesian.
We perform feature combination at decision level as
described above, instead of at feature level by concatenating the features, due to the small amount of data available.
Feature-level fusion increases the number of features during training, even though the amount of data remains thesame. Furthermore, any modification of features will leadto retraining of the entire model. On the other hand, the decision level fusion treats the modalities in a more modularway, allowing modification without the need of full model
retraining.
The use of semantic or lexical information for emotion
recognition commonly requires a large, often knowledgebased, language-specific database, i.e. word2vec [29] or
WordNet Aﬀect [30]. The construction of such model can beproblematic for many languages where resources are scarce,
such as Indonesian. In this work, in addition to the acoustic and visual features, we utilize semantic information ex-

IEICE TRANS. INF. & SYST., VOL.E101–D, NO.8 AUGUST 20182096

tracted purely from the available emotion corpus, without
the need of additional data. We exploit the TF-IDF score toweight the words to form the semantic representation.
7.

Experiments

7.1 Set-Up
In this experiment, we utlize the TV talk show corpus previously constructed to train the emotion recognizer. Due tothe relatively small amount of data, we simplify the emotion recognition problem by discretizing the aﬀective dimensions values into three classes: positive, neutral, andnegative. This simplification is a trade-oﬀ decision between
recognition granularity and model complexity. This discretization scheme has also been used in a number of previous studies, such as [31]. We believe that incremental steps
in tackling the problem will result in a better recognizer inthe long run, thus we focus on the less complex problem atthis stage of the research as the foundation for future experiments.
The emotion label of each segment is decided by majority voting. We first discretize the annotation into positive,
neutral, and negative, and perform majority voting afterwards. The positive class corresponds to the positive-valued
annotation (3 to 1), the neutral class to the zero-valued (0),and negative class to the negative-valued (−1 to −3). Segments with three diﬀerent votes are excluded to avoid potentially ambiguous emotion occurrences. We obtain 805(555 positive, 119 neutral, and 131 negative) segments forvalence and 820 (691 positive, 128 neutral, and 1 negative)
for arousal. We randomly divide the total with 80:10:10 ratio into training, development, and test sets.
We extract several feature sets to represent three fundamental modalities of communication: acoustic, semantic,
and visual. First, we extract global features of each utterance
using the openSMILE toolkit [32]. Two diﬀerent acoustic
feature sets are selected: INTERSPEECH 2009 baselinefeatures (IS09) and extended Geneva Minimalistic Acoustic Parameter Set (eGemaps) feature sets. The IS09 feature
set is described in Table 1. This feature set is widely used inemotion recognition research, thus providing comparability
to extensive related works.
On the other hand, the eGemaps feature set is proposed
as reduced acoustic feature set, containing only knowledgebased selected features that are 1) highly potential in indexing aﬀective signals, and 2) proven to be eﬀective in previous studies [33]. This feature set includes parameters related to frequency (pitch, jitter, formant), energy (shimmer,
Table 1Baseline feature of INTERSPEECH 2009 emotion challenge

LLD (16 · 2)

Functionals (12)(△) ZCR(△) RMS Energy(△) F0(△) HNR(△) MFCC 1-12

mean
standard deviation
kurtosis, skewness
extremes: value, rel. position, range
linear regression: oﬀset, slope, MSE

loudness, HNR), spectral (alpha ratio, Hammarberg index,
MFCC 1-4, spectral scope, format relative energy and bandwidth, spectral flux). Reduction of the total numbers of features is mostly contributed by the much fewer number offunctionals, mostly only involving arithmetic mean and coeﬃcient variation, with additional ones for selected features.
Secondly, to extract the semantic features of the utterances, we compute the TFIDF weighted vector of its transcribed speech. The TFIDF weight of a term t in a sentence
T is computed as:
TFIDF(t, T ) = Ft,T log

|T |
,
DFt

(1)

where Ft,T is defined as term frequency of term t in a sentence T , and DFt as total number of sentences that contains
the term t, calculated over the training set. Thus, the vector
for each sentence is the size of the corpus term vocabulary,with each term weighted according to Eq. (1).
Lastly, we extract facial features of each frame with the
OpenFace toolkit [34]. We extract 2D position of the facial
landmarks, as well as Action Unit (AU) intensities, and treat
them as two separate feature sets. We exclude frames withlow face detection confidence to exclude frames containing
other objects and uncertain recognition results. To obtainthe segment level features, we calculate 4 statistics from thecollection of frames for a given speech segment: mean, standard deviation, minimum, and maximum values. Since theframes are collected at rapid intervals, this helps neutralize
the eﬀect of multiple faces in a segment should it occur. Table 2 summarizes the total number of features in each featureset.
We train an SVM classifier for each feature set andcombination of the three modalities. We use the libSVM
library [35] in both of our unimodal and multimodal experiments. Prior to training, we scale the features into {0, 1}
range to avoid overpowering of features ranging in big values. Furthermore, we perform parameter optimization withgrid search to find the optimal value of C, the cost of misclassification, and γ, the free parameter of the Gaussian RBF
kernel. These steps are recommended in [36] and has beenshown to be eﬀective in SVM classification experiments.
For all models, unimodal or multimodal, we compare theirperformance by using the recognition precision, recall, andF1-score on the test set.
7.2

Results and Discussion

Table 3 presents the performance of unimodal emotion
recognition, measured in precision, recall, and F1 score. We
Table 2 Number of features in each feature set
Modality
Feature set
Number of features
Acoustic
VisualSemantic

gemapsIS09
AUlandmarks
TFIDF

62
384
72
336
2801

LUBIS et al.: CONSTRUCTION OF SPONTANEOUS EMOTION CORPUS FROM INDONESIAN TV TALK SHOWS AND ITS APPLICATION

2097
Table 3 Unimodal emotion recognition performance on test set in %. Highest number on each task
is boldfaced.
ArousalValence
Modality
Feature
Prec.
Rec.
F1
Prec.
Rec.
F1
eGemaps
90.02
90.24
89.12
87.23
83.95
85.19
Acoustic
IS09
93.28
92.68
91.84
90.23
86.42
87.87
AU
68.77
82.93
75.19
66.39
81.48
73.17
Visuallandmarks
68.77
82.93
75.19
66.39
81.48
73.17
Semantic
TFIDF
68.77
82.93
75.19
79.69
53.09
60.97

Table 4 Multimodal emotion recognition performance on test set in %. Highest number on each task
is boldfaced. *: higher than unimodal best.
Semantic

Features
AcousticeGemaps

TFIDF
IS09

VisualAU
landmarks
AUlandmarks

Prec.
68.76
68.76
93.27
93.27

average the measurements of the three classes with weighted
averaging based on class size to take into account the classimbalance. The unimodal recognition result shows thatin both tasks acoustic signals appear to be the most discriminative classification feature, suggesting that the richest
emotion clue in the conversational data is carried throughspeech. Interestingly, regardless of the much bigger number of features, the IS09 set outperforms the eGemaps set.
This suggest that some features that are present in IS09 but
absent in eGemaps are helpful in recognizing emotion in theconstructed Indonesian corpus.
For both tasks, the recognition performance using visual and TFIDF features is slightly lower than that of acoustic. The two visual feature sets yield identical performance,with precision score that are considerably lower than the recall. Reversely, the TFIDF feature set scores higher in precision than recall. This shows an imbalance in the predictionthat favors or misses the larger class. On the other hand,the acoustic features achieve high score for both recall andprecision.
Table 4 summarizes the result of multimodal emotion
recognition. We experimented with all possible combinations of the three modalities. The numbers suggest that thedetermining factor in the result of feature combination is thehighest performing feature among the unimodal ones. For
example, the decision of which visual features to use in the
combination does not aﬀect the performance of the end multimodal model, most probably due to its suboptimal performance in comparison to the other unimodal features.
Due to the lack of negative sample on the collected
data, the arousal models essentially learned only two classes(positive and neutral). In the arousal task the best combination of the modalities yields identical number as that of theunimodal. On the other hand, in the valence task, the combination is able to improve the recognition rate significantly.
To understand this diﬀerence better, we analyze the resultof the unimodal and multimodal recognition. For each pairof the best performing multimodal sets (i.e. set ABC becomes pairs AB, AC, and BC), we compute the correlation

Arousal
Rec.
82.92
82.92
92.68
92.68

F1
75.18
75.18
91.83
91.83

Prec.
84.12
84.12
91.79
90.25

Valence
Rec.
83.95
83.95
93.82
92.59

F1
78.45
78.45
92.46
91.26

coeﬃcient of the probability distributions to see the similarity of the predictions. We find that on the arousal task,
the unimodal predictions are highly correlated, making most
mistakes on the same examples. In consequence, they do
not provide additional information to each other when combined. In contrast, for the valence task, we observe varianceand diﬀerent trends of prediction. This means the modalities
contain complimentary information that is useful in bettermodeling the problem and classifying the data on the second stage of the model training.
The improvement of emotion recognition performanceafter considering multimodal information has been shownin previous studies in other languages as well. In a study
using an acted emotion corpus, Busso et al. reported an accuracy of 89% in recognizing four classes of emotion, a significant improvement compared to the accuracy of 70.9%
with speech-based system and 85% with facial expression
based system [37]. This finding is reaﬃrmed by Poria etal. who reported improvement when fusing three modalities into a single emotion recognizer [38]. Another study by
Nojavanasghari et al. focusing on spontaneous emotions inchildren also reported the same trend, with best overall performance at 69% [39].
However, comparison between emotion recognition results should be done carefully. Comparability has been along standing issue for emotion recognition tasks. The different methods and approaches in constructing an emotion
recognizer is diﬃcult to compare as they are rarely tested on
a common experimental condition [40]. One of the reasonof this problem is the absence of a database that can generalize to all facets of human emotion (e.g. culture, gender,
situation) to act as the standard corpus of emotion recognition works [41]. Each emotion-rich corpus is often aimed
to capture emotion in a specific set of circumstances, creating not only diﬀerences between the corpora, but also theworks based on them. Furthermore, diﬀerent emotion models result in a variety of annotation scheme (e.g. dimensional
traces or emotion classes) and recognition problems (e.g.
classification or regression). Readers are referred to litera-

IEICE TRANS. INF. & SYST., VOL.E101–D, NO.8 AUGUST 2018

2098

tures for comprehensive survey and comparison on emotion
recognition works [17], [41].
8.

Conclusion and Future Works

We presented an emotional audio-visual corpus in Indonesian and the subsequent eﬀort utilizing it in multimodal
emotion recognition. To construct the corpus, we exploited
television programs, in particular talk shows, to provide natural emotional data covering a broad range of emotions.
The corpus is the first of its kind in Indonesian, providing
a wide-range of opportunities of studies on emotion in thelanguage, especially as very few resources currently exist.
Subsequently, we construct an SVM based multimodal emotion recognizers utilizing the predictions of three modalities:
acoustic, semantic, and visual. We perfom decision-levelcombination and obtained a model with identical or better
recognition accuracy.
In this paper we have successfully utilized a corpus
constructed from spontaneous conversation to build emotion
recognizers for three levels of valence and arousal. We hope
to continue this work and move towards a finer-grain, more
precise quantification of emotion. Continuous developmentof the corpus is important, and we hope to collect more datafrom other situated dialogue to cover a wider scope emotion
and of social interactions. In the future, the content of theconversation should be considered during data collection toaddress the uneven distribution of emotion labels we havereported in this paper. We look forward to experiment withother features and methods to improve on the recognition
accuracy.
Acknowledgements
Part of this work was supported by JSPS KAKAENHI Grant
Numbers JP 17H06101 and JP 17K00237.
References[1] M. Schroder, E. Bevacqua, R. Cowie, F. Eyben, H. Gunes, D.
Heylen, M.T. Maat, G. McKeown, S. Pammi, M. Pantic, C.
Pelachaud, B. Schuller, E. de Sevin, M. Valstar, and M. Wollmer,“Building autonomous sensitive artificial listeners,” IEEE Trans. Affective Computing, vol.3, no.2, pp.165–183, 2012.
[2] K.J. Williams, J.C. Peters, and C.L. Breazeal, “Towards leveraging
the driver’s mobile device for an intelligent, sociable in-car robotic
assistant,” 2013 IEEE Intelligent Vehicles Symposium (IV), pp.369–
376, IEEE, 2013.
[3] D. Benyon, B. Gambäck, P. Hansen, O. Mival, and N. Webb, “Howwas your day? evaluating a conversational companion,” IEEE Trans.
Aﬀective Computing, vol.4, no.3, pp.299–311, 2013.
[4] B. Schuller, S. Steidl, and A. Batliner, “The INTERSPEECH 2009
emotion challenge,” INTERSPEECH, pp.312–315, Citeseer, 2009.
[5] B. Schuller, S. Steidl, A. Batliner, F. Burkhardt, L. Devillers, C.
Müller, and S. Narayanan, “The INTERSPEECH 2010 paralinguistic challenge,” INTERSPEECH, pp.2794–2797, 2010.
[6] B. Schuller, M. Valstar, F. Eyben, G. McKeown, R. Cowie, and M.
Pantic, “Avec 2011–the first international audio/visual emotion challenge,” Aﬀective Computing and Intelligent Interaction, pp.415–
424, Springer, 2011.

[7] B. Schuller, M. Valster, F. Eyben, R. Cowie, and M. Pantic, “Avec2012: the continuous audio/visual emotion challenge,” Proc. 14th
ACM international conference on Multimodal interaction, pp.449–
456, ACM, 2012.
[8] F. Ringeval, B. Schuller, M. Valstar, S. Jaiswal, E. Marchi, D.
Lalanne, R. Cowie, and M. Pantic, “Av+ ec 2015: The first aﬀect
recognition challenge bridging across audio, video, and physiological data,” Proc. 5th International Workshop on Audio/Visual Emotion Challenge, pp.3–8, ACM, 2015.
[9] M. Valstar, J. Gratch, B. Schuller, F. Ringeval, D. Lalanne, M.T.
Torres, S. Scherer, G. Stratou, R. Cowie, and M. Pantic, “AVEC
2016: Depression, Mood, and Emotion Recognition Workshop andChallenge,” AVEC ’16 Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, pp.3–10, ACM 2016.
[10] T. Polzehl, A. Schmitt, and F. Metze, “Approaching multi-lingual
emotion recognition from speech-on language dependency of acoustic/prosodic features for anger detection,” 2010.
[11] H. Sagha, J. Deng, M. Gavryukova, J. Han, and B. Schuller,“Cross lingual speech emotion recognition using canonical correlation analysis on principal component subspace,” 2016 IEEE International Conference on Acoustics, Speech Signal Process. (ICASSP),
pp.5800–5804, IEEE, 2016.
[12] Y. Pan, P. Shen, and L. Shen, “Speech emotion recognition using
support vector machine,” Int. J. Smart Home, vol.6, no.2, pp.101–
108, 2012.
[13] J.C.P. Gonzaga, J.A. Seguerra, J.A. Turingan, M.P.A. Ulit, and R.A.
Sagum, “Emotional techy basyang: An automated filipino narrative storyteller,” Int. J. Future Computer and Communication, vol.3,
pp.271–274, Aug. 2014.
[14] S. Sakti, A.A. Arman, S. Nakamura, and P. Hutagaol, “Indonesian
speech recognition for hearing and speaking impaired people,” INTERSPEECH, 2004.
[15] S. Sakti, M. Paul, R. Maia, S. Sakai, N. Kimura, Y. Ashikari, E.
Sumita, and S. Nakamura, “Toward translating Indonesian spoken
utterances to/from other languages,” Proc. O-COCOSDA, pp.137–
142, 2009.
[16] E. Cahyaningtyas and D. Arifianto, “HMM-based indonesian speechsynthesis system with declarative and question sentences intonation,” 2015 International Symposium on Intelligent Signal Processing and Communication Systems (ISPACS), pp.153–158, IEEE,
2015.
[17] M. El Ayadi, M.S. Kamel, and F. Karray, “Survey on speech emotion recognition: Features, classification schemes, and databases,”
Pattern Recognit., vol.44, no.3, pp.572–587, 2011.
[18] P.R. Shaver, U. Murdaya, and R.C. Fraley, “Structure of the indonesian emotion lexicon,” Asian journal of social psychology, vol.4,
no.3, pp.201–224, 2001.
[19] K.R. Scherer, R. Banse, and H.G. Wallbott, “Emotion inferencesfrom vocal expression correlate across languages and cultures,” J.
Cross-cultural psychology, vol.32, no.1, pp.76–92, 2001.
[20] F. Koto and M. Adriani, “Hbe: Hashtag-based emotion lexicons fortwitter sentiment analysis,” Proc. 7th Forum for Information Retrieval Evaluation, pp.31–34, ACM, 2015.
[21] J.E. The, A.F. Wicaksono, and M. Adriani, “A two-stage emotion
detection on indonesian tweets,” 2015 International Conference on
Advanced Computer Science and Information Systems (ICACSIS),
pp.143–146, IEEE, 2015.
[22] T.P. Tomo, G. Enriquez, and S. Hashimoto, “Indonesian puppet theater robot with gamelan music emotion recognition,” 2015 IEEE
International Conference on Robotics and Biomimetics (ROBIO),
pp.1177–1182, IEEE, 2015.
[23] L. Devillers, M. Tahon, M.A. Sehili, and A. Delaborde, “Inferenceof human beings emotional states from speech in human–robot interactions,” Int. J. Social Robotics, vol.7, no.4, pp.451–463, 2015.
[24] E. Douglas-Cowie, R. Cowie, C. Cox, N. Amier, and D. Heylen,“The sensitive artificial listener: an induction technique for generating emotionally coloured conversation,” 2008.

LUBIS et al.: CONSTRUCTION OF SPONTANEOUS EMOTION CORPUS FROM INDONESIAN TV TALK SHOWS AND ITS APPLICATION

2099[25] M. Grimm, K. Kroschel, and S. Narayanan, “The Vera am Mittag
German audio-visual emotional speech database,” Proc. IEEE International Conference on Multimedia and Expo (ICME), IEEE, 2008.
[26] J.A. Russell, “A circumplex model of aﬀect,” J. Personality and Social Psychology, vol.39, no.6, p.1161, 1980.
[27] E. Douglas-Cowie, R. Cowie, I. Sneddon, C. Cox, O. Lowry, M.
Mcrorie, J.C. Martin, L. Devillers, S. Abrilian, A. Batliner, N.
Amir, and K. Karpouzis, “The HUMAINE database: Addressing
the collection and annotation of naturalistic and induced emotionaldata,” Aﬀective computing and intelligent interaction, pp.488–500,
Springer, 2007.
[28] F. Ringeval, A. Sonderegger, J. Sauer, and D. Lalanne, “Introducing
the recola multimodal corpus of remote collaborative and aﬀective
interactions,” 2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), pp.1–8,
IEEE, 2013.
[29] B. Xue, C. Fu, and Z. Shaobin, “A study on sentiment computing
and classification of sina weibo with word2vec,” 2014 IEEE International Congress on Big Data, pp.358–363, IEEE, 2014.
[30] C. Strapparava and A. Valitutti, “Wordnet aﬀect: An aﬀective extension of wordnet,” LREC, pp.1083–1086, 2004.
[31] L. Tian, J. Moore, and C. Lai, “Recognizing emotions in spoken dialogue with hierarchically fused acoustic and lexical features,” 2016
IEEE Spoken Language Technology Workshop (SLT), pp.565–572,
IEEE, 2016.
[32] F. Eyben, M. Woellmer, and B. Schuller, “Opensmile: the munich
versatile and fast open-source audio feature extractor,” Proc. international conference on Multimedia, pp.1459–1462, ACM, 2010.
[33] F. Eyben, K.R. Scherer, B.W. Schuller, J. Sundberg, E. André,C. Busso, L.Y. Devillers, J. Epps, P. Laukka, S.S. Narayanan,and K.P. Truong, “The geneva minimalistic acoustic parameterset (GeMAPS) for voice research and aﬀective computing,” IEEE
Trans. Aﬀective Computing, vol.7, no.2, pp.190–202, 2016.
[34] T. Baltrušaitis, P. Robinson, and L.P. Morency, “Openface: An open
source facial behavior analysis toolkit,” 2016 IEEE Winter Conference on Applications of Computer Vision (WACV), pp.1–10, IEEE,
2016.
[35] C.C. Chang and C.J. Lin, “LibSVM: A library for support vector
machines,” ACM Trans. Intelligent Systems and Technology (TIST),
vol.2, no.3, p.27, 2011.
[36] C.W. Hsu, C.C. Chang, and C.J. Lin, “A practical guide to support
vector classification,” 2003.
[37] C. Busso, Z. Deng, S. Yildirim, M. Bulut, C.M. Lee, A.
Kazemzadeh, S. Lee, U. Neumann, and S. Narayanan, “Analysis
of emotion recognition using facial expressions, speech and multimodal information,” Proc. 6th international conference on Multimodal interfaces, pp.205–211, ACM, 2004.
[38] S. Poria, I. Chaturvedi, E. Cambria, and A. Hussain, “Convolutionalmkl based multimodal emotion recognition and sentiment analysis,”
2016 IEEE 16th International Conference on Data Mining (ICDM),
pp.439–448, IEEE, 2016.
[39] B. Nojavanasghari, T. Baltrušaitis, C.E. Hughes, and L.P. Morency,“Emoreact: A multimodal approach and dataset for recognizing
emotional responses in children,” Proc. 18th ACM International
Conference on Multimodal Interaction, pp.137–144, ACM, 2016.
[40] Z. Zeng, M. Pantic, G.I. Roisman, and T.S. Huang, “A survey ofaﬀect recognition methods: Audio, visual, and spontaneous expressions,” IEEE Trans. Pattern Anal. Mach. Intell., vol.31, no.1, pp.39–
58, 2009.
[41] C.N. Anagnostopoulos, T. Iliou, and I. Giannoukos, “Features andclassifiers for emotion recognition from speech: a survey from 2000
to 2011,” Artificial Intelligence Review, vol.43, no.2, pp.155–177,
2015.

Nurul Lubis
received the B.E degree (withdistinction) in 2014 from Bandung Insistute ofTechnology, Indonesia and the M.Eng degree in2017 from Nara Institute of Science and Technology (NAIST), Japan. She is currently a doctoral candidate at Augmented Human Communication Laboratory, NAIST, Japan. She is arecipient of the Japanese Ministry of Education, Culture, Sports, Science and Technology(MEXT) scholarship. She was a research internat Honda Research Institute Japan, Co. Ltd. Her
research interest include aﬀective computing, emotion in spoken language,
and aﬀective dialogue systems.

Dessi Lestari
was born in Bandung, Indonesia, in 1979. She received the B.E. degree in informatics engineering from Bandung Institute ofTechnology (ITB), Indonesia, in 2003, and theM.Eng. and Ph.D. degrees in computer sciencefrom the Tokyo Institute of Technology (Titech)
Tokyo, Japan, in 2007 and 2011, respectively.
In 2012, she joined the Department of Informatics Engineering, School of Electrical and Informatics Engineering, ITB, as a Lecturer. Since
February 2016, she became an Associate Professor in ITB. Her current research interests include speech signal processing, speech recognition, speech synthesizer, speaker recognition, emotionalrecognition, and broad domain of human computer interaction.

Sakriani Sakti
received her B.E. degree
in Informatics (cum laude) from Bandung Institute of Technology, Indonesia, in 1999. In 2000,
she received DAAD-Siemens Program Asia 21stCentury Award to study in Communication
Technology, University of Ulm, Germany, andreceived her MSc degree in 2002. During
her thesis work, she worked with Speech Understanding Department, DaimlerChrysler Research Center, Ulm, Germany. Between 2003–
2009, she worked as a researcher at ATR SLC
Labs, Japan, and during 2006–2011, she worked as an expert researcher
at NICT SLC Groups, Japan. While working with ATR-NICT, Japan, she
continued her study (2005–2008) with Dialog Systems Group University
of Ulm, Germany, and received her Ph.D. degree in 2008. She actively
involved in collaboration activities such as Asian Pacific Telecommunity
Project (2003–2007), A-STAR and U-STAR (2006–2011). In 2009–2011,
she served as a visiting professor of Computer Science Department, University of Indonesia (UI), Indonesia. From 2011, she has been an assistantprofessor at the Augmented Human Communication Laboratory, NAIST,Japan. She served also as a visiting scientific researcher of INRIA ParisRocquencourt, France, in 2015–2016, under “JSPS Strategic Young Researcher Overseas Visits Program for Accelerating Brain Circulation”. She
is a member of JNS, SFN, ASJ, ISCA, IEICE and IEEE. Her research interests include statistical pattern recognition, speech recognition, spoken
language translation, cognitive communication, and graphical modelingframework.

IEICE TRANS. INF. & SYST., VOL.E101–D, NO.8 AUGUST 20182100

Ayu Purwarianti
gained her bachelor andmaster degree at Informatics/Computer Science
Program, Bandung Institute of Technology. She
got her doctoral degree from Toyohashi University of Technology, Japan. Since 2008, she has
become a lecturer at School of Electrical Enginering and Informatics, Bandung Institute ofTechnology, Indonesia. Now, she is the Chair ofDoctoral Program of Electrical Engineering andInformatics at Bandung Institute of Technology,
Indonesia. Her research interest is on computational linguistics, mainly on Indonesian natural language processing. She
is now active as education oﬃcer at IEEE Indonesia and she is also activeat Indonesian Association for Computational Linguistics.

Satoshi Nakamura
is Professor of Graduate
School of Science and Technology, Nara Institute of Science and Technology, Japan, Project
Leader of Tourism Information Analytics Teamof RIKEN, Center for Advanced Intelligence
Project AIP, Honorarprofessor of Karlsruhe Institute of Technology, Germany, and ATR Fellow. He received his B.S. from Kyoto Instituteof Technology in 1981 and Ph.D. from Kyoto
University in 1992. He was Associate Professor of Graduate School of Information Scienceat Nara Institute of Science and Technology in 1994–2000. He was Director of ATR Spoken Language Communication Research Laboratories
in 2000–2008 and Vice president of ATR in 2007–2008. He was Director General of Keihanna Research Laboratories and the Executive Directorof Knowledge Creating Communication Research Center, National Institute of Information and Communications Technology, Japan in 2009–2010.
He is currently Director of Augmented Human Communication laboratory
and a full professor of Graduate School of Information Science at Nara
Institute of Science and Technology. He is interested in modeling and systems of speech-to-speech translation and speech recognition. He is one of
the leaders of speech-to-speech translation research and has been serving
for various speech-to-speech translation research projects in the world including C-STAR, IWSLT and A-STAR. He received Yamashita ResearchAward, Kiyasu Award from the Information Processing Society of Japan,
Telecom System Award, AAMT Nagao Award, Docomo Mobile ScienceAward in 2007, ASJ Award for Distinguished Achievements in Acoustics.
He received the Commendation for Science and Technology by the Minister of Education, Science and Technology, and the Commendation for
Science and Technology by the Minister of Internal Aﬀair and Communications. He also received LREC Antonio Zampolli Award 2012. He has
been Elected Board Member of International Speech Communication Association, ISCA, since June 2011, IEEE Signal Processing Magazine Editorial Board Member since April 2012, IEEE SPS Speech and Language
Technical Committee Member since 2013, and IEEE Fellow since 2016.