JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

1

Positive Emotion Elicitation in
Chat-based Dialogue Systems
Nurul Lubis, Non-Member, IEEE, Sakriani Sakti, Member, IEEE, Koichiro Yoshino, Member, IEEE,
Satoshi Nakamura, Fellow, IEEE

Abstract—We aim to draw on an important overlooked potential of affective dialogue systems: their application to promote
positive emotional states, similar to that of emotional support
between humans. This can be achieved by eliciting a more positive
emotional valence throughout a dialogue system interaction, i.e.,
positive emotion elicitation. Existing works on emotion elicitation
have not yet paid attention to the emotional benefit for theusers. Moreover, a positive emotion elicitation corpus does not
yet exist despite the growing number of emotion-rich corpora.
Towards this goal, firstly, we propose a response retrieval
approach for positive emotion elicitation by utilizing examples of
emotion appraisal from a dialogue corpus. Secondly, we efficiently
construct a corpus using the proposed retrieval method, by
replacing responses in a dialogue with those that elicit a morepositive emotion. We validate the corpus through crowdsourcing
to ensure its quality. Lastly, we propose a novel neural networkarchitecture for an emotion-sensitive neural chat-based dialoguesystem, optimized on the constructed corpus to elicit positive
emotion. Objective and subjective evaluations show that the
proposed methods result in dialogue responses that are morenatural and elicit a more positive emotional response. Further
analyses of the results are discussed in this paper.
Index Terms—natural language processing, emotion elicitation,affective computing, chat-based dialogue system, neural networks

I. I NTRODUCTION
It has been argued that humans impose the emotional aspectof social communications in their interaction with computers
and machines [1]. As the technology develops, the potentialof agents to improve the emotional well-being of users has
been growing as well. In particular, emotionally intelligent
systems could provide assistance and prevention measuresthrough various affective tasks.
Two of the most studied emotional competences for agentsare emotion recognition and emotion simulation. Emotion
recognition allows a system to discern the user’s emotions andaddress them in giving a response [2], [3], [4]. On the other
hand, emotion simulation helps convey non-verbal aspects tothe user for a more believable and human-like interaction, forexample to show empathy [5] or personality [6]. Acosta andWard [7] have attempted to connect the two competences tobuild rapport, by recognizing user’s emotion and reflecting it
All authors are with the Division of Information Science, Nara Institute of
Science and Technology, Japan, e-mail: {nurul.lubis.na4, ssakti,koichiro, s-nakamura}@is.naist.jp.
Koichiro Yoshino is with PRESTO, Japan Science and Technology Agency,
Japan.
Sakriani Sakti, Koichiro Yoshino, and Satoshi Nakamura are with the Center
for Advanced Intelligence Project AIP, RIKEN, Japan.
Manuscript received April 19, 2005; revised August 26, 2015.

in the system response. Although these competences addresssome of the user’s emotional needs [8], they are not sufficient
to provide emotional support in an interaction.
A number of studies have showed a consistent inclinationof humans to talk about and socially share their emotional
experiences, especially for an intense and/or negative emotion
exposure [9]. This is argued to be an essential part of theemotional processes [10]. On the contrary, the absence ofsuch support in times of need can result in more serious,
longer-term consequences. Unfortunately, there is a lack ofstudies examining negative emotion commonly encounteredin everyday life.
To provide support for healthy users, there exist technologies such as listening oriented systems [11], [12] and companion conversational agents [13]. However, these studies do
not consider negative emotional experiences nor recovery fromthem. On the other hand, there exist efforts in addressing a
number of clinical emotional disturbances, such as depression
and suicide risk, through affective computing, with speechprocessing techniques [14], or distress clues assessments [15].
Unfortunately, these works are not applicable for the larger,
general audience as they are focused on clinical circumstances.
Our work aims to draw on an important overlooked potentialof emotion in dialogue systems: its utilization to improve
emotional experience and promote positive emotional states,
similar to that of emotional support between humans forprompt recovery from negative emotion in everyday situations.
This can be achieved by actively eliciting a more positive
emotional valence throughout a chat-based interaction, i.e.,
positive emotion elicitation.
Skowron et al. have studied the impact of different affective
personalities in a text-based dialogue system [16], reporting
consistent impacts with the corresponding personality in humans. Hasegawa et al. constructed translation-based response
generators with various emotion targets to elicit a pre-definedemotional state [17]. A shortcoming of existing works is thatthey have not yet paid attention to the emotional experience of
the users. Instead, they focus on one-dimensional personalities
or target emotions on the system side to achieve the elicitation.
Furthermore, despite the growing number of emotion-richcorpora [18], [19], [20], a positive emotion elicitation corpus
does not yet exist.
Towards positive emotion elicitation in chat-based dialoguesystems, we first propose a response retrieval method which
exploits emotion appraisal in dialogue for an example-based
dialogue system (Section VI). We augment the response selection criteria with emotional parameters to select the response

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015that has the most potential to elicit positive emotion. Secondly,
we construct a positive emotion elicitation corpus by obtaining
new responses that elicit a more positive emotion with theproposed retrieval method (Section VII). We validate thecorpus through crowdsourcing to ensure its quality. Lastly, we
propose a neural chat-oriented dialogue system that capturesuser’s emotional state and considers it in generating a dialogue response (Section VIII). We achieve positive emotion
elicitation without the need of an explicit dialogue strategy by
optimizing the emotion-sensitive network with the constructed
corpus. The overview is illustrated in Figure 1.

Emotionally colorful
data: SEMAINE(Section V)

Response retrieval method
for positive emotion
elicitation (Section VI)

Positive emotion
elicitation data

Constructing positive
emotion elicitation data
(Section VII)

Example-based
dialogue system

Neural network based
method for affect sensitive
response generation(Section VIII)

Neural response generator

Fig. 1: Overview. White boxes with blue border show proposed
methods. Blue shapes show resulting models and data.

II. R ELATED W ORKS
Two main approaches in building chat-based dialogue systems are response retrieval and response generation. These
approaches operate in different ways and therefore have different advantages and disadvantages, which will be discussed
in this section. The retrieval method relies on a pre-defined setof example responses. The main question to be solved is themanner in which the system retrieves an appropriate response
from its pre-defined set, given a user input. In contrast, the generation approach produces the dialogue response sequentially,
from beginning to end, based on the probability distribution
it learned from the training data. This imposes an additional
challenge in training since the model is also required to learn
how to form a correct and grammatical response in addition
to it being coherent with the dialogue content.
One particular retrieval method is example-based dialogue
modeling (EBDM), which uses a semantically indexed corpus
of query-response examples instead of handcrafted rules orprobabilistic models [21]. At a given time, the system will
return the best response according to semantic similarity
between user input and the dialogue examples in its database.
This circumvents the challenge of domain identification andswitching – a particularly hard task in chat-oriented systems
where no specific goal or domain is predefined.
Despite its domain-free property, the EBDM approach has
not yet been studied to its full potential for affective dialoguesystems. An example is a micro-counseling dialogue systemthat uses the example-based approach to select a template forslot-filling response generation [3]. However, this approachstill relies on a small number of rule-based strategies. Strong
limitations of the EBDM approach include the lack of variety

2of responses and the difficulty in handling inputs that are notincluded in the example database, i.e., out-of-example cases.
With recent advancements in neural network research, endto-end approaches have shown promising results for non-goaloriented dialogue systems [22], [23], [24]. Instead of retrievinga response from an example database, neural dialogue systems
generate the response utterance, most commonly as a sequenceof words, based on the input query. These approaches rid
the need for explicit definition of dialogue states and action
spaces, allowing for a more dynamic model that mimics humanconversation contained in the training corpus. Furthermore,neural dialogue systems may still be able to produce a natural
response given a query that does not exist in the training
data, solving the out-of-example problem of the examplebased approach. These qualities are especially desirable fordomain-free, chat-oriented dialogue systems. Unfortunately, aswith response retrieval, application of this approach towards
incorporating emotion in the dialogue is still very lacking.
Only recently, Zhou et al. published their work addressing
the emotional factor in neural network response generation[25]. They examined the effect of internal emotional states on
the decoder, investigating 6 categories to emotionally color the
response, similar to that of emotion simulation. However, thisstudy has not yet considered user’s emotion in the response
generation process, nor attempted to utilize emotion to improve user experience. To the best of our knowledge, neuralnetwork approaches have not yet been utilized for affectsensitive response generation and emotion elicitation.
To bridge these gaps, we propose response retrieval andgeneration methods to incorporate emotion into chat-based
dialogue systems. We exploit emotion appraisal in dialogue foran example-based dialogue system. Modifying the traditional
response selection criteria with emotional parameters allows
us to select the response that has the most potential to elicit
positive emotion. For response generation, we propose a neuralchat-oriented dialogue system that captures user’s emotional
state and considers it while generating a dialogue response.
We train the latter with responses retrieved by the former toachieve end-to-end positive emotion elicitation.
III. E MOTION D EFINITIONIn this work, we define emotion using the circumplex modelof affect [26]. Two dimensions of emotion are defined: valence
and arousal. Valence measures the positivity or negativity ofemotion; e.g., the feeling of joy is indicated by positive valence
while fear is negative. On the other hand, arousal measures theactivity of emotion; e.g., depression is low in arousal (passive),
while rage is high (active). Figure 2 illustrates the valencearousal dimension in respect to a number of common emotion
terms.
This model describes the perceived form of emotion, andis able to represent both primary and secondary emotion.
Furthermore, it is intuitive and easily adaptable and extendable
to either discrete or other dimensional emotion definitions.
The long established dimensions are core to many works inaffective computing and potentially provide useful informationeven at an early stage of research.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

High arousal
Tense
NervousStressed

Alert

ExcitedElated
Happy

Upset

Negativevalence Sad

Content

Depressed

Positivevalence

SereneRelaxedLethargic
FatiguedCalm

Low arousal

Fig. 2: Emotion dimensions and common terms.
Henceforth, based on this scope of emotion, the term
positive emotion refers to the emotions with positive valence.
Respectively, a positive emotional change refers to the changeof position in the valence-arousal space where the value ofvalence after the movement is greater than before.
IV. D IALOGUE D EFINITION
Serban et al. have previously considered the two-hierarchy
view of dialogue [23], which we extend and adapt in thisstudy. First, we view a dialogue D as a sequence of dialogue
turns of arbitrary length M between two speakers. That is,
D = {U1 , ..., UM }. Each utterance in the m-th dialogue turnis a sequence of tokens of arbitrary length Nm . That is, Um ={wm,1 , ..., wm,Nm }.
Throughout the study and experiments, we utilize the dialogue triple format. A triple is a sequence of three dialogue
turns. That is, D = {U1 , U2 , U3 }. As we are focused on dyadic
dialogue, we consider U1 and U3 to be uttered by speaker A,
and U2 by speaker B. The triple format has been previously
utilized for considering context in response generation [27],and filtering multi-party conversations into dyadic snippets[28].
In this study, we assume two speaker orders in the triple
format. First, user-system-user, to observe emotional triggers
and responses in a conversation. The change of emotion
observed from U1 to U3 can be regarded as the impact of U2 .
Second, system-user-system, to observe both past and future
dialogue contexts of an emotion occurrence; U1 and U3 arethe contexts of the emotion occurrence in U2 .
V. E MOTIONALLY C OLORFUL C ONVERSATIONAL DATAIn this study, we utilize the SEMAINE database as
an emotion-rich conversational data [18]. The SEMAINE
database consists of spontaneous dialogues between a user anda sensitive artificial listener (SAL) in a Wizard-of-Oz fashion.
A SAL is a system capable of holding a multimodal conversation with humans, involving speech, head movements, andfacial expressions, topped with emotional coloring [29]. Thisemotional coloring is adjusted according to each of the SAL
characters; cheerful Poppy, angry Spike, sad Obadiah, andsensible Prudence. Each character is distinct and unchanging,i.e., Poppy is constantly happy, Spike is constantly angry, etc.,
independent of the dialogue content.
The corpus consists of a number of sessions, in which a user
is interacting with a wizard SAL character. Each user interacts

3

with all 4 characters, with each interaction typically lasting for
5 minutes. The topics of conversation are spontaneous, with alimitation that the SAL can not answer any questions.
The interactions are annotated using the FEELtrace system[30] to allow recording of perceived emotion in real time.
As an annotator is watching a target person in a video
recording, they would move a cursor along a linear scale
on an adjacent window to indicate the perceived emotional
aspect (e.g., valence or arousal) of the target. This results ina sequence of real numbers ranging from -1 to 1, called a
trace, that shows how a certain emotional aspect falls and rises
within an interaction. The numbers in a trace are provided withan interval of 0.02 seconds. Statistical analyses of validation
experiments have confirmed the reliability and indicated theprecision of the FEELtrace system [30].
In total, 95 sessions are provided, amounting to 475 minutesof material. In this study, we consider 66 sessions from thecorpus based on transcription and emotion annotation availability; 17 of Poppy’s sessions, 16 of Spike, 17 of Obadiah,
16 of Prudence. For every dialogue turn, we keep the speaker
information, time alignment, transcription, and emotion traces.
VI. R ESPONSE R ETRIEVAL : E LICITING P OSITIVEE MOTION BASED ON E XAMPLES OF H UMAN A PPRAISALEBDM is a data-driven approach that uses a semanticallyindexed corpus of query-response pair examples, i.e. {U1 , U2 },instead of handcrafted rules or probabilistic models [21]. In
traditional EBDM, the system will return a response of thebest example according to a semantic constraint between user
input and example queries, i.e. examples of U1 .
Lasguido et al. have previously examined the utilizationof cosine similarity for response retrieval in an examplebased dialogue system [28]. In their approach, the similarity iscomputed between term vectors of the query and the examples.
The vector for an utterance T is the size of the database term
vocabulary, where each term t is weighted by its TF-IDF score,computed as:
TFIDF(t, T ) = Ft,T log

|T |
,
DFt

(1)

where Ft,T is defined as term frequency of term t in sentence
T , and DFt as total number of sentences that contains theterm t, calculated over the example database. Cosine similarity
between two vectors a and b is computed as:
cossim (a, b) =a·b.
kak kbk(2)

Given a query, this cosine similarity is computed over all
example queries in the database and treated as the example
pair scores. The response of the example pair with the highest
score is then returned to the user as the system’s response.
This approach has a number of benefits. First, the TF-IDF
weighting allows emphasis of important words. Such quality isdesirable in considering emotion in spoken utterances. Second,as this approach does not rely on explicit domain knowledge,
it is practically suited for adaptation into an affective dialoguesystem. Third, the approach is straightforward and highly

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

4

reproducible. On that account, it serves as the baseline in thissection.



I failed the test.

A. Proposed Approach
Limitation of existing works on emotion elicitation [16],[17] is the oversight of emotion appraisal, which gives rise to
the elicited emotion in the first place. This entails the relationship between an utterance, which acts as stimulus evaluatedduring appraisal (emotional trigger), and the resulting emotion
by the end of appraisal (emotional response) [31]. By reversing
this loop, an appraisal-driven approach attempts to determine
the appropriate trigger to a desired emotional response. Thisknowledge is prevalent in humans and strongly guides how
we communicate with other people – for example, to refrainfrom provocative responses and to seek pleasing ones.
We attempt to elicit a positive emotional change by exploiting examples of appraisal in spontaneous dialogue. We
collect triples containing emotional triggers and responses toserve as examples in an EBDM. We augment the traditional
response selection criterion with emotional parameters: 1)
user’s emotional state, and 2) expected1 future emotional
impact of the candidate responses. These parameters represent
parts of the information that humans use in social-affective
interactions.
To take emotional aspects into account, in addition to thesemantic constraint in traditional EBDM, we formulate two
types of emotional constraints: 1) emotion similarity between
the query and the example queries, and 2) expected emotional
impact of the candidate responses. To allow observation ofthese constraints, we use a database of triples in place of
query-response pairs used in traditional EBDM. Within thecontext of triples in an example database, U1 is equivalent to
the example query, U2 to a candidate response, and U3 to the
future context. Figures 3a and 3b illustrate the general idea ofthe baseline and proposed approaches.
First, we measure emotion similarity by computing thePearson’s correlation coefficient between emotion vectors of
the query and the example queries in the database. Pearson’s
rab between two vectors a and b of length n is calculated as

My test result is bad.

Candidate response

Oh, again?

You will do better next time!

Cosine similarity

Perfect

i=1 (ai − ā)(bi − b̄)
qP
.
n
n
2
2
(a
−
ā)
(b
−
b̄)
i
i
i=1
i=1

(3)

In this case, a and b symbolize the emotion representations ofquery and example query. We utilize the trace provided in the
SEMAINE database as emotion representation (Section V).
As the length of emotion vector depends on the duration of
the utterance, prior to emotion similarity calculation, samplingis performed to keep the emotion vector in uniform lengthof n. For shorter utterances with fewer than n values in theemotion vector, we perform sampling with replacement, i.e.,a number can be sampled more than once. The samplingpreserves distribution of the values in the original emotion
vector. We calculate the emotion similarity score separately
1 Within the scope of the proposed method, we use the word expected forits literal meaning, as opposed to its usage as a term in probability theory.

The new book is great.

… What is it about?
Low

High

Oh, again?

(a) Selection with semantic similarity


Example query
Candidate response
Expected future
Cosine similarity
Emotional impact

I failed the test.
Tri-turn examples
I failed the test.

My test result is bad.

Oh, again?

You will do better next time!

Yeah…

Thank you.

Perfect
NegativearousalvalenceHigh
Positive

The new book is great.

…

What is it about?

It’s about space!
arousalvalencearousalLowvalence

Positive

You will do better next time!

(b) Selection with semantic similarity and emotion parameters

Fig. 3: Response selection in baseline and proposed systems.
for valence and arousal, and then take the average as the finalscore.
Secondly, we measure the expected emotional impact of thecandidate responses. In a triple, emotional impact of a response
U2 is computed according to the query U1 and future U3 using
Equation (4).
n
n
1X
1X
impact(U2 ) =emoU3 i −emoU1 i ,
n i=1
n i=1

(4)

where emoU1 and emoU3 are the emotion vectors of U1 andU3 . In other words, the actual emotion impact observed in
an example is the expected emotional impact during the realinteraction. As positivity of emotion is described by valence,for the expected emotional impact, we consider only valenceas the final score.

ExampleDatabasePn

rab = pP

Query-response examples

I failed the test.

Example query

Query

text
Semantic
similarity
scoring

emotionm-bestEmotioncorrelation
scoring

emotional changen-bestEmotional
impactscoring

best

best

Response(baseline)

Response(proposed)

Fig. 4: Steps of response selection.
Figure 4 illustrates the steps of response selection by thebaseline and proposed systems. We perform the selection inthree steps based on the defined constraints. For each step,a new score is calculated and re-ranking is performed onlywith the new score, i.e., no fusion with the previous score is
performed.
The baseline system will output the response of the triple
example with the highest semantic similarity score, calculatedwith Equation (2). In contrast, in the proposed system’s

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

response selection, we pass m examples with highest semantic
similarity scores to the next step and calculate their emotion
similarity scores with Equation (3). From n examples withhighest emotion similarity scores, we output the response of
the triple example with the most positive expected emotional
impact, calculated using Equation (4).
The scores are processed incrementally considering theimportance of each aspect. In selecting a response, we consider
semantic similarity to be most important as it ensures coherence of the response. If an example is not semantically relevantto the input, we should not consider it as a response regardlessof its emotional scores. Similarly, emotion similarity is important to guarantee consistent emotional impact with that of thedata; an utterance can result in different emotional responses
given different initial emotional states. Imposing the emotional
impact constraints in the reduced pool of semantically andemotionally similar examples will help achieve a more relevant
result. Additionally, this reduces the computation time since
the number of examples to be scored will be greatly minimized
after each step. When working with big example databases,
this property is beneficial for a timely response.
Consequently, there are three important points to noteregarding the proposed approach. First, U3 , the future context
of each triple, is not considered as a definite predictionof user response when interacting with the system. Instead,each triple acts as an example of human’s appraisal in aconversation with certain semantic and emotional contexts. In
real interaction, given similar semantic and emotional contexts
with an example triple’s U1 , when the system outputs U2 , we
expect the user to experience an emotional change similar tothat of U3 . The semantic and emotion similarities between user
input and U1 are important in ensuring an emotional responsethat is as close as possible to the example.
Second, this strategy does not translate to selection of the
response that expresses the most positive emotion. Our goal
is not to consistently output a “happy response.” There aresituations where such a response is inappropriate and could
cause the opposite effect, e.g., in times of grief. Instead, wewould like to reflect proper positive emotion elicitation given
semantic and emotional contexts. Even though there is no
explicit dialogue strategy to be followed, we expect the data
to reflect the appropriate situation to show negative emotion
to elicit a positive impact in the user, such as relating to
one’s anger or showing empathy. Lastly, we note that response
similarity with the target is not the objective of our retrieval
method. Our goal is not to replicate the target response throughthe example database, but instead to retrieve a response thathas most potential to elicit a more positive emotion.

5

We partition the recording sessions in the corpus into training
and test sets. The training set and test set comprise 29 (15Poppy, 14 Prudence) and 4 (2 Poppy, 2 Prudence) sessions,
respectively. We construct the example database exclusively
from the training set, containing 1105 triples.
We average as many annotations as provided in a session
to obtain the final valence and arousal label. We sample theemotion trace of every dialogue turn into 100-length vectors
to keep the length uniform. We utilize the transcription andemotion annotation provided within the corpus as informationof the triples to isolate the errors of automatic speech andemotion recognition. For the n-best filtering, we empiricallychose 10 for the semantic similarity constraint and 3 for theemotion.
C. Evaluation
To evaluate the proposed method, we perform subjective
evaluation to qualitatively measure perceived differences between the two response selection methods. From the test set,
we extract 198 test queries. For each test query, we generate
responses using the baseline and proposed systems explainedin Section VI-A. Queries with identical responses from thetwo systems are excluded from the evaluation. We further filter
the queries based on utterance length, to give enough context
to the evaluators; and emotion labels, to give variance in theevaluation. In the end, 50 queries are selected.
We perform subjective evaluation of the systems withcrowdsourcing. The queries and responses are presented inform of text. We ask the evaluators to compare the systems’
responses in respect to the test queries. For each test query, theresponses from the systems are presented with random ordering, and the evaluators are asked three questions, adapted from[16]. 1) Which response is more coherent? Coherence refers
to the logical continuity of the dialogue, 2) Which response
has more potential in building emotional connection between
the speakers? Emotional connection refers to the potentialof continued interaction and relationship development, and
3) Which response gives a more positive emotional impact?
Emotional impact refers to the potential emotional change theresponse may cause.
50 judgments are collected per query. Each judgment isweighted with the level of trust of the worker2 . The final
judgment of each query for each question is based on the
total weight of the overall judgements – the system with thegreater weight wins.
TABLE I: Percentage of wins of each system on all metrics.
Numbers in brackets show average agreement on queries
where the system wins for that metric.
SystemCoherenceBaseline
Proposed

0.34 (0.14)
0.66 (0.38)

B. Experimental Set Up
We gather the examples from the SEMAINE corpus by
extracting triples with user-system-user speaker sequence. An
analysis shows that Poppy and Prudence tend to draw theuser into the positive-valence region of emotion, resembling a
positive emotional impact. The same is not true for sessions
with Spike and Obadiah. Thus, we exclusively use sessionsof Poppy and Prudence to construct the example database.

Emotional
impact0.30 (0.19)
0.70 (0.32)

Emotional
connection
0.34 (0.17)
0.66 (0.36)

Table I presents the evaluation results. It is shown that incomparison to the baseline system, the proposed system is
2 A worker’s trust score is equal to their percentage of correct answers on aquiz, consisting of triples with gold standard answers. In this study, we only
employ workers with high level of trust.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

6

TABLE II: Candidate responses re-ranking based on threeconsecutive selection constraints: 1) semantic similarity with
example queries, 2) emotion similarity with example queries,and 3) expected emotional impact of the candidate response.
*: baseline response, **: proposed response.
Query: Em going to London tomorrow. (valence: 0.39, arousal: -0.11)
Candidate responses (U2 )* And where in Australia?
[laugh]Organized people need to have holiday.
It would be very unwise for us todiscuss possible external examiners.
[laugh]It’s good that sounds eh like a goodthing to do, although you wouldn’t
want to em overspend.
That sounds interesting you’ve
quite a lot going on so you needto manage your time.
Yes.
Mhm.
** That sounds nice.

semantic
1
2
3

ranking steps
emotionimpactTABLE III: Baseline and proposed responses for identicaltext with different emotional contexts. The proposed system
can adapt to user emotion, while the baseline method outputs
the same response.
Query : Thank you. (valence: 0.13, arousal: -0.18)
Baseline : Thank you very much that
Proposed : And I hope that everything goes exactly according to plan.
Query : Thank you. (valence: 0.43, arousal: 0.05)
Baseline : Thank you very much that
Proposed : It is always a pleasure talking to you you’re just like me.

1

4

5
6

7

2

8
9
10

3

1

perceived as more coherent, having more potential in building
emotional connection, and giving a more positive emotional
impact. We investigate this result further by computing theagreement of the final judgment using Fleiss’ Kappa formula.
We separate the queries based on the winning system andcompute the overall agreement of the 50 judgments accordingly. It is revealed that the queries where the proposedsystem wins have far stronger agreement than those where
the baseline system wins. Agreement between 0.10 and 0.20
can be interpreted as slight agreement, and that between 0.21
and 0.40 as fair agreement [32].
D. Analysis
We analyze the consequence of re-ranking and the effectof emotion similarity in the response selection using queries
extracted from the test set. Table II presents the 10-best
semantic similarity ranking, re-ranked and filtered into 3-bestemotion similarity ranking, and the candidate response thatpassed the filtering with the best emotional impact. The tableshows that the proposed method can select one of the candidate
responses that despite being not the best in semantic similarity
score, has a higher score in terms of emotion similarity andexpected impact compared to responses with semanticallymore similar examples.
Furthermore, the proposed selection method is able to generate different responses to identical textual input with different
emotional contexts. Table III demonstrates this quality. Thisshows that the system is able to adapt to the user’s emotion
in giving a response to elicit positive emotion.
VII. C ONSTRUCTING P OSITIVE -E MOTION E LICITING
DATA
Despite the existence of numerous emotion-rich data [18],[19], [20], there is not yet a conversational data demon-

strating positive emotion elicitation. Such data is valuablefor data-driven approaches of dialogue systems. However,the collection of emotional corpora is vastly expensive andlabor intensive. Collection of spontaneous emotional data isa sensitive matter, potentially raising moral and ethical issues[33]. On the other hand, portrayal of emotion often presents
a mismatch with real-life emotion occurrences.
Our goal is to construct a corpus that demonstrates positive
emotion elicitation. To efficiently collect responses that elicit
positive emotion, we enhance the emotion-rich SEMAINE
corpus through the procedure illustrated in Figure 5.

Database

Default
datasetHumanjudgement

Dialogue system

Systemresponse

Positive-emotion
eliciting dataset

Fig. 5: Obtaining references that elicit positive emotion.
First, we extract system-user-system triples and run themthrough the EBDM with the proposed response retrieval
method (Section VI-A) to obtain new candidate responses that
potentially elicit positive emotion. As noted previously, theresponse retrieval method makes a careful distinction between
the expressed emotion of a response and the emotion it may
elicit in the listener. Through this response selection method
we aim to reflect proper positive emotion in the corpus insteadof solely selecting “happy responses.”
Subsequently, we validate the data through crowdsourcing.
We ask human judges to decide which response in a triple, i.e.,
U3 , elicits a more positive emotional impact in the triple, thedefault or the system generated one. If neither is judged to doso, the human judge is asked to provide one that elicits positive
emotion. When more than one human-proposed responses areprovided for a triple, we manually select the best suited one
based on naturalness and potential positive emotional impact.
The result of this process is then used to replace the default
response from the corpus. These steps ensure the quality of
the new responses, aligning it to human standards.
We utilize 66 sessions based on the availability transcription
and emotion annotation. We fed a total of 2,349 triples to theentire process. For each triple, we obtain at least 3 humanjudgements, or more when ties occur. The final response isJOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

obtained by majority voting, with each vote weighted by thevoter’s trust score. In total, 419 crowd workers participated in
the judgment process with an average trust score of 0.93. The
average consensus of the voting is 0.78.
In the resulting corpus, 12.69% of the responses are humangenerated, 38.84% are SEMAINE default, 46.38% are systemgenerated, and 2.02% are cases where the default and systemgenerated responses are identical, and voted to elicit positive
emotion by the workers. The average word count for thehuman generated responses is 6.09 words.
VIII. R ESPONSE G ENERATION : E MOTION - SENSITIVEN EURAL D IALOGUE S YSTEM
The most commonly used neural architecture for dialogue
response generation is the recurrent neural network (RNN). An
RNN is a neural network variant that can retain information
over sequential data. In response generation, first, an encodersummarizes an input sequence into a vector representation.
An input sequence at time t is modeled using the information
gathered by the RNN up to time t − 1, contained in the hidden
state ht . Afterwards, a decoder recurrently predicts the output
sequence conditioned on ht and its output from the previous
time step. This architecture was previously proposed as neuralconversational model in [22].
Based on the two-hierarchy view of dialogue (Section IV),
the hierarchical recurrent encoder-decoder (HRED) extends
the sequence-to-sequence architecture [23]. It consists of three
RNNs, each with a distinct role. First, an utterance encoder
encodes a dialogue turn by recurrently processing each token
in the utterance. After processing the last token, the hidden
state of the utterance encoder hutt represents the entirety ofthe dialogue turn. This information is then passed on to the
dialogue encoder, which encodes the sequence of dialogue
turns hdlg . The utterance decoder, or the response generator,
takes hdlg , and then predicts the probability distribution over
the tokens in the next utterance. The advantages of thisarchitecture are: 1) summarization of dialogue history by the
dialogue encoder, containing common knowledge between thetwo speakers, and 2) reduced computational steps between
utterances, allowing a more stable optimization during thetraining phase.

71) Architecture: We utilize RNNs with gated recurrent unit
(GRU) cells as parts of the Emo-HRED. The information
flow of the Emo-HRED is as follows. After reading the input
sequence Um = {wm,1 , ..., wm,Nm }, the dialogue turn is
encoded into an utterance representation hutt .
utthutt = hutt
Nm = f (hNm −1 , wm,Nm ),(5)

where f represents one time step operation of RNN with
GRU. hutt is then fed into the dialogue encoder to model
the sequence of dialogue turns into dialogue context hdlg .
dlg
hdlg = hdlg
m = f (hm−1 , hutt ).

(6)

In Emo-HRED, the hdlg is then fed into the emotion
encoder, which will then be used to model the emotion context
hemo .
hemo = f (hemom−1 , hdlg ).

(7)

We consider utilizing additional information in modeling theemotion context. In particular, by using a fully connectedneural network to encode the dialogue turn’s acoustic feature
audm into haud and feeding it into the emotion encoder. In
such a case, Equation (7) is modified as follows.

hemo = f hemo
m−1 , concat(hdlg , haud ) .

(8)

The generation process of the response, Um+1 , is conditioned by the concatenation of the dialogue and emotion
contexts.
Pθ (wn+1 = v|w≤n ) =
exp g(concat(hdlg , hemo ), v)
 . (9)P0v 0 exp g(concat(hdlg , hemo ), v )Figure 6 shows a schematic view of this architecture. To thebest of our knowledge, this constitutes the first neural network
approach for affect-sensitive response generation.
𝑤2,1… 𝑤2,𝑁2

ℎ𝑑𝑙𝑔

𝑤3,1

… 𝑤3,𝑁3ℎ𝑒𝑚𝑜

A. Proposed approachℎ𝑢𝑡𝑡

The HRED architecture holds a property which is essential
in positive emotion elicitation: retaining dialogue history atturn level. This allows the observation of both past and futurecontexts of an emotion occurrence, i.e., U1 and U3 are thecontexts of the emotion occurrence in U2 . We propose toincorporate an emotion encoder into the HRED architecture,
placed in the same hierarchy as the dialogue encoder. The
emotion encoder captures emotion information at dialogueturn level and maintains the emotion context history throughout the dialogue. We propose to incorporate this informationin generating a dialogue response with an emotion-sensitive
hierarchical recurrent encoder-decoder (Emo-HRED).

ℎ𝑎𝑢𝑑

𝑤1,1

… 𝑤1,𝑁1 𝑎𝑢𝑑1

𝑤2,1

… 𝑤2,𝑁2 𝑎𝑢𝑑2

Fig. 6: Emo-HRED architecture.
The emotion encoder has its own target vector, which isthe emotion label of the currently processed dialogue turnemo
Um
. The emotion encoder uses hyperbolic tangent activation
function. We modify the definition of the training cost toincorporate the prediction error of the emotion encoder anduse this cost to jointly train the entire network. We defineJOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 20158

costemo as the mean squared error between the target emotion
context and the predicted emotion context.
The training cost of the Emo-HRED is a linear interpolation
between the response generation error costutt (i.e. negative
log-likelihood of the generated response) and the emotion label
prediction error costemo with a decaying weight α. The final
cost is then propagated to the network and the parameters are
optimized as usual with the optimizer algorithm.
cost = α · costemo + (1 − α) · costutt .

(10)2) Pretraining and selective fine-tuning: Availability oflarge-scale data is an ongoing challenge for emotion-related
research because of the difficulties in capturing life-likeemotion occurrence and annotating it reliably. Due to thelimited amount of conversational data available with emotion
information, training a full end-to-end dialogue system fromscratch is unlikely to yield a high quality result. To that extent,
pretraining the Emo-HRED with a large scale conversational
corpus is essential to infer content and syntactic knowledge
prior to training its emotion-related parameters.
We propose selective fine-tuning of the Emo-HRED, limiting the parameter updates to the emotion encoder and utterance
decoder only. We hypothesize that the encoding ability has
converged during pretraining by utilizing the large amount of
data, and will potentially destabilize when fine-tuned using
the much smaller, emotion-rich data. As emotion is not yet
involved during encoding, we further hypothesize that the pretrained encoders can be used for the affect-sensitive response
generation task as is.
B. Experimental Set Up
1) Pretraining: Previous works have demonstrated the effectiveness of large scale conversational data in improving thequality of dialogue systems [34], [35], [23]. In this study,
we make use of SubTle, a large scale conversational corpus,
to learn the syntactic and semantic knowledge for response
generation. The use of movie subtitles is particularly suitable
as they reflect natural human conversation and are available
in large amounts.
The SubTle corpus [35] contains conversational pairs extracted from movie subtitles expanding four genres: horror,
science fiction, western, and romance. High-quality movie
subtitles are obtained using movie identifiers shared by movie
cataloging websites. The corpus consists of 6,072 subtitle files
in total. The subtitles are then automatically processed toobtain conversation pairs similar to Query-Answer format.
In our experiments, we utilize the HRED trained on the
SubTle corpus as our starting model. The data preprocessing
steps are performed as in [23]. The processed SubTle corpus
contains 5,503,741 query-answer pairs in total. The triple
format is forced onto the pairs by treating the last dialogue turnin the triple as empty. The 10,000 most frequent tokens aretreated as the system’s vocabulary, and the rest as unknowns.
The model is pretrained by feeding this dataset sequentiallyinto the network until it converges, taking approximately 2days to complete. In addition to the model parameters, we
also learn the word embeddings of the tokens. We use wordembeddings of size 300, utterance vectors of size 600, anddialogue vectors of size 1200. The parameters are randomly
initialized, and then trained to optimize the log-likelihood of
the training triples using the Adam optimizer.
2) Fine-tuning: All the models considered in this study are
the result of fine-tuning the pretrained model with the emotionrich data, fed sequentially into the network. To investigate
the effectiveness of the proposed approach, we train multiple
models with combinations of set ups.
Model. We propose the Emo-HRED architecture in place of
HRED which serves as the baseline. As emotion information
for the Emo-HRED, we use the valence and arousal traces
provided by the SEMAINE corpus as emotion context. For a
dialogue turn, we sample with replacement a vector of length
100 from each trace. We concatenate the valence and arousal
vectors to form the final emotion label, resulting in an emotion
vector of length 200. To accommodate this additional information during fine-tuning, we append new randomly initialized
parameters to the utterance decoder. These parameters aretrained exclusively during the fine-tuning process.
Selective fine-tuning scheme. We propose selective finetuning of the Emo-HRED. In the standard fine-tuning, we finetune all the parameters of the model. In the proposed selective
fine-tuning scheme, we fix the parameters of the utterance
and dialogue encoders, and train only the emotion encoderand utterance decoder (Section VIII-A2). We hypothesize thatthe selective fine-tuning will produce a more stable model.
The SubTle and SEMAINE corpus have a number of majordifferences that may cause straight-forward fine-tuning to notwork optimally: the sizes of the corpora differ significantly(5.5M vs. 2K triples), and the conversations are of different
nature (human-human vs. human-wizard, acted speech vs.
spontaneous speech).
Positive emotion elicitation data. We propose an implicit
positive emotion elicitation strategy via training data. We
compare fine-tuning with two datasets: the default SEMAINE
dataset, using the dialogue turns as provided by the SEMAINE
corpus; and the positive SEMAINE dataset, containing positive
emotion eliciting responses, i.e., U3 produced through theprocess previously described (Section VII). We hypothesizethat the positive corpus will cause the model to elicit morepositive emotion. For both datasets, we consider all 66 sessions
from the SEMAINE corpus. We partition the data as follows:
58 sessions (1985 triples) for training, 4 (170) for validation,and 4 (194) for test.
Audio encoder. We propose utilizing acoustic features incombination with dialogue state for emotion encoding. We
suspect that some affective information that is essential indetermining emotional context is lost when the observation
is limited to text only. When acoustic features are included,
Equation (8) is used in place of Equation (7). We extract
the INTERSPEECH 2009 emotion challenge baseline features[36] using the OpenSMILE toolkit [37]. The audio encoder isof size 200, randomly initialized and exclusively trained duringfine-tuning. Audio information is solely used as additional
information for emotion encoding. Since speech recognitionis out of the scope of this paper, we use the transcription as
the text input for Emo-HRED.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

9

TABLE IV: Evaluation results. Each of the proposed methods is incrementally compared. Objective evaluation is measured
in “Perplexity.” Subjective evaluation is measured in “Naturalness” and “Emotional impact.” Best number for each metric isbold-faced. On subjective evaluation, * denotes significant difference (p<0.05) with best model (No. 10). Highlighted systems(No. 3, 4, 8, and 10) are further analyzed in the following subsection.
No.

1. Model1
2
3
4

Baseline
HRED

5
6
7
8
9
10

2. Selective
fine-tune

Proposed
Emo-HRED

No
Yes

Yes

Yes

3. Positive
data
No
Yes
No
Yes
No
Yes
No
Yes
Yes

4. Audio
encoder

5. Useprediction

No

No

No
No
Yes
No
Yes

Emotion encoder prediction. We investigate the effect ofdifferent emotion inputs for the utterance decoder duringfine-tuning: feeding the target emotion vector into utterance
decoder, or using the prediction of emotion encoder instead.
Note that in either scenario, the emotion prediction is used forevaluation. We suspect that using the prediction of emotion
encoder leads to better optimization; when the emotion target
is used during training, the error of emotion encoder is
not propagated to the output, creating a disconnect between
generated response and emotion prediction.
C. Evaluation and Analysis
We perform two evaluations to confirm the effectiveness ofthe proposed method. First, we perform objective evaluationof the systems by computing the model perplexity. Second,
we perform subjective evaluation to measure the naturalness
and emotional impact of the generated responses. The result ofboth evaluations is summarized in Table IV. To obtain deeper
insight into the evaluation results, analyses are provided at theend of this section.
1) Objective evaluation: Perplexity measures the probability of exactly regenerating the reference response in a
triple. This metric is commonly used to evaluate dialoguesystems that rely on probabilistic approaches [23] and has beenpreviously recommended for evaluating generative dialoguesystems [38]. We evaluate the models using the test set ofthe positive SEMAINE data, as we assume this dataset to be
the one that fulfills our main goal of an emotionally positive
dialogue. The “Perplexity” column of Table IV presents the
perplexity of the models with different fine-tuning set ups. In
the interest of space and readability, we iteratively choose thebest option of the proposed set ups. That is, we keep a set up
fixed when it has shown consistent improvement on a numberof systems.
We test the effect of the parameter update on both positiveand default datasets by keeping the model fixed to HRED. We
observe significant improvements when fine-tuning only the
decoder compared to fine-tuning the entire network (rows No.
1-4). This supports the hypotheses that we have previously
made: it is better to utilize the encoders pretrained using thelarge dataset as is, rather than to fine-tune them further using

Yes

Perplexity
185.66121.44151.77100.94
41.30
42.26
42.38
37.42
35.92
20.35

avg

Naturalness
std. dev

Emotional impact
avg
std. dev
n/a2.71 *
3.26 *
3.38 *
3.27 *
3.49 *
3.72
3.43 *
3.75

0.31
0.22
0.39
0.38
0.34
0.25
0.31
0.22

2.56 *
3.22 *
3.22 *
3.39 *
3.36 *
3.70
3.51 *
3.78

0.29
0.25
0.35
0.30
0.28
0.21
0.29
0.17

the small emotion-rich data. We train the rest of the set ups
with the selective fine-tuning scheme.
We test the impact of the emotion encoder by comparing
HRED and Emo-HRED (rows No. 3-6). We found that withidentical starting model and fine-tune set up, the Emo-HRED
architecture converges to significantly better models comparedto the HRED. This suggests two things: incorporation of theemotion prediction error helps the model to converge to abetter local optimum, and that the emotion information helps
in generating a response closer to the training reference.
We suspect that partly tuning the parameters through thesmaller valence-arousal space helps the model to infer useful
information for response generation through the simpler emotion recognition task. The relationship between semantic andemotional content is not arbitrary, and thus utilizing them incombination could benefit the learning process of the model.
We found that the models trained with positive SEMAINE
data tends to yield lower perplexity than those trained withthe default dataset (rows No. 1-8), with only an exception
between rows 5 and 6. The model perplexity further shows thatthe incorporation of audio information for emotion encoding
allows significant improvement when positive data is used(rows No. 6 and 8), but not when default data is used (rows No.
5 and 7). This suggests that the audio information could allow
emotion encoder to form a representation of the emotional
context that further helps model the data better. We see
consistent improvement by using the prediction of the emotion
encoder for utterance decoding during fine-tuning (rows No.
6 to 9, and 8 to 10), reaching the best perplexity of 20.35.
The fine-tune set up of row No. 1 is equivalent to the
SubTle bootstrap approach proposed in [23]. However, there
are differences that are important to highlight, summarizedin Table V, which made it not possible to straightforwardly
compare our results with those reported in [23]. Nonetheless,this comparison demonstrates the ability of Emo-HRED to efficiently take advantage of emotion information, consequently
decreasing model perplexity despite of small data size, which
is often a challenge in affective computing works.
2) Human subjective evaluation: We perform human subjective evaluation via crowdsourcing. We exclude systems not
fine-tuned with the selective scheme due to the poor qualityof the generated responses. We present human judges with

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

TABLE V: Model comparison.
Pretraining
Fine-tune and testdata
# triplesArchitecture
Emotion
Audio
Perplexity

Serban, et al. [23]This research
SubTle bootstrap
MovieTriples
Positive SEMAINE
245,492
HRED Bidirectional
No
No
26.81

2,349
Emo-HRED
Yes
No
Yes
35.92
20.35

a dialogue triple and ask them to rate the response on two
criteria. The first is naturalness, which evaluates whether the
response is intelligible, logically follows the dialogue context,
and resembles real human response. The second is emotional
impact, which evaluates whether the response elicits a positive
emotional impact or promotes an emotionally positive conversation.
We evaluate 100 triples from the full test set, where eachis judged by 20 human evaluators. Each triple is presented
in A-B-A format, the first two dialogue turns are held fixedaccording to the test set, and the last turn is the response generated by the evaluated model. Evaluators are asked to judge theresponses by stating their agreement to two statements: 1) A
gives a natural response, and 2) A’s response elicits a positive
emotional impact in B. The agreement is given using a Likert
scale, ranging from 1 (strongly disagree) to 5 (strongly agree).
Columns “Naturalness” and “Emotional impact” in Table
IV show the results of the subjective evaluation. With identicalfine-tune set ups, compared to HRED, Emo-HRED is consistently perceived as more natural and eliciting a more positive
emotional impact (rows No. 3-6). We observe consistentimprovements when comparing models trained on the default
and positive SEMAINE data (rows No. 3 and 4, 5 and 6, 7 and
8). The subjective ratings further show that utilization of audio
encoder gives perceptible improvements (rows No. 5-8). Whenemotion prediction is used during fine-tuning, significant improvements are observed for models without emotion encoder(rows No. 6 and 9), and slight improvement is observed formodels with encoder (rows No. 8 and 10). We also noticethat the standard deviation of the ratings to diminish as themodel improves. We perform t-test to measure whether the
improvement of the best model in subjective evaluation issignificant. For both naturalness and emotional impact, model
No. 10 shows significant improvement (p<0.05) compared to
all models, except model No. 8.
Finally, we perform the same subjective evaluation to themodel proposed in Section VI and compare the results with thebest proposed Emo-HRED (model No.10). The EBDM-basedmodel achieves a score of 3.32 ± 0.30 and 3.20 ± 0.25 fornaturalness and emotional impact, respectively. T-test reveal
statistically significant difference (p<0.05) between the twomodels in terms of subjective perception, i.e. the responses
generated by the Emo-HRED approach is perceived as morenatural and elicit a more positive emotional impact.
3) Analysis: We analyze the generated responses to reason
for the objective and subjective evaluation results. In thisanalysis, we consider models No. 3 and 4 as baseline models,and 8 and 10 as best proposed models. These models are

10highlighted in Table IV.
Table VI shows an example of a test triple along withresponses generated by the models. This example demonstrates
the disconnect between perplexity and subjective perception ofa dialogue response. While the response from model No. 10 ismore similar to the target response, both are arguably naturaland have potential in eliciting positive emotional impact. This
could explain the different trends between subjective andobjective evaluation results (Table IV).
TABLE VI: Comparison of system responses for a triple in
the test set.
U1
U2
U3 SEMAINE
U3
positive
SEMAINE
Model No. 3
Model No. 4
Model No. 8
Model No. 10

that’s so cool you must be so proud of yourself.
ah yeah i am i am very proud because it’s like ididn’t think it was gonna go this far [laugh].
yeah.
that’s good yes.
cause you don’t really want to go out with people.
yeah so you have to be inside really for the best.
it’s good to hear that.
that’s good i hope that.

We found that on average, the Emo-HRED models generated responses that are shorter compared to that of HRED
(5.54 vs. 8.19 words). Consequently, the Emo-HRED responses amount to a smaller vocabulary than the HRED.
However, this smaller vocabulary contains larger proportionsof positive-sentiment words. For example, with systems No.
3, 4, 8, and 10 respectively, the word ”good” makes up
2.4%, 4.9%, 25.6%, and 26.3% of the evaluated responses,
excluding stop-words. Table VII lists the top 10 words fromthese vocabularies in order of frequency, as well as the positive
SEMAINE vocabulary for reference.
TABLE VII: 10 most frequent words in the responses, excluding stop words. Positive sentiment words are bold-faced.
No. 3
No. 4
No. 8
No. 10positive
SEMAINEtell, well, like, good, think, make, else, go, get, knowtell, good, think, nice, well, sensible, see, meet, know,
reallygood, able, yeah, tell, well, hear, oh, ok, nice, ahagood, hope, happy, makes, yeah, nice, meet, well, ok,ahagood, think, laugh, oh, well, like, things, else, excellent, tell

It is interesting to note that we also observe the same tendency in the responses we collected from human annotators forthe positive SEMAINE dataset. This actually follows humanstrategy when promoting positive emotional experiences inconversations with only limited context provided – by using
general responses that contain positive-sentiment words.
Furthermore, we observe similar phenomena on the subjective evaluation results. As the response length grows, so does
its likelihood to carry grammatical or logical errors. This leads
to both poor naturalness and uncertain emotional responses
upon human perception. The responses generated from the
proposed model are short and sweet, enough to sustain generalconversation with short context (in this case, two previous
dialogue turns), similar to that of human daily small talks.
These tendencies observed from the pos_sem dataset andJOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

Emo-HRED model could explain the lower perplexity when
one of them is employed, and lowest when both are.
To clarify, this is not to say that short, generic responses
are always desirable. This is a standing problem for neuralnetwork based response generation [39] – moving toward
longer, context-specific responses will lead to a more engaging
interaction. However, we note that there are circumstances
for which the implicit strategy of the proposed method issuitable, as previously discussed. We look forward to expand
the conversational ability of the model to accommodate longer
context and content-specific information in future works.
IX. C ONCLUSION
In this paper we presented a research effort towards positive
emotion elicitation in chat-based dialogue systems. First, we
utilized examples of human appraisal in an example based
dialogue system. We augment the traditional response selection criteria to take into account the emotional context of the
dialogue and the emotional impact of the candidate responses.
Secondly, we employed the proposed retrieval method toefficiently alter an emotion rich conversational data into a
positive emotion elicitation data by replacing the dialogue
responses with ones that elicit a more positive emotion. The
data is validated through crowdsourcing to ensure its quality.
Lastly, we proposed a hierarchical neural dialogue systemwith an emotion encoder to capture the emotional context ofthe dialogue. This information is then used in the response
generation process to produce an affect-sensitive response.
We optimize the network on the constructed corpus to train
a neural response generator that elicits positive emotion.
Objective and subjective evaluations show that the proposed
methods result in dialogue responses that are more natural andelicit a more positive emotional response.
We acknowledge that evaluation through real user interaction needs to be carried out in the future to test the effectiveness of the positive emotion elicitation in longer, continuous
conversations. We also hope to further improve the qualityof the proposed system, both in terms of response quality
and user’s emotional experience. We believe that collection ofemotionally rich conversational data is crucial and will highly
benefit this research effort, widening the scope of the data tocover larger conversational scenarios. In terms of elicitation
strategy, we would like to define an explicit training goal
to maximize the positive emotional effect of the generated
response. We look forward to apply reinforcement learning
to positive emotion elicitation task. Lastly, we would like toconsider longer dialogue history for a more context-specificresponse generation.
ACKNOWLEDGMENT
Part of this work was supported by JSPS KAKENHI Grant
Numbers JP17H06101 and JP17K00237.
R EFERENCES[1] B. Reeves and C. Nass, How people treat computers, television, and new
media like real people and places. CSLI Publications and Cambridge
university press, 1996.

11[2] K. Forbes-Riley and D. Litman, “Adapting to multiple affective statesin spoken dialogue,” in Proceedings of the 13th Annual Meeting of theSpecial Interest Group on Discourse and Dialogue. Association for
Computational Linguistics, 2012, pp. 217–226.
[3] S. Han, Y. Kim, and G. G. Lee, “Micro-counseling dialog system
based on semantic content,” in Natural Language Dialog Systems andIntelligent Assistants. Springer, 2015, pp. 63–72.
[4] M. Tielman, M. Neerincx, J.-J. Meyer, and R. Looije, “Adaptive
emotional expression in robot-child interaction,” in Proceedings of the2014 ACM/IEEE international conference on Human-robot interaction.
ACM, 2014, pp. 407–414.
[5] R. Higashinaka, K. Dohsaka, and H. Isozaki, “Effects of self-disclosureand empathy in human-computer dialogue,” in Proceedings of Spoken
Language Technology Workshop. IEEE, 2008, pp. 109–112.
[6] A. Egges, S. Kshirsagar, and N. Magnenat-Thalmann, “Generic personality and emotion simulation for conversational agents,” Computer
animation and virtual worlds, vol. 15, no. 1, pp. 1–13, 2004.
[7] J. C. Acosta and N. G. Ward, “Achieving rapport with turn-by-turn,user-responsive emotional coloring,” Speech Communication, vol. 53,
no. 9-10, pp. 1137–1148, 2011.
[8] R. W. Picard and J. Klein, “Computers that recognise and respond touser emotion: theoretical and practical implications,” Interacting withcomputers, vol. 14, no. 2, pp. 141–169, 2002.
[9] O. Luminet IV, P. Bouts, F. Delie, A. S. Manstead, and B. Rimé,“Social sharing of emotion following exposure to a negatively valenced
situation,” Cognition & Emotion, vol. 14, no. 5, pp. 661–688, 2000.
[10] B. Rime, B. Mesquita, S. Boca, and P. Philippot, “Beyond the emotional
event: Six studies on the social sharing of emotion,” Cognition &Emotion, vol. 5, no. 5-6, pp. 435–465, 1991.
[11] T. Meguro, Y. Minami, R. Higashinaka, and K. Dohsaka, “Learning
to control listening-oriented dialogue using partially observable markov
decision processes,” ACM Transactions on Speech and Language Processing (TSLP), vol. 10, no. 4, p. 15, 2013.
[12] T. Yamaguchi, K. Inoue, K. Yoshino, K. Takanashi, N. G. Ward, and
T. Kawahara, “Analysis and prediction of morphological patterns ofbackchannels for attentive listening agents,” in Proc. 7th International
Workshop on Spoken Dialogue Systems, 2016, pp. 1–12.
[13] M. Cavazza, R. S. De La Camara, and M. Turunen, “How was your day?:a companion eca,” in Proceedings of the 9th International Conference
on Autonomous Agents and Multiagent Systems: volume 1-Volume
1. International Foundation for Autonomous Agents and Multiagent
Systems, 2010, pp. 1629–1630.
[14] N. Cummins, S. Scherer, J. Krajewski, S. Schnieder, J. Epps, and T. F.
Quatieri, “A review of depression and suicide risk assessment using
speech analysis,” Speech Communication, vol. 71, pp. 10–49, 2015.
[15] D. DeVault, R. Artstein, G. Benn, T. Dey, E. Fast, A. Gainer, K. Georgila,
J. Gratch, A. Hartholt, M. Lhommet et al., “Simsensei kiosk: A virtualhuman interviewer for healthcare decision support,” in Proceedings ofthe 2014 international conference on Autonomous agents and multiagent systems. International Foundation for Autonomous Agents andMultiagent Systems, 2014, pp. 1061–1068.
[16] M. Skowron, M. Theunis, S. Rank, and A. Kappas, “Affect and social
processes in online communication–experiments with an affective dialog
system,” Transactions on Affective Computing, vol. 4, no. 3, pp. 267–
279, 2013.
[17] T. Hasegawa, N. Kaji, N. Yoshinaga, and M. Toyoda, “Predicting andeliciting addressee’s emotion in online dialogue.” in Proceedings of
Association for Computational Linguistics (1), 2013, pp. 964–972.
[18] G. McKeown, M. Valstar, R. Cowie, M. Pantic, and M. Schroder, “TheSEMAINE database: Annotated multimodal records of emotionally colored conversations between a person and a limited agent,” Transactions
on Affective Computing, vol. 3, no. 1, pp. 5–17, 2012.
[19] M. Grimm, K. Kroschel, and S. Narayanan, “The Vera am Mittag
German audio-visual emotional speech database,” in Multimedia andExpo, 2008 IEEE International Conference on. IEEE, 2008, pp. 865–
868.
[20] E. Douglas-Cowie, R. Cowie, I. Sneddon, C. Cox, O. Lowry,
M. Mcrorie, J.-C. Martin, L. Devillers, S. Abrilian, A. Batliner et al.,“The humaine database: addressing the collection and annotation ofnaturalistic and induced emotional data,” in Affective computing andintelligent interaction. Springer, 2007, pp. 488–500.
[21] C. Lee, S. Jung, S. Kim, and G. G. Lee, “Example-based dialog modeling for practical multi-domain dialog system,” Speech Communication,
vol. 51, no. 5, pp. 466–484, 2009.
[22] O. Vinyals and Q. Le, “A neural conversational model,” arXiv preprint
arXiv:1506.05869, 2015.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015[23] I. V. Serban, A. Sordoni, Y. Bengio, A. Courville, and J. Pineau,“Building end-to-end dialogue systems using generative hierarchicalneural network models,” in Thirtieth AAAI Conference on Artificial
Intelligence, 2016.
[24] L. Nio, S. Sakti, G. Neubig, K. Yoshino, and S. Nakamura, “Neuralnetwork approaches to dialog response retrieval and generation,” IEICE
Transactions on Information and Systems., 2016.
[25] H. Zhou, M. Huang, T. Zhang, X. Zhu, and B. Liu, “Emotional chatting
machine: Emotional conversation generation with internal and external
memory,” arXiv preprint arXiv:1704.01074, 2017.
[26] J. A. Russell, “A circumplex model of affect.” Journal of personality
and social psychology, vol. 39, no. 6, p. 1161, 1980.
[27] A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell,
J.-Y. Nie, J. Gao, and B. Dolan, “A neural network approach tocontext-sensitive generation of conversational responses,” arXiv preprint
arXiv:1506.06714, 2015.
[28] N. Lasguido, S. Sakti, G. Neubig, T. Toda, and S. Nakamura, “Utilizing human-to-human conversation examples for a multi domain chatoriented dialog system,” Transactions on Information and Systems,
vol. 97, no. 6, pp. 1497–1505, 2014.
[29] M. Schröder, E. Bevacqua, R. Cowie, F. Eyben, H. Gunes, D. Heylen,
M. T. Maat, G. McKeown, S. Pammi, M. Pantic et al., “Building
autonomous sensitive artificial listeners,” Transactions on Affective Computing, vol. 3, no. 2, pp. 165–183, 2012.
[30] R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, and
M. Schröder, “‘FEELTRACE’: An instrument for recording perceived
emotion in real time,” in ISCA tutorial and research workshop (ITRW)
on speech and emotion, 2000.
[31] K. R. Scherer, A. Schorr, and T. Johnstone, Appraisal processes inemotion: Theory, methods, research. Oxford University Press, 2001.
[32] J. R. Landis and G. G. Koch, “An application of hierarchical kappatype statistics in the assessment of majority agreement among multiple
observers,” Biometrics, pp. 363–374, 1977.
[33] M. El Ayadi, M. S. Kamel, and F. Karray, “Survey on speech emotion
recognition: Features, classification schemes, and databases,” Pattern
Recognition, vol. 44, no. 3, pp. 572–587, 2011.
[34] R. E. Banchs and H. Li, “Iris: a chat-oriented dialogue system basedon the vector space model,” in Proceedings of the ACL 2012 System
Demonstrations. Association for Computational Linguistics, 2012, pp.
37–42.
[35] D. Ameixa, L. Coheur, P. Fialho, and P. Quaresma, “Luke, I am yourfather: dealing with out-of-domain requests by using movies subtitles,”in International Conference on Intelligent Virtual Agents. Springer,
2014, pp. 13–21.
[36] B. Schuller, S. Steidl, and A. Batliner, “The INTERSPEECH 2009
emotion challenge.” in INTERSPEECH, vol. 2009, 2009, pp. 312–315.
[37] F. Eyben, M. Wöllmer, and B. Schuller, “OPENsmile: the munich
versatile and fast open-source audio feature extractor,” in Proceedingsof the international conference on Multimedia. ACM, 2010, pp. 1459–
1462.
[38] O. Pietquin and H. Hastie, “A survey on metrics for the evaluation ofuser simulations,” The knowledge engineering review, vol. 28, no. 1, pp.
59–73, 2013.
[39] J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan, “A diversitypromoting objective function for neural conversation models,” in Proceedings of NAACL-HLT, 2016, pp. 110–119.

Nurul Lubis received the B.E degree (with distinction) in 2014 from Bandung Institute of Technology,
Indonesia and the M.Eng degree in 2017 from Nara
Institute of Science and Technology (NAIST), Japan.
She is currently a doctoral candidate at Augmented
Human Communication Laboratory, NAIST, Japan.
She is a recipient of the Japanese Ministry of Education, Culture, Sports, Science and Technology(MEXT) scholarship. She was a research intern at
Honda Research Institute Japan, Co. Ltd. Her research interest include affective computing, emotionin spoken language, and affective dialogue systems.

12

Sakriani Sakti received her B.E. degree in Informatics (cum laude) from Bandung Institute of Technology, Indonesia, in 1999. In 2000, she received
DAAD-Siemens Program Asia 21st Century Award
to study in Communication Technology, University
of Ulm, Germany, and received her MSc degreein 2002. During her thesis work, she worked with
Speech Understanding Department, DaimlerChrysler
Research Center, Ulm, Germany. Between 20032009, she worked as a researcher at ATR SLC Labs,
Japan, and during 2006-2011, she worked as anexpert researcher at NICT SLC Groups, Japan. While working with ATRNICT, Japan, she continued her study (2005-2008) with Dialog Systems Group
University of Ulm, Germany, and received her PhD degree in 2008. She actively involved in collaboration activities such as Asian Pacific Telecommunity
Project (2003-2007), A-STAR and U-STAR (2006-2011). In 2009-2011, she
served as a visiting professor of Computer Science Department, Universityof Indonesia (UI), Indonesia. In 2011-2017, she was an assistant professor at
the Augmented Human Communication Laboratory, NAIST, Japan. She served
also as a visiting scientific researcher of INRIA Paris-Rocquencourt, France, in
2015-2016, under JSPS Strategic Young Researcher Overseas Visits Programfor Accelerating Brain Circulation. Currently, she is a research associate
professor at NAIST, as well as a research scientist at RIKEN, the Centerof for Advanced Intelligent Project AIP, Japan. She is a member of JNS,
SFN, ASJ, ISCA, IEICE and IEEE. She is also the officer of ELRA/ISCA
Special Interest Group on Under-resourced Languages (SIGUL) and a board
Member of Spoken Language Technologies for Under-Resourced Languages(SLTU). Her research interests include statistical pattern recognition, graphical modeling framework, deep learning, multilingual speech recognition &synthesis, spoken language translation, affective dialog system, and cognitive
communication.

Koichiro Yoshino received his B.A. degree in 2009
from Keio University, M.S. degree in informatics in2011, and Ph.D. degree in informatics in 2014 from
Kyoto University, respectively. From 2014 to 2015,
he was a research fellow (PD) of Japan Society for
Promotion of Science. From 2015 to 2016, he was a
research assistant professor of the Graduate Schoolof Information Science, Nara Institute of Science andTechnology (NAIST). Currently, he is an assistantprofessor of NAIST. He is also a researcher of
PRESTO, JST, concurrently. He is working on areasof spoken and natural language processing, especially on spoken dialoguesystems. Dr. Koichiro Yoshino received the JSAI SIG-research award in 2013.
He is a member of IEEE, ISCA, IPSJ, and ANLP.

JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2015

Satoshi Nakamura is Professor of Graduate Schoolof Science and Technology, Nara Institute of Scienceand Technology, Japan, Project Leader of Tourism
Information Analytics Team of RIKEN, Center for
Advanced Intelligence Project AIP, Honorarprofessor of Karlsruhe Institute of Technology, Germany,
and ATR Fellow. He received his B.S. from Kyoto
Institute of Technology in 1981 and Ph.D. from
Kyoto University in 1992. He was Associate Professor of Graduate School of Information Scienceat Nara Institute of Science and Technology in
1994-2000. He was Director of ATR Spoken Language Communication
Research Laboratories in 2000-2008 and Vice president of ATR in 20072008. He was Director General of Keihanna Research Laboratories and the
Executive Director of Knowledge Creating Communication Research Center,
National Institute of Information and Communications Technology, Japan in
2009-2010. He is currently Director of Augmented Human Communicationlaboratory and a full professor of Graduate School of Information Science at
Nara Institute of Science and Technology. He is interested in modeling andsystems of speech-to-speech translation and speech recognition. He is one of
the leaders of speech-to-speech translation research and has been serving forvarious speech-to-speech translation research projects in the world including
C-STAR, IWSLT and A-STAR. He received Yamashita Research Award,
Kiyasu Award from the Information Processing Society of Japan, Telecom
System Award, AAMT Nagao Award, Docomo Mobile Science Award in
2007, ASJ Award for Distinguished Achievements in Acoustics. He received
the Commendation for Science and Technology by the Minister of Education,
Science and Technology, and the Commendation for Science and Technology
by the Minister of Internal Affair and Communications. He also received
LREC Antonio Zampolli Award 2012. He has been Elected Board Memberof International Speech Communication Association, ISCA, since June 2011,
IEEE Signal Processing Magazine Editorial Board Member since April 2012,
IEEE SPS Speech and Language Technical Committee Member since 2013,
and IEEE Fellow since 2016.

13