言語処理学会 第30回年次大会 発表論文集（2024年3月）

Dialogue Response Generation
Using Personal Facts and Personality Traits
Weiwen Su1 Naoki Yoshinaga2 Yuma Tsuta1 Masashi Toyoda2
1 The University of Tokyo
2 Institute of Industrial Science, The University of Tokyo{su-w,ynaga,tsuta,toyoda}@tkl.iis.u-tokyo.ac.jp
Agent Persona

Abstract

1. I live in Tokyo

Persona-based chatbots assuming a speciﬁc persona forchatbots can generate consistent responses given the persona. Existing persona-based dialogue datasets such as
PersonaChat and Multi-Session Chat (MSC), however,contain mainly personal facts (e.g., “I like cats.”) but
lack personality traits (“I am extraverted.”). We thus
automatically annotate the MSC dataset with personality traits to train persona-based chatbots using personal
facts and personality traits. Experimental results on the
personality-augmented MSC datasets conﬁrmed that ourchatbot improves personality consistency scores, when using a personality-aware reranking.

1

Introduction

Open-domain dialogue systems such as Siri and ChatGPT have become more common in our daily lives. As adaily conversation partner, we expect chatbots to converse
like humans with a consistent persona. However, since theconversation data used to train chatbots usually compile
conversations from various persons, the resulting chatbots
are likely to generate inconsistent responses.
To address those inconsistent responses by data-driven
chatbots, researchers consider the identity of speakers togenerate responses [1, 2, 3, 4, 5, 6, 7, 8]. To model andcontrol the chatbot persona explicitly, Zhang et al. [3] built
PersonaChat, which provides speaker proﬁles as text descriptions (e.g., “I have a dog.”), and most of the following
studies on persona-based chatbots utilized this dataset orits extension, Multi-Session Chat (MSC) [9]. Although theexisting datasets for persona-based chatbots contain various proﬁles to describe a speaker persona, those proﬁles
are mainly personal facts such as personal tastes, relatives, social status, and experiences, and barely include

User: Where do you live now?
Agent: I’m living in Tokyo.
+ Personal Fact

Agent Persona
1. I live in Tokyo
2. I'm full of enthusiasm
and spontaneity

User: Where do you live now?
Agent: Oh, I'm currently living it up in
Tokyo!
+ Personal Fact &Personality Traits

Figure 1

An example of a generation with or without person-

ality traits.

personality traits such as agreeableness and extraversion.
Meanwhile, Saha et al. [10] predict Big-Five personality
traits for speakers in several dialogue datasets to control
the style of generated responses. However, their datasets
do not contain personal facts.
In this study, aiming to investigate the impact of personality traits in persona-based chatbots, we automatically annotate personality traits to existing persona-based datasets,
MSC, using Big-Five personality predictor trained on Pandora dataset [11]; we then train and evaluate a personabased chatbot using the proﬁles on personal facts and theestimated personality traits of the speaker. An issue here is
how to represent predicted personality traits (category withintensity). We adopt the same short text descriptions as theoriginal proﬁles on personal facts to maintain interpretability and ﬂexibility. To enhance personality consistency, we
incorporate a response reranking model [7] to compute theconsistency between the personality proﬁles and the generated utterance to choose the response with the highest
personality consistency.
We use the personality-augmented MSC dataset to evaluate the impact of using both proﬁles on the original personal
facts and predicted personality traits in a persona-based
chatbot. The automatic and human evaluation conﬁrmed
the eﬀectiveness of personality-based proﬁles, when we
use the proposed reranking model.

― 1481 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

27-class

Related Work
Persona-based Response Generation

I'm a bit extraverted.

The prob-

Strong Extraversion(> u+0.5σ)

lem of inconsistent responses of data-driven chatbots was

3-class
I'm extraverted.

ﬁrst pointed out by Li et al. [1]. To address this problem, they trained the model with user embeddings fromthe speakers’ dialogue histories to generate more consistent responses. Meanwhile, Zhang et al. [3] proposed Per-

Concrete

Enjoys being the center of attention.
Likes to start conversations.
Finds it easy to make new friends.

Figure 2

A schema of the three personality verbalization methods. A case of extraversion intensity beyond 𝑢 + 0.5𝜎 is shown.

sonaChat, the most commonly used dataset for personabased chatbots; it compiles conversations between a pairpersonality traits 𝑃personality and personal facts 𝑃fact , theof speakers that role-play given persona, a series of text

model optimizes the generation of response 𝑅 to maximize

descriptions (proﬁles). To address the scarcity of persona

𝑃(𝑅|𝐶, 𝑃personality , 𝑃fact ) according to the dataset.

in PersonaChat, Majumder et al. [12] expand the original
proﬁles using commonsense knowledge. In the context oflong-term conversation, the persona may change over time.

3.1

Annotating Personality Traits

To augment the existing persona-based dialogue dataset

Xu et al. [9] thus extends PersonaChat with future conver-with personality traits, we train a personality detector using

sation sessions, referred to the resulting dataset as MSC.

Pandora [11] dataset, which consists of Big-Five personal-

However, the proﬁles in the PersonaChat and MSC containity traits and their intensities for 1500+ Reddit users. We

mainly personal facts such as personal tastes, relatives, anddevelop a RoBERTa [16]-based regression model to pre-social status, but lack personality traits.

dict the target user’s intensities of all personality traits at

Personality Controlled Dialogue GenerationIn

once from the target user’s utterances. Following previous

dialogue modeling, personality traits such as Big-Five per-work [10], this detector predicts a target user’s personality

sonality are considered to aﬀect speaking styles. In thefrom accumulated target user’s utterances, rather than ag-

early stage, Mairesse and Walker [13] leverage a statisti-

gregating the predictions from each utterance. Therefore,

cal generation model, focusing on extraversion personal-

this model processes several utterances at once like batchity. Recently, Wang et al. [14] proposed a seq2seq model

processing.

for Big-Five personality-conditioned response generation.

In this study, we explore an eﬀective way to verbalize
the detected intensity of personality traits. Speciﬁcally,

Saha et al. [10] leverage Big-Five personality and discourse
intents as stylistic control codes to generate stylistic responses. Although personal facts and personality traits areverbalization approaches to create personality proﬁles as

two important factors that characterize a speaker, no study

shown in Figure 2 are explored in our work:

attempts to model both persona information into accountin response generation, due to the absence of datasets.1）

based on the distribution of the personality traits, three

3-class verbalization

trait into three classes (positive, neutral, and nega-

We thus add personality traits as text descriptions to thetive) with 0.5 standard deviation 𝜎 from the mean 𝑢

MSC dataset to train and evaluate persona-based chatbotsof this dataset. Then verbalize the personality traits

using both personal facts and personality traits.

3

Split each Big-Five personality

in the same format as the personal fact proﬁles in the

Approachdataset using the adjectives of each Big-Five personality and its opposite expression (e.g. “Extraverted”

In this section, we describe our method to generate aand “Introverted”). We do not add personality proﬁlesdialogue response based on not only personal facts but
also personality traits in chat conversation.

The task

is deﬁned as, given the dialogue context 𝐶, and agent

for the neutral class.
7-class verbalization

1） Very recently, RealPersonaChat [15] datasets have been constructed, including massive Japanese conversations with both personal facts and personality traits provided by the speakers. However,the datasets have not been released at the moment.

― 1482 ―Rather than the three classes, using 0.5, 1, and 2 𝜎 from 𝑢 as the threshold, we further divide each personality trait into seven classes byadding adverbs (“a bit”, “quite” and non-adverb) of
the degree to create more reﬁned personality proﬁles.

This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Speciﬁcally, three adverbs with original and opposite

model could predict the consistency score between them.

adjectives result in six classes in addition to the neutral

4class.
Concrete verbalization

Experiments

Instead of the original and op-

In this section, we train chatbots on the augmented MSC

posite adjectives of Big-Five personality, we takedataset to conﬁrm the eﬀectiveness of using both person-

more concrete descriptions of the people owning suchality traits and personal facts with our reranking method.

personality traits from psychology websites2）as the
personality proﬁles. Speciﬁcally, we sampled the concrete descriptions according to the predicted intensity

4.1

Settings

Datasets

We use the MSC dataset [9], augmented by(33%, 66%, and 100% descriptions among all thepredicted personality traits for evaluation. Since individual

descriptions used and combined).

dialogue sessions, a series of consecutive utterances, in

Response generation based on personal facts and personality traits

the MSC datasets can be generated by diﬀerent pairs ofThough the personal facts and the personality traits areboth in the format of text descriptions, the elements they

we ignore multi-session settings in the dataset and handle
individual sessions as independent dialogues to guarantee

inﬂuence in the conversation are diﬀerent. Rather thanthe consistency of speakers’ personalities.

3.2

the basic facts and demographic features, the personality

speakers even if they maintain the same personal facts,

ModelsWe adopt DialoGPT3）[18] as a backbone oftraits inﬂuence mainly the action pattern, speaking style,persona-based chatbots, which ﬁnetunes a GPT-2 [19] on

and more complicated aspects. Thus, we ﬁrst concatenate

Reddit comment chain data. The models to be comparedindividual descriptions in the personal facts and personal-

in the experiment are as follows:

ity traits respectively, then concatenate the two sequences

Baseline

We ﬁnetuned the DialoGPT model on the orig-of descriptions with a special token. We feed the wholeinal MSC datasets with and without the personal facts

descriptions concatenated before the context to the decoder.

as baselines. We hereafter referred to them as Base-

Personality-aware Reranking

Naively incorporat-

ing personality traits may not facilitate the model to fullyline and +person. facts, respectively.
Proposed

We ﬁne-tuned three DialoGPT models on theutilize the given personality traits. Thus, inspired by [7]personality-augmented MSC dataset by combining

using reranking to improve consistency between personal

three personality verbalization methods (3-class, 7-

facts and generated responses leveraging a model trainedon DNLI dataset [17], we propose to reuse the augmentedclass, and concrete verbalization). We referred themto as 3-class, 7-class, concrete respectively.

training data of MSC dataset to train a personality consis-

We performed the reranking of 5 response candidates

tency prediction model for response reranking. In this task,
the response reranking model using RoBERTa [16] regres-for the three proposed models.
Metrics

We use perplexity (ppl.) and BLEU-1/2 [20]sion model calculates the consistency between personality

as basic metrics. We also evaluate the Distinct-1/2 (DIST-

traits and the generated responses. For the training data, we1/2) [21] to show whether the generated responses exhibit

make positive sample pairs of personality traits with utter-a certain degree of diversity.

ances in the original dialogue and negative sample pairs of

In addition to these generic metrics, following the pre-personality proﬁles with utterances from other dialogues,

vious studies, we evaluate the personal fact consistency

and use the cosine similarity between the Big-5 personality

by consistency score (C. score) [22] which is a textual

intensity of those personality traits and intensity of person-

entailment score computed using a RoBERTa [16] model

ality traits of the utterance. We use the triplet of the ﬁrst

trained on the DNLI dataset [17]. As for personality con-part of personality traits, the second part of utterances, andsistency, inspired by the utterance-level Pearson correla-

the cosine similarity as training data. In the inference stage,
given personality proﬁles and one candidate utterance, thetion used by [10], we compute the Pearson correlation of
dialogue-level personality detected from gold and gener-2） https://www.verywellmind.com/
the-big-five-personality-dimensions-2795422

3） https://huggingface.co/microsoft/DialoGPT-small― 1483 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Table 1

Automatic results. BLEU-1/2 and Distinct 1/2 arescaled by multiplying 100. Pearson correlation is with a 𝑝-value< 0.05.
Setting
ppl. BLEU-1/2 DIST-1/2 C. score Pers. Corr.
Baseline
18.81 10.86/1.99 2.87/28.70 0.332+person. facts 18.52 10.86/2.06 2.83/28.73 0.443

0.386
0.390

Proposed (Baseline +person. facts and personality traits)
3-class
18.51 11.39/2.17 2.83/28.72 0.450
0.646
7-class
18.46 11.11/2.04 2.82/28.75 0.429
0.664
concrete18.56 11.14/2.11 2.73/28.34 0.443
0.638

Table 3

Ablation studies. BLEU-1/2 and DIST-1/2 are scaledby multiplying 100. Pearson correlation is with a 𝑝-value < 0.05.
Setting

ppl. BLEU-1/2 DIST-1/2 C. score Pers. Corr.

Proposed w/o Personality-aware Reranking
3-class 18.51 11.02/1.99 2.80/28.52 0.446
7-class 18.46 10.88/1.98 2.86/28.74 0.426
concrete 18.56 11.01/1.98 2.75/28.11 0.426

0.400
0.387
0.395

only personal facts (+person. facts) achieved better resultsin Coherence and Informativeness. The use of personality
traits does not show signiﬁcantly negative inﬂuence on the

Table 2

Human evaluation for response quality.

Setting

response quality.

Fluency Coherence Infomative. Consist.

+person. facts
Proposed (7-class)

3.85
3.88

2.77
2.69

2.65
2.57

0.17
0.18

Human

4.88

4.78

4.53

0.66

Ablation Studies

To examine the inﬂuence of the reranking of response
candidates on personality consistency, we also conducted
an ablation study as shown in Table 3.

ated responses as personality correlation (Pers. Corr.)
In human evaluation, as for response quality, we choose
Fluency, Coherence, and Informativeness following thesetting of [7], and ask human subjects to annotate 5-point
Likert scales for the three dimensions, where 1 point means
bad quality, 3 points mean moderate and 5 points mean aperfect performance. As for personal fact consistency, apoint within -1,0,1 is assigned to each response, whichmeans contradicted, neutral, or related to given personal
facts.

4.2

4.3

The resultsof the ablation test without reranking (w/o Personalityaware Reranking) allow us to observe the huge impact of
response-candidate reranking on personality consistency.
Compared with previous results of Personal Facts (in Table 1), the model trained with personality traits could not
improve the Personality Correlation. This is probably because the model may have a strong emphasis on personal
facts due to the process of creating the MSC dataset, andnaively incorporating personality traits did not contributeto the personality consistency. At the moment, personality-

Main Results

aware reranking is vital to improve personality consistency.

Table 1 list the automatic evaluation results of the generated responses. We can observe that the personality
traits did not change perplexity, Distinct-1/2, and personal
fact consistency (C. score) (Proposed vs. +person. facts).
As for BLEU scores, we can see a consistent increase of
BLEU-1 with personality traits. As for Personality Correlation, we could observe that the proposed models gain
great increase and the 7-class verbalization of personality
traits achieves the best personality consistency. In short, theproposed models improve the consistency with assigned
personality proﬁles while maintaining other performancewith +person. facts.
Table 2 shows the human evaluation for response quality. Considering the automatic results, we chose the bestperforming models with the 7-class verbalization of per-5

Conclusions
In this study, we explore the use of personality traits inaddition to personal facts in persona-based chat response
generation. We augment the existing MSC dataset [9] withpersonality traits using a personality detector trained on the
Pandora dataset. To fully leverage the predicted personality
traits, we explore personality verbalization and propose apersonality-aware reranking method to pick response candidates with better personality consistency. Experimental
results on the personality-augmented MSC dataset show
an improvement in personality consistency.
As for future work, we plan to design a more eﬀectivemodel structure to fully utilize personal facts and personality traits at the same time.

sonality traits for evaluation. The results show that the 7class setting slightly outperformed +person. facts in termsof Fluency and Consistency, and the model trained with― 1484 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Acknowledgement
This work was partially supported by the special fundof Institute of Industrial Science, The University ofTokyo, by JSPS KAKENHI Grant Number JP21H03494,
JP21H03445, and by JST, CREST Grant Number JPMJCR19A, Japan.

References[1] Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis,
Jianfeng Gao, and Bill Dolan. A persona-based neural conversation model. In Proceedings of the 54th Annual Meeting of the
Association for Computational Linguistics (Volume 1: Long
Papers), pp. 994–1003. Association for Computational Linguis-

tics, August 2016.
[2] Shoetsu Sato, Naoki Yoshinaga, Masashi Toyoda, and Masaru Kitsuregawa. Modeling situations in neural chat bots. In Allyson Ettinger, Spandana Gella, Matthieu Labeau, Cecilia Ovesdotter Alm,
Marine Carpuat, and Mark Dredze, editors, Proceedings of ACL
2017, Student Research Workshop, pp. 120–127, Vancouver,
Canada, July 2017. Association for Computational Linguistics.
[3] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam,
Douwe Kiela, and Jason Weston. Personalizing dialogue agents:
I have a dog, do you have pets too? In Proceedings of the56th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pp. 2204–2213. Associ-ation for Computational Linguistics, July 2018.
[4] Qiao Qian, Minlie Huang, Haizhou Zhao, Jingfang Xu, and Xiaoyan Zhu. Assigning personality/proﬁle to a chatting machinefor coherent conversation generation. In Proceedings of theTwenty-Seventh International Joint Conference on Artiﬁcial
Intelligence, IJCAI-18, pp. 4279–4285. International Joint Con-

ferences on Artiﬁcial Intelligence Organization, 7 2018.
[5] Zhengyi Ma, Zhicheng Dou, Yutao Zhu, Hanxun Zhong, and JiRong Wen. One chatbot per person: Creating personalized chatbots based on implicit user proﬁles. In Proceedings of the 44th
International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’21, p. 555–564.

Association for Computing Machinery, 2021.
[6] Haoyu Song, Yan Wang, Kaiyan Zhang, Wei-Nan Zhang, and Ting
Liu. BoB: BERT over BERT for training persona-based dialogue
models from limited personalized data. In Proceedings of the59th Annual Meeting of the Association for Computational
Linguistics and the 11th International Joint Conference on
Natural Language Processing (Volume 1: Long Papers), pp.

Association for Computational Linguistics, May 2022.
[10] Sougata Saha, Souvik Das, and Rohini Srihari. Stylistic response
generation by controlling personality traits and intent. In Proceedings of the 4th Workshop on NLP for Conversational AI, pp.
197–211. Association for Computational Linguistics, May 2022.
[11] Matej Gjurković, Mladen Karan, Iva Vukojević, Mihaela Bošnjak,
and Jan Snajder. PANDORA talks: Personality and demographics
on Reddit. In Proceedings of the Ninth International Workshop on Natural Language Processing for Social Media, pp.
138–152. Association for Computational Linguistics, June 2021.
[12] Bodhisattwa Prasad Majumder, Harsh Jhamtani, Taylor BergKirkpatrick, and Julian McAuley. Like hiking? you probably
enjoy nature: Persona-grounded dialog with commonsense expansions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp.
9194–9206. Association for Computational Linguistics, November
2020.
[13] François Mairesse and Marilyn Walker. PERSONAGE: Personality generation for dialogue. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pp. 496–503. Association for Computational Linguistics, June

2007.
[14] Wanqi Wu and Tetsuya Sakai. Response generation based on thebig ﬁve personality traits. 2020.
[15] Ao Guo Shota Mochizuki Tatsuya Kawahara Ryuichiro Higashinaka Sanae Yamashita, Koji Inoue. Realpersonachat: A
realistic persona chat corpus with interlocutors’ own personalities. In Proceedings of the 37th Paciﬁc Asia Conference on
Language, Information and Computation, 2023.
[16] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi,
Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, andVeselin Stoyanov. Roberta: A robustly optimized bert pretraining
approach. ArXiv, Vol. abs/1907.11692, , 2019.
[17] Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho.
Dialogue natural language inference. In Proceedings of the 57th
Annual Meeting of the Association for Computational Linguistics, pp. 3731–3741. Association for Computational Linguis-

tics, July 2019.
[18] Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris
Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan.
Dialogpt: Large-scale generative pre-training for conversational
response generation. In ACL, system demonstration, 2020.
[19] Alec Radford, Jeﬀ Wu, Rewon Child, David Luan, Dario Amodei,
and Ilya Sutskever. Language models are unsupervised multitask
learners. 2019.
[20] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.
Bleu: a method for automatic evaluation of machine translation. In
Proceedings of the 40th Annual Meeting of the Association
for Computational Linguistics, pp. 311–318. Association for

167–177. Association for Computational Linguistics, August 2021.
[7] Junkai Zhou, Liang Pang, Huawei Shen, and Xueqi Cheng.
SimOAP: Improve coherence and consistency in persona-based
dialogue generation via over-sampling and post-evaluation. In

Computational Linguistics, July 2002.
[21] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill
Dolan. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of theProceedings of the 61st Annual Meeting of the Association
for Computational Linguistics (Volume 1: Long Papers), pp.

North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110–

9945–9959. Association for Computational Linguistics, July 2023.
[8] Qian Liu, Yihong Chen, Bei Chen, Jian-Guang Lou, Zixuan Chen,
Bin Zhou, and Dongmei Zhang. You impress me: Dialogue generation via mutual persona perception. In Proceedings of the119. Association for Computational Linguistics, June 2016.
[22] Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale
Fung. Personalizing dialogue agents via meta-learning. In Pro-

58th Annual Meeting of the Association for Computational
Linguistics, pp. 1417–1427. Association for Computational Lin-

ceedings of the 57th Annual Meeting of the Association for
Computational Linguistics, pp. 5454–5459. Association for

Computational Linguistics, July 2019.

guistics, July 2020.
[9] Jing Xu, Arthur Szlam, and Jason Weston. Beyond goldﬁsh memory: Long-term open-domain conversation. In Proceedings of
the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 5180–5197.

― 1485 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

A

Appendix

A.1

Human Evaluation Point Assignment
Instruction

We show the concrete point assignment instruction ofhuman evaluation about response quality (Fluency, Coherence, Informativeness) as follows:Flue.

• 1 point: Extremely diﬃcult to understand, withfrequent language issues.
• 2 points: Communication is often unclear, with
noticeable language challenges.
• 3 points: Communication is generally clear, withoccasional disruptions in ﬂuency.
• 4 points: Communication is very clear, with minimal interruptions or language issues.
• 5 points: Extremely ﬂuent communication, very
natural and easy to understand.

Coh.

• 1 point: The text lacks any coherence, making it
extremely diﬃcult to follow.
• 2 points: Coherence is often lacking, and the text
lacks logical ﬂow.
• 3 points: Text is generally coherent, but there aresome logical breaks or disruptions.
• 4 points: The text is very coherent, with strong
logical connections between sentences and ideas.
• 5 points: Text is extremely coherent, with seamless logical ﬂow and strong connections.

Info.

• 1 point: Provides very limited information, offering little to no assistance.
• 2 points: Provides limited information, withsome helpful aspects.
• 3 points: Provides suﬃcient information, but
may lack depth or detailed explanations.
• 4 points: Provides ample information, with moderate depth, and is helpful to the user.
• 5 points: Provides highly detailed and comprehensive information, greatly assisting the user.

― 1486 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).