PEACOK: Persona Commonsense Knowledge for Consistent and Engaging Narratives
Year: 2023

Abstract:
Sustaining coherent and engaging narratives requires dialogue or storytelling agents to understand how the personas of speakers or listeners ground the narrative. Specifically, these agents must infer personas of their listeners to produce statements that cater to their interests. They must also learn to maintain consistent speaker personas for themselves throughout the narrative, so that their counterparts feel involved in a realistic conversation or story. However, personas are diverse and complex: they entail large quantities of rich interconnected world knowledge that is challenging to robustly represent in general narrative systems (e.g., a singer is good at singing, and may have attended conservatoire). In this work, we construct a new large-scale persona commonsense knowledge graph, PEACOK, containing ∼100K human-validated persona facts. Our knowledge graph schematizes five dimensions of persona knowledge identified in previous studies of human interactive behaviours, and distils facts in this schema from both existing commonsense knowledge graphs and largescale pretrained language models. Our analysis indicates that PEACOK contains rich and precise world persona inferences that help downstream systems generate more consistent and engaging narratives.1

heading: 1 Introduction
text: Interlocutors or storytellers in narrative scenarios often exhibit varying behaviours, which are affected by their own diverse personas, but also the personas of the counterparts they are interacting with. For example, an adventurous architect may be interested in talking about outdoor explorations with his friends who have similar hobbies, but may prefer to discuss architectural design ideas with his
*Equal contribution. †Corresponding author. 1We release our data and code to the community at https:
//github.com/Silin159/PeaCoK
colleagues at work. Narrative systems must know when such behaviours should be exhibited, requiring them to learn and represent the rich personas of characters based on self-introductions, biographies and other background profiles.
This goal of modeling diverse persona attributes is at the heart of research in the areas of personagrounded dialogue (Zhang et al., 2018; Zhong et al., 2020; Xu et al., 2022), story generation (Chandu et al., 2019; Zhang et al., 2022) and narrative understanding (Brahman et al., 2021). However, the complex nature of real-world personas, which involve rich world knowledge, and the countless ways in which they might interact, is challenging to reliably learn purely from data. For instance, as shown in Figure 1, a singer preparing an album may have studied music at university at one point, which would allow them to share their experience with a student majoring in composition, who may study music as a daily routine.
Prior work takes first steps at improving the persona knowledge representations available in narrative systems. Mazare et al., 2018 extract self-
6569
comments from Reddit websites to expand the scale of background persona profiles that can be used in downstream narrative settings. However, their collected profiles are fragmented and ignore the interconnections between personas that govern interactions. Meanwhile, Majumder et al., 2020 use knowledge generators (Bosselut et al., 2019) to expand the persona profiles with commonsense inferences, but these commonsense expansions are limited to general social commonsense (Hwang et al., 2021), and do not form a systematic personacentric knowledge frame. Consequently, the lack of world-level persona commonsense knowledge resource hinders progress in learning the systematic persona representations necessary to sustain consistent and engaging narratives.
In this work, we propose a Persona-grounded Commonsense Knowledge graph (KG), PEACOK, which represents world-level persona knowledge at scale. Building off the persona concept initially proposed in human-computer interaction (Cooper, 1999; Mulder and Yaar, 2006; Cooper et al., 2007) and on behaviour analysis literature for human leisure conversations (Dunbar et al., 1997), we define a persona frame that formalizes five common aspects of persona knowledge: characteristics, routines and habits, goals and plans, experiences, and relationships. Using this knowledge frame, we construct a large-scale graph of persona commonsense knowledge by extracting and generating persona knowledge from both existing hand-crafted commonsense KGs and large-scale pretrained language models (LMs). We validate the knowledge graph via a joint human-AI majority voting scheme that integrates large pretrained LMs into the loop of crowdsourcing, and efficiently mediates the disagreements between human annotators.
Our resulting KG, PEACOK contains ∼100K high-quality commonsense inferences (i.e., facts) about personas whose connectivity in the KG reveals countless opportunities to discover common ground between personas. A neural extrapolation from the KG (Hwang et al., 2021) also shows that PEACOK’s annotated personas enable the development of effective persona inference generators. Finally, the extended knowledge provided by PEACOK enables a downstream persona-grounded dialogue system to generate more consistent and engaging responses in conversations, particularly when more interconnections between the interlocutor personas are found in PEACOK.

heading: 2 Related Work
text: Commonsense Knowledge Graphs Commonsense KGs such as ConceptNet (Liu and Singh, 2004; Speer et al., 2017), ATOMIC (Sap et al., 2019a), ANION (Jiang et al., 2021) and ATOMIC2020 (Hwang et al., 2021) are widely used in NLP applications that involve integrating implicit world knowledge, e.g., question answering (Talmor et al., 2019; Sap et al., 2019b; Chang et al., 2020; Shwartz et al., 2020) and text generation (Lin et al., 2020). However, despite the importance of persona knowledge in modeling human behavior — a crucial component for building reliable narrative systems (Zhang et al., 2018; Chandu et al., 2019) — no commonsense KG explicitly focuses on representing human persona knowledge. We present PEACOK to open the field of developing commonsense knowledge graphs around personas. Persona-Grounded Narratives Integrating personas to improve consistency and engagement of narratives is an important goal in dialogue (Song et al., 2020; Liu et al., 2020) and storytelling (Chandu et al., 2019; Zhang et al., 2022) systems. One representative work that greatly contributed to the development of faithful persona emulation, PERSONA-CHAT (Zhang et al., 2018), constructs a crowdsourced dialogue dataset by asking participants to perform conversations based on their assigned persona profiles — five statements of selfintroduction. More recent work improves persona modeling in narrative systems by generating persona profiles from online resources (Mazare et al., 2018), training persona detectors (Gu et al., 2021) and predictors (Zhou et al., 2021), and distilling persona knowledge from commonsense inference engines (Majumder et al., 2020). However, while these works align characters in narratives with persona profiles, they only implicitly model the areas of interaction between personas. In contrast, PEACOK explicitly represents interconnections between persona profiles, enabling persona interaction modeling in narrative systems.

heading: 3 PEACOK Knowledge Frame
text: To construct a systematic representation of persona knowledge, we distill five common aspects of personas from classical persona definitions.
In the field of human-computer interaction, a persona is a fictitious example of a user group that is conceptualized to improve interactive design in areas such as marketing, communications, and
service product development (Soegaard and Dam, 2012). From the perspective of goal-directed design (Cooper, 1999; Cooper et al., 2007), personas encapsulate user needs and goals when interacting with a product, along with their intrinsic character traits and past experiences (Randolph, 2004) that contextualize the interaction. Using these attributes of goals, traits, and experiences as the foundation of personas, we also leverage prior studies in human conversational behaviour that explore which topics of conversation are often broached in relaxed human social interactions. After conducting observational studies, Dunbar et al. (1997) categorized the topics of human conversations into bins: personal relationships (i.e., experiences or emotions rising from social interactions), personal experiences (i.e., factual events or circumstances experienced by a person), future activities (i.e., arrangements of meetings or events), leisure activities (e.g., hobbies), interests (e.g., culture, politics, religion), and work (e.g., daily routines).
To select our persona dimensions, we discard certain controversial categories from the above studies (i.e., culture, politics, and religion), as well as temporary dimensions of persona (i.e., emotion, which is well covered by prior work; Gupta et al., 2017; Chatterjee et al., 2019; Rashkin et al., 2019). Our final persona frame consists of five relations for each persona, each with multiple attributes attached to it. We describe the five relations below:
Characteristics describe an intrinsic trait, e.g., a quality or a mental state, that the persona likely exhibits. For example, as shown in Figure 1, good at singing describes a talent of a singer, which is one of the singer’s characteristics.
Routines or Habits describe an extrinsic behaviour that the persona does on a regular basis,
e.g., a singer may regularly write songs.
Goals or Plans describe an extrinsic action or outcome that the persona wants to accomplish or do in the future, e.g., a singer may aim to win a Grammy award some day.
Experiences describe extrinsic events or activities that the persona did in the past. For instance, a singer may have studied music at college.
Relationships encode likely interactions of the persona with other people or social groups. Note that this relation can be overlapped with other relations in PEACOK. For example, a singer may want to have more fans, which connotes a relationship between singer and fans, but also a future goal or plan of singer.

heading: 4 PEACOK Construction
text: We use our persona frames to construct a knowledge graph of persona commonsense where personas are treated as head entities in the graph, frame relations constitute edge type relations, and attributes are tails in a (head, relation, tail) structure. Then, we devise a three-step procedure to construct the frames that make up PEACOK, as shown in Figure 2. First, we search existing commonsense KGs to select entities that can serve as head personas. Then we query these KGs and prompt pretrained LMs to collect tail attributes that are potentially associated with the personas via the five relations defined in Sec. 3. Finally, we use crowdsourcing with large LMs in the loop to classify whether these persona inferences are valid.

heading: 4.1 Persona Selection
text: We select entities that can represent head personas using ATOMIC2020 (Hwang et al., 2021), a common-
sense KG covering knowledge about physical objects, daily events, and social interactions. We assume that entities related to personas should be about human beings, rather than other animals or non-living objects. Therefore, we first over-sample living entities from ATOMIC2020 which have animated behaviours, by extracting head entities that possess the CapableOf relation (i.e., are capable of doing something), e.g., an actor who is capable of performing, as shown in Figure 2. Then we filter out non-human beings in our extracted living entities, by removing entities that appear in the Animal Appendix of Wiktionary.2 We also manually filter out other inappropriate entities which are too generic (e.g., man) or unrealistic (e.g., devil).
This initial procedure provides us with a diverse collection of initial coarse personas (e.g., actor, singer). To enlarge our persona set with finegrained personas (e.g., actor who acts in movies vs. actor who acts in plays), we collect additional persona candidates using three types of event-based entities derived from our initial persona set: a) entities containing the initial persona in a more complex context, e.g., X becomes an actor associates with the process of becoming an actor, rather than being an actor, b) entities that can be linked to the initial persona through the ATOMIC2020 CapableOf relation, e.g., X acts in play is linked to actor, and c) entities that are returned by Sentence-BERT retrieval (Reimers and Gurevych, 2019) for the initial persona, e.g., X becomes a movie star. For the latter two types of derived event-based entities, we prompt InstructGPT-3 (Ouyang et al., 2022) to filter out extended personas which do not entail their initial seed persona, e.g., X wants to be a lawyer is not entailed by a X is a judge, as X would already be a lawyer if they were a judge. Finally, we extract 3.8K personas, which are converted to persona statements and integrated in PEACOK.3

heading: 4.2 Attribute Induction
text: We derive the attribute knowledge for our collected set of head personas using both hand-crafted KGs and large language models pretrained on natural language corpora (which contain many narratives with implied persona information).
2https://en.wiktionary.org/wiki/Appendix: Animals
3Details regarding head entity conversion and the prompt for InstructGPT-3 entity filtering are in Appendix A.
KG-Based Approach We first select 10 commonsense relations in ATOMIC2020 KG which are potentially related to persona knowledge.4 For each persona entity selected in Sec. 4.1, we extract potential attributes by taking 1-hop inferences of the persona along one of our selected ATOMIC2020 relations. As ATOMIC2020 may have a limited coverage of commonsense knowledge, we also use a knowledge model, COMET (Bosselut et al., 2019), pretrained on ATOMIC2020, to generate potential attributes of each persona as well. We append each selected ATOMIC2020 relation to the persona entity, and feed each persona-relation pair to COMET to generate 5 new potential attributes.
LM-Based Approach To mine more persona knowledge implied in natural language corpora, we also prompt InstructGPT-3 to generate new persona attributes. Using each of the five relations defined in Sec. 3, we prompt InstructGPT-3 with our persona statements and generate 5 new attributes for each relation. For example, for the Experience relation, we instruct the model to guess distinctive activities that an individual fitting the persona might have done in the past. We adapt InstructGPT3 using 5 manually created in-context examples for each type of relation.5

heading: 4.3 Relation Classification
text: Once we have a large-set of initial candidate knowledge tuples to compose our persona frames, we use crowdworkers from Amazon Mechanical Turk to verify every collected relationship consisting of a head persona, relation, and tail attribute. Because we observe that a fine-grained labeling schema can
4Appendix A lists our selected 10 ATOMIC2020 relations and their descriptions.
5We provide our instruction and few-shot examples for InstructGPT-3 attribute generation in Appendix A.
Intrinsic
Extrinsic
Past PresentFuture
One-Off
Regular
Characteristic
ExperienceGoal or Plan Experience
Routine or Habit
Main Relation
help workers better distinguish different relations and yield more precise annotations, we task workers with classifying fine-grained underlying features of the relations. For each attribute, we independently ask two workers to judge whether it describes: a) an intrinsic or extrinsic feature of the persona, b) a one-off or regular attribute of the persona, c) a past, present or future attribute of the persona, d) an attribute of only the persona itself, or describing the persona’s relationship with others (interactivity). Finally, for each attribute in the persona frame, we ask workers whether the attribute is distinctively associated with the persona or generically associated with many potential personas (distinctiveness). As an example, in Table 1, we see that get tips from customers is distinctively associated as a common routine of a waiter. Meanwhile, get better is a generic attribute that would not be strongly associated with runner, as many personas can have the goal of self-improvement.
We follow Figure 3 to map the first three dimensions of the feature labels to one of the first four relations defined in Sec. 3, which we define as the main relation label of each persona-attribute pair. The other two dimensions of feature labels, i.e., interactivity (containing the fifth relation in Sec. 3) and distinctiveness, are defined as two additional relation labels. If a worker judges that an attribute is not associated with the persona at all, we instead ask the worker to label the relation as Not Persona.
Majority Voting with LM in the Loop To mediate the disagreements between two crowdworkers without introducing more human labour (i.e., a third worker), we use InstructGPT-3 and the two workers in a majority vote scheme to determine the final relation labels of some persona-attribute mappings. For each attribute collected in Sec. 4.2, we prompt InstructGPT-3 to produce additional labels for the relation of the attribute with respect to the persona. We prompt InstructGPT-3 on three label-
ing tasks corresponding to the three dimensions of relation labeling schema shown in Figure 3. For the main dimension, we set the labeling classes to include the four main relation labels, and also a negative class (No Persona) indicating that the attribute is not a persona attribute or too generic (e.g., living a happy life). We prompt InstructGPT-3 with 2 examples of each class for the main dimension (i.e., 10 manually labeled in-context examples).
For the interactivity and distinctiveness dimensions, we ask InstructGPT-3 to predict a binary label for each dimension. For these predictions, we provide InstructGPT-3 with 4 examples of each class (i.e., 8 manually labeled in-context examples for each dimension).6
For each dimension of the relation labeling schema shown in Figure 3, we determine the final label as the majority label given by InstructGPT-3 and the two workers. We set the final label as Controversial if no unique majority label is found, e.g., InstructGPT-3 and two workers all give different labels. Finally, each persona-attribute pair forms a persona fact triple with its annotated relation labels in PEACOK. Table 1 shows some examples of PEACOK facts.7
6We include our designed instruction and few-shot examples for InstructGPT-3 relation labeling in Appendix A.
7We list more PEACOK persona facts in Appendix B.

heading: 5 PEACOK Analysis
text: Our statistics of the final PEACOK relations are shown in Table 2, where we construct 102,097 facts with valid persona knowledge inferences. We stratify PEACOK statistics based on the two persona collection approaches (KG-based and LM-based) described in Sec. 4.2. We find that the KG-based distillation (which extracts information initially annotated by human workers) results in more imbalanced persona knowledge. A large proportion (∼57%) of Routine or Habit relations dominate the extracted persona relations, and there are fewer Relationship and Distinctive facts, as well. This indicates that hand-crafted social commonsense KGs contain a narrower view of real-world persona knowledge, highlighting the importance of also distilling a balanced set of persona knowledge from large pretrained LMs. However, the repurposed knowledge from the KG was initially written by humans, and contains diverse persona inferences less likely to be generated by LLMs.
Persona Interconnectivity In addition to containing diverse knowledge from multiple sources, PEACOK also contains interesting interconnections among personas, which potentially indicate engaging points of common ground for characters of narratives. For example, as shown in Figure 1, a professional singer’s experience of studying music at college is also the routine of a music-major student, which shows a common topic for these two persona to discuss. Among 40,665 distinctive attributes in PEACOK, we find that 9,242 attributes
are connected to two or more personas, forming 239,812 bridges, i.e., pairs of personas connected via a shared common attribute.8

heading: 5.1 Attribute Disagreements
text: One of our innovations in this work is to introduce InstructGPT-3 as a third annotator to resolve disagreements between human annotators via majority voting. We analyze the disagreements between workers across the annotations as in Table 3, and observe that labels from InstructGPT-3 effectively solve many disagreements between human workers. For the main dimension labeling, ∼73% of the disagreements are solved by adding InstructGPT3 as a third annotator. However, ∼27% of labels remain Controversial when both annotators and GPT3 all disagree in different ways. These controversial labels enable further research on the ambiguities in real-world persona types and the potential stereotypes in persona judgments. In the interactivity and distinctiveness dimensions where the labeling schema is binary, disagreements of workers are fully solved by the majority voting with InstructGPT-3, though ambiguous cases may still remain.
Expert Study However, one question that naturally arises, when employing a majority voting with InstructGPT-3 in the loop, is whether this classification decision remains accurate. To evaluate this, two experts from our research group manually re-annotate the relations of 825 persona facts
8The number of bridges grows combinatorially with the number of personas sharing an attribute.
in PEACOK, and then compare their annotations to the majority voting results to check the voting accuracy. The 825 persona facts consist of 275 samples from each of the three PEACOK subsets where majority voting is employed, that is, when InstructGPT-3 agrees with one of the workers but not the other, and when both workers agree with each other but not with InstructGPT-3. Experts are required to pass a qualification test by performing 20 test annotations correctly. Furthermore, in the case of disagreements (7% of cases), a third expert re-checked the annotations of the two experts and resolved the disagreement cases.9
Table 4 presents the accuracy and F1 of the majority voting results, compared to the re-annotations from experts as ground truth labels. We stratify the results into two cases: the two workers disagree with each other but InstructGPT-3 agrees with one of them, and both workers agree with each other but not with InstructGPT-3. We observe a high agreement between the experts and the majority vote, with an average accuracy and F1 of 0.874 and 0.865, respectively. These results validate majority voting with InstructGPT-3 in the loop, showing that InstructGPT-3 serves as a reliable third annotator when disagreements arise. Moreover, the integration of InstructGPT-3 in the verification loop costs less in terms of time and money compared to adding more human annotators.
However, we note that InstructGPT-3 is not a panacea on its own. While the model effectively resolves worker disagreements, we find that its individual predictions are only correct with ∼60% macro-F1, which is far from the ∼85% macro-F1 with majority voting, indicating that not all PEACOK persona relations are known by large-scale language models, and that human crowdsourcing
9To ensure fairness, the experts do not see the relation labels predicted by crowdworkers and InstructGPT-3.
is still necessary to ensure data quality.

heading: 6 Generalizing Persona Knowledge
text: Following the neural KG analysis method proposed by Hwang et al., 2021, we assess whether PEACOK could be used to train inference generators that hypothesize persona knowledge. We train a BART-based (Lewis et al., 2020) COMET (Bosselut et al., 2019) knowledge generator (COMET-BART) based on a held-out training set (∼65K facts) of PEACOK, where the model learns to generate the tail attribute of a fact given its head persona and relation. We evaluate COMET-BART on a test set from PEACOK containing 3030 facts with unique head-relation combinations. As baselines, we compare to a few-shot GPT-3 (Brown et al., 2020) that uses 5 randomly sampled training facts (with same relation as the testing fact) to prompt the tail knowledge generation and a zero-shot GPT-3.5 (text-davinci-003) baseline model. These baselines compare PEACOK training to larger LMs that use both in context-learning and instruction tuning. We conduct both automatic and human evaluations on the knowledge generators, with results shown in Tables 5 and 6.10
Compared to few-shot GPT-3, COMET-BART trained on PEACOK achieves overall better automatic evaluation results on various NLG metrics, despite being a much smaller model.11 In the human evaluation, we find that facts generated by COMET-BART receive a high acceptance rate by
10We include more implementation details of our neural KG analysis in Appendix C.
11GPT-3 and COMET-BART have 175B and 440M parameters, respectively.
crowdworkers for plausibility, slightly beating fewshot GPT-3. We also find that zero-shot GPT-3.5 model, although more advanced than the GPT-3 baseline model, scores, on average, ∼15.3% and ∼9.3% lower than COMET-BART in terms of automatic metrics and human acceptance, respectively. All above results indicate that PEACOK can serve as a reliable persona knowledge base, which enables light-weight LMs to learn knowledge generation capabilities comparable to large-scale LMs.

heading: 7 Enhancing Dialogue Systems
text: As our knowledge graph PEACOK covers rich world persona knowledge, we validate whether acccess to this knowledge enables better persona modeling in downstream narrative systems. Using PEACOK, we augment a persona-grounded dialogue model P2BOT (Liu et al., 2020) developed on the ConvAI2 (Dinan et al., 2020) PERSONA-CHAT (Zhang et al., 2018) dataset. We link facts from PEACOK to PERSONA-CHAT dialogues, thereby extending P2BOT’s persona perception and augmenting its dialogue response generation.12
We evaluate our models based on both original and revised interlocutor profiles provided in the ConvAI2 PERSONA-CHAT dataset, and measure the perplexity (PPL), word-level F1, and cumulative 4-gram BLEU (Papineni et al., 2002) of the generated responses compared to the references. We also follow ConvAI2 to measure Hits@1, i.e., the probability that real response is ranked the high-
12Downstream application details are in Appendix D.
est by the model among 20 candidates.
Persona Knowledge Linking We link PEACOK knowledge to interlocutors based on both their PERSONA-CHAT profiles and their utterances in the dialogue. For each interlocutor, we extract all statements in their profile, as well as first-person sentences in their utterances. Then, we follow a commonsense fact linking benchmark, ComFact (Gao et al., 2022), to link relevant facts from PEACOK to each extracted statement or sentence. We remove linked facts that are labeled as Generic in the distinctiveness dimension, i.e., have little effect on distinguishing this persona from others.
For each interlocutor, we randomly sample 5 PEACOK facts that are linked to their PERSONACHAT profile,13 and convert them into natural language statements to form their extended persona knowledge.14 Our augmented model is denoted as P2BOT + PEACOK. To compare PEACOK’s persona-centric knowledge augmentations with general commonsense augmentations, we also evaluate another baseline model P2BOT + ATOMIC2020, where we follow Majumder et al., 2020 to extend interlocutor personas with 5 randomly sampled commonsense inferences from the COMET-ATOMIC2020 model (Hwang et al., 2021).
Results In Table 7, we show that P2BOT + PEACOK significantly outperforms P2BOT on PPL
13Due to the model capacity limitation of the baseline P2BOT, we only sample a subset of linked PEACOK facts as the extended persona knowledge for each interlocutor.
14Fact preprocessing details are in Appendix C and D.
and Hits@1,15 and has comparable F1 and BLEU scores. Compared to P2BOT+ ATOMIC2020, P
2BOT + PEACOK also demonstrates a clear improvement across all metrics, indicating the importance of augmenting narrative systems with persona-grounded commonsense knowledge.
Human Evaluation Automatic metrics are not fully reliable for evaluating dialogue systems (Liu et al., 2016; Novikova et al., 2017), so we also conduct human evaluations on the dialogue responses. We make pairwise comparisons between P2BOT + PEACOK and other baseline models, based on their generated responses to 200 randomly sampled dialogue histories (100 each with original and revised PERSONA-CHAT profiles). Two expert annotators from our research group manually compare four aspects of the response generation quality: fluency, whether the response is fluent and understandable, consistency, where the response is consistent with the dialogue history, engagement, whether the response is engaging and interesting, and persona expression, whether the response demonstrates persona information related to the interlocutor’s profile. To ensure the fairness and reliability of our human evaluation, similar to Sec. 5.1, we require each expert to pass a qualification test on 10 pairwise comparisons, and also include a third qualified expert to re-check the evaluation results. We note that both expert annotators do not see the source model from which each response is generated.
The human evaluation results in Table 8 show that P2BOT + PEACOK generates more consistent and engaging dialogues compared to other neural baselines, demonstrating that persona commonsense knowledge is a key contributor to the conversation consistency and engagement. However, P2BOT + PEACOK still has room for improvement compared to human performance.
Perhaps most interestingly, though, we find that PEACOK’s impact on the consistency and engagement of dialogues is most pronounced when there are interconnections between the personas of the interlocutors. We stratify the pairwise comparison between P2BOT + PEACOK versus P2BOT from Table 8 based on the overlap of the two interlocutors’ linked PEACOK knowledge. In Table 9, we show the results of this stratification across the cases where the interlocutors have 0, 1 or more than 1 shared attributes. Specifically, we find that
15significant at p<0.02 and p<0.01, respectively, in paired sample t-test
the winning rates of P2BOT w/ PEACOK on dialogue consistency and engagement increase as the overlap of the two speakers’ linked PEACOK personas becomes larger, demonstrating that more connections between interlocutors leads to more consistent and engaging conversations, and highlighting the importance of learning interconnected world persona knowledge in narratives.

heading: 8 Conclusion
text: In this work, we propose a persona commonsense knowledge graph, PEACOK, to complement the real-world picture of personas that ground consistent and engaging narratives. PEACOK consists of ∼100K persona commonsense inferences, distilled from existing KGs and pretrained LMs, across five dimensions of persona knowledge identified in prior literature on human interactive behaviours. Our analysis and experiments demonstrate that PEACOK contains high-quality inferences whose connectivity provides many instances of common ground between personas, improving the consistency and engagement of downstream narrative systems.
Limitations
We acknowledge a few limitations in this work. First, PEACOK cannot be comprehensive. Persona knowledge is very broad and our resource cannot cover all dimensions of personas, nor all attributes of these dimensions. We select five dimensions of personas that we found salient from background literature in human interaction, and we distill attributes for these dimensions from ATOMIC2020, COMET and InstructGPT-3. These resources, while rich in knowledge, only represent a subset of possible background resources for the construction
of PEACOK(among other KGs and pretrained language models). Furthermore, the primary language of these three resources is English, making PEACOK a solely English resource. Finally, in downstream narrative experiments, the usage of our augmented persona knowledge is constrained by the capacity of baseline model, which leaves for future work the exploration of downstream persona knowledge augmentation on a larger scale.
Ethics Statement
Our work is approved by our institution’s human research ethics committee to conduct human-centric or ethics-related experiments, e.g., crowdsourcing and human evaluations. Topic-wise, our research develops a knowledge graph of commonsense knowledge about personas to augment understanding of characters and their interactions in diverse narratives. Given that some of the attributes are extracted from previous KGs or generated by LMs, we cannot guarantee our knowledge graph does not contain attribute alignments with negative connotations that could provide undesired information to a downstream system. However, we took the following steps to mitigate this effect. First, the set of personas we include in PEACOK was manually filtered to not include stereotypical and harmful roles, thereby limiting the negative associations of the personas themselves. Second, we explicitly prompted the LM to generate optimistic attributes about personas, which has been shown in prior work to reduce the toxicity of outputs (Schick et al., 2021). Finally, each attribute in PEACOK is explicitly validated by two human workers for toxicity, providing a final opportunity for workers to flag problematic content. However, we acknowledge that none of these safeguards are perfect, as language models may still produce toxic outputs and annotators may have differing opinions on what constitutes toxic content (Sap et al., 2022).

heading: Acknowledgements
text: We thank Gail Weiss, Syrielle Montariol, Graciana Aad and Mete Ismayil for reading and providing comments on drafts of this paper. We also gratefully acknowledge the support of Innosuisse under PFFS-21-29, the EPFL Science Seed Fund, the EPFL Center for Imaging, Sony Group Corporation, and the Allen Institute for AI.

heading: A PEACOK Construction Details
text: Head Persona Selection Table 10 shows our designed prompt for InstructGPT-3 head persona filtering described in Sec. 4.1. We preprocess our extracted human and event-based entities to make them fit into the prompt. Specifically, we fill each human entity into the template “I am a(n) ___.” to convert it into a natural language sentence. We also replace the general token “PersonX” in each evenbased entity with the pronoun “I”, and lemmatize the third person singular in its verbs. To build the integral statement (final head persona in PEACOK) that combines a human entity with each of its derived event-based entity, we instead replace the even-based entity’s “PersonX” token with “who”, and then append it to the converted sentence of its human entity. Note that for each human entity itself or event-based entity that contains a human entity (i.e., the first type of derived event-based entities),
we directly include its converted sentence alone as one of the head persona statements in PEACOK.
KG-Based Tail Attribute Collection We use ATOMIC2020 as the background resource for KGbased tail attribute collection described in Sec. 4.2. This advanced KG contains 1.33M general social commonsense inferences based on a rich variety of entities, including 0.21M inferences about physical objects, 0.20M inferences centered on daily events, and other 0.92M inferences based on social interactions. Table 11 lists the 10 ATOMIC2020 relations that we consider as potentially related to persona knowledge, which we use to query tail attributes from ATOMIC2020 KG and COMET, based on each original entity collected in the head persona selection (Sec. 4.1).
LM-Based Tail Persona Collection Tables 12 and 13 show the prompts provided to InstructGPT3 tail to generate attributes for each persona (Sec. 4.2), based on each converted persona statement derived from the head persona selection (Sec. 4.1). We use 2 different sets of in-context examples to prompt the InstructGPT-3 generation. Specifically, examples under the Simple Head Personas block are used for head statements converted
from human entities or event-based entities that directly contain human entities (the first type of derived event-based entities). While examples under the Complex Head Personas block are used for event-based entities that do not contain human entities (the second and third types of derived eventbased entities), where the event-based entity is combined with its source human entity to form a integral statement.
Crowdsourcing Relation Classification We conduct a worker qualification for our persona relation classification described in Sec. 4.3. To select native English speakers, we focus on the group of workers whose locations are in the USA. We test workers with 10 head personas, each with 2 tail personas (i.e., totally 20 head-tail persona pairs), and select workers who can reasonably annotate 18 or more (i.e., ≥90%) relations between the given head and tail personas. Finally, 72 out of 207 workers are selected as qualified. We pay each worker $0.30 for doing every 5 annotations. The average hourly wage for each worker is about $18.00, which is in the acceptable range of hourly wage suggested by Amazon Mechanical Turk. Figure 4 and 5 shows the screenshots of our acceptance policy, privacy policy, and task instruction used for crowdsourcing.
Majority Voting Table 14, 15 and 16 show the prompts provided to InstructGPT-3 to label relations as the majority vote among worker disagreements (Sec. 4.3). Similar to the InstructGPT-3 tail attribute generation (Sec. 4.2), we use 2 different sets of in-context examples to handle the complexity differences in the head persona statements. The verbalizers that we use for each labeling class are characteristic, routine, plan, experience & no in the main dimension; relationship & self in the interactivity dimension; and distinctive & generic in the distinctiveness dimension.

heading: B PEACOK Analysis Details
text: Table 17 shows the fine-grained statistics of persona relations included in PEACOK. Each PEACOK fact’s relation consists of three dimensions of labels as shown in Figure 3. The combinations of Routine or Habit, Self and Distinctive labels is the most frequent relation in PEACOK, which implies that individual daily activities might be the most common topic involved in human interactions. Table 18 shows several examples of persona facts in PEACOK, which showcases our knowledge graph’s rich commonsense inferences on persona-grounded knowledge.

heading: C Neural KG Analysis Details
text: Fact Preprocessing We develop neural knowledge generator based on the PEACOK facts whose relations are labeled as Distinctive in the third (distinctiveness) dimension. We preprocess these distinctive PEACOK facts to facilitate knowledge generation. In particular, we follow Table 19 to map each fact’s relation into a textual description, and then concatenate it with the fact’s head and tail personas. If the relation is labeled as Relationship in the second (interactivity) dimension, we also append its description in Table 19 to the fact’s maindimension label description, i.e., one of the other four descriptions in Table 19. For example, (I am a waiter, Characteristic and Relationship, skilled at customer service) is converted into I am a waiter,
here is my character trait related to other people or social groups, skilled at customer service.
Evaluation Details We split our preprocessed facts into three sets, with size 64853, 8913 and 14112 for training, validation and testing, respectively. Note that the three sets of facts do not have overlapped head personas with each other. We evaluate tail persona generation on the 3030 unique head-relation combinations in the testing set, with the 14112 gold tail personas serving as references. Several NLG metrics are adopted for the automatic evaluation, including cumulative 4- gram BLEU (Papineni et al., 2002), ROUGE-L (Lin, 2004), METEOR (Banerjee and Lavie, 2005) and SkipThoughts (Kiros et al., 2015). For human evaluation, we use the same group of workers qualified for PEACOK relation classification described in Appendix A. Each fact with generated tail is evaluated by one Amazon Mechanical Turk worker, following our instruction shown in Figure 6. We pay each worker $0.20 for evaluating every 5 facts, which keeps similar hourly wage as compared to PEACOK relation classification.
Model Training We use Kogito (Ismayilzada and Bosselut, 2022) toolkit to train the COMET-BART
knowledge generator, with the default hyperparameters suggested by the toolkit. One NVIDIA TITAN X Pascal GPU is used to train the model for 7 epochs, which costs about 1 hour to get the highest ROUGE-L score on the validation set. For the 5-shot GPT-3 generation, we prompt the davinci endpoint with default hyperparameters suggested by the OpenAI GPT-3 platform.
We also train a DeBERTa (He et al., 2020) discriminator to re-rank the facts generated by COMET-BART and GPT-3. For each training fact, we create one negative example by replacing its tail persona with a randomly sampled one from another training fact, which have a different head persona but same relation. We train the DeBERTa model to discriminate true facts versus negative samples based on a binary classification loss, with hyperparameters suggested by the ComFact (Gao et al., 2022) benchmark. Four NVIDIA TITAN X Pascal GPUs are used to train the model for 6 epochs, which costs about 21 hours to get the highest F1 score on the validation set. Finally, for both COMET-BART and GPT-3, we evaluate their top-1 of 5 generated facts re-ranked by our DeBERTa discriminator, with their default decoding methods, i.e., beam search for COMET-BART and nucleus
sampling for GPT-3, with 1.0 top-p sampling rate and 0.9 temperature value.

heading: D Persona Dialogue Agent Implementation Details
text: Our downstream dataset, ConvAI PERSONA-CHAT, contains 17878 and 1000 crowdsourced dialogues for training and validation, while 1015 testing dialogues are not public. In each dialogue sample, two speakers are pre-given their own persona profiles, i.e., four or five sentences of self-introductions, to conduct conversations. Based on the persona profiles, P2BOT uses a reinforcement learning (Sutton et al., 1999) approach to build mutual persona perception between speakers, which enhances the quality of personalized dialogue generation.
Persona Knowledge Linking We first link candidate facts from PEACOK via the pattern matching and embedding similarity heuristics introduced in ComFact, and then use a DeBERTa (He et al.,
2020) entity linker trained on ComFact to select relevant facts from the candidates. We use the DeBERTa entity linker (instead of fact linker) to check the relevance of each fact’s head and tail personas independently, without considering their in-between relations. This is because the DeBERTa fact linker from ComFact is trained on ATOMIC2020 relations, which cannot well identify the new relation sets of PEACOK. We link persona facts from PEACOK whose head and tail personas are both relevant to the extracted PERSONA-CHAT statement or sentence. We also include an additional set of persona facts which only have relevant tail, since the high-level head personas are not always revealed in the dialogue. Similar to the fact preprocessing described in Appendix C, we convert each linked persona fact into a natural language statement, by first following Table 19 to map each fact’s relation into a textual description, and then concatenate it with the fact’s head and tail personas.
Model Training We train our knowledge augmented models (i.e., P2BOT w/ PEACOK and P2BOT w/ ATOMIC2020) with the same hyperparameters and early stopping settings as the original P2BOT model. Two NVIDIA TITAN X Pascal GPUs are used, which takes about 20 hours to get convergence (early stopped) on the validation set.
Human Evaluation For each pairwise comparison, we show the experts two responses generated by different models, with the gold dialogue history and the interlocutor persona profiles. We ask the experts to compare the two responses with regard to our four evaluation aspects (i.e., fluency, consistency, engagement and persona expression). To guide the experts to better understand our evaluation criteria, we interpret each evaluation aspect as a specific question, as shown in Table 20. Downstream Dialogue Generation Examples Table 21 presents an example of our downstream dialogue generation results, where we show the response generated by each model along with the dialogue history and the persona profile associated with the speaker of the response. The linked PEACOK knowledge (i.e., fact) that involved in the response generation is also presented. We find that the involved PEACOK fact help identify a potential role of the speaker, i.e., breeder inferred from milking cows and farmland, and also explain the speaker’s persona of having a pet canine, i.e., dog. Therefore, compared to other baseline models, P2BOT w/ PEACOK generates a more consistent and engaging response, which is well associated with the counterpart’s last utterance in the dialogue history, and also simultaneously conveys a related persona of the speaker.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Refer to Limitations
A2. Did you discuss any potential risks of your work? Not applicable. No obvious potential risk is observed.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Refer to Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? All sections, our paper creates a new knowledge base.
B1. Did you cite the creators of artifacts you used? Not applicable. Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Our knowledge base is not public yet.
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Refer to Appendix A: Claim of Usage
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? The privacy protection is guaranteed by the data collection platform we use, i.e., Amazon Mechanical Turk.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Refer to Section 3, Limitations, Appendix B
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Refer to Appendix B and Appendix C – Evaluation Details
C 3 Did you run computational experiments? Refer to Section 5.3 and Section 6
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Refer to Footnote 9, Appendix C – Model Training and Appendix D – Model Training
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Refer to Appendix C – Model Training and Appendix D – Model Training
7 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? We follow the experimental settings of previous works, which did not provide related statistics for making comparisons.
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Refer to Section 5.3, Section 6, Appendix C and Appendix D
D 3 Did you use human annotators (e.g., crowdworkers) or research with human participants? Refer to Section 4.3, Section 5.3 and Section 6.2
3 D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Refer to Appendix A – Crowdsourcing Relation Classification, and Appendix C – Evaluation Details
3 D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Refer to Appendix A – Crowdsourcing Relation Classification
3 D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Refer to Appendix A – Crowdsourcing Relation Classification
3 D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Refer to Ethics Statement
3 D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Refer to Appendix A – Crowdsourcing Relation Classification

heading: Future Work
text: Finally, in downstream narrative experiments, the usage of our augmented persona knowledge is constrained by the capacity of baseline model, which leaves for future work the exploration of downstream persona knowledge augmentation on a larger scale