Large Language Models Meet Harry Potter: A Dataset for Aligning Dialogue Agents with Characters
Authors: Nuo Chen
Yan Wang
Haiyun Jiang
Deng Cai
Yuhan Li
ziyang chen
Longyue Wang
Jia Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Personalized dialogue systems
dataset

Abstract: In recent years, Dialogue-style Large Language Models (LLMs) such as ChatGPT and GPT4 have demonstrated immense potential in constructing open-domain dialogue agents. However, aligning these agents with specific characters or individuals remains a considerable challenge due to the complexities of character representation and the lack of comprehensive annotations. In this paper, we introduce the Harry Potter Dialogue (HPD) dataset, designed to advance the study of dialogue agents and character alignment. The dataset encompasses all dialogue sessions (in both English and Chinese) from the Harry Potter series and is annotated with vital background information, including dialogue scenes, speakers, character relationships, and attributes. These extensive annotations may empower LLMs to unlock character-driven dialogue capabilities. Furthermore, it can serve as a universal benchmark for evaluating how well can a LLM aligning with a specific character. We benchmark LLMs on HPD using both fine-tuning and in-context learning settings. Evaluation results reveal that although there is substantial room for improvement in generating high-quality, character-aligned responses, the proposed dataset is valuable in guiding models toward responses that better align with the character of Harry Potter.

Large Language Models Meet Harry Potter:
A Bilingual Dataset for Aligning Dialogue Agents with Characters
Nuo Chen ; Yan Wang Haiyun Jiang', Deng Cai Yuhan Lit, Ziyang Chen', Longyue Wang and Jia Lits *Tencent AI Lab
iDSA, Hong Kong University of Science and Technology (Guangzhou). Hong Kong University of Science and Technology
'chennuo2 60 gmaal.l....m com, (yanwang lyanwang. brragnnnggnaadddgagaaaaaa com, jialee hk)
HPD Benchmark: https: http////coccchc...///// httpeeennkkuggthhuuu.e github. Ho/HHDD .ereD.gitte githubtthhu...... io
Abstract
Datasct
Sce.
Att.
Re.
Dy.
SI.
In recent years, Dialogue-style Large Language Models (LLMs) such as ChatGPT and GPT4 have demonstrated immense potential in con- structing open open-domain dialogue agents. How- ever, aligning these agents with specific charac- ters individuals remains considerable chal- lenge due to the complexities of character rep- resentation and the lack of comprehensive an- notations. In this paper, introduce the Harry Potter Dialogue (HPD) dataset, designed to ad- vance the study of dialogue agents and char- acter alignment. The dataset encompasses all dialogue sessions (in both English and Chinese) from the Harry Potter series and is annotated with vital background information, including dialogue scenes, speakers, character relation- ships, and attributes. These extensive annota- tions may empower L.MSs to unlock character- driven dialogue capabilities. Furthermore, serve as universal benchmark for evalu- ating how well can LLM aligning with a spe- cific character. We benchmark I.LMss HPD using both fine-tuning and in on-oonexxt learning settings. Evaluation results reveal that although there substantial room for improvement in generating high-quality, character-alignat re- sponses, the proposed dataset is valuable in guiding models toward responses that better align with the character of Harry Potter.
PchatbotW (2021)
PeDialog (2019)
KyPI (2020)
P-CHAT (2018)
WOW (2019)
V
X
Fri-QA (2019)
X
X
Focus (2021)
UlraChat(2233)
X
X
LaMP(2023)
X
X
Ours
Table I: Datasets Comparison. Sce., Att., Re., Dy. and SL. denote Scenes, Attributes, Relations, Dynamic and Storyline, separately.
towards aligning dialogue agents with characters in a story. Throughout this process, we discover that despite the assistance of super-powerful LLMs such as ChatGPT and GPT4, we encounter numer- ous challenges: (1) Firstly, the knowledge of LLMs is primarily dominated by real-world knowledge. which may diverge
contradict the char- acter's story setting. In fact, ChatGPT and GPT- 4 have developed substantial understanding of the Harry Potter series and the wizarding world behind it, yet they still occasionally produce hal- lucinations that contradict the story setting. For instance, GPT4-act Harry might attempt to visit Ron's 's house using the Hogwarts Express, despite the fact that this train exclusively operates between King's Cross Station and Hogwarts. (2) Secondly, LLMs encounter difficulties interpreting intricate character relationships, particularly when multiple relationships coexist (e.g., friendships, romantic partnerships and competitors). (3) Finally, the most significant challenge stems from the LLMs' inabil- ity to accurately represent the impact of time on characters, leading to potential inconsistencies in their portrayal and development within the story.
1 Introduction
With the emergence of dialogue-centric Large Lan- guage Models (LLMs) such as ChatGPT and GPT- 4 (OpenAI, 2023), there has been growing inter- est among researchers in exploiting the capabilities of these models to develop open-domain dialogue agents. particularly exciting and challenging as- pect of this pursuit involves aligning the behavior of these agents with a distinct character or individ- ual (Kirk et al., 2023; Salemi et al., 2023).
In this paper, using the Harry Potter series fic- tion prime example, we take significant leap
The dialogue on the right side of Figure effec- tively illustrates the difficulties (2) and (3). We all know that Harry and Ron are very close friends, but
Equal Contribution. Indicates Corresponding authors.

---

Scene (summarized version):
Timing: Book 4-Chopler
Harry Hermione are Hogameade Village.
Harry and Ron bad sectous aryument due to the events suroonding the Trtwzardd Tounnamest.
Relations and A ttributes
Come on, please just take off your Cloak for a bit, no one's 's going to bother you here
Altribettes
Aflection:
Famillarity:
Oh yeah? Look behind you. She's staying in the village. I het she's coming watch the first task.
Classmale
Mffecton:
Pamilioriy::
Why don't we go to find Roa have butterbeer the Three Bracensiilks, it's bit cold, isn't
Classnate
Affectoon: Pamiliariyy
Affectiont -3
Famuliartyy
What will Harry say next?
ChatG A Iright, let's go find Ron and have pint of butterbeer together.
Spilt
Exp repliimees Lome
PT-44: Fine, let's have a glass butterbeer together and go check if Ron is around.
Edoedi
banakiiyy Cask
Attrivates
Expected A nswer: A Aright, I can go to have pint of butterbeer, but I really don't want to Ron right
Figure I: A conversation between Harry and Hermione selected from Book 4-Chapter 19 in the Harry Potter Series. In this example, we present the scene, timing of the conversation, relations and attributes of speakers. Texts in red refer to the expected response.
they sometimes experience conflicts well, like most adolescents. In Book 4-Chapter 19, Harry and Ron have heated disagreement over the "Tri- wizard Tournament" registration (Ron believes that Harry has hidden the truth from him, but in real- ity, Harry has not), leading to a period of mutual avoidance. At this point, when Hermione suggests to Harry that they should find Ron for a Butterbeer, Harry's ' normal response would be reluctance to see Ron. However, even after providing an extensive context prompt, both ChatGPT and GPT-4 still generate responses that portray Harry and Ron as good friends, which is clearly inconsistent with the situation at that time.
ter novels (English and Chinese versions) 1. In total, we obtain 1042 dialogue sessions for training (containing positive response only) and 149 ses- sions for testing (containing 1-3 positive responses and 9 negative responses in average). We also an- notate each conversation with essential background information that we believe is useful for aligning dialogue agents with Harry, including dialogue scenes, speakers, character relationships. and attributes. With the goal of giving a full pic- ture of the speakers in dialogue, we have labeled each speaker with 12 types of relationships with Harry Potter and 13 types of attributes. Please note that even if the speakers in two different dialogues are identical, their relationships and attributes may change due to the story's 's progression.
The appearance of the previously mentioned challenges is unsurprising, considering that cur- rent dialogue datasets intrinsically lack accurate annotations and guidelines for handling dynamic character relationships and attributes. As shown in Table 1, while some existing datasets (Zhang et al., 2018; Zheng et al., 2019; Jang et al., 2021; Ding et al., 2023) do include basic information about scenes, relationships, and attributes, the is- sue is that this information remains static and does not change over time. For example, if John and Harry are labeled as "friends", their relationship will remain as friends across all dialogue sessions.
The main contributions of this paper can be sum- marized into three aspects:
We introduce the HPD dataset, designed facilitate the study of aligning dialogue agents with characters. For each dialogue session in the Harry Potter novels, we provide all the background information that we believe may be helpful in assisting models to generate Harry Potter-aligne responses.
This is a bilingual dialogue dataset that does not rely on machine translation. The only difference between languages is that they are
In this paper, we propose Harry Potter Dialogue (HPD), a dataset for facilitating the study of Di- alogue Agents Character aligning. This dataset encompasses all dialogue sessions from Harry Pot-
'Due to the space limits, we only introduce the English experimental results in this paper.

---

based on Harry Potter novels in different lan- guages. As a result, the data quality in both languages is comparable, making it suitable for investigating the impact of language differ- ences on the task of aligning dialogue agents with characters.
dialogue agents with characters in story. Un- fortunately, so far there are currently no publicly available datasets that provide information about the dialogue scene and participants. To facilitate the study of this task, we construct a new dataset from the popular fictions Harry Potter Series, in the hopes of creating Harry Potter-aligned dialogue agent. All dialogue sessions that Harry partici- pates in are collected in our dataset, along with fine-grained annotated dialogue scenes and partici- pant information. We recruit four avid Harry Potter fans (professional annotators, no crowd-sourcingg for the annotation work in this study.
The experimental results show that HPD can help LLMs, such as ChatGPT, better align with the behaviors of Harry Potter. However, the degree of alignment is still far from the level of human experts, indicating ample room for further improvement.
2 Task Definition
We collect three parts of information to construct our dataset, as shown in Figure I: 1) The dialogue part (Section 3.1) contains all utterances in the di- alogue sessions, well as the speaker's name of each utterance. 2) The scene part (Section 3.2) in- cludes the summarization of the text around the dialogue session. 3) Finally, the speaker informa- tion part (Section 3.3), which consists of attributes and relations of characters, is shown in the left part of Figure 1. Please note that these scenes, attributes, and relations are time-sensitive, they may change as the storyline go on, so we should annotate them session by session.
We use the Harry Potter novels our test-bed with the aim of aligning dialogue agents with Harry Potter in a story. The generated responses of such a dialogue agent should be not only relevant to the context, but also seem like something Harry would say at the time and scene.
Figure shows some main factors that affect behaviors of Harry in conversation. The first fac- tor is the conversation history, which is the most important factor that determines Harry's 's response, The scene, which is the second factor, provides de- tails about the motivation (Hermione invites Harry to have a butterbeer with Ron in the Three Brrom- sticks) of this dialogue. The third factor is the par- ticipants' information (attributes and relations). obviously, Harry will say very different things to different characters, such as Malfoy and Hermione. The latter two factors belong to the background information and are dynamically determined by the timing of this dialogue (Book 4-Chapter 19), and they are continuously varied over the storyline. Formally, the task of aligning dialogue agents with characters in a story can be defined as fol- lows: Given a dialogue history H, corresponding dialogue scene S and participants information P as input, which evolve depending the development of storyline. The dialogue agent is supposed to generate a response Y
3.1 Dialogue Construction
Dialogue sessions in the books are divided into a training set and a test set. Their main difference is that each training session contains only one posi- tive response, while each test dialogue session con- sists of at least positive response Hluman-written ground-truth) and 9 negative responses. We hope this test set may support the evaluation of both dialogue generation and retrieval.
Training Set We request the annotators to extract all multi-turn dialogues from the books. Besides, the speaker name of each utterance in the session is labeled well.
Test Set The effectiveness of dialogue models should be evaluated using a well-designed test set. However, if we directly select the test dialogue ses- sions from the books, it may meet serious knowl- edge leakage problems, i.e., the fact that we evalu- ate in the test set also presents in the pre-training corpus. To prevent this problem, we deliberately design a test set in the following steps:
{yr, 92, ::
y argmax P(Y P(Y
Y is supposed to be not only fluent and natural, but also highly relevant to S and P.
3 Dataset Construction
A high-quality dataset including all pertinent infor- mation in Section 2 is the prerequisite for aligning
First, we manually select some raw dialogues that meet the following requirements: (i) Dia-

---

Timing: Book 1-Chapter 12
Gender
Male
Talents
Parsetinguu
Affection: Familiarity:
Attribate
None
Belongings
Wand
Friend, Clasmaate
Attribute
Aftiction: 5
Familianty
Relations
Attribate
Adlectio::
Familiarity
Age
Friend, Chastnate Collengue, Teammaate
Ganieer
Male
Talents
Parseltungue, Quidditch, Defense Against Dark Arts.
A Affection: Familiarit: 8
Spells
Lumos, Expecto Patronam,
Attribate
Belongings
invisibility Cloak, Fireboll,
Affiliaton
Timing: Book 6-Chapter
Data examples from two different timings: Book 1-Chapter 12 and Book 6- 6-Chapter 7. Words in color denote the changed relations and attributes.
Figure 2:
logues with only one speaker, which contain only one or two sentences; (ii) Dialogues, in which there is no response from other speak- ers the last question. For these samples, we pick out dialogues that are relevant Harry and can be answered from Harry's perspective to construct the test set.
quality of annotated answers when disagreements appear. These strategies alleviate spam and bias, and thus get high-agreement dataset. Finally, we manually double-check and revise mistakes fur- ther ensure the data quality.
3.2 Scenes Construction
In order to offer accurate location information and textual details for each dialogue, we further anno- tate scenes. We assume that the that surround and immediately relate the dialogue in the novel provide ample scene information. Therefore, we initially instruct the annotators to extract these texts from the novel. Subsequently, to leverage the ca- pabilities of GPT-4, we employ it to summarize the extracted texts. To ensure the utmost precision and accuracy, two skilled annotators required to meticulously calibrate the summarized texts, sulting the final scene data.
Second, since directly composing high- quality Harry-like response is quite challeng- ing for the annotators, we try to alleviate the difficulty with the help of LLMs. Specifi- cally, we utilize ChatGPT and GPT-4 to gen- crate potential responses for each selected di- alogue session under in-context learning set- tings. Subsequently, each model is needed to predict responses the candidates least.
Third, we ask three annotators to select the most reasonable responses from the predic- tions the positive response (ground-truth)* while the remaining samples are taken as neg- atíves. A holdout annotator is responsible for integrating the annotations. If the selected re- sponse not comprchensive enough or lacks certain information, rewriting is required. Fur- thermore, if all predictions are considered un- reasonable, the holdout annotator would write response from scratch.
It is worth noting that we deliberately abstained from directly utilizing the extracted texts as scenes, primarily due to two compelling reasons: Firstly, the inherent length of the original text exceeds the maximum limit imposed by most LLMs, rendering impractical for purposes. Secondly, in real- world scenarios, the attainment of such exception- ally high- gighitualtty scenes is highly improbable, as scene information typically originates from video captions or rule-based templates.
This setup has advantages over employing sin- gle annotator to label predictions: One annotator may regard some predictions as positive responses while the other may label them as negatives. Hence, the holdout annotator can be used to guarantee the Some questions may have multiple valid answers
3.3 Attributes and Relations Construction
One of the most important and appealing proper- ties of our benchmark is the fine-grained annotated character information, which includes the attributes and relations of the characters. With the goal

---

providing in-depth and comprehensive character information, we collect 13 fine- seee-graied attributes and 12 relations. We take into account all of these attributes and relationships that are dynamically evolving as the story goes on. Thus, we annotate them chapter by chapter. Considering not all char- acters are essential to understanding and driving the story in Harry Potter series, we target 113 im- portant characters annotate their attributes and relations, such as Harry, Ron.
Affection
10
Parents and relativva died fortlarr.
Coumplex Harry's pererts
Choractare who are very doee with Harry and cave Hanyy's
kxampiee Has kook-3
Characters who are love Harry.
Complen Clary
Chasadtere help
Sicrrgiee Durblledao
Chanadtrrs who selethvvty hicenll Harry
Esurplion Nevi
Nermal Teammatee...............
Dxamples Wood
Nermel CleamottesTeecherr
Exempfer Brown
We divide the attributes into two categories: (1) inborn: (2) nurture. The former denotes some in- nate attributes or abilities, which contains [Gender, Age, Lineage, Talents, and Looks). The latter refers to properties acquired through efforts, including {Achievement, Title, Belongings, Export, Hobby, Character, Spells and Nickname) (some cases are presented in Figure 2). In total, we collect 13 at- tributes for each character, which basically cover most properties in the Harry Potter series.
Pirst Heeding
Darrppes Harry fost and indiioee
Scampies
Dellberttely bewt//iggeeeeeeeeelyyygoooooo
Maticinaly targeting and herm.
Exampler Draco Mailey Book
-8
Ietontionnall inftec burnn.
Reorepee ta tettiage in Beck-
-10
101 Heryy'
Ceamples Voldemsrt
Figure 3: Affection Definition and Examples.
Familiarity
The relations between Harry and other charac- ters can be classified into binary relations and dis- crete relations. The former include 8 types, which are {Friend, Classmate, Teacher, Family, Lover, Opponent, Teammate, Enemy). Multiple binary relations can exist between two characters. Harry and Ron, for instance, are friends, classmates, and teammates in the Quidditch team (In Book-6 only).
together years tamiper.
Examptex Bon in Book
famiiarr with
Exampiex
fomelkrr with beckgrounst intormatiion Exampiis dirius in Book-
Chameteers whe
FriendaTeachhrso moet rupt times and are stightly fumiltar with each other
Eearmplow Predessee McCorsagal
Famillar with character's background but don't know Ninuher.
In order to fully represent the relationship be- tween two characters, we need to know not only the type of relationship (the binary relations) they have but also their familiarity and affection for each other. We annotate affection and familiarity as 4 types of discrete relations: (1) Harry's Familiarity with someone, (2) Harry's s Affection for someone, (3) someone's 's Familiarity with Harry, and (4) some- onc's Affection for Harry.
Mast flxx dieee
Figure 4: Familiarity Definition and Examples.
with Ron are relatively low (1 and 2, respectively), and he isn't 't aware of any spells. As the story pro- gresses, however, in Book 6-Chapter 7, he is a full-fledged wizard, and Ron is his best friend. So their affection and familiarity are high at this time (7 and 8, respectively). Harry also masters a lot of spells such as Expecto Patronum and Expelliarmus, and has some equipments such as his broomstick Firebolt and the invisibility cloak.
The difference between these types can be illus- trated by the following two examples: 1) Draco Malfoy hates Harry, but he is also familiar with Harry. So his Affection for Harry is low but his Familiarity with Harry is high. 2) In addition, ever since Harry lost his parents, Dumbledore has shown great concern for Harry. Hence, his Familiar- ity and Affection for Harry are high, while Harry's Familiarity with Dumbledore is relatively low.
A Affection Definition Affection is rated on 21 lev- cls, ranging from -10 to 10, where -10 and 10 indi- cate the lowest and highest affection, respectively. A positive affection level means the character has positive relationship with Harry and vice versa. Figure 3 shows the detailed definition and some examples of different Affection levels. For exam- ple, 1 refers to "someone meets Harry for the first
Figure 2 is another example of how attributes and relationships have changed over the course of the story. Book 1-Chapter 7, Harry has just met Ron on the train to Hogwarts and is an alien in the wizarding world. So his affection and familiarity

---

Statistics
Train
Test
3.4 Data Statisties
per dialogue
The detailed statistics of dialogues are shown in Table 2. The training set and test set contain 1042 and 149 dialogue sessions, respectively. Of note, initially collect 1471 dialogues for constructing the training set, and we filter out those dialogues that are without Harry, leading to 1042 conversa- tions for considcration. Most of the conversations involve multiple speakers, with the maximum one including up to 20 speakers. It is obviously a seri- ous challenge to the dialogue agents.
Average Turn
13.8
7.8
Maximum Speakers
20
8
Minimum Speakers
2
2
per utterance
Average Length
32.9
28.3
Maximum Length
77
Minimum Length
3
Total Dialogues
1042
149
Table 2:
Data statistics of collected dialogues.
time", when Harry first met Ron and Hermione in Book 1, Harry's is Affection to both 1. As for 10, it means the deepest hatred, and the most obvious example is Harry and Voldmort.
Experiments
In this section, we conduct extensive experiments to investigate 1) whether Annotations HPD can assist LLMs in aligning with characters, and 2) if so, whether the degree of alignment can reach human-level performance. We will discuss these questions in section 4.3.
Familiarity Definition Similarly, also rate Familiarity with 11-point scale, which ranges from 10, where 10 is the highest and is the lowest. We present detailed definitions for each Familiarity level Figure 4. Concretely, 0 denotes strangers, and 10 denotes very close friends who stay together for many years and are very familiar with each other's 's habits, secrets, and temperaments. Ron in Book 7 meets this condition.
4.1 Baselines
We build multiple strong baselines in our exper- iments, which can be divided into two types: generation- based and retrieval-based systems. Due to the page limit, here we only introduce the gener- ation task in HPD. For details of retrieval tasks in HPD, please refer to Appendix and D.
During annotation, we ask annota- tors to annotate Attributes and Relations following the above definitions and examples. To eliminate the effects of individual bias, we ask three of them to repeatedly label the Attributes and Relations chapter by chapter. And a holdout annotator (se- nior) is responsible for the quality of annotated data when the former three annotators have disagree- ments. Finally, we manually re-check all annotated data and revise some controversial annotations to further control the data quality.
Annotation
Models We implement different generative mod- els in two ways: fine-tuning and in-context learn- ing. For the former, we fine-tune Alpaca² (6B) and ChatGLM-6B on our dataset. For the latter, we deliberatcly design prompts for GPT3 (Brown et al., 2020) fext-davinci-002), ChatGPT (gpt3.5- turbo). and ChatGLM (Zeng et al., 2022) (chat- glm-130B) Furthermore, to explore the effect of an- notated fine-grained background knowledge, some methods (Alpaca, GPT3, and ChatGPT) are imple- mented in two different settings: 1) base setting with a prompt that only includes task description, one dialogue example, and dialogue history: 2) rich-persoo setting (denoted as Per-Model) with a prompt contains all annotated background infor- mation in HPD as in-context learning exemplars. The detailed prompts can be found in Appendix C (Table 8 for base setting, and Table 10 for rich- persona setting).
Claim Notice that we hope to provide as rich character information as possible for the commu- nity, even if some of them seem redundant in this work. Therefore, we collect 13 types of attributes for each character and 12 types of relations in the collected HPD. We leave plenty of opportunity for other research communities to investigate which information is helpful in their work. For example, it can support other tasks such as the sentiment analysis of Harry Potter. One may not need to in- clude all the Enn-grained annotated information in his/her study, because it is still challenging.

---

Automatic Evaluation
GPT-4 Evaluation Relv.Att.
Haman Evaluation
Model
Bleu-1
Dist.1
Rough-L
Relv.Sce.
Relv.Re.
Relv.Sce.
Relv.Att.
Relv.Re.
Fine-tuning
CGLM*-6B
1.2
18.4
3.4
1.21
1.10
1.01
1.34
0.67
0.67
Alpaca
2.0
30.2
10.8
3.37
1.19
1.51
2.03
1.19
1.51
Per-Alpaca
12.2
20.4
13.8
3.51
1.34
1.51
4.04
1.34
1.51
In-context learning
CGLM
14.2
25.1
15.7
6.71
6.01
6.04
7.38
6.04
5.37
GPT3
9.8
23.0
14.6
2.68
2.68
3.36
4.70
4.70
4.03
Per-GPT3
22.6
20.1
16.5
12.75
6.04
3,36
12.08
6.71
5.37
ChatGPT
33.1
19.0
20.2
33.56
38.92
37.58
32.89
34.23
32.89
Per-ChatGPT
33.6
19.4
22.4
37.58
42.95
46.98
35.57
45.63
48.99
Table 3: A comprehensive evaluation of all baselines HPD test Per-Model means the model with prompts in rich-persona setting. Here we report the percentage of generated responses ranked as the best one for each dialogue session (top-1 ranking) in GPT-4 and human evaluation. CGLM refers ChatGLM. We report the average ranks of these LLMs in Figure 5. Dist.1 is short for Distinct-1.
4.2 Evaluation Metric
to human-evaluation results. During the annotation, annotators were allowed to consult the original text for reference when ranking. Moreover, as long as at least one annotator among the three made ad- justments to GPT-4's ranking results, we would adopt the modified results as the human evaluation results. If multiple annotators made revisions to single result, would take the average of their modifications as the final human evaluation result.
To comprehensively measure whether the texts gen- erated by various models exhibit a Harry Potter- aligned tone, we employ three different kinds of evaluation methods: beased (automatic), GPT-4 based, and human-based. Although some existing studies (Liu al., 2023; Peng et al., 2023) claims that GPT-4 good-enough to assess the quality of generated texts, we found that human judgment is still the most thorough and realistic assessment of whether the generated text is Harry Potter-aligned in our task.
4.3 Results
Whether the HPD can assist LLMs in aligning with characters? To answer this question, we present the results of these baselines in Table 3, and observe that across all GPT-4 and human-bases metrics, methods perform significantly better in the rich- rich-persona setting compared to their performance in the base setting. This improvement, obviously. should be attributed the additional background information provided by the HPD.
Reference-based Metrics We employ Rough- L (Lin, 2004), Bleu-1 (Papineni et al., 2002), Distinct-1 as our metrics to validate the relevance and diversity of the generated responses.
GPT-4 Evaluation We evaluate the persona's consistency based on the three criteria: Relevance with the Scene (Rely.cce.). Relevance with the At- tributes (Relv.Att.) and Relevance with the Rela- tions (Relv.Re.). Considering the potential draw- backs of having GPT-4 to assign scores for all re- sponses individually, such as the risk of confusing GPT-4's comprehension,
Additionally, for ChatGPT, the improvement in persona consistency-related metrics, including Relv.Att. and Relv.Re., is more pronounced (for example, compared to ChatGPT, Per-ChatGPT has increased by 11.4% and 16. in these two met- rics, respectively). This is mainly because, most cases, the dialogue history has some overlap with the scene, and ChatGPT can be consistent with the scene by merely using the dialogue history. For GPT-3 and Alpaca, their understanding of dialogue history not strong as that of ChatGPT, their improvements Relv.Sce. are still noticeable.
have opted for an al- ternative approach. We instruct GPT-4 to rank the generated texts based on above distinct crite- ria. This ranking process allows us evaluate the performance different models effectively. The detailed prompts are in Appendix C, Table 9,
Human Evaluation In light of the discovery of some poor GPT-4 annotation cases, we further in- structed our annotators to revise and rectify the ranking results GPT-4 for each test data, leading
Models VS. Human Experts Intuitively, we may further raise a question: How good are the gener-

---

Category
Relv.Sce.
Relv.Att.
Relv.Re.
Ave.
4.4
Case Study
WIN
77.85
77.18
73.15
76.06
As the case in Table 5, the scene of selected dia- logue is about "Harry finds he is able to converse with snakes." " and Harry is facing questioning from Pears, who often bullies Harry with Dudley. He dis- likes and even feels somewhat afraid of the Dudley family and Pears. At that time, Harry doesn't know anything about the wizarding world, and hasn't yet known about or gone to Hogwarts. Hence, Harry is likely trying to downplay the situation, keep his sccret (he can talk with snakes) and avoid poten- tial conflict with Piers and Dudley. However, all models cannot capture such complicated relation- ship and fail to generate real Harry Potter-aligned responses. Most models generate responses that start with "Yes, I was..." or "I was", which are dull and clearly contradict Harry's actual intention. Additional examples can be found in Appendix E, Table 11. They demonstrate that when dialogues necessitate deep understanding of the current context involve a sudden shift in Harry's 's lationships with other characters, current LLMs' behavior significantly deviates from Harry's 's ac- tual actions. This is, obviously, far from satisfac- tory. Although the Harry Potter series is already included in the training corpus of most LLMs (they know the basic information about characters and their world), relying solely on 'next word predic- tion' may be insufficient to fully understand the nuanced knowledge embedded within the story be- hind Harry Potter.
TIE
1.34
4.03
8.05
4.47
LOSE
20.81
18.79
18.79
19.46
Table 4: Comparison between Per-ChaiGPT and Human experts. The win/tie/lose proportions of human-wriin written responses on different metries are reported.
Harry and his family visit the reptile house at the z00, where Harry strikes up a conversation with a boa constrictor from Brazil.. Harry is blamed for talking to the snake and causing the incident.
Scene:
Dialogue Position: Book -chaptcc
Speakers: Dudley, Harrr...Piers
Dialogue: Dudley; Make it move...
the snake: get that all the time.
Harry: know, Where do you come from, anyway?
the snake: Boa Constrictor, Brazil.
Harry: Was it nice there? Oh, I see so you' 'e never been to Brazil?
keeper of the reptile house: DUDLEY! MR. DURSLEY! COME AND LOOK AT THIS SNAKE! YOU WON'T BELIEVE WHAT IT'S DOING!
the snake: Brazil, here| come. ... Thanksss, amigo.
Piers: Harry was talking to it, weren't 't you, Harry?
GPT-3: was just asking it where it came from. It's not like I was trying to make friends with it or anything.
Per-GPT3: didn't mean to cause any trouble. was just curious about where the snake came from. didn't t know it would escape.
ChatGPT: Yes, I was just curious about where it came from and it had ever been to Brazil. But Dudley, you shouldn't be rude the keeper of the reptile house. He was just trying show something amazing.
Per GPr:C Yes, I was. But I didn't make the glass disappear or tell the snake to escape. It was just a harmless conversation. I didn't 't mean for any of this to happen.
Annotated Response: No, wasn't 't talking to it. I was just looking at it and reading the sign about where it came from. You two must have imagined things.
5 Conclusion
In this paper, we propose a new benchmark named Harry Potter Dataset (HPD) to promote aligning dialogue agents with characters in a story. Unlike existing datasets, HPD not only contains interesting dialogues, but also scenes, character attributes and relations that are dynamically changed as the story- line goes on. It also provides a well-designed test set to facilitate the evaluation of both generation- based and vetrieval-based dialogue agents. Results and case studies show that powerful LLMs are still far from human expectations, proving there is am- ple room for improvement. Generally, HPD offers several open research problems in character align- ing, such as how to build automatic evaluation metrics for personalized text generation, exploring effectively prompting L.LMs for character aligning in a story. We hope HPD can play an crucial role in moving through cracking them.
Table 5:
Examples of generated responses by base- lines. Here, we have omitted the attributes and rela- tionship information hetween the characters.
ated texts? Are they good as the ground-truth? To explore this, we further recruit another three annotators to compare the Per- ChatGGT--geeerttee responses with the human-written responses in the test set. The experimental results, as shown in Table 4, are surprising. Contrary to our common sense that ChatGPT has already reached human- level performance in conversations, humans show clear advantage in the task of aligning with Harry Potter. In all metrics, the proportion of Pre- ChatGPT outperforming or being on par with hu- mans even lower than 30%. As we analyzed in
Section 1, the existing LLMs are still far from be- ing able to align with specific characters.

---

Ethical Statement
Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1ly Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. CoRR, 0etr2005.14166
To avoid the potential issue of using Harry Potter novels, we promise the annotated dataset is devel- oped for non-commercial use. Moreover, we only provide the line number and page number of each collected dialogue in Harry Potter rather than the detailed content of each dialogue session. We fur- ther supply the script to extract corresponding raw dialogue data from the novels according to the pro- vided line and page numbers, in which the data format is the same as the data examples in Table 11. As for the annotated character attributes and re- lations, we have our copyright and will release for research communities.
Nuo Chen, Hongguang Li, Yinan Bao, Junqing He, Xinshi Lin, Qi Yang. Jianfeng Liu, Ruyi Gan, Ji- axing Zhang, Baoyuan Wang. et al. 2023. Orca: A few-shot benchmark for chinese conversational machine reading comprehension. arXiv preprint arXiv:2302. 13619.
Nuo Chen, Chenyu You, and Yuexian Zou. 2021. Self- supervised dialogue learning for spoken conversa- tional question answering. In Interspeech, pages 231-235. ISCA.
Cristian Niressuu-icuulsc--Mizii and Lillian Lee. 201 I. Chameleons in imagined conversations: new ap- proach to understanding coordination of linguistic style in dialogs. In CMCL@ACL, pages 76 87. As- sociation for Computational Linguistics.
Limitations
@
The main target of this paper is towards building dialogue agents for characters in a story. In this paper, we present a new benchmark named Harry Potter Dialogue (HPD) in the hope of creating a Harry Potter-aligned dialogue agent. The signif- icant feature of HPD is that it contains detailed scenes and fine-grained attributes and relations of each speaker which are dynamically changed as the storyline goes on. More generally, we expect the core idea of this paper can give insights into other research communities that want to build effec- tive person-like chatbots in the virtual world. Our well-designed test sets could even support RLHF training of characterized dialogue agents. Our fine- grained annotated knowledge also can be used to build other tasks such as sentiment analysis and reading comprehension of Harry Potter. Admit- tedly, the data in the proposed dataset from the Harry Potter Series is restricted to a specific area, that is, Harry Potter Magic World. Considering the high cost of annotation, our character relation annotation work is restricted to Harry Potter. These concerns warrant further research and considera- tion when utilizing this work build intelligent person-like dialogue systems in the virtual world.
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019. Wizard of wikipedia: Knowledge-powered conversational agents. In ICLR (Poster). OpenReview.net.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin. Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conver- sations. arXiv preprint arXiv:2305. 2005:23005.1233..
Edward J Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint 10Xi:2100600685..
Yoonna Jang, Jungwoo Lim, Yuna Hur. Dongsuk Oh, Suhyune Son, Yconsoo Lec, Dong-Hoon Shin, Se- ungryong Kim, and Heuiseok Lim. 2021. Call for customized conversation: Customized conver- sation grounding persona and knowledge. CORR, abs/2 12.08619.
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. 2023. Personalisation within bounds: A risk taxonomy and policy framework for the align- ment of large language models with personalised feedback. arXiv preprint arXiv:2303 333.00333..
Satwik Kottur, Xiaoyu Wang, and Vítor Carvalho, 2017. Exploring personalized neural conversational models. In IJCAI, pages 3728-3734, ijcai.org.
References
Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and William B. Dolan. 2016. persona- neural conversation model. In ACL (1). The Association for Computer Linguis- tics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess,
Chin-Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Sunmariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.

---

Yongkang Liu, Shi Feng, Daling Wang, Yifei Zhang, and Hinrich Schlitze. 2023. Evaluate what you can't t evaluate: Unassessable generated responses quality. arXiv preprint arXiv:2305. /4658.
Chenyu You, Nuo Chen, and Yuexian Zou. 2021. Self- supervised contrastive cross- modality representation learning for spoken question answering. In EMNIP (Findings), pages 28-39, Association for Computa- tional Linguistics.
OpenAl. 2023. Gpt-4 technical report.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuayi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et 2022. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- lational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311-318. ACL.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Per- sonalizing dialogue agents: have dog, do you have pets too? In ACL. (1), pages 2204-2213. Association for Computational Linguistics.
Baolin Peng, Chunyuan Li, Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with gpt-4, arXiv preprint arXiv:2304.03277.
Hongjin Qian, Xiaohe Li, Hanxun Zhong, Yu Guo. Yueyuan Ma, Yutao 7hu, Zhanliang Liu, Zhicheng Dou, and Ji-Rong Wen. 2021. Pchatbot: large- scale dataset for personalized chatbot. In SIGIR, pages 2470-2477. ACM.
Wei-Nan Zhang, Qingfu Zhu, Yifa Wang. Yanyan Zhao, and Ting Liu. 2019. Neural personalized response generation as domain adaptation. World Wide Web, 22(4):1427-146.
Yinhe Zheng, Guanyi Chen, Minlie Huang, Song Liu, and Xuan Zhu. 2019. Personalized dialogue genera- tion with diversified traits. CoRR, 0es//191.096777
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2023, Lamp: When large lan- guage models meet personalization. arXiv preprint arXiv:2304. 1 1406.
Yinhe Zheng, Rongsheng Zhang, Minlie Huang, and Xiaoxi Mao. 2020. A pre-training based personalized dialogue generation model with persona-sparse data. In AAAI, pages 9693 9700. AAAI Press.
Haoyu Song, Yan Wang, Weinan Zhang, Zhengyu Zhao, Ting Liu, and Xiaojiang Liu. 2020. Profile consis- tency identification for open-domain dialogue agents. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 6651 16662.
Tengtao Song, Nuo Chen, Ji Jiang, Zhihong Zhu, and Yuexian Zou. 2023. Improving retrieval- based dialogue system via syntax-informed attention. In ICASSP 2023-2023 IEÉE International Confer- Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEEE.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019. Transfertransfo: A transfer learning approach for neural network based conver- sational agents. CORR, abs/1901.081499
Chen Xv, Piji Li. Wei Wang, Haoran Yang, Siyun Wang. and Chuanghai Xiao. 2022. Cosplay: Concept set guided personalized dialogue generation across both party personas. In Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 201- 211.
Zhengzhe Yang and Jinho D. Choi. 2019. Friendsqa: Open-domain question answering on TV show tran- scripts. In SIGdial, pages 188-197.7 Association for Computational Linguistics.
Chenyu You, Nuo Chen, Fenglin Liu, Dongchao Yang, and Yuexian Zou. 2020. Towards data distillation for end-to-end spoken conversational question answer- ing. CoRR, abs/2010.08923.

---

A Related Work
standing of the dialogue background information.
Recently, building personalized dialogue systems draw lot of attention from research communities. Aiming for promoting this area, several efforts and benchmarks (Chen et al., 2021; Ganescu-Niculesee Mizil and Lee, 201 I; Zheng et al., 2019; Yang and Choi, 2019; Song et al., 2020; Zheng et al., 2020; You et al., 2020; Chen et al., 2023; You et al., 2021; Xu et al., 2022) have been made, demonstrating promising results for endowing personal style into dialogue systems. Some initial efforts (Danescu- Niculescu-Ming and Lee, 201 aimed modeling characters from movies.
B Baselines Setup
In the following, we briefly introduce each model and describe the training and test details.
B.1 Baselines
BERT-FP is commonly-user strong retrieval- based dialogue system, which devises several post- training objectives. When fine-tuning BERT-FP, given n-1 utterances in each dialogue session, the model is required to find the ground-truth response from candidate answers. Concretely, we first post- train BERT-FP in Harry Potter novels and then fine-tune the resulting model in the collected HPD.
Further developments provide personas via two types: implicit and explicit personalization. In the former streams (Kottur et al., 2017; Li al., 2016; Zhang et al., 2019), each speaker's personal- ity information can be compressed as the persona embeddings. In this manner, the existing issue of these methods makes it hard to explain their effec- tiveness. For the latter (Wolf et al., 2019; Song et al., 2020), the personal information are provided as: (1) dense personas, such as speaker profile text-described personas; (2) sparse personas, including some personality traits. For example, personas from (Zheng al., 2019) are formulated as key-value pairs: "Age:xx, Gender:xx, Location: xx".
B.2 Experimental Setup
Our experimental settings can see in Table 6. Notice that, considering the memory cost, we uti- lize LoRA-tuning (Hu et al., 2021) and prompts don't tt contain any examples.
C Prompts
In this section, we present three different prompts in our experiments: Table shows the prompts of the base setting which only include task descrip- tion, a dialogue example and the input dialogue. Table 9 presents the ranking prompts for GPT-4 evaluation. Table 10 illustrates the rich-persona setting prompts.
More recently, several efforts (Dinan et al., 2019; Yang and Choi, 2019; Jang et al., 2021; Song et al., 2023) incorporated scenes and relations knowl- edge into each dialogue session for encouraging more real personalized conversation. Yang and Choi (2019) presented open-domain question answering dataset excerpted from Friends Series, where each dialogue involves multiple speakers and their relations. Jang et al. (2021) proposed FoCus dataset where the customized responses are gen- erated based on the user's persona and Wikipedia background knowledge.
D Retrieval Task
Automatic Metrics For evaluating the retrieval- based model, we also employ some common met- rics: MAP (mean average precision). MRR (mean reciprocal rank), and P@ (precision one). Re- call also be considered, which is used as R10@k, which implies that the correct response exists among the top k candidates out of the ten candidate responses.
In this paper, our goal to align dialogue agents with characters in a story, which requires modeling scenes and speaker information that are dynami- cally changed as the storyline goes on. However, the personality settings of current studies are static, and are not changed with scenes times changing. Therefore, present HPD: Harry Potter Dialogue Dataset, aiming for creating Harry Potter-alignd di- alogue agent. detail, we annotate detailed scenes, attributes and relations each speaker over given dialogues to help the model have a deeper under-
Automatic Evaluation BERT-FP also performs poorly P@1 score (25.9%) and MAP score (46.8%) in the retrieval-based task. The results show the current state-of-the-art retrieval-based model also can not handle the challenge of our benchmark, and thus, there is ample room for fu- ture improvement.

---

Parameter
Alpaca
Per-Alpaca
ChaiGLM-6B
BERT-FP
Batch size
4
32
Learning Rate
le-
le-
3e 3e-5
8e-5
Epoch
5
20
Max Length
512
256
Table 6: Hyper-parameters setup in fine-tuning.
CharoPt
Chetury
OP22
Apace
Nrt Abacc
(a) GPT-4 Evaluation
Per coaorrr
Per Charort
For chettrr
Pvr
For
ce
cems
cm
Chencindd
Betx.cee
Ret
Rehu
(b) Human Evaluation
Figure Average ranks of generated texts in terms of human evaluation and GPT-4 evaluations.
Retrieval-based
However, it is intriguing to observe that Per- Per- and ChatGPT diverge from this pattern and reject Cho Chang's 's expression of affection. Furthermore, the responses from GPT-3 and CharGPT-6B are dis- appointingly concise, failing to adequately capture the nuanced blend of shyness and excitement exhib- ited by Harry in response to Cho Chang's 's heartfelt confession.
Model
MAP MRR P@1
R1061
R10@5
BERT-FP 0,468 0.468
0.259
0.259
0.788
Table 7: Automatic evaluation results of the retrieval- based model.
E
More Case Study
Intuitively, higher pursuit of our task is expect the dialogue system can generate the logical response according attributes and relations between char- acters. Chapter 21 of the fifth book, there is a significant scene where Cho Chang presents Harry with a Christmas gift and confesses her feelings. The annotated character relationships and story de- velopment depicted in the Table 11 reveal that Harry has harbored feelings for Cho Chang all along. Confronted with Cho Chang's heartfelt con- fession, it is reasonable to expect Harry to respond with mix of shyness and excitement, ultimately accepting her advances. This aligns seamlessly with the subsequent progression of the story.
Within this dialogue scenario, it is noteworthy analyze the responses generated by large mod- els such as GPT3, Per-GPT3 and ChatGPT. They all convey a similar sentiment of reciprocation, ex- pressing an inclination towards liking Cho Chang.

---

Prompts:
Your task is to act as a Harry Potter-like dialogue agent in Magic World. There is a dialogue between Harry Potter and others. You are required to give a response to the dialogue from the perspective of Harry Potter.
Here is an example:
Dialogue: "Petunia: Bad news, Vemon, Mrs. Figg's broken her leg. She can take him. Now what?",
I'm warning now, boy any funny business, anything at all - and you' 'll be in that cupboard from now until Christmas."
"Vernon: I' I'm warning
Harry's 's Response: I know, I will obediently obedient, and I won't cause you trouble. Keep in mind the following requirements:
Before generating the response, should read and understand the dialogue content carefully.
Input:
Dia ogue: (Input Día logue)
Onput:
Earry's Response:
Table Prompts of the base setting our experiments.
You are J.K. Rowling who is the author of the Harry Potter Novels. Here scene featuring a conversation hetween Harry Potter and other characters in Harry Potter Novels. Given 8 potential responses from the perspective of Harry Potter to the scene, you are required to rank the quality of these responses based on the following criteria, respectively:
Prompts:
(I) coherence with relations between Harry and other characters. (short for Coh.Rel);
(2) coherence with Harry's 's attributes. (short for Coh:Att);
(3) coherence with the scene (short for Coh.Sce).
To help you rank these responses.
additionally provide some background information, including 'Dialogue Position', 'Speaker's altributes' and 'Speakers relations with Harry'.
You should generate the response format with "Coh. Rel: R8s...R1; Coh.Att: R8*...RR:; Coh.Sce: R8 and then give several sentenees explain your opinion.
Input:
a ogue Position: (Dialogue Position)
Speakers relations with Earry: (Speaker relations]
Harry's s attributes: (Rarry's attributes]
Scene:
(Scene)
Dialogue: (Input Dialogue)
R1: (Responsel)
R8: (Response8)
Output:
Table 9: Ranking Prompts for GPT-4 in our experiments.

---

Prompts:
"character": "None". Your task is as Harry Potter-like like dialogue agent in the Magic World. There is dialogue between Harry Potter and others. You required to give response the dialogue from the perspective of Harry Potter. To do this, you can write out your thought and answer with "Harry's response" at the end. To better help you mimic the hehavior of Harry Potter, we additionally provide the following background information the dialogue:
Dialogue position, which represents timeline of the dialogue Happy Potter Novels, For example, "Dialogue Position: BookS-chapter28" means the dialogues occurs Chapter28, Book5.
Dialogue speakers.
Harry Potter's attributes, which refers basic properties of Harry Potter when the dialogue happens. can contains 13 categories: Gender, Age, Lineage, Talents, Looks, Achievement, Title, Belongings. Export, Hobby. Character, Spells Nickname.
Speaker relations with Harry, such as whether he was friend, classmate, or family member; Harry's 's Familiarity to the speaker, which ranges from 0 to 10. Concretely, denotes stranger, and denotes close friends who often stay together for many years and are very familiar with each other's 's habits, secrets and temperaments, where Ron meets this condition Book 7.
6. Harry's Affection to the speaker, which ranges from -10 to 10. refers to speaker met Harry for the first time, For instance, when Hary first met Ron and Hermione in Book 1, Harry's 's Affection to them are hoth set to 1, And -10 means the speaker killed Harry's 's parents, where Voldemort meets this condition in the novels.
Here is an example:
Dialogue position: Book1- 1chap-eha
Dialogue speakers: Harry, Petunia, Vernon
Harry's 's attributes:
{ "nname:: "Harry",
"nickname": "The bay who lived", "gender": "male",
"age": "age 11",
"looks": "Very thin, black hair, emerald green eyes, wearing glasses, knife injury with lightning shape at the forchead", "hobbies": "None".
"talents": "None".
"export": "None".
"belongings": "None", "aftiliation": "None". "fincage": "None",
"title": "The boy who lived".
"spells": "None"|
Speakers relations with Harry: Vemon is Harry's uncle and Petunia Harry's aunt.
Harry's Familiarity Vernon: 8
Harry's Affection Vernon: -4
Harry's Familiarity Petunia:
Harry's Affection Petunia: -4
Dialogue: "Petunia: Bad news, Vemon, Mrs. Figg's broken her leg. She take him. Now what?",
"Vernon: I'm warning you, I'm warning you now, boy any funny business, anything at all and you 'Il be that cupboard from now until Christmas."
Thought: Let's think step by step. According the conversation history, Vernon warned Harry not to spoil the special day. According to Harry Potter's attributes.
still very thin, does not know any spells, and has not gone to Hogwarts yet. So currently incapable resisting them. At same time, based on his affection for them -4, means that he relatively doesn't 't like them, and may even be little scared. Therefore, Harry possiblely says: I know, will obediently obedient, and won't It cause you trouble.
Harry's 's Response: know, will obediently obedient, and won't cause you trouble.
Keep in mind the following requirements:
1. Before generating the response, you should read the above information and dialogue content carefully.
You can not generate the response that is against Harry Potter's attriuutes and Harr's 's relations with the speaker.
Not every component in the background information may be useful, you should choose some of them to help you generate more concise and comprehensive responses that satisfy the behavior of Harry Potter in the dialogue.
4. Not every speaker have relations, familiarity ad affection to Harry. At that time, you can directly predict what would Harry say only based on the dialogue context..
Input:
Dialogue Fosition: (Dialogue Position)
Speakers relations with Harry: (Speaker relations)
Harry's s attributes: (Harry's attributes)
Scene:
(Scene)
Dialogue: (Input Dialogue)
Output:
Table 10: Prompts of the rich-persona setting in our experiments.

---

Scene: Harry stays behind in the common room hoping to receive Merry Christmas from Cho. When they alone, Cho starts crying and Harry tries to comfort her. She apologizes and mentions Cedric's 's death, but Harry tells her that was good magic that Voldemort would have killed him anyway. Cho compliments Harry on his teaching skills and they share moment under the mistletoe where Cho confesses that she really likes him. Dialogue Position: BookS-chapter21
Speakers:Cho Chang. Harry Harry's 's attributes:
"nickname": "The boy who lived", "gender": "male",
"age": "15 years old",
"looks": "Very thin, black hair,
emerald green eyes, wearing glasses, knife injury with lightning shape at the forehead", "hobbies"; "None".
"character": "None".
"talents": "Snake cavity, Quidiqi",
"export": "None".
"helongings": "Winter green wond phoenix feathers wand, owl, stcalth jacket, sleeve spare mirror, crossbow flying broom, golden eggs, three strong cups, live maps, fake Galon", "aftiliation": "Hogwarts, Dumbledore", "lineage": "Mixed wizard".
"titl": "Boys who do not die, warriors, teacher Dumbledore". "spells": "Except for your weapons, fluorescent flashes, separation from left and right, calling god guards, bustling, four-point cracking. pouring force, funny bones, fainting to the ground, pointing me, obstacles, recovery early" Speakers' relations with Harry:
Cho Chang is Harry's 's friend and classmate.
Harry's ' affection to her: 7.0,
Harry's 's familiarity with her: 4.0,
His affection Harry 7.0,
His familiarity with Harry: 4.0
Dialogue: Harry: Wha ? What's up?. Cho Chang: I'm sorry. suppose ... it's just... learning all this stuff. It just makes
wonder whether if he'd known it all. he'd still be alive
he could never have got the middle of that maze. But if Voldemort really kill you. you don't stand chance.,
Harry: He was really good
Cho Chang: You survived when you were jerejust a baby.. Harry: Yeah, well, I dunno why, nor does anyone else, so it's nothing he proud of.,
Cho Chang: I'm really sorry get upset like this. didn't mean to ... Me mentioning Cedric, I know must be horrible for you, when you saw him dic. ... I suppose you just want forget about it... You're You're r-really good teacher, you know, I've never been able to Stun anything before.,
Harry: Thanks..
Cho Chang: Mistletoe.,
Harry: Yeah, It's probably full nargles. though.,
Cho Chang: What are nargles?,
Harry: No idea, You'd have ask Loony. Luna, mean..
Cho Chang: really like you, Harry.?
Alpaca: really like you too, Cho.
ChatGL-M-BB really like you too, Cho. But I'm not if I'm ready for relationship right now. I'm still dcaling with lot stuff from the past, and don't want to bring that into new relationship. hope you understand.
ChaiGLM- 130B: it's okay. know you't still grieving Cedric's death, and understand why you't 're feeling upset. here you you need talk.
GPT-3: 1 like you too.
Per-GPT3: I'm glad could help, Cho. Cedric was a really great guy and it's 'ss hard to lose someone like that. I just try to
keep busy and not think about too much.
ChatGPT: like you too, Cho. You's re a good person and appreciate your kind words. But have to be honest with you, I don't 1 think I'm ready for relationship right now. still have a lot things to figure out and deal with. Can we just he friends for now?
Per-Ch Per-ChatGPT: Thanks, Cho. You're a good student and I'm glad can help you with your spells, As for Cedric, it's not something can forget easily, but I try not to dwell on it. And about the mistletoe, it's just silly tradition. Don't 't worry about it too much.
Annotated Response: I, um, I really like you too, Cho, I said, feeling my face grow hot. My heart was racing, and I couldn't 't believe this was actually happening. I just... I didn't 't know if you felt the same way.
Table 11: Examples of our generated responses by baselines.


Reviews:
[{"review_id": "AxwqelsH5p", "reviewer": "Reviewer_Jmrm", "summary": "", "questions": "1. See the reason for rejection. \n2. Could you explain more about the scene summarization? What is the window size for summarization? How informative? How is the hallucination problem now even if it is relevant to the scene? Please clarify the metric of relevancy to the scene and limitations of current methods.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "First and foremost, we appreciate the time and effort you dedicated to reviewing our paper and providing valuable feedback. In response to the concerns raised by the reviewer, we offer the following responses.\n\n\n**Q1**: It is unclear how annotations will perform to extend to other characters. One needs to do heavy annotation again to support a different character. This paper didn't give an efficient way for prompt engineering.\n\n>**A1**: We sincerely appreciate your comments regarding the annotation effort and costs in our paper, we want to clarify our intentions with:\n\n-  Firstly, ****our core motivation is to explore the capabilities of current LLMs in character alignment within stories****. To achieve this, ****we create a benchmark called HPD with detailed annotations, aiding the study of character alignment****. Our experiments show that such finely annotated data significantly improves the model's performance in this regard.  Notably, our paper represents the initial step in character alignment, and we have indeed experimented with various prompting methods. However, the pursuit of more efficient prompt engineering or annotation is reserved for our future work, presenting a challenge we leave to the research community.\n\n- Secondly, please note that we have emphasized our goal of providing as much background information as possible to assist researchers in character alignment (Line 381-392). As a result, we have allocated funds to annotate any information we believe could be useful and have integrated it into the dataset. One of our intentions is to offer insights into what kind of information might be necessary to test and enhance the character-aligning capabilities of LLMs. Therefore, in researchers' own practical scenarios, they can annotate information that suits the application's context as needed. ****It is not essential to carry out annotations at the same detailed level as we have done.****\n\n- Lastly, the cost of annotating our dataset is less than 10k US dollars, a remarkably economical investment even for industrial companies. Notably, this cost is significantly lower than the salary of a researcher (which exceeds 100k US dollars) dedicated to this project. In real-world scenarios, we firmly believe that if such finely annotated data can be leveraged to create a dialogue agent that aligns closely with game NPCs, ****the associated annotation efforts and costs would be negligible for game companies even if we need to extend the heavy annotation to different characters.****\n\n**Q2**: Also, it is unclear whether this could also help with arbitrary chat prompts used by arbitrary users in the real world. Because users will not give those informative prompts, which makes the application not quite clear and useful.\n\n>**A2**: We appreciate your review and we would like to take this opportunity to respectfully address the concern you raised regarding the clarity and usefulness of our dataset's application. \n\nIn various practical contexts, there is often a demand for intricate background information and prompts. Please allow us to take the LLM-driven NPCs in a gaming scenario as an example to describe the needed background information at least:\n\n- **Complex Game Numerical System**: In actual gaming scenarios, intricate numerical systems are in place such as Location,  Mission, Records, Level, Experience, Health Points, Magic Points, Attack, Defence, Damage, which is much more complex than the Scene and Relation information exhibited within our paper.\n- **Elaborate Skill and Equipment System**: Taking World of Warcraft as an example, there are 12 roles and 13 races in the game, with each class having over a hundred skills. Additionally, each character can have 18 pieces of equipment, with each piece of equipment having hundreds to thousands of options. This is a system with a complexity far beyond the 'attribute information' of this dataset.\n- **Dynamic Timeline**: In exploration and mystery-solving games, players progress at different rates. Therefore, when  NPCs respond, they must not reveal future information to maintain the players' immersion and experience.\n- **Character Relationship**: This includes affiliations, gender, and more.\n\nAll this background information significantly influences dialogues between NPCs and players in specific contexts. Thus, it becomes evident that the annotations in our paper serve as a simulation of real-world application scenarios. ***Consequently, our labeled data is not \"excessively granular or informative\" compared to genuine scenarios; Rather, it remains \"insufficiently granular or informative\" by comparison***. That is, real-world applications may need more complex informative prompts.\n\nFurthermore, please note that we've emphasized our goal of providing as much background information as possible to explore the advancement in **fine-grained character alignment of LLMs** .  Accordingly, we allocate resources to annotate and incorporate potentially beneficial information into our dataset. Researchers are afforded the autonomy to select information tailored to their specific needs. If a specific piece of annotation isn't applicable in their potential use case, they're free not to use it. That is, they don't need to use such informative prompts as ours.\n\n**Q3**: Explain more about the scene summarization, including window size and hallucination problem.\n\n>**A3**: Thanks for your consideration of our scene summarization. Our scene summarization encompasses the following key points:\n\n- Our prompts for scene summarization are: \"\"Below, there will be a scene from the Harry Potter novels along with its chapter reference. Your task is to summarize what happens in the scene within several sentences. Please be comprehensive and fluent.\"\"\n- When utilizing GPT-4 (4k max lengths) to summarize the scene, the maximum output tokens is 500.\n- To address potential issues of hallucination, we employ two expert annotators to inspect and calibrate the summarized scenes, as illustrated in our paper LIne 270-273. However, during our evaluation process, we observe that GPT-4 exhibits minimal instances of hallucination when performing the task of scene summarization.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It proposed a useful benchmark that may help future research on aligning LLMs with characters (Harry Potter). The annotated dynamic scene, relationship, and attributes seems to provide a good informative framework for the alignment challenges.", "reasons_to_reject": "1. The annotated dynamic scene, relationship, and attributes seem the key to making LLM aligned, which is heavily engineered. \nTake the same Harry Potter Story,  it is unclear how this will perform to extend to other characters. One needs to do heavy annotation again to support a different character. This paper didn't give an efficient way for prompt engineering. \n2. Also, it is unclear whether this could also help with arbitrary chat prompts used by arbitrary users in the real world. Because users will not give those informative prompts, which makes the application not quite clear and useful. ", "questions_to_authors": "1. See the reason for rejection. \n2. Could you explain more about the scene summarization? What is the window size for summarization? How informative? How is the hallucination problem now even if it is relevant to the scene? Please clarify the metric of relevancy to the scene and limitations of current methods.", "paper_topic_and_main_contributions": "This paper investigates the problem of aligning LLMs with character. It mainly introduces a new benchmark dataset(HPD) which is designed to generate and retrieve Harry Potter’s response given a dynamic dialogue history, dialogue scene, and participate information. They conduct instruct-fintuning on open-sourced, in-context learning on closed-sourced LLMs. Both automatic metrics “reference-based automatic metrics and GPT-4 ” and human evaluation are conducted to evaluate the generated dialogue. Experimental result shows that the proposed dataset could help LLMs to align with character, however, it is still far from human expectation. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zqZDuM7Ht2", "reviewer": "Reviewer_ZU76", "summary": "", "questions": "A. The dataset only contains train/test but not validation. How the model selection is performed?\n\nB. In scenes construction, when extracting the text surrounding the dialogue, how long is the extracted text? What is the prompt to summarize the text? How long is the summarized text?\n\nC. Does the surrounding text provide enough information to generate appropriate utterances, or does the model need to know more historical information as well? For example, in sec 4.4, lines 520-528 provide more information about the recap and Harry's mental activity, but this information is not included in the scene information of Table 5. Without this information, it's not surprising that models generate utterances starting with \"yes, I was…\"\n\nD. Why the comparison results between ground-truth and model-generated utterances are surprising? The ground-truth is based on the best one of the five GPT-generated response candidates, meaning at least 80% of the model output would be worse than ground-truth. \n\nE. Why not use the ground-truth selection data to train a scoring model to choose from model-generated output candidates?\n\nF. What is the contribution of each type of annotation to the model improvement?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our manuscript and for providing invaluable feedback. In response to your concerns and questions, we offer the following clarifications:\n\n**Q1**: The conclusion that \"powerful LLMs are still far from human expectations\" is questionable. It's not clear whether this is due to the insufficiency of the provided scene information (Question C) or the lack of careful selection of model outputs (Questions D, E).\n\n>**A1**: We arrive at the aforementioned conclusion primarily because even with approaches like fine-tuning or in-context learning, rich persona settings of LLMs, such as Per-Alpaca, Per-GPT3, and Per-ChatGPT, fall considerably short of human performance in both automated and human evaluations (see Table 3) during experiments. The comparison between Person-ChatGPT and human experts presented in the Table 4 further reinforces this observation. \n\nPlease refer to **A7**, **A8**, **A9** and **A10** for more details.\n\n**Q2**: The dataset's size is relatively small, and its scope is confined to the Harry Potter series. Some annotations only cover interactions between Harry and others, neglecting relationships involving other characters.\n\n>**A2**: Thanks for pointing this out! Allow us to elaborate on our standpoint. \n\n- Firstly, we have considered the interactions among different characters in our annotated dataset, which is evident in the following aspects: \n    - We have applied attribute annotations to all characters. \n    - Initially, we collected a total of 1471 dialogues between characters, from which we selected 1042 as our training set. (Line 397 in the main paper)  \n\nAnnotated attributes of all characters and the remaining dialogue data have been preserved for future endeavors, which encompasses the modeling of other characters. \n\n- Secondly, it is pivotal to acknowledge that our annotations focus exclusively on the dynamic interactions between Harry Potter and other characters, which is primarily driven by the substantial annotation costs involved.  ***An attempt to annotate interactions between all 113 significant characters would result in a labeling cost and complexity that is 113x112 = 12656 times the current expense, which is unfeasible***. Therefore, the focal point of this paper is to utilize Harry Potter as the subject to delve into the current LLMs' characters aligning's abilities.  If our dataset draws lots of attention from research communities, we are happy to go on annotations to extend other characters. \n\n**Q3**: The proposed approach relies on human annotation to improve model performance, which is impractical in real-world applications. A potential solution is to train a model to automatically generate the annotated information.\n\n>**A3**: We appreciate your review and would like to take this opportunity to respectfully address the concern regarding the clarity and usefulness of our dataset's application.\n\n In various practical contexts, there is often a demand for intricate background information and prompts. Please allow us to use a gaming example of constructing intelligent language model-driven NPCs to describe the needed background information:\n\n- **Complex Game Numerical System**: In actual gaming scenarios, intricate numerical systems are in place such as Location,  Mission, Records, Level, Experience, Health Points, Magic Points, Attack, Defence, Damage, which is much more complex than the Scene and Relation information exhibited within our paper.\n- **Elaborate Skill and Equipment System**: Taking World of Warcraft as an example, there are 12 roles and 13 races in the game, with each class having over a hundred skills. Additionally, each character can have 18 pieces of equipment, with each piece of equipment having hundreds to thousands of options. This is a system with a complexity far beyond the 'attribute information' of this dataset.\n- **Dynamic Timeline**: In exploration and mystery-solving games, players progress at different rates. Therefore, when  NPCs respond, they must not reveal future information to maintain the players' immersion and experience.\n- **Character Relationship**: This includes affiliations, gender, and more.\n\nAll this background information significantly influences dialogues between NPCs and players in specific contexts. Thus, it becomes evident that the annotations in our paper serve as a simulation of real-world application scenarios. ***Consequently, our labeled data is not \"excessively granular\" compared to genuine scenarios; rather, it remains \"insufficiently granular\" by comparison***. Therefore, real-world applications may need more complex human annotations.\n\nIn terms of training a model for automatic annotation, it's something that's receiving attention in both academia and industries. The intricate labeling in our dataset is hard to replace by model-generated annotations to this moment. Human involvement is definitely still crucial. Nevertheless, we hold the belief that following our meticulous manual annotation process, we can venture into training an annotation model utilizing our curated annotations. This approach would entail a fusion of annotations generated by the model and subsequent manual refinements, with the aim of cutting down the manual annotation effort. This is an avenue we're eager to explore in our future work.\n\n**Q4**: The paper lacks a discussion of data quality and IAA. The holdout annotator is the major way to solve annotation disagreement, which can introduce that annotator's bias.\n\n>**A4**: Thank you for your valuable feedback on data quality and inter-annotator agreement (IAA). \n\nWe have thoroughly considered these concerns and integrated the following measures in our paper.  Within our project, there is a dedicated researcher responsible for ensuring and overseeing the quality of the annotation data, who implements the following strategies:\n\n\n- **Rigorous Training**: Our annotators are all professionals who are also ardent fans of the Harry Potter series. Prior to annotation, we require them to thoroughly familiarize themselves with the plot of each book in the Harry Potter series. This proactive step significantly improves IAA  and ensures a comprehensive and faithful annotation process.\n- **Trial Annotation**: Following the training phase, we conduct a trial annotation phase. Annotators receive clear guidelines and annotated examples (e.g., Figure 3 and Figure 4 in our main paper). This lets them perform a practice annotation on a smaller dataset, helping us gauge quality. If annotators don't meet the required standards, additional training is given.\n- **Regular IAA Checks**: Annotators receive consistent feedback on their work. For instance, if there is a significant disparity between annotations from one annotator and the rest, a warning and corrective measures are provided. This iterative process ensures prompt identification and rectification of any discrepancies or errors.\n- **Regular Quality Checks**: We uphold stringent quality control by regularly reviewing and evaluating annotators' work chapter by chapter. This ensures consistent, high-quality annotations.\n- **Author's Review**: Finally, as authors, we individually review the annotated content for each book and hold focused discussions on areas that require modifications. This includes addressing inaccuracies in relationship annotations, ensuring comprehensive attribute tagging, and other relevant aspects.  This feedback is then conveyed to the annotators, requesting revisions on the annotated content. This process continues until a unanimous consensus is reached among the authors, with no further modifications needed.\n\nThese steps collectively underscore our commitment to maintaining high data quality and mitigating IAA-related issues. \nWe believe that these efforts will contribute to the credibility and reliability of our dataset.\n\n**Q5**: The dataset only contains train/test but not validation. How the model selection is performed?\n\n>**A5**: During the training process, we divided the initial training dataset randomly into training and development sets using an 8:2 ratio. At the end of every epoch, we assessed the loss of the checkpoint model on the development set. Upon completion of training, we chose the checkpoint corresponding to the lowest loss on the development set as our ultimate model.\n\n**Q6**: In scenes construction, when extracting the text surrounding the dialogue, how long is the extracted text? What is the prompt to summarize the text? How long is the summarized text?\n\n>**A6**：Thanks for your suggestion. Our scene summarization encompasses the following key points:\n\n- Our prompts for scene summarization are: ** Below, there will be a scene from the Harry Potter novels along with its chapter reference. Your task is to summarize what happens in the scene within several sentences. Please be comprehensive and fluent.**\n-  The extracted texts for summarizing scenes are consistently composed of multiple paragraphs. Considering the length of dialogues within them, we follow two guiding principles: The first is that the longest texts cannot be longer than one chapter, and the second is that extracted texts can only contain one dialogue session.\n- When utilizing GPT-4 (4k max lengths) to summarize the scene, the maximum output tokens is 500.\n\n**Q7**: Does the surrounding text provide enough information to generate appropriate utterances, or does the model need to know more historical information as well? \n\n>**A7**: Thank you for your insightful question regarding the adequacy of the surrounding text in our paper as the scene for generating appropriate utterances. \nWe would like to assure you that we have taken this aspect into careful consideration during the course of our research.\n\n***In the data collection phase, the length of these summarized scenes is determined by the maximum allowable input length of state-of-the-art models at that time.*** For instance, when we collect the data, ChatGPT has a maximum length of 4k tokens. It's worth noting that other large language models might have significantly shorter maximum input lengths due to memory constraints, like Llama and Alpaca.\n\n In our experiments, model inputs are beyond just the scene, which includes various components such as dialogues, character relations, and attributes, as well as a timeline (see Table10 and Table 11). **These supplementary elements were meticulously crafted to enrich the context available to the model. However, they may also lead to exceeding the model's maximum input length.** Therefore, we limit the length of summarized scenes in our dataset as we encompass not only the scene but also the conversational dynamics and character interactions in inputs, although sometimes summarized scenes can not convey enough contextual information.\n\nGiven the recent advancements in language model capabilities, including the extended input length of up to 16k tokens in ChatGPT, we acknowledge the opportunity to further enrich our scene annotations. We plan to leverage these advancements to improve the depth and breadth of contextual scenes provided to the model.\n\n**Q8**: Why the comparison results between ground-truth and model-generated utterances are surprising? The ground-truth is based on the best one of the five GPT-generated response candidates, meaning at least 80% of the model output would be worse than ground-truth.\n\n>**A8**：Thanks for your careful consideration of our experiments in Table 4.  \n\n **In annotation**, we use vanilla prompts (see Table 8) without any persona and scene information. In annotation, it became evident that the responses generated by these models in this setting were notably subpar, often necessitating subsequent manual revisions.\n\n**In experiments**, prompts of ChatGPT are more complex and with rich personal settings (denoted as Per-ChatGPT in the paper). \n\n***Thus, their prompts are different.*** The experimental results, contrary to our common sense that ChatGPT has already reached human-level performance in conversations, show a clear advantage in the task of aligning with Harry Potter.  So we argue it is surprising. \n\n**Q9**: Why not use the ground-truth selection data to train a scoring model to choose from model-generated output candidates? \n\n>**A9**：Thanks for your kind suggestion.  In this paper, we mainly focus on the construction of HPD and the validation of some baseline methods.  In our future work, as you've suggested, we plan to delve deeper into more effective training methods for character alignment.\n\n**Q10**: What is the contribution of each type of annotation to the model improvement?\n\n>**A10**: As our paper explains, evaluating model performance most accurately involves manual assessment. ***However, manual evaluation is complex and costly, and ensuring consistent scoring for ablation studies is challenging.*** This is why we don't present ablation study results for each annotation type in our paper.\n\nStill, our analysis uncovers interesting insights. Among all types of annotations, scene annotations clearly have the most impact on model responses. Notably, in specific cases like the one shown in Figure 1, attributes and relations also influence performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-organized. \n\nThe problem addressed in this study is important.\n\nThe introduced dataset can be interesting as it includes annotations of dynamic attributes and relationships.", "reasons_to_reject": "The conclusion that \"powerful LLMs are still far from human expectations\" is questionable. It's not clear whether this is due to the insufficiency of the provided scene information (Question C) or the lack of careful selection of model outputs (Questions D, E).\n\nThe dataset's size is relatively small, and its scope is confined to the Harry Potter series. Some annotation only covers interactions between Harry and others, neglecting relationships involving other characters.\n\nThe proposed approach relies on human annotation to improve model performance, which is impractical in real-world applications. A potential solution is to train a model to automatically generate the annotated information.\n\nThe paper lacks a discussion of data quality and IAA. The holdout annotator is the major way to solve annotation disagreement, which can introduce that annotator's bias.", "questions_to_authors": "A. The dataset only contains train/test but not validation. How the model selection is performed?\n\nB. In scenes construction, when extracting the text surrounding the dialogue, how long is the extracted text? What is the prompt to summarize the text? How long is the summarized text?\n\nC. Does the surrounding text provide enough information to generate appropriate utterances, or does the model need to know more historical information as well? For example, in sec 4.4, lines 520-528 provide more information about the recap and Harry's mental activity, but this information is not included in the scene information of Table 5. Without this information, it's not surprising that models generate utterances starting with \"yes, I was…\"\n\nD. Why the comparison results between ground-truth and model-generated utterances are surprising? The ground-truth is based on the best one of the five GPT-generated response candidates, meaning at least 80% of the model output would be worse than ground-truth. \n\nE. Why not use the ground-truth selection data to train a scoring model to choose from model-generated output candidates?\n\nF. What is the contribution of each type of annotation to the model improvement?", "paper_topic_and_main_contributions": "This paper introduces the Harry Potter Dialogue (HPD) dataset, which consists of bilingual dialogue sessions from the Harry Potter series, enriched with background information such as dialogue scenes, speakers, character relationships, and attributes. The primary focus is on the task of generating character-specific dialogues, and the results demonstrate that the provided annotations significantly enhance the quality and relevance of the generated dialogues.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ma32skL6Ij", "reviewer": "Reviewer_R7XD", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we sincerely thank you for dedicating your time and effort to review our paper and provide valuable feedback. In response to the concerns raised, we would like to offer the following clarifications:\n\n**Q1**: The number of professional annotators for dialogue construction is too small. I also understand crowd workers could not handle this work, but I still have concerns about dataset bias from the small workers.\n\n>**A1**: Thank you for your thoughtful concerns. We appreciate the importance of addressing the concern about the limited number of professional annotators for our dataset annotations.\n\nBased on our annotation experience, there's a **Trade-Off** between annotation quality and the number of annotators, especially in tasks requiring high expertise and detailed instructions: \n\n*Having fewer annotators allows for better control over annotation quality and inter-annotator agreement through one-on-one training and testing. As the number of annotators increases, providing individualized instruction and testing becomes challenging, potentially leading to lower annotation quality and reduced inter-agreement among annotators.*\n\nIn scenarios with high expertise requirements, like annotating our dataset, the quality of annotators takes precedence. Within our project, there is a dedicated researcher responsible for ensuring the data quality and reducing dataset bias:\n\n- **Rigorous Training**: Our annotators are all professionals who are also ardent fans of the Harry Potter series. Prior to annotation, we require them to thoroughly familiarize themselves with the plot of each book in the Harry Potter series. This proactive step significantly improves IAA  and ensures a comprehensive and faithful annotation process.\n- **Trial Annotation**: Following the training phase, we conduct a trial annotation phase. Annotators receive clear guidelines and annotated examples (e.g., Figure 3 and Figure 4 in our main paper). This lets them perform a practice annotation on a smaller dataset, helping us gauge quality. If annotators don't meet the required standards, additional training is given.\n- **Regular IAA Checks**: Annotators receive consistent feedback on their work. For instance, if there is a significant disparity between annotations from one annotator and the rest, a warning and corrective measures are provided. This iterative process ensures prompt identification and rectification of any discrepancies or errors.\n- **Regular Quality Checks**: We uphold stringent quality control by regularly reviewing and evaluating annotators' work chapter by chapter. This ensures consistent, high-quality annotations.\n- **Author's Review**: Finally, as authors, we individually review the annotated content for each book and hold focused discussions on areas that require modifications. This includes addressing inaccuracies in relationship annotations, ensuring comprehensive attribute tagging, and other relevant aspects.  This feedback is then conveyed to the annotators, requesting revisions on the annotated content. This process continues until a unanimous consensus is reached among the authors, with no further modifications needed.\n\nThese steps collectively underscore our commitment to maintaining high data quality and mitigating IAA-related issues. We believe that these efforts will contribute to the credibility and reliability of our dataset.\n\n**Q2**: It is not realistic to have well-defined background information for fine-grained character alignment in real scenarios. Thus, the generalizability (Availity) of the dataset is limited.\n\n>**A2**: We appreciate your review and we would like to take this opportunity to respectfully address the concern you raised regarding the clarity and usefulness of our dataset's application.\n\n In various practical contexts, there is often a demand for intricate background information and prompts. Please allow us to use a gaming example of constructing intelligent language model-driven NPCs to describe the needed background information:\n\n- **Complex Game Numerical System**: In actual gaming scenarios, intricate numerical systems are in place such as Location,  Mission, Records, Level, Experience, Health Points, Magic Points, Attack, Defence, Damage, which is much more complex than the Scene and Relation information exhibited within our paper.\n- **Elaborate Skill and Equipment System**: Taking World of Warcraft as an example, there are 12 roles and 13 races in the game, with each class having over a hundred skills. Additionally, each character can have 18 pieces of equipment, with each piece of equipment having hundreds to thousands of options. This is a system with a complexity far beyond the 'attribute information' of this dataset.\n- **Dynamic Timeline**: In exploration and mystery-solving games, players progress at different rates. Therefore, when  NPCs respond, they must not reveal future information to maintain the players' immersion and experience.\n- **Character Relationship**: This includes affiliations, gender, and more.\n\nAll this background information significantly influences dialogues between NPCs and players in specific contexts. Thus, it becomes evident that the annotations in our paper serve as a simulation of real-world application scenarios. Consequently, our labeled data is not \"excessively granular\" compared to genuine scenarios; rather, it remains \"insufficiently granular\" by comparison.\n\nFurthermore, please note that we've emphasized our goal of providing as much background information as possible to explore the advancements in **fine-grained character alignment of LLMs** .  Accordingly, we allocate resources to annotate and incorporate potentially beneficial information into our dataset. ***Researchers are afforded the autonomy to select information tailored to their specific needs. If a specific piece of annotation isn't applicable to their potential use case, they're free not to use it.***", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They present a whole-new dataset containing rich annotations to study character alignment.\n    1. Especially dynamic and multiple-character relationships have not been explored in the research community.\n2. The annotation schemes are designed deliberately, and the experimental results show that the information contributes the better character alignment.\n3. The paper reveals that role-playing within a complex context is not a trivial problem for current LLMs.", "reasons_to_reject": "1. The number of professional annotators for dialogue construction is too small. I also understand crowd workers could not handle this work, but I still have concerns about dataset bias from the small workers.\n2. It is not realistic to have well-defined background information for fine-grained character alignment in real scenarios. Thus, the generalizability (Availity) of the dataset is limited.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper introduces the 'character alignment' problem; a dialogue agent aligns itself to a specific character by demonstrating its challenges of handling dynamic and multiple character relationships and attributes. To facilitate the study, this paper introduces Harry Potter Dialogue (HPD) dataset. It consists of 1k dialogue sessions obtained from Harry Potter novels. They annotate rich background information such as dialogue scenes, speakers, character relationships, and attributes. Experimental results demonstrate that rich information plays a crucial role in character alignment compared to its baselines only leveraging dialogue history and context.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Rough-L ⇒ ROUGE-L"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a richly annotated dataset (in English and Chinese) for character alignment based on Harry Potter, aiming to provide a benchmark and also to boost research in this field. The reviewers generally appreciate the importance and usefulness of the dataset as well as clarity of presentation but also raise concerns regarding the amount of effort that would be needed to train a character-aligned system (especially one that is not aligned with Harry Potter), the small size of the dataset, and some concerns regarding annotation. The authors have provided responses to address most concerns and therefore I recommend accepting this paper in Findings.
Decision: Accept-Findings