Venue: ACL 2023

# Personality Understanding of Fictional Characters during Book Reading

 Mo Yu\({}^{1,*}\) Jiangnan Li\({}^{2,**}\) Shunyu Yao\({}^{3}\) Wenjie Pang\({}^{1}\)

Xiaochen Zhou\({}^{4}\) Xiao Zhou\({}^{1}\) &Fandong Meng\({}^{1}\) Jie Zhou\({}^{1}\)

###### Abstract

Comprehending characters' personalities is a crucial aspect of story reading. As readers engage with a story, their understanding of a character evolves based on new events and information; and multiple fine-grained aspects of personalities can be perceived. This leads to a natural problem of **situated and fine-grained** personality understanding. The problem has not been studied in the NLP field, primarily due to the lack of appropriate datasets mimicking the process of book reading. We present the first labeled dataset PersoNet for this problem. Our novel annotation strategy involves annotating user notes from online reading apps as a proxy for the original books. Experiments and human studies indicate that our dataset construction is both efficient and accurate; and our task heavily relies on long-term context to achieve accurate predictions for both machines and humans.1

Footnote 1: Available at [https://github.com/Grovev/personet_acl23](https://github.com/Grovev/personet_acl23).

+
Footnote †: footnotemark: 

+
Footnote †: footnotemark: 


challenges in dataset creation, especially the latter. To accurately mimic the human reading process, annotators would need to read entire books, which is not practical due to the significant time required.

We overcome this annotation difficulty and create a large-scale dataset for personality prediction in the reading processes. To achieve this goal, we propose a novel annotation strategy that utilizes publicly available book notes. Recent online reading apps such as WeRead2 (shown in Figure 1) allow users to take notes while reading a book. As users read, they can add notes at the current reading position. These notes are linked to specific text in the book, which is highlighted with a dotted underline, referred to as **underlined texts** throughout this paper. This mechanism ensures that the notes accurately reflect the thoughts the user had while reading the surrounding text of the underlined text.

Footnote 2: [https://weread.qq.com/](https://weread.qq.com/).

Based on this resource, we propose our strategy of annotating _user notes as a delegate of the book reading process_. Specific to our task of personality prediction, this corresponds to (1) identifying if a user note discusses the personality trait of a character; and (2) associating the trait label to the context at the note location. We take user notes that contain at least a character name and a personality trait word, and ask human annotators to confirm if the trait is a modifier of the character in the note text (_i.e._, the user note mentions that the character has the trait). The verified notes serve as nature labels of character personalities reflected by the surroundings of the underlined texts. By using this approach, we collect labeled data that only requires annotators to read short notes, without the need for knowledge about the books themselves.

With our new strategy, we create our situated personality prediction dataset, PersoNet, that contains \({\sim}32\)K instances from 33 books in the classic literature domain. We prove that our annotation strategy is _efficient_ as each worker only requires a median of 23.7s to finish one sample. The whole annotation process costed in total $2,400 and 471.8 hours (distributed to 20 working days by 11 annotators). It is also _accurate_ evidenced by both the over 88% inter-annotator agreement. In addition, we make the dataset bilingual in both English and Chinese with automatic book sentence alignment and manual character alignment.

We conduct experiments with our dataset in two folds. First, we develop various improvement over the standard pre-trained models, including enabling the models to use different types of long contexts, equipping the models with oracle history trait information, and task-oriented unsupervised training. Second, we conduct extensive human studies with people who have read the books (_i.e._, with the knowledge of the book history) and not. Our results show that (1) our task is challenging as humans with knowledge of book history can achieve more than 70% accuracy, compared to the best model accuracy of \({\sim}45\%\); (2) our task heavily requires the long context modeling, as introducing characters' history information significantly improves the model accuracy; and humans without the book history can only perform on par with models.

We make the following contributions:

\(\bullet\) A dataset, PersonNet, that is the first benchmark of _situated reading comprehension_ and of _fine-grained personality prediction_ on books. We prove that our dataset is a valid assessment to long context understanding for both machines and humans without significant shortcuts.

\(\bullet\) A novel dataset creation approach for book comprehension problems based on user notes, which is efficient and accurate.

\(\bullet\) Task-oriented unsupervised training and character history enhancement methods that improve on our task, providing insights to future work.

## 2 Related Work

Story book understanding has been recognized as a challenging and rewarding direction Piper et al. (2021); Sang et al. (2022). Many evaluation benchmarks on various narrative understanding tasks have been developed, such as plot structure analysis Saldias and Roy (2020); Papalampidi et al. (2019), question answering Richardson et al. (2013); Kocisky et al. (2018); Xu et al. (2022), summarization Ladhak et al. (2020); Kryscinski et al. (2021); Chen et al. (2022), character identification and relationship extraction Elson et al. (2010); Elsner (2012); Elangovan and Eisenstein (2015); Iyyer et al. (2016); Chaturvedi et al. (2016); Kim and Klinger (2019); Sang et al. (2022).

All of the prior work takes the entire long story as input to a model for predictions. None of them considers the _situated_ reading process like ours.

Existing strategies of dataset construction over **long stories** fall into the following categories: \(\bullet\)A straightforward way is **to have labelers read the entire stories**. Because of the huge efforts, it only 

works for short stories for young children (Xu et al., 2022) or simpler tasks like coref Bamman et al. (2019), NER Bamman (2020) and quotation attribution Vishnubhotla et al. (2022). **Using the book summaries as proxy** of the original stories, _e.g._, the creation of book-level question answering task Kocisky et al. (2018). The created data usually only covers abstract and major events in the book, as shown in Mou et al. (2021). Thus the types of comprehension skills that can be assessed with this strategy are limited. **Exploiting Web resources created by fans or experts**Flekova and Gurevych (2015) used fans' rated MBTI types to create a classification task for book characters; Ladhak et al. (2020); Kryscinski et al. (2021) created a book chapter summarization task based on summaries on the English learning websites; and Thai et al. (2022) created a book retrieval task based on quotes in literature reviews. The drawback of this strategy is that the tasks can be supported are limited by the available resources. **Automatically created clozte tests** is a traditional strategy. With specifically designed techniques, the clozes can be made resolvable only with global context, _e.g._, Rae et al. (2020); Sang et al. (2022); Yu et al. (2022). The problem of this method is that the created datasets usually have unclear assessment goals.

The limitations of these strategies make them insufficient to create datasets for our task of situated personality understanding.

## 3 Problem Definition

Our PersoNet is the first task on situated prediction of characters' personality traits in book contexts. That is, we aim to predict the traits reflected by a local snippet of book, given all the previous book content as the background (Figure 1).

Formally, we consider a local book snippet \({\cal S}^{(i)}=\{s_{k^{(i)}_{1}},s_{k^{(i)}_{2}},...,s_{k^{(j)}_{J}}\}\). Each \(s_{k^{(i)}_{j}}\) is a sentence from the book, with \(k^{(i)}_{j}\) the absolute position of the sentence in the book. Each \({\cal S}\) in our task depicts a character's personality. Therefore, it is associated with a pair of \((c,t)\), where \(c\) is a character name or alias and \(t\) is the personality trait of \(c\) that reflected by \({\cal S}\). Note that different pairs may share a same snippet. Our task is then to predict:

\[P(y=t|c,{\cal S}^{(i)},{\cal H}^{(i)}=s_{1:k^{(i)}_{1}}), \tag{1}\]

where \(s_{1:k^{(i)}_{1}}\) refers to all the sentences before \({\cal S}^{(i)}\) in the book. We split the books into training, dev and test sets, so that the evaluation characters are unseen during training. For evaluation, we adopt a multi-choice setting. For each instance, we sampled 4 negative candidates, two from the top-20 most frequent traits and the rest from the whole list. Combining the negative choices with \(t\), we have a candidate set \({\cal T}\). Our data thus form a tuple \(({\cal S},{\cal H}=s_{1:t^{(i)}_{k}},c,t,{\cal T})\).

## 4 Our PersonNet Dataset

### Data Source

List of Personality TraitsFollowing previous work Shuster et al. (2019), we use the list of 818 English personality traits from the MIT Ideonomy project.3 We translate the traits into Chinese with Youdao dictionary,4 then ask human annotators to select all the translated meanings that depict personality in Chinese. There are 499 English traits and 565 Chinese traits left that are bilingually aligned.

Footnote 3: [http://idonomy.mit.edu/essays/traits.html](http://idonomy.mit.edu/essays/traits.html).

Footnote 4: [https://cidian.youdao.com/](https://cidian.youdao.com/).

Books and NotesWe collect 100 public books available in the Gutenberg project. For each book, we find all its Chinese-translated versions on the WeRead app and collect all their user notes. We kept notes that (1) contain any traits, (2) contain any person names5 and (3) with lengths less than 100 words (relatively shorter notes can improve human annotation efficiency). We filtered out the books with less than 100 notes left, leaving 33 books and 194 of their Chinese translations. These books have 110,114 notes that contain 140,268 traits in total.

Footnote 5: We use Spacy (zh_core_web_1q) for NER.

Note ClusteringIt is common for multiple users to comment on the same part of a book, discussing the same character. When these users express similar opinions about a character, it leads to duplication. To remove this duplication for data annotating, we group the notes according to their positions, defined as the center token offset of its associated snippet \({\cal S}^{(i)}\) (_i.e._, its underlined text). Notes with distances smaller than 100 tokens are grouped, leading to 27,678 note clusters. We take the unique traits within each cluster for human labeling, which corresponds to 113,026 samples as defined in Section 3. The notes are anonymized for human annotation.

Extension of the SnipetsThe lengths of underlined texts can vary significantly, which means they may not always provide a representative context 

for reflecting a character's personality, particularly when the texts are very short. We address this issue by extending each \({\cal S}^{(i)}\) from the underlined text to a window of 480 tokens. This window is generally large enough to encompass a scene and ensures that the context relevant to the user note is included. The reason for choosing this window size is that it is typically longer than one page displayed by the WebRead App (as shown in Figure 1) -- users often write notes on the same page while reading the context, rather than flipping through previous or subsequent pages.6

Footnote 6: Therefore, if the context is not covered by the window, it suggests that the note should not be taken on that page.

### Dataset Construction

Our dataset construction consists of two major steps: (1) human annotation of user notes; (2) projection of labeled data from Chinese to English. In addition, we show that (3) our data construction strategy enables to build an accurate note classifier for automatic weakly-supervised data labeling.

**Step 1: Human Annotation** This step requires the annotators to read each user note, and determine if it discusses the personality of a character. We present the annotators with notes that contain at least one trait word in our vocabulary in Section 4.1. The note is paired with the _underlined book content_, which is optional to read, if they think the note itself is ambiguous. The annotators are then asked to (1) judge if the note is indeed about a certain character's trait; then (2) marked the target character name with the trait from the note.

The first step takes most of the human efforts. We wrote concrete guidelines (Figure 4 in Appendix A) for the decision making process. The annotators are citizens in China who have received at least high school education (which, in the Chinese education system, covers most of the general knowledge about classic literature). Therefore it is more convenient for them to work in Chinese; and Figure 4 lists both the original guidelines in Chinese and their English translations.

Our annotation interface (with English translations) is shown in Appendix A. Once the annotators confirm that the given trait word describes some characters, they are required to annotate the character name by dragging from the note text. If not, the character name will be left empty.

**Step 2: Bilingual Projection** The human annotation step has created a personality prediction dataset in Chinese. Next we project the data to English. Since the same English book may have multiple translated books in Chinese, their labeled data scattered. By projecting the labeled data to English books, the book version is unified and the annotations become dense.

According to Section 3, to create an English version of our dataset, we only need to project the traits \(t\), the characters \(c\) and the snippets (positions) \({\cal S}\). The trait \(t\) is already from a bilingual vocabulary, so we only need to focus on the latter two.

\(\bullet\)**Book Alignment** The projection of \({\cal S}\) is equivalent to finding each labeled instance's sentence positions in the English book, which is essentially a sentence alignment problem. Specifically, we sentencize the books firstly with Spacy; then utilize the _vecalign_[5] toolkit to achieve sentence alignments among books. We represent each sentence with the default number (10) of its consecutive sentences, and employ the multilingual sentence embedding _LASER7_ to embed the sentences. After that, _vecalign_ performances sentence alignments with dynamic programming based on the embeddings.

Footnote 7: [https://github.com/facebookresearch/LASER](https://github.com/facebookresearch/LASER).

With bilingual sentence alignment, the position of each labeled instance can be mapped to the corresponding position in the English book, _i.e._, \({\cal S}_{\text{en}}=\{a(s)|\forall s\in{\cal S}\}\), where \(a(s)\) refers to aligned position of the Chinese sentence \(s\) in the English book. For most of the \({\cal S}\) in our dataset, we can find consecutive \({\cal S}_{\text{en}}\) as the aligned results. There are a few instances mapped to empty. We excluded these cases in our English dataset. There are also a few instances mapped to inconsecutive English sentences, sometimes in a wide range. For this situation, we take the median position of the mapped English sentences and include the consecutive context in a window as the projection.

\(\bullet\)**Character Name Projection** We manually project the list of 377 frequent (appear >10 times in our labeled data) character names to English. We askeds two annotators to find the English names of these characters; and resolved all the inconsistency after they complete their own annotation jobs.

**Step 3: Weakly-Supervised Data** Our method reduces the problem of annotation over books to annotation over notes. This makes it possible to build a note classifier for automatic data augmentation.

We collect another 65,521 notes from the same book collection that consists of at least one trait 

word and one person name. By pairing traits with names within the same notes, we create 154,030 examples. Then we train a binary _roberta-wwm-ext_Cui et al. (2020) classifier over our human-labeled data to determine if the note discusses the character's trait, _i.e._, the same task in human annotation but without the need of marking target characters. For each human annotated note, if the note is recognized as describing a trait of a character, it is used as a positive example. For those labeled as irrelevant to character traits, _i.e._, no characters are annotated, we denote them as negative examples. Cross-validation on the human-labeled data shows that our classifier is accurate: 91.1% and 90.2% on the dev and test set. Applying our classifier to these unlabeled examples, we recognize 31,346 examples as describing characters' traits.

### Quality of the Annotated Data

This section proves the accuracy of our data construction method via human study.

**Correctness of Book Notes** First of all, we need to prove that the user notes are indeed an accurate delegate of books. That is, when a note mentions a personality of a character, whether it is highly consistent to what the book content reflects.

This study requires annotators who have read the books to make the correct judgement. We selected four books with two annotators who have read and are familiar with them. Each annotator labeled two books. We sampled in total 431 notes from these books. The annotators are required to judge if the note is accurate about the character or not. We present the corresponding underlined content along with the note, so that the annotators can identify which part the note is commenting. The results in Table 1 show that 89.1% of the notes are accurate understanding of the books. There are 9.7% _ambiguous_ examples, meaning the annotated traits are implied by the current place of the books, but might be falsified later, _e.g._, the authors may intend to mislead the readers to create surprisal or tension. These ambiguous labels give valid data for our problem of _dynamic personality prediction_, according to our description at the beginning of Section 1 and Eq. (1).

**Accuracy of Human Labels** Next, we proved that our annotation process leads to accurate human labels. This accuracy is verified in two ways. First, we compute the inter-annotator agreement, with a duplicated set of 3,000 notes during annotation. 88.67% of the duplicated samples receive consistent labels. The Cohen's Kappa Cohen (1960) is 0.849, which indicates nearly perfect agreement Viera et al. (2005). Second, as shown in the Step 3 in Section 4.2, a fairly accurate note classifier can be trained on our human-labeled data (91.1% and 90.2% accuracy on dev and test).

Both tests confirm the accuracy of our annotation strategy. Considering the relevance of the book notes (Table 1), this gives an estimation of overall accuracy around 87.6\(\sim\)89.1%. The two endpoints are computed with inter-annotator agreement and classifier accuracy accordingly. It confirms that our dataset is overall accurate.

Table 7 in Appendix B gives some difficult examples that created disagreements. There are two major sources of difficulties: (1) the trait word has multiple meanings in Chinese and the usage does not represent the sense of the trait; (2) a trait word is used to recall the general impression or history behavior of a character in an implicit way.

**Accuracy of Cross-Lingual Alignment** Finally, we evaluate the quality of the bilingual alignment. We randomly sampled 200 labeled instances for human study. We present to the annotators the snippet \(\mathcal{S}\) of each instance in the Chinese book and their aligned sentences from the English books. The human annotators were asked to rate the alignments into four grades: _perfect/high overlap/low overlap/no match_, _i.e._, all/\(>\)50%/\(<=\)50%/none of the Chinese sentences have their translations in the paired English sentences. Table 2 show that \(>\)97% of the cases fall into the _perfect_ and _high overlap_ categories. When taking texts from the median 

position of the sentences for model inputs, these categories both can make accurate projections of annotations to the English books.

### Data Statistics and Visualization

Data StatisticsTable 3 shows the statistics of our PersoNet. We give the full list of books in Appendix C. We can also see that our dataset contains a wide range of book characters. In the annotated training set, approximately 41% of the notes are about positive traits, 36% are about negative traits, and 23% are about neutral traits. This distribution reveals a slight bias, which can be attributed to the fact that users are more inclined to write notes when they have strong sentiments or opinions about a character.

Visualization of Our DatasetFigure 2 visualizes the major traits and the polarity of traits over time for two of the most popular characters. It can be found that the major traits match readers' common impressions; and the trends well align with the common feelings of readers during reading. This further confirms the quality of our data.

Detailed explanations of the figures and more examples can be found in Appendix D and Figure 7.

## 5 Models for Persona Prediction

We design models based on two different types of pre-trained models, BERT Devlin et al. (2019) and Longformer Beltagy et al. (2020). We use the latter model to investigate the strength of models that are pre-trained to handle long contexts.

### Input to the Reader Models

Our data instance consists of a tuple \(({\mathcal{S}},{\mathcal{H}},c,t,{\mathcal{T}})\). Here \({\mathcal{S}}\) is a book snippet that expresses a personality trait \(t\) of a character \(c\). \({\mathcal{H}}\) is the previous history of \({\mathcal{S}}\) in the book. \({\mathcal{T}}\) is a set of candidate traits with \(t\) as an element. The task is to rank \(t\) to the top within \({\mathcal{T}}\) given \(({\mathcal{S}},{\mathcal{H}})\) and \(c\). We represent the input \(({\mathcal{S}},{\mathcal{H}},c)\) with the following format options:

* **No history** represents the input as \(x=[c\texttt{~{}[SEP]}\,{\mathcal{S}}]\), _i.e._, does not use the history \({\mathcal{H}}\).
* **Extended history:**\(x=[c\texttt{~{}[SEP]}\,{\mathcal{S}}\,\texttt{~{}[SEP]}\,{\mathcal{H}}_{\text{ prev}}]\), where \({\mathcal{H}}_{\text{prev}}\subset{\mathcal{H}}\) includes sentences that are adjacent to \({\mathcal{S}}\), truncated by models' length limited.
* **Character history:**\(x=[c\texttt{~{}[SEP]}\,{\mathcal{S}}\,\texttt{~{}[SEP]}\,{\mathcal{S}}_{c}]\). \({\mathcal{S}}_{c}\subset{\mathcal{H}}\) includes snippets to the left of \({\mathcal{S}}\) that contains the character \(c\) in our dataset.

### Model Architectures

Our methods compute the score of an input \(x\) having a trait \(t\), based on the siamese model.

Text EncodingFirstly, we use a pre-trained LM (PLM, either BERT or Longformer) to encode \(x\) and \(t\) to the embedding space. The encoded contextualized embeddings of input and output are denoted as \({\mathbf{X}}=\text{PLM}(x)\in\mathbb{R}^{l_{x}\times d}\), where \(l_{x}\) is the length of \(x\) and \(d\) is the size of hidden states; and \({\mathbf{T}}=\text{PLM}(t)\in\mathbb{R}^{l_{t}\times d}\), with \(l_{t}\) the length of \(t\).

Baseline Siamese ModelAs our baseline models, we compute a weighted sum over \({\mathbf{X}}\) to get a vector representation of the input. Specifically, we use a linear model to compute the attention score over each token of \(x\):

\[A=\text{Attention}({\mathbf{H}}),\,\alpha=\text{Softmax}(A).\]

The attention \(\alpha_{x}\) is then used to summarize the hidden states \({\mathbf{X}}\) a vector \({\mathbf{x}}={\mathbf{X}}^{T}\alpha\).

The sequence of a trait \(t\) is usually short (_e.g._, a single word's BPE tokenization). Therefore we simply take the average \({\mathbf{t}}=\text{mean}({\mathbf{T}})\). The model makes prediction with \(t=\arg\max_{t\in{\mathcal{T}}}<\!\!{\mathbf{x}},{\mathbf{t}}>\).

Contextualization with HistoryWhen the input \(x\) contains the extended or character history as defined in Section 5.1, we need to separate the information of the history from the current context. We maintain a mask \(H\in\mathbb{R}^{l_{x}\times 1}\), such that \(H[j]=1\) if the \(j\)-th word belongs to the appended history and 0 otherwise. Two attention vectors are computed for the current snippet and the history:

\[\alpha_{s}=\text{Softmax}(A\odot(1-H)),\,\alpha_{h}=\text{ Softmax}(A\odot H).\] 

The corresponding summarized vectors are \({\bf s}={\bf X}^{T}\alpha_{s}\) and \({\bf h}={\bf X}^{T}\alpha_{h}\). The prediction function is then modified with a gating function \(\sigma({\bf s})\) added:

\[t=\arg\max_{i\in\mathcal{T}}\sigma({\bf s})<{\bf s,t}>+(1-\sigma({\bf s}))<{\bf h ,t}>. \tag{2}\]

### Unsupervised Training

Finally, we propose an unsupervised training task to improve personality prediction. The unsupervised task is used to pre-train the classifiers, before they are fine-tuned on our labeled data. The task mimics the problem definition in Section 3 and constructs tuples of \((\mathcal{S},t)\). We first extract sentences that contain trait words. If a sentence \(s_{j}\) contains a trait \(t\), we keep a local window of it as the book snippet, with the sentence itself removed. That is, \(\mathcal{S}^{(i)}=\{s_{j-w},\cdots,s_{j-1},s_{j+1},\cdots,s_{j+w}\}\). Intuitively, since \(\mathcal{S}\) provides the context of \(s_{j}\), it is informative for inferring the appearance of the trait described in \(s_{j}\). Therefore this unsupervised task helps to find narrative clues of traits thus can help to better pre-train the encoders.

The method has the limitation of not being character-specific, hence not compatible with our character-history-based models. We leave it to future work.

## 6 Experiments

### Experimental Settings

We use _bert-base-uncased_ and _longformer-base-4096_ as backbones for English experiments; and _Roberta-wwm-ext_ for the Chinese experiments.

#### Hyperparameters

For our siamese models with and without history, the most important hyperparameter is the lengths of \(\mathcal{S}\) and \(\mathcal{H}\). We set the maximal length of \(\mathcal{S}\) to 480 tokens for most of the models. For models with history we set the maximum of \(|\mathcal{S}|+|\mathcal{H}|\)=1,600. To show the better performance of our usage of history, we also compare with Longformer with a maximum \(|\mathcal{S}|\)=2K tokens (the best a single A100 GPU can handle).

The batch size is 40 for BERT-based models; and 8 for Longformer-based models with gradient accumulation every 5 batches. Each epoch of BERT and Longformer models takes \(\sim\)7 and \(\sim\)40 minutes respectively on a single A100 GPU. We set the learning rate to 2\(e^{-5}\). We conduct early-stopping on the dev set; and run 5 times to compute the average and stand derivation for all the methods.

#### Additional Baselines

Besides the models in Section 5, we further compared with the follows: \(\bullet\)**Models with Oracle Traits in History**, which uses the character's history traits in replace of the history texts. For each instance, we take its target character \(c\)'s other instances prior to it, and concatenate their groundtruth traits as a sequence to replace \(\mathcal{H}\) in the model of Eq. (2). \(\bullet\)**Char-Majority**, which always predicts the most frequent trait for a character. This is used to show the diversity of traits for the same character (_i.e._, necessity of situated prediction). \(\bullet\)**GPT-davinci** (text-davinci-003), the few-shot instruct-GPT (Ouyang et al., 2022). \(\bullet\)**ChatGPT**, which conduct zero-shot prediction on our task thus can take longer inputs. We test \(|\mathcal{S}|\)=480 and 1.6K as in our experiments with trained models. \(\bullet\)**Humans:** we present the same format of our instances with maximal \(|\mathcal{S}|\)=480 to humans to get their performance.

Furthermore, we added LoRA Hu et al. (2022) fine-tuning of the **LLAMA**Touvron et al. (2023) and **WeLM**Su et al. (2022) on our PersoNet as additional baselines. The fine-tuning of large language models and the usage of ChatGPT reflect the latest state-of-the-arts in concurrence with our work.

### Overall Results

Our main results are shown in Table 4. First, all the three models without the usages of history achieve similar results. The Longformer with a 2K window does not give better performance, showing that simply increasing the length of input without including useful history information is not helpful for our task. Second, our model with character history achieves the best results. Replacing the character history with extended history slightly reduces the dev performance but lead to significant test performance drop (according to the standard derivation). Among all the supervised-only methods, this model is the only on that maintains consistent dev and test accuracy. Third, our unsupervised training significantly improve the accuracy for all the models.

Fourth, the oracle history traits improve the supervised accuracy with a large margin. Yet for Longformer, adding character history and unsupervised training makes the gap smaller. Finally, the best human performance with knowledge of story history greatly outperforms all the models with and without oracle information with 20\(\sim\)23%, showing the challenges and great potential of our PersoNet. These results highlight the importance of incorporating history information in solving our 

task; and reveal that characters exhibit dynamic personalities that evolve over time, thus solely relying on history traits (even oracle) is limited.

The two methods based on large language models, namely GPT-davinci and ChatGPT, performed worse than the models trained on our dataset. This indicates that our task is still a challenge for these general-purpose models. Moreover, although ChatGPT performed better than GPT-davinci, it was not better overall to use the longer context length of 1.6K as compared to using shorter contexts. This suggests that ChatGPT may not have been trained to effectively utilize long context in our situated reading setting.

Chinese Task PerformanceTable 5 shows results on the Chinese version of PersoNet. The results are in general higher than those in the English setting for two reasons: (1) during annotation we have the semantic space of traits in Chinese, so their English translations may not be the most commonly used words. (2) the user notes tend to reuse words in the books, so there is higher change that some traits explicitly appear in Chinese books.

Performance of Fine-Tuned LLMsTo fine-tune the LLMs, we adopt the same setup in the ChatGPT experiments, where the same prompts serve as inputs and the ground truth answers are used as outputs. The optimization focuses on minimizing perplexity concerning the outputs. Regarding hyperparameter tuning, we specifically adjust the rank \(r\), weight \(\alpha\), and number of training epochs. For model selection, we rely on the accuracy on the development subset utilized in our human study, which sets \(r=8\), \(\alpha=1\) and 10 training epochs.

The results in Table 4 and 5 show that the finetuned LLM achieves slightly better results compared to our proposed baselines. However, it still significantly lags behind human performance by a considerable margin. Interestingly, unlike the other models and humans, the fine-tuned LLM perform better on the testing subset compared to the development one. Our hypothesis is that the testing book _Notre-Dame de Paris_ is more popular on the Internet, thus may be more sufficiently trained during the pre-training stages of LLaMA and WeLM.

The LLM fine-tuning results can be potentially improved by employing a contrastive training approach similar to our proposed models. We leave this to future study.

### Human Study

We conduct human study to understand the challenges of our task. We sampled instances from the two books that have most instances from the development and testing sets; and asked human annotators (who are co-authors of the paper but have not seen the labeled data before) to complete our multi-choice task. There are two types of annotators: Type-I who have not read the books before (_human w/o history_); and Type-II who have read the books (_human w/ history_).

We have annotated in total 160 samples. Each sample is guaranteed to be annotated by two humans, one with history and one without history.

Ratio of Ambiguous InstancesSometimes an event in a book can depict multiple aspects of personality. When the sampled negative choices share similarity to these personality traits, it leads to ambiguous cases with more than one correct answers. To investigate these cases, we require the Type-II

\begin{table}
\begin{tabular}{l c r r} \hline \hline
**System** & **Len** & \multicolumn{3}{c}{**Accuracy**} \\  & & **Dev** & **Test** \\ \hline Random & – & 20.00 & 20.00 \\ Frequent Traits & – & 14.10 & 12.75 \\ \hline BERT (no-hist) & 480 & 45.01\(\pm\)0.64 & 42.96\(\pm\)1.07 \\ + unsup & 480 & 46.18\(\pm\)0.49 & 44.93\(\pm\)1.01 \\ Longformer (no-hist) & 480 & 45.02\(\pm\)0.45 & 42.75\(\pm\)0.97 \\ Longformer (no-hist) & 2K & 45.00\(\pm\)0.44 & 42.42\(\pm\)0.39 \\ Char-Hist-Longformer & 1.6K & 45.50\(\pm\)0.54 & 45.33\(\pm\)1.11 \\ + unsup & 1.6K & **46.39\(\pm\)**0.63 & **45.85\(\pm\)**0.72 \\ w/ extend-hist & 1.6K & 45.46\(\pm\)0.67 & 43.44\(\pm\)0.72 \\ + unsup & 1.6K & 45.93\(\pm\)0.52 & 44.54\(\pm\)1.49 \\ \hline \hline \end{tabular} 
\begin{tabular}{l c c} \hline \hline
**w/ Oracle Information** & \multicolumn{3}{c}{_W/ Oracle Information_} \\ BERT + hist traits & 480 & 50.15\(\pm\)1.01 & 50.02\(\pm\)1.03 \\ Longformer + hist traits & 2K & 48.66\(\pm\)0.96 & 48.11\(\pm\)1.21 \\ Char-Majority & – & 16.10 & 17.25 \\ \hline GPT-davinci 5-shot* & 480 & 34.88 & 31.51 \\ ChatGPT 0-shot* & 480 & 33.72 & 42.47 \\ ChatGPT 0-shot* & 1.6K & 36.05 & 36.99 \\ LLaMA + LoRA-sft* & 1.6K & 47.67 & 49.32 \\ Human w/o history* & 480 & 44.19 & 40.54 \\ Human w/ history* & 480 & 69.77 & 65.75 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Overall performance (%) on our PersoNet-en task. (*) Results were conducted on a subset of the dataset.

 

annotators to mark the instances that they believe have ambiguous labels.8 There are 41 ambiguous samples recognized, _i.e._, \(\sim\)25% of the cases have more than one correct answers. This indicates an \(\sim\)**87.2% approximated upperbound** accuracy of our task, if we consider each ambiguous instance has two choices that are correct.

Footnote 8: Because these people have memory of the books, they can accurately distinguish the ambiguous cases from those can be disambiguated by the history.

In the future, we can leverage our note clusters to mitigate this ambiguity by ensuring that negative candidates do not appear in the cluster from which the snippet originates.

Main FindingsThe knowledge of book history is not only important to models, but also to humans. Table 6 compares humans performance with and without history. There is an \(\sim\)25% performance gap. Furthermore, human performance without history is only comparable to the best model performance (selected according to dev accuracy, which performs 47.18% and 47.21% on the full dev and test data). These results confirm that our task raises the core challenge of long context understanding.

Detailed results show that the Type-I annotators labeled \(\sim\)35% of cases that they believe unsolvable because of their lacking of the book history. After verification by Type-II annotators, there are 37 cases left for close examination. It reveals that the history information is critical for these cases for two major reasons: (1) there are multiple possible answers given the snippets but with the knowledge of the characters' history behavior the incorrect traits can be resolved (17 of 37); (2) the plots in the snippets cannot be understood and linked to any personality without book history (11 of 37). There is a third difficult category (9 of 37), where reasoning is required to draw connections, _e.g._, consequence or analogy between the current snippet and a character's previously demonstrated personality. Examples of these categories can be found in Table 10 in Appendix E.

### Analysis

Learning CurveFigure 3 plots the learning curve of our PersoNet task. The curves shows that the size of our dataset is large enough as the curves become flat after the point of 30K. More importantly, the results justify the accuracy of our data construction strategy. As adding weak supervision (all) significantly outperforms training with only human-labeled data (dotted lines).

Difficult Trait TypesWe examine the traits that appear more than 20 times in the dev set. The most difficult types include _Confident_ (0.00%), _Mature_ (5.56%), _Liberal_ (7.41%), _Humorous_ (7.69%), _Impressionable_ (8.82%), _Gentle_ (9.09%), _Optimistic_ (10.81%), _Rational_ (11.36%), _Important_ (14.29%) and _Insincere_ (16.00%). It can be found that most of the difficulty types are abstract, which are usually not explicit depicted in the books but require reasoning from characters' behaviors.

## 7 Conclusion

We propose a dataset PersoNet for the new problem of situated personality understanding of book characters. We overcome the difficulty in dataset construction with a new strategy of annotating the user notes as a proxy for the original books. Our dataset constuction method maintains both efficiency and accuracy. Experiments show that the task raised challenges of long-text understanding for both humans and machines.

### Limitations

Our propose annotation strategy can be applied to labeling other MRC problems, no matter situated comprehension ones or not. However, when generalizing to other problems other than personality prediction we studied here, the accuracy of the user notes may vary with the difficulty of tasks. Additional human verification on the correctness of

\begin{table}
\begin{tabular}{l c c} \hline \hline
**System** & \multicolumn{2}{c}{**Data**} \\  & **All** & **Unamb** \\ \hline Best model & 48.75 & 49.58 \\ \hline GPT-davinci 5-shot & 33.33 & 38.46 \\ ChatGPT zero-shot & 37.74 & 41.03 \\ LLaMA + LoRA-sft & 48.43 & 52.63 \\ \hline Human w/o history & 42.50 & 50.42 \\ Human w/ history & 67.92 & 73.50 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Comparison of performance among models and humans. The _Unambiguous subset is annotated by annotators who have read the books.

Figure 3: Learning curves with varying sizes of training data.

 

notes like in our Section 4.3 need to be conducted.

Our unsupervised training technique does not support the Longformer reader with character history (Char-Hist Longformer) yet. Therefore, the improvement from unsupervised training for our this model is smaller.

While Longformer is common in benchmarking for long story understanding tasks. There are other families of models [12, 50, 69, 78] handling long text encoding. We leave the comparison with these models to future work.

Potential RisksLike the other work that based on the similar set of books [1, 69, 78, 79], the classic literature may be limited by the time of writing, thus raise fairness considerations. However, please note that our dataset construction strategy is not limited to these books, but can work with any books on WebRead to create a sampled book set without such biases. The main reason we stick with the current list of books is for reproducibility since they are publicly available.

## References

* Ainslie et al. (2020) Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Li Yang. 2020. Etc: Encoding long and structured inputs in transformers. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 268-284.
* Bamman (2020) David Bamman. 2020. Litbank: Born-literary natural language processing. _Computational Humanites, Debates in Digital Humanities (2020, preprint)_.
* Bamman et al. (2013) David Bamman, Brendan O'Connor, and Noah A Smith. 2013. Learning latent personas of film characters. In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 352-361.
* Bamman et al. (2019) David Bamman, Sejal Popat, and Sheng Shen. 2019. An annotated dataset of literary entities. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2138-2144.
* Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. _arXiv preprint arXiv:2004.05150_.
* Chaturvedi et al. (2016) Snigdha Chaturvedi, Shashank Srivastava, Hal Daume III, and Chris Dyer. 2016. Modeling evolving relationships between characters in literary novels. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 30.
* Chen et al. (2022) Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. Summscreen: A dataset for abstractive screenplay summarization. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 8602-8615.
* Cohen (1960) Jacob Cohen. 1960. A coefficient of agreement for nominal scales. _Educational and psychological measurement_, 20(1):37-46.
* Cui et al. (2020) Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Shijin Wang, and Guoping Hu. 2020. Revisiting pre-trained models for Chinese natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_, pages 657-668, Online. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of NAACL-HLT 2019_, pages 4171-4186.
* Elangovan and Eisenstein (2015) Vinodh Krishnan Elangovan and Jacob Eisenstein. 2015. "you're mr. lebowski, i'm the dude": Inducing address term formality in signed social networks. In _The 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1616-1626.
* Elsner (2012) Micha Elsner. 2012. Character-based kernels for novelistic plot structure. In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_, pages 634-644.
* Elson et al. (2010) David K. Elson, Nicholas Dames, and Kathleen R. McKeown. 2010. Extracting social networks from literary fiction. In _ACL 2010, Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 138-147. The Association for Computer Linguistics.
* Flekova and Gurevych (2015) Lucie Flekova and Iryna Gurevych. 2015. Personality profiling of fictional characters using sense-level links between lexical resources. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1805-1816.
* Hu et al. (2022) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In _The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022_. OpenReview.net.
* Iyyer et al. (2016) Mohit Iyyer, Anupam Guha, Snigdha Chaturvedi, Jordan Boyd-Graber, and Hal Daume III. 2016. Feuding families and former friends: Unsupervised learning for dynamic fictional relationships. In _Proceedings 

of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1534-1544.
* Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. pages 874-880.
* Kim and Klinger (2019) Evgeny Kim and Roman Klinger. 2019. Frowning frodo, wincing leia, and a seriously great friendship: Learning to classify emotional relationships of fictional characters. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 647-653.
* Kocisky et al. (2018) Tomas Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. _Transactions of the Association for Computational Linguistics_, 6:317-328.
* Kryscinski et al. (2021) Wojciech Kryscinski, Nazneen Rajani, Divvansh Agarwal, Caiming Xiong, and Dragomir Radev. 2021. Booksum: A collection of datasets for long-form narrative summarization. _arXiv preprint arXiv:2105.08209_.
* Ladhak et al. (2020) Faisal Ladhak, Bryan Li, Yaser Al-Onaizan, and Kathleen McKeown. 2020. Exploring content selection in summarization of novel chapters. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 5043-5054.
* Mou et al. (2021) Xiangyang Mou, Chenghao Yang, Mo Yu, Bingsheng Yao, Xiaoxiao Guo, Saloni Potdar, and Hui Su. 2021. Narrative question answering with cutting-edge open-domain QA techniques: A comprehensive study. _Trans. Assoc. Comput. Linguistics_, 9:1032-1046.
* Myers and McCaulley (1988) Isabel Briggs Myers and Mary H McCaulley. 1988. _Myers-Briggps type indicator: MBTI_. Consulting Psychologists Press Palo Alto.
* Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _arXiv preprint arXiv:2203.02155_.
* Pang et al. (2022) Bo Pang, Erik Nijkamp, Wojciech Kryscinski, Silvio Savarese, Yingbo Zhou, and Caiming Xiong. 2022. Long document summarization with top-down and bottom-up inference. _arXiv preprint arXiv:2203.07586_.
* Papalampidi et al. (2019) Pinelopi Papalampidi, Frank Keller, and Mirella Lapata. 2019. Movie plot analysis via turning point identification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1707-1717.
* Piper et al. (2021) Andrew Piper, Richard Jean So, and David Bamman. 2021. Narrative theory for computational narrative understanding. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 298-311.
* Rae et al. (2020) Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Richardson et al. (2013) Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCtest: A challenge dataset for the open-domain machine comprehension of text. In _Proceedings of the 2013 conference on empirical methods in natural language processing_, pages 193-203.
* Saldias and Roy (2020) Belen Saldias and Deb Roy. 2020. Exploring aspects of similarity between spoken personal narratives by disentangling them into narrative clause types. In _Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, NUSE@ACL 2020, Online, July 9, 2020_, pages 78-86. Association for Computational Linguistics.
* Sang et al. (2022a) Yusi Sang, Xiangyang Mou, Jing Li, Jeffrey Stanton, and Mo Yu. 2022a. A survey of machine narrative reading comprehension assessments. In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022_, pages 5580-5587. ijcai.org.
* Sang et al. (2022b) Yusi Sang, Xiangyang Mou, Mo Yu, Dakuo Wang, Jing Li, and Jeffrey Stanton. 2022b. MBTI personality prediction for fictional characters using movie scripts. In _Findings of the Association for Computational Linguistics: EMNLP 2022, Abu Dhabi, United Arab Emirates_, pages 6715-6724. Association for Computational Linguistics.
* Sang et al. (2022c) Yusi Sang, Xiangyang Mou, Mo Yu, Shunyu Yao, Jing Li, and Jeffrey Stanton. 2022c. Tyshowguess: Character comprehension in stories as speaker guessing. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4267-4287. Association for Computational Linguistics.
* Shuster et al. (2019) Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. 2019. Engaging image captioning via personality. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 12516-12526.
* Su et al. (2022) Hui Su, Xiao Zhou, Houjin Yu, Yuwen Chen, Zilin Zhu, Yang Yu, and Jie Zhou. 2022. Welm: A well-read pre-trained language model for chinese. _CoRR_, abs/2209.10372.
* Thai et al. (2022)Katherine Thai, Yapei Chang, Kalpesh Krishna, and Mohit Iyyer. 2022. Relic: Retrieving evidence for 

literary claims. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
* Thompson and Koehn [2019] Brian Thompson and Philipp Koehn. 2019. Vecalign: Improved sentence alignment in linear time and space. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 1342-1348, Hong Kong, China. Association for Computational Linguistics.
* Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. _CoRR_, abs/2302.13971.
* Viera et al. [2005] Anthony J Viera, Joanne M Garrett, et al. 2005. Understanding interobserver agreement: the kappa statistic. _Fam med_, 37(5):360-363.
* Vishnubhotla et al. [2022] Krishnapriya Vishnubhotla, Adam Hammond, and Graeme Hirst. 2022. The project dialogism novel corpus: A dataset for quotation attribution in literary texts. In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_, pages 5838-5848. European Language Resources Association.
* Xiong et al. [2021] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty, Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. 2021. Nystromformer: A nystrom-based algorithm for approximating self-attention. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 35, pages 14138-14148.
* an authentic dataset for narrative comprehension. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 447-460. Association for Computational Linguistics.
* Yu et al. [2022] Mo Yu, Yisi Sang, Kangsheng Pu, Zekai Wei, Han Wang, Jing Li, Yue Yu, and Jie Zhou. 2022. Few-shot character understanding in movies as an assessment to meta-learning of theory-of-mind. _arXiv preprint arXiv:2211.04684_.

 

Annotation Guidelines and Interface

We show our guidelines in Figure 4; and the annotation interface with translations in Figure 5.

## Appendix B Notes that Are Difficult or Ambiguous to Label

Table 7 in Appendix B gives some difficult examples that created disagreements. There are two majors sources of difficulties. First, the trait word has multiple meanings in Chinese and the usage does not represent the sense of the trait. In the first example, "(frightening enemy)" in Chinese usually means "an very-strong enemy that is hard/impossible to beat", _i.e._, a terrible enemy. The enemy, here refers to the protagonist Dantes, does not necessary has the _brightening_ personality. Similarly, in the second example, the annotators have disagreement because some people believe in Chinese, "(extraordinary)" can be used as a personality trait only when a person possesses exceptional characteristics. While some annotators think the trait can also describe a person with exceptional abilities.

Second, a trait word is used to recall the general impression or history behavior of a character in an implicit way. In the third example, the user wanted to expresses that Elizabeth used to be clear-headed but becomes a fool at the dance party. This recall of the general impression _clear-headed_ is not explicit, but can be inferred from the next sentence that this note is commenting on a snippet of the dance party. Therefore the user aims to comment the _foolish_ trait on the snippet instead of _clear-headed_.

## Appendix C Full Book List

Table 9 shows the detailed information of each book included in our PersoNet.

## Appendix D Visualization

Trait CloudsFigure 6 include more word clouds for different characters.

Sentiment PlotsOur trait vocabulary contains in total 818 traits with polarity annotations. Specifically, there are 234 positive traits, 292 neutral traits and 292 negative traits. Figure 7 visualizes readers' sentiments towards four popular characters through the lens of traits. We map the labeled traits to their sentiments, _i.e._, positive or negative, and then plot the sentiment along time. Here the x-axis is the position of an note with trait label, normalized by the lengths of the books. The curves are smoothed within a window of 50 for _The Count of Monte Cristo_ and 20 for _Notre-Dame de Paris_, depending on the sparsity of the samples.

The trends well align with the common feelings of readers during the reading process. For example, the character _Albert_ is in general a brave and decent person. Most readers liked his personality until he recklessly challenged _Dantes_ for a duel. Then the character's reputation is saved after he found out that his father framed many people including _Dante\({}^{\text{s}}\)_ and decided to give up the duel and live off his father's ill-gotten gains. One the other hand, _Claude Frollo_ received monotone decreased rates, who appeared first as a pious and highly knowledgeable man then turned to be evil and morbid because of his obsessive for _Esmeralda_.

\begin{table}
\begin{tabular}{||l l|} \hline \hline
**Target Trait:** & **[**]** \\  **Note Text (Chinese:** & **[**]** \\  **Note Text (Chinese:** & **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**]**]** \\  **[**]** \\  **[**]** \\  **[**]**]** \\  **[**]** \\  **[**]** \\  **[**]**]** \\  **[**]** \\  **[**]** \\  **[**]** \\  **[**] 

Figure 4: Our annotation guidelines. Top: the original Chinese guidelines. Bottom: the English translation.

Figure 5: Our annotation interface (with English translations in _blue words_).

 

A Case StudyWe assessed the model performance on the points where people's ratings of _Albert_ have dramatic fluctuations (around \(x\)=0.8). Specifically, we compared three models: the baseline BERT model without any history, the BERT model enhanced with our unsupervised objective, and the Char-Hist Longformer, which can leverage longer historical information. The results are shown in Table 8.

Our findings revealed that both the enhanced models--BERT with the unsupervised objective and Char-Hist Longformer--achieved a similar level of improvement over the BERT baseline when considering the entire evaluation set of _Albert_. These results align with our experimental observations from the comprehensive evaluation data. However, it is noteworthy that the model incorporating the unsupervised objective exhibited a significantly greater enhancement at the slump of the curve. As mentioned earlier in this section, the author explicitly portrayed Albert's reckless personality through his actions and dialogues in this particular case. Even without prior knowledge of the events leading up to this point, humans can intuitively grasp Albert's personality traits. Our unsupervised task aims to capture the correlation between personality and the external expressions manifested within the narrative. This is why it proves to be more effective in this specific case.

## Appendix E Examples of Cases that Require History Information

The cases where history information is necessary to solve can be roughly categorized into three types according to our human study in Section 6.3. We include examples for each type in Table 10.

 

* [14] M. C. G. Kristian, and J. A. L. O'Brien, _The New Journal of Computer Science_**20**(3):679-852, 2000.

[MISSING_PAGE_POST]

 

- Napoleon ii.? " ~ no, louis xviii. " ... dates 'whole attention was riveted on a man who could thus forget his own informates while occupying himself with the destines of others. " yes, yes, " continued he, " \({}^{\prime}\) well be the same as it was in england. after charckes i., crownull ; after cornwell, charcyles it., and then jumps iii., and then some son
- law or relation, some prince of orange, a statholder who becomes a king, then new concessions to the people, then a constitution, then liberly. ah, my friend! " said the abbe, turning towards dante, and surveying him with the kindling gaze of a prophet, " you are young, you will see all this come to pass...."**

* room with that sort of official stiffness which precludes a formal communication. among worldly people manner is contagious. madamze de wilfeort received them with equal solemity. valentine entered at this moment, and the formalities were resumed. ... Category: (3) The current snippet can be associated to some previous plot where the character demonstrates a personality trait Target Character: Esmeralda Groundtruth Trait: simple Distractors: rational, mature, emotional, egocentric Snippet: is she to being yonder? " '' fool! t'is here that she is to make her apology in her shift! the good god is going to cough latin in her face! that is always done here, at midday. if 'tis the gallows that you wish, go to the greve. " " i will go there, afterwards. " " tell me, la boucourary? is it true that she has refused a confessor? " " it appears so, Ia bedengine. " " you see what a qapan she is! " "'tis the custom, moinsier. the baiffiff of the courts is bound to deliver the maleflector ready judged fore execution if he a layman, to the provers of paris ; if a clerk, to the official of the bighoric. " " thank you, sir. "" oh, god! " said flexur
- lys, " the poor creature! " this thought filled with sadness the glance which she cast upon the populence. the captain, much more occupied with her than with that pack of the rabble, was anorously rumpling her glarle behind. she turned round, entracting and smiling. " please let me alone, phoebus! if my mother were to return, she would see your hand! " at that moment, midday rang slowly out from the clock of nore
- dame. a murmur of satisfaction broke out in the crowd. the last vibration of the twelfth stroke had hardly died away when all heads surged like the waves beneath a squall, and an immense shout vent up from the pavement, the windows, and the roofs, " there she is! "fieur
- lys pressed her hands to her eyes, that she might not see. " charming girl, " said phoebus, " do you wish to withdraw? " " no, "she replied ... Table 10: Example of cases that require history information to solve. The red texts are underlined text of the notes that used to construct the labeled instance. In the first example, according to the snippet, both _simple_ and _impressionable_ are possible traits to explain the character's behavior. Only from the history that _Dantes_ is a brave and determined person, we can select _simple_ as the correct answer. In the second example, only when the readers know that _Noiriter_ (_The elder_) aims to help Valentine get immunity from the poisoned juicy, they can understand the character's wisdom. In the third example, _Esmeralda_ is not present. However, the scene of love between Phoebus and Fleur-de-Lys is quite similar to her story with Phoebus, illustrating that she was easily deceived by the man.
 



**ACL 2023 Responsible NLP Checklist+**

Footnote †: margin: The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.



C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? _Section 6.1_.
* C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? _Section 6.2. We report mean-std results with 5 runs_.
* C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? _Section 4_.
* Did you use human annotators (e.g., crowdworkers) or research with human participants? _Section 4_.
* D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? _Appendix A_.
* D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants' demographic (e.g., country of residence)? _Section 1 and Section 4.2 Step 1_.
* D3. Did you discuss whether and how consent was obtained from people whose data you're using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? _Section 4.2 Step 1_.
* D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? _Not applicable. The data is from social network thus the study follows IRB_.
* D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? _Section 4.2 Step 1_.