Multi-level Contrastive Learning for Script-based Character Understanding
Authors: Dawei Li
Hengyuan Zhang
Yanran Li
Shiping Yang
Venue: EMNLP 2023 Main (2023)
Keywords: Natural Language Processing
Character Understanding
Contrastive Learning

Abstract: In this work, we tackle the scenario of understanding characters in scripts, which aims to learn the characters' personalities and identities from their utterances. We begin by analyzing several challenges in this scenario, and then propose a multi-level contrastive learning framework to capture characters' global information in a fine-grained manner. To validate the proposed framework, we conduct extensive experiments on three character understanding sub-tasks by comparing with strong pre-trained language models, including SpanBERT, Longformer, BigBird and ChatGPT-3.5. Experimental results demonstrate that our method improves the performances by a considerable margin. Through further in-depth analysis, we show the effectiveness of our method in addressing the challenges and provide more hints on the scenario of character understanding. We will open-source our work in this \href{https://github.com/David-Li0406/Script-based-Character-Understanding}{URL}.

Multi-level Contrastive Learning for Script-based Character Understanding
Dawei Li', Hengyuan Zhang?, Yanran Li,, Shiping Yang'
Haliiioglu Data Science Institute, University of California, San Diego
?Shenzhen International Graduate School, Tsinghua University
'Independent Researcher
School of Computer Science, Beijing University of Posts and Telecommunications dal0340ucsd. edu, Thang-hy22@maal tsinghu.. edu. cn, yanranli. summe mummerdgmaill 1. com,
y com,
Abstract
Character
Sheldon
Jennifer
Story Title
TBET
The Test
Dataset
TVSHOWGUESS
ROCStories
In this work, tackle the scenario of under- standing characters in scripts, which aims to learn characters' personalities and identi- ties from their utterances. We begin by ana- lyzing several challenges in this scenario, and then propose multi-level contrastive learning framework capture characters' global infor- mation in a fine- mne-grained manner. To validate the proposed framework,
Text Length
528832
41
Character's
Sheldon:
Jennifer has big exam tomorrow.
Related
take Koothrap- paliand his dog. Re- ally give ourselves challenge."
Text
Jennifer felt bitter- sweet about it.
Table 1: Comparison between a script from TVSHOWGUESS (Sang et al., 2022b) and narrative from ROCStories (Mostafazadeh et al., 2016).
conduct exten- sive experiments on three character understand- ing sub-tasks by comparing with strong pre- trained language models, including SpanBERT, Longformer, BigBird and ChatGPT-3.5. Fx- perimental results demonstrate that our method improves the performances by considerable margin. Through further in-depth analysis, we show the effectiveness of method in ad- dressing the challenges and provide more hints on the scenario character understanding. will open-source work in this URL.
Although pre- pre-trained language models (PLMs) have demonstrated their effectiveness language and vision research fields (Qiu et al., 2020; Min et al., 2023), script-based character understand- ing is yet a hard task, as shown in our experi- ments. Here we highlight two challenges. The first one is text type. As scripts mainly consist of conversations between different characters, the core of script-based character understanding is conversation understanding. Especially, scripts often involve multi pallt-pprtty conversations where mul- tiple characters talk and interact with each other in a single scene. Considering other common is- sues in conversation understanding. it is non-trivial for PLMs to comprehend characters based on fine- grained conversation information (Li and Zhao, 2021: Ma al., 2022: Li al.. 2022; Tu et al., 2022). The other challenge of upplying PLMs script-based character understanding is text length. Table shows a comparison between a script from TVSHOWGUESS (Sang et al., 2022b) and a short story from ROCStories (Mostafazadeh et al., 2016). Typically, scripts are very long with even billion of words (Chen and Choi, 2016), and in turn character information are distributed globally throughout the entire script (Bai al., 2021; Inoue al., 2022). However, PL.Ms are ineffective in capturing such global information due to the sensitiveness con- text modeling (Liu et al., 2019; Joshi et al., 2020)
1 Introduction
As one essential element in stories, character com- prehension is a popular research topic in literary, psychological and educational research (McKee, 1997; Currie, 2009; Paris and Paris, 2003; Bower, 1978). To fully understand characters, individuals must empathize with characters based on personal experiences (Gernsbacher et al., 1998), construct profiles according to characters' identities, and inference about characters' future actions (Fiske al., 1979; Mead, 1990).
According to the data modality and format, char- acter comprehension can be categorized into sev- eral classes (Sang et al., 2022a). In this work, we fo- cus character understanding in scripts (Chen and Choi, 2016; 6: Sang et al., 2022b). Scripts writ- ten text for plays, movies, or broadcasts (Onions al., 1966). Typically, scripts are often structured with several text fields, including scene description, conversation, transition and summary (Saha, 2021).

---

and the limitation of input length (Dai al., 2019; Beltagy al., 2020).
benchmark the character linking and coreference resolution tasks on TV show scripts. Brahman al. (2021) collect dataset with storybooks and their summaries, and define the character description generation and character identification tasks. Sang et al. (2022b) extend the character guessing task into multi-character scenario on TV show scripts. Additionally, some works attempt combine tradi- tional self-supervised learning methods (Mikolov et al., 2013) with language models (Liu et al., 2019) to learn contextual character embeddings and apply them in downstream tasks (Azab et al., 2019; Inoue et al., 2022).
To address the aforementioned challenges, we propose a multi-level contrastive learning frame- work and capture both fine-grained and global in- formation using two devised contrastive losses. For fine-grained character information, we build cunnery-conversationn contrastive loss by com- paring character representations from different sources. Specifically, we leverage two text fields in scripts, i.e., summary and conversation, and then extract character representations from the corre- sponding field. The representations of the same character are then treated as the postive pairs, while those of different characters are negative pairs. To model the global information, we also propose novel cross-sample contrastive loss as inspired by (Bai al., 2021; Inoue al., 2022). By aligning the same character's 's representation in different sam- ples, the model overcomes the input length limita- tion and learns the global information of each char- acter. To validate the effectiveness of our frame- work, we benchmark the performances of several PLMs. including SpanBERT, Longformer. BigBird, and ChatGPT-3.5, three widely-adopted charac- ter understanding tasks.
In this work, we focus on character understand- ing tasks in scripts. While some works benchmark sunmary-based tasks in narratives (Chen et al.. 2022; Brahman et al., 2021), we are the first to leverage script summaries as auxiliary data and learn fine- finegrrined and global character representa- tions in a novel way.
2.2 Contrastive Learning
In recent years, contrastive learning is widely used in various NLP tasks (Zhang et al., 2022b), includ- ing sentence representation (Gao et al., 2021; Kim et al., 2021), machine translation (Pan et al., 2021; Vamvas and Sennrich, 2021), text generation (Lee et al., 2020; Shu et al., 2021; Zhang et al., 2022a, 2023), and etc. Literatures in multimodal research field adopt contrastive learning for languael model training, constructing positive pairs with im- ages and their corresponding captions (Li et al., 2020; Radford et al., 2021; Yang et al., 2022). In work, also regard characters in summaries and conversations two different views of the same target and align them for better representa- tion.
In general, our contributions are as follows:
We identify two critical challenges for charac- ter understanding in scripts and propose a multi- level contrastive learning framework to address them.
Through extensive experiments, demonstrate the effectiveness of our method across multiple datasets and downstream tasks.
With further analysis, we provide some insights into script-based character understanding. All codes will be open-sourced for future research.
2
Related Work
Moreover, some works aim to construct positive pairs in global manners. Both Qin al. (2021) and Hogan et al. (2022) conduct document-level contrastive learning in the relation extraction task to align the representation of the same entity and relation. Pan et al. (2021) propose an aligned aug- mentation method that generates more positive sentence pairs in different languages to improve translation performances in non-English directions. Similarly, Qin al. (2022) acquire multilingual views of the same utterance from bi-lingual dictio- naries. Following this line of research, we propose the cross saosle contrastive learning in addition to the in-sample contrastive loss to learn character
2.1 Character Understanding
Character understanding has long been the subject considcrable interest and scrutiny. Some early works propose to extract keywords as characters' features from movies (Bamman et al., 2013) and novels (Flekova and Gurevych, 2015). Other works attempt to learn the relationship between characters in both supervised (Massey et al., 2015; Kim and Klinger, 2019) and unsupervised ways (Krishnan and Eisenstein, 2015; lyyer al., 2016).
Recently, more challenging tasks in character un- derstanding have emerged. Chen and Choi (2016)

---

Summary Encoding
Multi-levl Contrastive Learning
Sample 1
Lum Levon
Sample
Atlenioe Bosol Layer
Summary
PLM lincodeer
Charactor Embroding
Enkaneed Cluracter
Embbeldinggs
Cress-cmple Maximize
Conversation
Cros sample
Lan
Sum Coay Masiniee
Mininize
Conversation
Inference with Embedding
The overview pipeline of method. Each color represents character entity embedding. The conversation and summary encoding parts correspond to Section 4. and 4.2 respectively. The multi-level contrastive learning part corresponds Section 4.3, The inference with character embedding part corresponds Section 4.4.
Figure 1:
representations globally.
paradigm that applies different losses in different learning stages. Figure illustrates an overview pipeline of our method.
3 Preliminaries
Generally, character understanding tasks require the model to predict character's information given a segment of text. For script-based character under- standing, the provided texts often consist of con- versations within scripts. In this work, we also leverage script summaries as an additional source, We provide detailed examples in Appendix A. In practice, the model first generates character's embeddings e in the representation learning step. Subsequently, feed-forward network FFN is often adopted as the classifier with the cross-entropy loss:
4.1 Character Representation in Conversation
To obtain character representations from the conver- sation field in the scripts, we first concatenate each utterance (Joshi et al., 2020; Beltagy et al., 2020) and utilize a pre-trained language model PLM produce the encoding of the whole text H:
H = PLM((1; u2: uz!....;
(3)
Then, the character embeddings C1, C2, Cn are extracted from the contextual encoding H. After that, follow previous works (Bai et al., 2021; (1) Sang et al., 2022b) and use an attention-based layer to share the character-level information among each embedding?:
LSup
Σ
yi log(p)
El....n = Extract(H)
(4)
4 Method
C1. Attention(el....e)))
(5)
Our work presents multi-level contrastive learn- ing framework for character representation learn- ing. Firstly, follow general encoding process obtain character representations from conversa- tions and summaries. Then, we describe two novel contrastive losses capture fine-grained and global information at both in-sample and cross-sample levels. Finally, we propose a two-stage training
However, the conversations in the scripts complex and thus the character embeddings solely based on the conversations are often insufficient for fine-grained character understanding.
Wittout loss of generalization, we adopt several PLMs experiments.
We provide further details Appendix

---

4.2 Character Representation in Summary To supply more information, we leverage scripts' summaries as auxiliary data and apply contrastive learning to capture the character intricacies.
character oi we randomly select one ea; and ea respectively.
By applying the sunmary-conversation con- trastive loss, we are able to learn fine- mme-grained char- acter representations from both summary and con- versation texts.
Similar with conversation encoding, given a sum- mary S contains a group of character mentions {emf, cm- ...y cm also encode the whole summary and extract the character representations:
4.3.2 Cross-sample Contrastive Learning
In addition fine-grained information, global-level information is also crucial for character representa- tion learning (Bai al., 2021; Inoue et al., 2022). To this end, we also propose a cross-sample con- trastive learning to align the same character repre- sentation in different samples within batch:
H, = PL
eg = Lsttrt; + lend; <=i<==
where tstart, and fend, are the first and last tokens of the ith character mention emf in the summary, After that, we follow (Bai et al., 2021) and use a mention-level self-attention (MLSA) layer 3 to gather information for each character embedding:
K
eap sim Eag
LCross
log
(10)
K Dド=1 crp
)
sim(e!, 16/)
.i..... S MLSA(ei. ex,,)
(8)
and the last layer's output eş is treated as the char- acter's representation from the summary.
SI(E + SI(Ee))
(11)
where SI(e) means the sample index of the charac- ter representation e When there are multiple rep- resentations of one given character in batch, we randomly select two from them. For cross- sample learning, impose constraint that restricts edd and e originate from different samples. K is the number of characters appearing at least two diffcrent samples within a batch. To this end, the cross-sample contrastive loss forces the model to utilize global information in a batch and thus obtain comprehensive understanding of the characters.
4.3 Multi-level Contrastive Learning
To enhance the character representations learned from the conversation and the summary, we de- velop a novel multi-level contrastive learning to capture both fine-grained and global information.
4.3.1 Sunnaayyoonversation Contrastive Learning
the local sampee level, we develop summary- conversation contrastive loss align representa- tions of the same character. This gives the model an additional perspective on character represen- tation and encourages to find a general space where different representations of the same charac- ter are closer. Concretely, the loss function for the summary-cooversation contrastive learning is:
4.4 Two-stage Training
To fully train the model, further propose two- stage training paradigm apply different losses in different learning stages.
Concretely, the first stage, we combine the two contrastive losses with the supervised loss together, and post-train the trained language model. The supervised loss serves as guidance to facilitate the contrastive learning, and stabilize the training at the very beginning. The total loss of the first stage is:
exp samlensaaa) Jimlec, (2g)
Lsum Σ
log
1=1 Σド=1 crp
where denotes the ith character, and P here is the number of characters that appear in both scripts and summaries. Also, 7 is a temperature hyper- parameter, and sim(,) stands for the similarity function". Note that in samples where conversation and summary contain multiple representations of
LTotal = x Lsup a * Lsam + 3+ LCross (12)
where X, a, 3 are and we will analyze their effects Section 6.3. After the first stage, only the supervised loss is
task ratios,
'It transformer encoder layer with repeated block. Please refer Bai et al. (2021) 1) for more details. 'Here we use Cosine similarity.
'e generally represents any character emhedding.

---

kept to train the model in the second stage. This makes the model concentrate on the downstream supervision signals.
character linking, we choose SpanBERT (Joshi et al., 2020), raanformer-architeeete pre-trained model with the contiguous random span mask strat- egy in the pre-training stage. We also adopt C?, which combines coreference resolution and char- acter linking together and achieves the SOTA per- formance in both two tasks. For character guess- ing, we use BigBird (Zaheer et al., 2020) and Longformer (Beltagy al., 2020), as they are specialized for long-form document input. We fol- low Sang al. (2022b) and add character-specific attentive pooling layer upon the the model encoders and denote them as BigBird-P and Longformer-P. Notably, we also design zero-shot and one-shot in- struction prompts and evaluate ChatGPT-3.5 (gpt- 3.5-turbo) via its official AP1 as another strong large language model baseline.
5
Experiments Setup
5.1 Tasks and Datasets
We evaluate the proposed method on three char- acter understanding tasks, i.c., coreference res- olution (Chen and Choi, 2016), character link- ing (Chen and Choi, 2016), and character guess- ing (Sang et al., 2022b).
Coreference Resolution Given conversation in scripts that contains multiple utterances and char- acter mention entity C1 C2.--.; On within it, the ob- jective of the coreference resolution task is to as-
semble all mention entities that refer to the same character in a cluster.
Character Linking The input of the character link- ing task is the same as coreference resolution. Un- like coreference resolution, the goal of character linking is to accurately classify each mention en- tity to the character in pre-defined character set {21, .2..... }m
5.3 Evaluation Metrics
For coreference resolution, we follow the previous works (Zhou and Choi, 2018; Bai et al., 2021) and use B3, CEAF44, and BLANC as our evaluation metrics. These three metrics are first proposed by the CoNNL' 12 shared task (Pradhan et al., 2012) to measure the clustering performance of the coref- erence resolution task. For character linking and character guessing, use Macro and Micro Fl evaluate the models' classification performances.
Character Guessing Distinct from previous tasks, the character guessing task focuses on identifying the speaker for each utterance in scripts. In this task, each utterance within scene is segmented and fed into the model. The speaker's name pre- ceding each utterance masked and replaced with special token. The same speaker within scene is represented by the same special token. The ob- jective of the character guessing task to predict the identity of the speaker for each special token. Datasets We choose two TV show datasets to con- duct experiments. For coreference resolution and character linking, use the latest released version of the Character Identification dataset". For char- acter guessing, we adopt the TVSHOWGUESS dataset? conduct experiments. We follow all the training, development, and testing separation pro- vided by the original datasets. The dataset statistics are given in Table 13 in Appendix.
5.4 Implementation Details
We employ both the base and large sizes of each model, and implement our proposed method on them. For sunmary-conversation contrastive loss, summary corpus collected by Chen et al. (2022). We follow the hyper-parameter settings in the original papers to reproduce each baseline's 's result. We repeat each experiment 3 times and report the average scores. For ChatGPT prompts and other implementation details, please refer to Appendix C and Appendix D. We will open-source all codes in this work.
Results and Analysis
6.1 Main Results
5.2 Baseline Models
Table 2 and Table 3 present the automatic evalua- tion results on the three tasks. Surprisingly, even with specialized instruction and one-shot demon- stration, ChatGPT-3.5 performs the worst among all the baselines on each task. This implies that
Following previous works, we adopt several state- of-the-art (SOTA) models in character understand- ing as baselines and apply the proposed frame- work on them. For coreference resolution and
httpss//ggithub..oommmmmrrytt cwt./moryymm/////////////m. deraacter-deetiiiic fication hhttps: h/ttt::://gthubb. www./iisiSangg///SoOWGUILA
http://pppaa.rmooopaa.a... www.pppppewnnnnneeeeoeeeoo com/docs/ api- rei-refferecc/commee pwteffrrenne/coompeeiioo///rrrtr

---

Coreference Resolution
Character Linking
MODEL
B3
CEAF44
BLANC
MICRO
MACRO
PREC.
REC.
FI
PREC.
REC.
F1
PREC.
REC.
F1
DhaGGPT-Zero-Shie
63.43
59.51
61.41
68.39
64.37
66.32
80.39
77.74
78.97
74.7
64.3
ChatGPT-One-Shee
66.43
62.54
64.43
68.47
64.44
66.40
82.19
79.40
80.70
76.2
63.6
SpanBERT-base
77.40
82.67
79.94
74.69
67.93
71.15*
84,80*
89.96
87.20
85.0F
78.4
SpanBERT-Boe (Ours)
79.95*
84.71
82.26
76.67
70.38
73.39
87
91.26
89.26
86.3
78.9*
SpanBEKT-large
81.92
85,56
83.69*
77.85
74.74
76 76.25*
88.61
91.91
90.20
87.2*
82,8*
SpanBEKT-large (Ours)
83.55
87.38*
85.42*
79.83
76.29
78.02*
89.18*
93.00
91.00
88.2
83.7
Cd -u-base
80.75
84.77*
82.71*
76.97
71.78
74.28
82,22*
91.52
89.80*
85.6
80.4*
C² base (Ours)
83.35
85.12"
84.23
76.88*
74.97
75.91
90.48
91.85*
91.15
86.4
81.1
C large
84.98
86,92*
85.94
79.63
78.16*
78.89
90.87
93.05*
91.93
87.6*
82.5*
large (Ours)
86.42
86.44*
86.24
78.82
80.42*
79.61
91.77*
93.13
92.45*
88.0*
83.2*
Table 2: Automatic evaluation results on coreference resolution and character linking. The best results are in bold. We follow previous works to present the results of coreference resolution in a 2-digital decimal and the results of character linking in a I-digital decimal. denotes that p I 0.01 in the statistical significance test.
Model
MICRO
MACRO
6.2 Ablation Studies
ChatGPT-Zero-S5
48.58
42.17
We also conduct an ablation study to examine the contributions of the two novel contrastive losses, i.e., the cross-sample loss and summary- conversation loss. To implement, SpanBERT-base and SpanBERT-large backbone models and implement model variants by removing one two contrastive losses in the training phases.
ChatGPT-One-She
51.57
44.05
BigBird-P-beet
71.01
70.32
BigBird---base (Ours)
72.61
73.00
select
BigBird-P-large
7
75.24
Bigrird-P-large (Ours)
77.68*
76.41
Longformer-PP P-hase
71.80
73.75
Longformer-PP Srreformer-P-base (Ours)
73.65*
74.22
ongformer-P-laree
77.58
75.92"
Table presents the results of our ablation study on the coreference resolution and character link- ing tasks. Compared with the vanilla SpanBERT- base and SpanBERT-large, adding one or two con- trastive losses yield better performances. Addition- ally, we observe that when applied separately, mod- els with the summary-conversatton loss work better than models with the cross-sample loss only. More importantly, it is evident that the models trained with both contrastive losses together outperform the models with only one loss, indicating the ne- cessity of our multi-level contrastive framework as well its effectiveness in addressing the two challenges, i.e., text type and text length.
Longformer-PP
Parrfrrerr-----ggeee- P-large (Ours)
78.92
76.52
Table Automatic evaluation results on character guessing. The best results are in hold.
character understanding is still hard and complex solve for large language models. Among the three tasks, models perform worse on character guessing than on coreference resolution and char- acter linking tasks. In particular, ChatGPT achieves extremely low scores of 44.05 Macro-F1 in charac- ter guessing. Since character guessing requires deeper understanding of each character and more varied narrative comprehension skills (Sang et al., 2022b), this suggests that the current pre-trained models, especially LLMs, have room for im- provement in tasks that require global and in- depth learning for specific individual.
We also conduct an ablation study on the two- stage learning strategy. Table shows the exper- iment results on C2-base using character linking and coreference resolution. While the one-stage multi-task training can also improve the baseline model's performance, we found it leads to a sub- optimal result compared with that using our two- stage learning strategy. This observation leads us to the conclusion that supervision-only fine-tuning is also very important in our method, consistently en- hancing baseline models' performance. This aligns with the findings of prior research, which advocate for task-specific fine-tuning following multi-task
Despite the discrepancies in model architecture and size, the proposed method brings significant improvements each baseline model on almost ev- ery metric, except for B3 and CEAF31 in C² -laree model. These results indicate the effectiveness and compatibility of our method.

---

cross-sample loss, respectively, Accordingly, the first block (Row 1) presents the vanilla SpanBERT- Coreference Resolution
Character Linking
MODEL
B3
CEAF64
BLANC
MICRO MACRO
PREC.
REC.
F1
PREC.
REC.
F1
PREC.
REC.
F1
SpanBERT-bar base
77.40
82.67
79.94
74.69
67.93
71.15
84.80
89.96
87.
85.0
78.4
SpanBERT-buse (Ours)
79.95
84.71
82.26
76.67
70.38
73.39
87.44
91.26
89.26
86.3
78.9
w/o cross-sample loss
79.48
83.06
81.23
74.72
71.07
72.85
87.68
90.59
89.08
86.1
80.0
w/o summary-converrstiion loss
79.00
83.36
81.11
74.68
70.33
72.44
85.45
90.61
87.85
85.6
78.8
SpanBERT-large
81.92
85.56
83.69
77.85
74.74
76.25
88.61
91.91
90.20
87.2
82.8
SpanreeT (Ours)
83,55
87.38
85,42
79.83
76.29
78.02
89.18
93.00
91.00
88.2
83.7
w/o cross-sample loss
83.85
86.68
85.24
79.44
76.37
77.88
90.68
92.47
90.65
87.4
83.6
w/o summaryy-comerrsation loss
85,29
83.96 84.62
74.65
79.20
76.69
91.45
91.15
90.80
87.9
82.8
Table 4: Ablation study results on two contrastive losses. The experiment is conducted using character resolution and character linking.
Coreference Resolution
Character Linking
80
Global & In-depth Evidence
B3
[CEFF44
BLANC
MICRO MACRO
Local & Contextual Evidence
Our method
+2.7%
+1.6%
C2-base
82.71
74.28
89.80
85.6
80.4
76
C2 O2-baac-OS
83.58
74.28
90.63
86.1
80.8
C2-base-TS
84.23
75.91
91.15
86.4
81.1
MICORFF
+1.1%
+2.4%
Table 5: Ablation study results on two-stage learning strategy. -OS and -TS represents the one-stage training and two-stage training respectively. For one-stage train- ing, we remove the second supervised loss-only stage and adopt the multi-task training only.
70-
66
Longformer-P-oe
Longformer- P-large
Model
post-training (Guan et al., 2020; Han et al., 2021).
Figure 2: Evidence type analysis result.
CEAF44
BLANC MICRO MACRO
83.69
76.25
90.20
87.2
82.8
1.0
0.0
0.0
83,98
76.12
90.75
87.8
82.2
0.0
1.0
1.0
85.04
77.72
90.77
86.4
78.6
1.0
1.0
85.42
78.02
91.00
88.2
83.7
0.5
1.0
1.0
85.
14
78.15
91.02
88.4
83.2
large performance w/o our framework, and the second block (Row 2 and Row 3) shows the model variants with only supervision loss or con- trastive losses. Comparing the first and second block we can see, there is no obvious improve- ment when only keeping the supervised loss, a.k.a susupen
1.0
0.5
0.5
85.23
79.00
90.96
88.1
82.0
Table 6: Hyper-parameter analysis results corefer- ence resolution and character linking. For coreference resolution, we report the F1 scores of the B3, CEAF44 and BLANC metrics.
À 1.0, a 0.0, 3 0.0 in the first stage. More- over, when is to 0.0, the model trained without supervised loss also exhibits inferior performances. e.g., there a notable decrease in Macro FI (from 82.8 to 78.6). This finding supports our hypothesis that the task-specific supervision signal plays a crucial role in guiding the two contrastive learn- ing. When examining the last block (Row 4-6), we observe that the models w/ our framework under different task ratios consistently surpasses the oth- ers (except only one MARCO metric). This further demonstrates the robustness of our method on the task ratio
6.3 Analysis on Hyper-Parameters
The task ratio setting is also an important compo- nent of our method. In this section, we investigate their impacts by testing various task ratios in the first training stage. We employ the SpanBERT- large model and perform experiments on the coref- erence resolution and character linking tasks.
The results of the hyper-paramend analysis are presented in Table As defined in Equation 12, X, a, and 3 represent the ratios of task-specific Soadan supervised loss, cummary-conversation loss and

---

6.4 Resource Availability Analysis
The proposed summary--onvvrsattin contrastive learning relies on well-organized seript datasets that include a summary of each scene. This pre- requisite could potentially limit the applicability of our approach to datasets in other languages or domains. To address this constraint, we conduct an experiment in which we replaced the manually collected summary dataset with an automatically generated onc, produced by ChatGPT. As depicted in Table 7, our results indicate that when using the auto-genee generated corpus in Hummary-conversaaee contrastive learning, a significant improvement is still observed when compared to the vanilla basc- line. This discovery further validates the adaptabil- ity of our method, irrespective of whether golden or generated summaries are used.
(a) Longforme--PLarge
Large (Ours)
Figure Character emhedding visualization result.
& Textual evidence (1.1% and 1.6% for the base and large models respectively). Based on these results, we safely conclude that our framework is effective in facilitating character information modeling, especially for global information.
6.6 Visualization
The core of our method is to learn fine-grained and global character representations. To this end, we also visualize the learned character embeddings in the character guessing task. Specifically, we use character embeddings in the test set of the "FRIENDS" subset of TVSHOWGUESS dataset) and randomly choose embeddings for each char- acter from different samples.
Coreference Resolution Character Linking
B3
CEAF
BLANC MICRO
MACRO
C2-base
82.71
74.28
89,80
85.6
80.4
C2 (2- 84.14]
76.06
90.89
86.1
80.9
C2-base-G
84.23
75.91
91.15
86.4
81.1
Table 7: Experiment results with automatically gener- ated summarization. -L.M and -G denote the model trained on summaries generated by ChatGPT and those trained using the dataset provided by (Chen et al., 2022).
Figure 3 shows the visualization results using T-SNE (Van der Maaten and Hinton, 2008). We compare the character embeddings generated by Longformer-P-Larg w/ and w/o our framework. One thing note is that without our framework, some character embeddings of Ross overlap with those of Rachel. This is because that in the TV show "FRIENDS", Ross and Rachel are partners and together appearing and engaging in many scenes. In contrast, this overlapping phenomenon is greatly mitigated. Overally speaking, our frame- work encourages the embeddings belonging to the same character exhibit more compact clustering pattern. This finding provides new perspective to understand the effectiveness of our proposed method in character comprehension tasks.
6.5 Breakdown to Evidence Type
To better understand when and how our method works on each sample, we conduct an evidence type analysis on the character guessing task based on the fine-graine aneeggrinedd annotation provided by Sang et al. (2022b). To remedy the scarcity issue in the original annotations, we merge the fine-grained annotation categories into two broader categories: Global & In-depth Evidence and Local & Textual Evidence. More details evidence type merging is described in Appendix E.
The results of evidence type analysis are pre- sented Figure 2. Note that our framework works better when Local & Textual evidence is required for character guessing than Global & In-depth ev- idence. This finding aligns with our intuition that Global & In-depth evidence is more challenging for the model to comprehend. It is also worth noting that our framework yields larger increases for sam- ples requiring Global In-depth evidences (2.4% and 2.7% for the base and large size models re- spectively), as compared to those requiring Local
6.7 Case Study
We also choose challenging sample from "The Big Bang Theory" subset of TVSHOWGUESS in the character guessing task, and analyze the pre- dictions from Longformer----aatt w/o and w/ our method, well as that from ChatGPT. As shown in Table 8, all the predictions from ChatGPT are wrong, indicating ChatGPT lacks a fine-grate -ne-ggai understanding of each character, Be-

---

sides, the only difference between the vanilla model w/ and w/o our framework is whether the speaker Pl is predicted correctly or not. this case, predict- ing P1 is particularly challenging, as few utterances are spoken by this character. Hence, it is a must for the models to guess Pl's identity using other details in the scene. By understanding the relationships be- tween P1 and other characters, our method is able correctly predict that P1 is Sheldon's partner, Amy. This demonstrates that our method bene- fits the fine-grained understanding on character relationships in script-based character under- standing, e.g., character guessing tasks.
the character information globally. For example. in character guessing, anonymous speakers some- times need to be identified with some global evi- dence, like linguistic style and the character's 's rela- tionship with others. These subtle cues are usually not included in the current sample and thus require the model to learn them globally from other sam- ples (Sang et al., 2022b). However, due to the fine- tuned unavailabilityy of ICI., LLMs can only utilize local information from the current sample and lim- ited demonstrations to make inferences. We believe this is the reason that LLMs don't perform well in our script-based character understanding scenario. Additionally, we notice ICL also falls short in some other tasks that involve learning domain-specific entity or individual across multiple samples, like knowledge graph completion (Yao et al., 2023).
PO: Hey. sorry about that
P1: : No. we're sorry. We never should have been comparing relationships in the first place.
P2: : Why? We won. You know, I say, next, we take on Koothrappali and his dog. Really give ourselves a challenge.
This shortcoming in the global learning scenario. which is similar to hallucination (Yang et al., 2023) and the reverse problem (Berglund et al., 2023), can limit LLMs' application in many downstream tasks.
P3: I just want to say one more thing about this. Just because Penny and I are very different people does not mean that we'ne re bad couple.
P2 : The answer is one simple test away. Hmm? You know, it's like when thought there was possum in my closet. Did I sit around wondering? No, I sent Leonard in with pointy stick and a bag. P3: killed his Chewbacca slippers.
It appears that augmenting the number of demon- strations in the prompt could be a potential strategy for enhancing the capabilities of LLMs in these global learning tasks. Nonetheless, it's essential to note that incorporating an excessive number of relevant samples as demonstrations faces practical challenges, primarily due to constraints related to input length and efficiency considerations. the future, more efforts are needed to explore optimal ways of harnessing the ICL method of L.MSs in such global learning scenarios.
PO: Let's just take the test.
P3 No, no, no, I don't want
PO: Oh, well. 'cause you know we'ne re gonna do bad.
P3 : Because it doesn't 't matter. don't 't care if we're two.
P2: Or a one. A one possible.
P3 : Marriage is scary. You're You're scared, I'm scared. But
it doesn't 't make me not want do it. It, just makes me want to hold your hand and with you. PO: Leonard.
Pl: It makes me happy you said things like that.
P2: : We got an right-point--we Trust me, you't re happy.
ChatGPT:PO: Leonard, Pl: Sheldon, P2: Penny, P3:Howard
Vanilla: PO: Penny, Pl: Howard, P2: Sheldon, L3:Landddd
8 Conclusions
Ours: PO: Pemny, Pl: Amy, P2: Sheldon, P3:Laadard
In this work, we focus on addressing two key challenges, text length and text type in script- based character understanding. To overcome these challenges, we propose a novel multi-level con- trastive framework that exploits in-sample and cross-sample features. The experimental results on three tasks show that our method is effective and compatible with several SOTA models. We also conduct in-depth analysis to examine our method detailedly and provide several hints in the character understanding tasks.
Golden: PO: Penny, Pl: Amy, P2: Sheldon, P3:Leonard
Table 8: An example chosen from "The Big Bang The- ory" in the character guessing task. We analyze the predictions made by ChatGPT (one-shot), Longformer- P-Large (vanilla and with our framework).
7 Discussion about LLMs on Character Understanding
In this section, we go deeper to discuss the unsatis- fied performance when adopting the ICL of LLMs to perform character understanding tasks. One pos- sible reason for this is the suript-based character un- derstanding we focus on requires the model to learn
In the future, we plan to apply contrastive learn- ing to other long-form document understanding tasks, such as long document matching (Jiang et al., 2019) and fiction understanding (Yu et al., 2023).

---

9 Limitations
60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 1502-8615.
Our framework depends on pre-trained large lan- guages (PLMs) to encode conversations and sum- maries, and requires gradient information to tune the PLMs' parameters. This makes it challenging to apply our approach to language models with gigantic sizes. In this work, we demonstrate the generalization of our method in the experimental section the base and large size, as well as the in- capability of ChatGPT-3.5 on character understand- ing tasks. Nevertheless, it remains unclear how well our framework will fit to 3B+ encoder-decoder PLMs or decoder-only LLMs. As our experiments suggest, there is still room for improvement in char- acter understanding tasks.
Yu-Hsin Chen and Jinho D Choi. 2016. Character identi- fication on multiparty conversation: Identifying men- tions of characters in tv shows. In Proceedings of the 17th annual meeting of the special interest group on discourse and dialogue, pages 90-100.
Gregory Currie. 2009. Narrative and the psychology of character. The journal of aesthetics and art crificism, 67(1):61-71.
Zihang Dai, Zhilin Yang. Yiming Yang. Jaime G Car- bonell, Quoe Le, and Ruslan Salakhutdinov. 2019. Transformer-xl. Attentive language models beyond fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988.
Susan T Fiske, Shelley E Taylor, Nancy Etcoff, and Jessica K Laufer. 1979. Imaging, empathy, and causal attribution. Journal of Experimental Social Psychology, 15(4):356-31 -377.
References
Mahmoud Azab, Noriyuki Kojima, Jia Deng, and Rada Mihalcea. 2019. Representing movic characters in dialogues, In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL),
Lucie Flekova and Tryna Gurevych. 2015. Personal- ity profiling of fictional characters using sense-level links between lexical resources. In Proceedings of the 2015 Conference on Empirical Methods in Natu- ral Language Processing, pages 1805-1816.
pages 99-109.
Jiaxin Bai, Hongming Zhang, Yangqiu Song, and Kun Xu. 2021. Joint coreference resolution and character linking for multiparty conversation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 539-548.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. Simese: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 6894 6910.
David Bamman, Brendan O'Connor, and Noah A Smith. 2013. Learning latent personas of film characters. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 352-361.
Morton Ann Gernshacher, Brenda Hallada, and Rachel RW Robertson. 1998. How automatically do readers infer fictional characters' emotional states? Scientific studies of reading, 2(3):271-300.
Iz Beltagy, Matthew E Peters, and Arman Cohan, 2020. Longformer: The long-do document transformer. arXiv prepriat wxXiv2200.005150.
Jian Guan, Fei Huang, Zhihao Zhao, Xiaoyan Zhu, and Minlie Huang. 2020. A mnowledge-enhaneee pre- training model for commonsense story generation. Transactions of the Association for Computational Linguistics, 8:93-108
Lukas Berglund, Meg Tong, Max Kaufmann, Mikita Balesni, Asa Cooper Stickland, Tomasz Korbak, and Owain Evans. 2023. The reversal curse: Llms trained on" is b" fail to learn" b is a". arXiv preprint arXiv:2309.1 12288.
Janghoon Han, Taesuk Hong, Byoungjae Kim, Youngjoong Ko, and Jungyun Sco. 2021. Fine- grained post-training for improving retrieval-based dialogue systems. Proceedings of the 2021 Con- ference the the North American Chapter of the Asso- ciation for Computational Linguistics: Human Lan- guage Technologies, pages 1549-1558
Gordon H Bower: 1978. Experiments on story compre- hension and recall. Discourse Processes, 1(3):211- 231.
Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, and Snigdha Chaturvedi. 2021. "let your characters tell their story": A dataset for character-centtic narrative understanding. In Findings of the Association for Computational Lin- guisties: EMNLP 2021, pages 1734-1752.
William Hogan, Jiacheng Li, and Jingbo Shang. 2022. Fine- Fine-grained contrastive learning for relation extrac- tion. In Proceedings of the 2022 Conference on Em- pirical Methods in Natural Language Processing, pages 1083-1095.
Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. Summscreen: dataset for abstrac- tive screenplay summarization. In Proceedings of the
Naoya Inoue, Charuta Pethe, Allen Kim, and Steven Skiena. 2022. Learning and evaluating character

---

representations in novels. In Findings of the Associa- tion for Computational Linguistics: ACL 2022. pages 1008-1019.
Yiyang Li and Hai Zhao. 2021. Self-and pseudo-selff supervised prediction of speaker and key- key-utterance for multi-party dialogue reading comprehension. In Findings of the Association for Computational Lin- guistics: EMNLP 2021, pages 2053-2063.
Mohit lyyer, Anupam Guha, Snigdha Chaturvedi, Jor- dan Boyd-Graber, and Hal Daumé III. 2016. Feuding families and former friends: Unsupervised learning for dynamic fictional relationships. In Proceedings of the 2016 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1534 1544.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprini arXiv: 1907::9907111999
Jyun-Yu Yu Jiang, Mingyang Zhang, Cheng Li, Michael Bendersky, Nadav Golbandi, and Mare Najork. 2019. Semantic text matching for long-form documents. In The world wide web conference, pages 795-806.
Xinbei Ma, Zhuosheng Zhang. and Hai Zhao. 2022. Structural characterization for dialogue disentangle- ment. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume I: Long Papers), pages 285-297.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre- pre-training by representing and predict- ing spans. Transactions of the Association for Com- putational Linguistics, 8:64-77.
Philip Massey, Patrick Xia, David Bamman, and Noah A Smith. 2015. Annotating character relationships in literary texts. arXiv preprint arXiv: 1512.00728.
Robert McKee. 1997. Story: style, structure, substance, and the principles of screenwriting. Harper Collins.
Evgeny Kim and Roman Klinger. 2019. Frowning frodo, wincing leia, and a scriously great friendship: Learn ing to classify emotional relationships of fictional characters. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 647-653.
Gerald Mead. 1990. The representation of fictional character: Style, pages 440-452.
Tomas Mikolov, lya Sutskever, Kai Chen, Greg S Cor- rado, and Jeff Dean. 2013. Distributed representa- tions of words and phrases and their compositionality. Advances in neural information processing systems. 26.
Tacuk Kim, Kang Min Yoo, and Sang-goo Lee. 2021. Self-guided contrastive learning for bert sentence representations. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 2528-2540.
Bonan Min. Hayley Ross, Elior Sulem, Amir
Pouran Ben Veyseh. Thien Huu Nguyen, Oscar Sainz, Kneko Agirre, lana Heintz, and Dan Roth. 2023. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Computing Surveys, 56(2):1-40.
Vinodh Krishnan and Jacob Eisenstein. 2015, "you're 'e mr. lebowski, i' 'm the dude": Inducing address term formality in signed social networks. In Proceedings of the 2015 Conference of the North American Chap- ter of the Association for Computational Linguisties: Human Language Technologies, pages 1616-1626.
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839-849.
Seanie Lee. Dong Bok Lee, and Sung Ju Hwang. 2020. Contrastive learning with adversarial perturbations for conditional text generation. In International Con- ference on Learning Representations.
Charles Talbut Onions, George Washington Salisbury Friedrichsen, Robert William Burchfield, et al. 1966. The Oxfort dictionary of English etymology, volume 178. Clarendon Press Oxford.
Dawei Li, Yanran Li, Jiayi Zhang, Ke Li, Chen Wei, Jianwei Cui, and Bin Wang- 2022. C3kg: A chi- nese commonsense conversation knowledge graph. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1369-1383.
Xiao Pan, Mingxuan Wang. Liwei Wu, and Lei Li. 2021. Contrastive learning for many -toy--to-mn to-many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol- ume 1: Long Papers), pages 244-258.
Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, et al. 2020. Oscar: Object- semantics aligned pre-training for lasionelanguage tasks. In Computer ESCVn 2020: 16th Euro- pean Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer.
Alison H Paris and Scott G Paris. 2003. Assessing narrative comprehension in young children. Reading Research Quarterly, 38(1))36-76

---

Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll- 2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint conference on EMNLP and CoNLL-shared task, pages 1-40.
Jannis Vamvas and Rico Sennrich. 2021. Contrastive conditioning for assessing disambiguation in mt: A case study of distilled bias. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10246-10265.
Libo Qin, Qiguang Chen, Tianbao Xie, Qixin Li, Jian- Guang Lou, Wanxiang Che, and Min- Yen Kan, 2022. GI-clef: A global- local contrastive learning frame- work for cross-lingual spoken language understand- ing. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (V (Volume 1: Long Popers), pages 2677-2686.
Laurens Van der Maaten and Geoffrey Hinton. 2008
Visualizing data using t-sne. Journal of machine learning research, 9(11).
Jianwei Yang. Chunyuan Li, Pengchuan Zhang, Bin Xiao. Ce Liu, Lu Yuan, and Jianfeng Gao. 2022. Uni- fied contrastive learning in image-text-labed space. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 19163- 19173.
Yujia Qin. Yankai Lin. Ryuichi Takamobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021. Erica: Improving entity and relation understanding for pre-trained language models via contrastive learning. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguisties and the 1 11th International Joint Confer- ence on Natural Language Processing (Volume 1: Long Papers), pages 3350-3363.
Shiping Yang, Renliang Sun, and Xiaojun Wan. 2023. A new benchmark and reverse validation method for passage-level hallucination detection. arXiv preprint 1rXiv:2310.06-490
Liang Yao, Jiazhen Peng. Chengsheng Mao, and Yuan Luo. 2023. Exploring large language mod- els for knowledge graph completion. arXiv preprint arXiv:2308. 13916.
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, 63(10):1872- 1897.
Mo Yu, Jiangnan Li, Shunyu Yao, Wenjie Pang. Xi- aochen Zhou, Zhou Xiao, Fandong Meng, and Jie Zhou. 2023. Personality understanding of fictional characters during book reading. arXiv preprint arXiv:2305.10156
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763. PMLR.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tanon, Philip Pham, Anirudh Ravula, Qifan Wang. Li Yang, et al. 2020. Big bird: Transformers for longer sequences. Advances in neural information processing systems, 33:17283-1722 172977
Aveek Saha. 2021. Movie Script Database.
Yisi Sang. Xiangyang Mou, Jing Li, Jeffrey Stanton, and Mo Yu. 2022a. A survey of machine narrative reading comprehension assessments. arXiv preprint arXiv:2205.00299.
Hengyuan Zhang, Dawei Li, Yanran Li, Chenming Shang, Chufan Shi, and Yong Jiang. 2023. Assisting language learners: Automated trans-lingual defini- tion generation via contrastive prompt learning. In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023), pages 260-274, Toronto, Canada. Association for Computational Linguistics.
Yisi Sang, Xiangyang Mou, Mo Yu. Shunyu Yao, Jing Li. and Jeffrey Stanton. 2022b. Tvshowguess: Char- acter comprehension in stories as speaker guessing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4267-4287
Hengyuan Zhang, Dawei Li, Shiping Yang. and Yan- ran Li. 2022a. Fine-grained contrastive learning for definition generation. In Proceedings of the 2nd Con- ference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th Interna- tional Joint Conference on Natural Language Pro- cessing, pages 1001-1012.
Chang Shu, Yusen Zhang, Xiangyu Dong, Peng Shi, Tao Yu, and Rui Zhang. 2021. Logic-consistency text generation from semantic parses. In Findings of the Association for Computational Linguistics: ACL- IJCNLP 2021, pages 441 4414-
Rui Zhang, Yangfeng Ji, Yue Zhang, and Rebecca J Passonneau. 20226. Contrastive data and learning for natural language processing. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies: Tutorial Abstracts. pages 39-47.
Quan Tu, Yanran Li, Jianwei Cui, Bin Wang, Ji-Rong Wen, and Rui Yan. 2022. Misc: mixed strategy- aware model integrating comet for emotional support conversation. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers), pages 308-319.

---

Ethan Zhou and Jinho D Choi. 2018. They exist! in- troducing plural mentions to coreference resolution and entity linking. In Proceedings of the 27th Inter- national Conference on Computational Linguistics, pages 24-34.

---

A
Example of Script-based Character Understanding Task
Input
PO: Hey, sorry about that Pl: No, we're sorry. We never should have been comparing relationships in the first place. P2: Why? We won. You know, I say, next, we take on Koothrappali and his dog. Really give ourselves a challenge. P3: just want say more thing about this. Just because Penny and very different people does not mean that we're bad couple. P2: The answer is one simple test away. Hmm? You know, it's like when thought there was possum in my closet. Did sit around wondering? No, I sent Leonard with a pointy stick and a bag. P3: killed his Chewbacca slippers. PO: Let's just take the test. P3: No, no, no, I don't 't want to. PO: Oh, well, 'cause you know we're gonna do bad. P3: Because it doesn't 't matter. I don't I care if we're a ten or two. P2: Or one. A one is possible. P3: Marriage is scary. You're scared, I'm scared. But it doesn't make me not want do it. It, it just makes me want to hold your hand and do it with you. PO: Leonard. P1: It makes me so happy if you said things like that. P2: We got an cight-point-two. Trust me, you're 'r happy. PO: Penny, Pl: Amy, P2: Sheldon, P3: Leonard
Label
Table A example from character guessing task.
Input
Ross: I told mom and dad last night, they seemed to take pretty well. Mon- ica: Oh really, so that hysterical phone call got from a woman at sobbing 3:00 A.M., "I'll never have grandchildren, I'll never have grandchildren. was what? A wrong number? Ross: Sorry. Joey: Alright Ross, look. You'l re feeling lot of pain right now. You're re angry. You're hurting, Can I tell you what the answer is?
Label-CR
Ross: told mom and dad last night, they seemed to take pretty well. Mon- ica: Oh really, so that hysterical phone call got from a woman at sobbing 3:00 A.M., "I'll never have grandchildren, I'll never have grandchildren." was what? A wrong number? Ross: Sorry. Jocy: Alright Ross, look. You's 'r feeling a lot of pain right now. You're 're angry. You't 're hurting. Can I tell you what the answer is?
Label-CL
I: Ross Geller, mom: Judy Geller, dad: Jack Geller, 1: Monica Geller, woman: Judy Geller, I: Monica Geller, Monica Geller, Ross: Ross Geller, You: Ross Geller, You: Ross Geller, You: Ross Geller, Joey Tribbiani,
you: Ross Geller
Table 10: A example from coreference resolution and character linking tasks. For coreference resolution, the goal of the task is to cluster the coreferences that refer the same character in one group (we the same color represent).
B Details of Character Embedding Generation
Here give detailed formulations of our character embedding extraction and character-level attention process.
Coreference Resolution & Character Linking Given the context encoding H, we follow (Bai et al.,
2021) to initialize the mention-level character embedding:
tstart, lend, + Espeaker,
(13)
where Extart, and fend; are the contextualized representation of the beginning and the end tokens

---

mention i, and the espeaker; is the speaker embedding for the current speaker of the utterance where the ith mention belong to. The speaker embeddings are randomly initialized before training. Character Guessing We follow (Sang et al., 2022b) to extract speaker-level character embedding from the context encoding H:
A Attention(H)
(14)
0g Softmax(A M,)
(15)
C HHH'a,
(16)
where Attention(.) ) is a one-layer feedforward network to compute the token-level attention weight. Mj is a token-level mask such that M = 1 if the juh word belongs to an utterance of the ith anonymous speaker and Mx[j] otherwise. a; is the token weight used to pool the hidden states to summarize a character representation.
After obtaining character embedding, we adopt the MLSA layer we mentioned in Section 4.2 to gather information for each character embedding:
C1, C1, Cn = MLSA(eI,....C.)
(17)
C Prompts for ChatGPT
For character linking, as Table 15 shows, we provide the original scripts' content for ChatGPT, followed by the position of the mention to be inferenced and all the optional characters. For coreference resolution, we tried several different prompts ask ChatGPT to do clustering. However, there is always omitting of mentions in the models' output which leads to very poor performance on coreference resolution. So we just use the model's output character linking as the clustering results of each character and calculate the corresponding metries for coreference resolution.
For character guessing, we provide the show name of the scripts and optional characters to the model. We concatenate them in front of the script's ss content and input the prompt to ChatGPT to do inference as Table 16 shows. For zero-shot, we try asking ChatGPT to guess one character in each request (represent (ommpt-Character and guess all characters in each request (represent as Prompt-Sample) and find the latter performs better as Table shows. One possible reason for that is the model would attend to more information from other characters' utterances given Prompt-Sample which is exactly the key perform well in character guessing, For one-shot, we only adopt the Prompt-Sample due its superior performance in the zero-shot setting.
For the one-shot setting, we additionally provide the model a sample together with its label as a demonstration. For both tasks, after getting the output of the model, we also use the RE module provided by Python to map the raw text to the most similar label.
MICRO
MACRO
Prompt- Charat-Carrrtte Prompt-Sample
43.77
37.87
48.58
42.17
Table II: Comparison of the two prompt methods in character guessing with zero-shot.
Detailed Training Settings
For every available positive pair 10 of the same character, we randomly choose one conduct contrastive learning. We set the X, and /3 to 1.0, 0.5, 0.5 respectively in the first stage of training for all three tasks. We test SpanBERT and C2 in coreference resolution and character linking and Longformer-P and Bird-r in character guessing. We ChatGPT in all three tasks. Table 12 gives parameter settings of the learning rate, batch size, and training epochs in the two stages of learning. We use Pytorch-1.8.
"For example, there are mentions clustered in sample but the model's output just contains the clustering result of them.
10 "Excluddig the situation that character show in the dialogue hut not in the summary, and vice versa.

---

deep learning framework and Transformers-4.122 library for our experiment. We train our models on a single A40 GPU. It takes about hours train SpanBert-base/ C2-base model and 20 hours train a Longformer-P-benn
BigBird-P-bass model. Large-version model training takes twice the time. Table shows the detailed statistics of the datasets we use. We also give detailed information about models in the base and large size we use in the experiment in Table 14.
Stage Batch Size
Second Stage
Dataset
LR
Epocch
Batch Size
Epoch
Coreference Resolution
lc-5
30
2c-5
100
Character Linking
Character Guessing
4e-6
20
2c-5
40
Table 12: Other hyper-parameee parameters settings our experiments. Note that for coreference resolution and character
linking, we follow the previous works (Chen and Choi, 2016; Bai et al., 2021) and incorporate every sample inside scene in a batch.
Task
Train
Validation
Test
Total
Character Linking & Coreference Resolution
987
122
192
1301
Character Guessing
10071
819
823
11713
Table 13: Detailed information about the datasets for each task.
Number of Transformer Layer
Number of Attention Head
Size
Parameter Size
Hidden Size
Base
125M
12
12
768
Large
354M
24
16
1024
Table 14: Detailed information about models in the base and large size we use in the experiment.

---

Monica Geller: Tell him.
Rachel Green: No.
Phoebe Buffay: him, tell him.
Monica Geller: Just...please tell him.
Rachel Green: Shut up!
Chandler Bing: Tell me what?
Monica Geller: Look at you, you won't even look at him.
Chandler Bing: Oh, come on tell me. could use another reason why women won't
look at me.
Rachel Green: All right, all right, all right. Last night, had a dream that, uh, you and I, were...
Scripts
Phoebe Buffay: Doing it on this table.
Chandler Bing: Wow!
Jncy Tribbiani: Exellent dream score
Ross Geller: Why, why, why would you dream that?
Chandler Bing: More importantly, was I any good?
Rachel Green: Well, you were pretty damn good.
Chandler Bing: Interesting, cause in dreams, I'm allways surprisingly inadequate.
Rachel Green: Well, last night you seemed to know your around the table.
Ross Geller: love it, when we share. Chandler Bing: You're re okay there?
Ross Geller: can't 't believe you two had sex in her dream.
Chandler Bing: I'm sorry, was one-time- thing. was very drunk and was somebndy
clse's s subconscious..
Here is an example:
<Demonstration>
Following this example and read the following conversation:
<Scripts>
Prompt
The NO.1 "him" in the utterance "Tell him ." refer to which character? you should choose answer from Ross Geller, Rachel Green, Chandler Bing. Monica Geller, Jocy Tribbiani, Phoebe Buffay, Emily, Richard Burke, Carol Willick, Ben Geller, Peter Becker, Judy Geller, Barry Farber, Jack Geller, Kate Miller, #OTHER# #GENERAI
Table 15: Original scripts and the prompt we input ChatGPT in coreference resolution and character linking. We replace the original scripts in the prompt with special token <Scripts> to save space. We use red color to represent the additional prompt and demonstration adopted one-shot settings.

---

PO: hey, frasier, i'm glad caught you, did you just get home?
Pl no, i've been here a while, can't bring myself to in. not with her in there.
Pl she's getting better.
PO: look, did you favor. my lawyer drew up this document, it releases you from all liability, if you just get ann sign it.
Pl oh, roz, there's way i 'm going to her to sign this, but i have better plan: i've just hooked passage for her and her mother on two week cruise to alaska, that way i'll get her out of my home, but we' Il still feel like we're re friends.
PO: hmm, not had idea, good luck with that.
Pl thank you. roz, i've been meaning to ask you: how did you ever become friends with ann? mean, she's really not your type, is she?
PO: oh, we're not really friends. rear ended her in 1989, PI oh, the tickets arrived. Pl: well, i'd hoped she would be. P1 you told her taking you? Pl: ann...
Pl: hold that thought, while bunny goes and pours himself big ol' glass o' wine. Pl caroline.
P1 uh...
P1 uh...just neighbor.
P1 [pained] no. since we made our plans, caroline, met someone else,
Scripts
Pl: just go!
: [darkly] don't know what i'm gonna with you either.
: oh. it's nothing. just some work stuff.
Pl: no.
Pl: no!
Pl: no, never!
PI: realize that you're angry now, bunny...
Pl: right, fine! go ahead and sue! am up with this charade! this was an accident! have cared for you, have waited on you, have pumiced your heels and set your hair! well, if that's not enough for you, so be it! i don't care anymore, i will not beg! you can take me to the cleaners but you cannot take my dignity! P1 oh dear god, please, no! pleas, no, no, please! please, please don't 't sue me! mh...thinggs, my beautiful, beautiful things. love them so... Pl: [weeping] no.
Pl you will?
Pl: thank you, ann. i'm sorry it had to come down to all this, this logal business. if it were up to me, would tear this piece of paper and forget everything that's happened here. Pl : and, uh, here.
Pl : aere..
Here is an example: >Demonstration>>
Following this example and tell me PO below are which character from TV show "Frasier",
please choose from frasier, roz, niles, martin, dapliee and bob:
Prompt-Characee
<Scripts>
Tell me P1 below are which character from TV show "Frasier", please choose from frasier, roz, niles, martin, duphne and bob: <Scripts>
Here is an example:
<Demonstration>
Prompt-Sampie
Following this example and tell me PO, P1 below are which character from TV show "Frasier", please choose from frasier, roz, niles, martin, daphne and bob:
<Scripts>
Table 16: Original scripts and the prompt we input to ChatGPT in character guessing. We replace the original scripts the prompt with a special token <Scripts> to save space. use red color to represent the additional prompt and demonstration adopted in one-shot settings.

---

E Evidence Type Merging
Sang et al. (2022b), the annotations divide the evidence types of guessing characters into extremely detailed categories, and several psychologists are asked to assign a category of evidence to each character, In specific, there are 9 types of evidence totally for guessing character identification according to Sang et al. (2022b). They are attribute, relation, status, background, exclusion, mention, linguistic, memory, and personality.
However, one drawback of the subdivided categories is the scarcity of certain categories. To address the high variance issue caused by the scarcity, we merge the fine- ane-grained annotations into broader ones, Based on the definition of them in the original paper"), we split them into 2 big categories as Table 17 shows. Here Global & In-depth evidence means cases which the character can only be predicted according to his/her global information (like one's relationship with others) or some subtle clues (like one's Linguistic style). Local & Textual evidence means cases in which a character can be easily predicted only using the content in the local sample (like background information appearing in other characters' utterances) or something very direct (like calling one character's name directly). Note that abandon the evidence exclusion because it is more like guessing technique rather than evidence.
Merged Category
Original Category
Attribute
Relation
Global & In-depth evidence
Status
Linguisties
Memory
Personality
Background
Local & Textual evidence
Mention
Table 17: Merged result of the subdivided evidence type.
Plecese refer to the original paper for more details.


Reviews:
[{"review_id": "x30Qo5IIN4", "reviewer": "Reviewer_zjYA", "summary": "", "questions": "1.\tHow is the $Extract$ layer/ method implemented in Equation 4?\n2.\tHow do you determine when the first stage training is completed?\n3.\tIn Table 5, is the hyper-parameter analysis conducted on the test set or the validation set?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your review of our work and would like to address some points you've raised, including the presentation issue and details of our method.\n\n__Definition to character understanding tasks__\nFor the first rejection reason, we acknowledge our work incorporates various script-based character understanding tasks in the experiments. To highlight our novel contrastive losses and enhance the fluency of the methodology section, we use a separate section (Section 5.1) to define each of them in detail. We value your suggestion to add an example in the preceding sections, and will give a more detailed illustration of character understanding in the methodology section. \n\n__Rationale statement in our paper__\nRegarding the second rejection reason, we need to clarify that we give the analysis of the two challenges of script-based character understanding in Section 1, spanning lines 042-072. Subsequently, we explain how the two contrastive losses help to mitigate the aforementioned challenges in lines 073-092. Additionally, we give further intuitive rationales for the two contrastive losses in Section 4.3.1 and 4.3.2 respectively.\n\n__Notation issue__ \nConcerning the notation issue, as we mentioned in Section 4.1, lines 203-204, $n$ here represents the number of characters number in the script's conversation. Additionally, in lines 250-251, $P$ is expressly defined as the number of characters that appear in both conversation and summary, which is also the maximum positive pairs we can build in summary-conversation contrastive learning. They just refer to different numbers and here we have $n$>=$P$.\n\n__Character embedding extraction__\nConcerning the extraction equation issue, we use a series of sub-tasks to validate our method and each of them has its own character embedding extraction process. Therefore, we omit the detailed extraction equations to streamline our paper in that part and highlight the two novel contrastive learning losses we proposed. We give the detailed equations below and will include them in the appendix of the next edition of our paper.\n\n__First stage's epoch number__\nAs for the third rejection reason, we give detailed epoch numbers for the first and second stages of each single task in Appendix B, Table 8. The specific signal of the epoch number chosen in the first stage is the convergence of the two contrastive losses.\n\n__Our contribution__\nAs for the final rejection reason, we would like to underscore the contributions of our work beyond mere quantitative performance enhancements. In this work, we are the first to analyze two challenges in character understanding and address them with novel contrastive losses. In the experiment section, we validate the effectiveness and compatibility of our method with a series of baselines and sub-tasks. In Sections 6.4, 6.5 and 6.6, we also provide fine-grained and in-depth analysis to demonstrate the success of our method in addressing the text type and length challenges in character understanding. Besides, according to our statistics of each model's improvement with our method, we think 1% somehow underestimates our method's performance.\n\n| Model      | Improvemnt |\n|------------|:----------:|\n| SpanBERT   | 1.8%       |\n| C2         | 1.1%       |\n| Bigbird    | 2.7%       |\n| Longformer | 1.6%       |\n\n__Statiscally significance result__\nBelow we provide the statistically significance result of our main experiment.  ** denotes that p ≤ 0.01.\n\n| Model                 | B3-PREC | B3-REC  | B3-F1   | CEAFφ4-PREC | CEAFφ4-REC | CEAFφ4-F1 | BLANC-PREC | BLANC-REC | BLANC-F1 | MICRO  | MACRO  |\n|-----------------------|---------|---------|---------|-------------|------------|-----------|------------|-----------|----------|--------|--------|\n| SpanBERT-base         | 77.40   | 82.67** | 79.94   | 74.69       | 67.93      | 71.15**   | 84.80**    | 89.96     | 87.20    | 85.0** | 78.4   |\n| SpanBERT-base (Ours)  | 79.95** | 84.71   | 82.26   | 76.67       | 70.38      | 73.39**   | 87.44      | 91.26     | 89.26    | 86.3   | 78.9** |\n| SpanBERT-large        | 81.92   | 85.56   | 83.69** | 77.85**     | 74.74      | 76.25**   | 88.61**    | 91.91     | 90.20    | 87.2** | 82.8** |\n| SpanBERT-large (Ours) | 83.55** | 87.38** | 85.42** | 79.83       | 76.29      | 78.02**   | 89.18**    | 93.00     | 91.00    | 88.2** | 83.7** |\n| C2-base               | 80.75   | 84.77** | 82.71** | 76.97       | 71.78      | 74.28     | 82.22**    | 91.25     | 89.80**  | 85.6   | 80.4** |\n| C2-base (Ours)        | 83.35   | 85.12** | 84.23   | 76.88**     | 74.97      | 75.91     | 90.48      | 91.85**   | 91.15    | 86.4   | 81.1   |\n| C2-large              | 84.98   | 86.92** | 85.94   | 79.63       | 78.16**    | 78.89     | 90.87**    | 93.05**   | 91.93    | 87.6** | 82.5** |\n| C2-large (Ours)       | 86.42   | 86.44** | 86.24** | 78.82       | 80.42**    | 79.61     | 91.77**    | 93.13     | 92.45**  | 88.0** | 83.2** |\n\n| Model                     | MICRO   | MACRO   |\n|---------------------------|---------|---------|\n| BigBird-P-base            | 71.01   | 70.32** |\n| BigBird-P-base (Ours)     | 72.61   | 73.00   |\n| BigBird-P-large           | 75.43** | 75.24   |\n| BigBird-P-large (Ours)    | 77.68** | 76.41   |\n| Longformer-P-base         | 71.80   | 73.75   |\n| Longformer-P-base (Ours)  | 73.65** | 74.22   |\n| Longformer-P-large        | 77.85   | 75.92** |\n| Longformer-P-large (Ours) | 78.92** | 76.52** |\n\n__Detailed character embedding extraction__\n\nFor coreference resolution and character linking, we follow Bai et al 2021.,[1] to initialize the mention-level character embedding:\n\\begin{equation}\n    e_i = t_{start_i} + t_{end_i} + e_{speaker_i}\n\\end{equation}\nwhere $t_{start_i}$ and $t_{end_i}$ are the contextualized representation of the beginning and the end tokens of mention $i$, and the $e_{speaker_i}$ is the speaker embedding for the current speaker of the utterance where the $i_{th}$ mention belong to. The speaker embeddings are randomly initialized before training.\n\nFor character guessing, we follow Sang et al 2022.,[2] to extract speaker-level character embedding from the context encoding $\\mathbf{H}$:\n\\begin{equation}\n    A = {\\rm Attention}(\\mathbf{H})\n\\end{equation}\n\\begin{equation}\n    a_i = {\\rm Softmax}(A \\odot M_i)\n\\end{equation}\n\\begin{equation}\n    e_i = \\mathbf{H}^Ta_i\n\\end{equation}\n\nwhere Attention($\\cdot$) is a one-layer feedforward network to compute the token-level attention weight. $M_i$ is a token-level mask such that $M_i[j]=1$ if the $j_{th}$ word belongs to an utterance of the $i_{th}$ anonymous speaker and $Mx[j]=0$ otherwise. $a_i$ is the token weight used to pool the hidden states to summarize a character representation.\n\n__Hyper-parameter analysis__\nAs for the third question, we conduct hyper-parameter analysis on the test set.\n\nWe appreciate other suggestions for our work and believe these presentation-related concerns can be effectively rectified in the subsequent version of our paper. Finally, we kindly request your reconsideration of our paper in light of the aforementioned points.\n\n[1] Bai, Jiaxin, et al. \"Joint Coreference Resolution and Character Linking for Multiparty Conversation.\" Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.\n\n[2] Sang, Yisi, et al. \"TVShowGuess: Character Comprehension in Stories as Speaker Guessing.\" Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper proposes a contrastive learning and two stage training method for improving character understanding task with LLMs\n\n2.\tSufficient empirical evaluation is performed to show effectiveness of the proposed approach", "reasons_to_reject": "1.\tThe paper presentation and writing can be improved. The character understanding task is not well defined in Section 3 and 4 and hard to follow for unfamiliar readers. It becomes more clearer only in experiments. I suggest giving a concrete example of a specific task at the beginning (e.g. character guessing/ character linking) and using it as a running example to explain the method.\n2.\tThe paper also jumps into the proposed method without providing sufficient intuition as to why this is necessary. The notations are a bit confusing too. As an example, from equation 3, the input sequence is of length $T,$ $(u_1,..., u_T)$ which is encoded by the PLM. As per equation 4, from this PLM encoding $n$ character representations are extracted. How is this extraction performed? No equations are provided to explain this and Figure 1 is unclear. In Section 4, the number of characters is again represented by a different variable $P$\n3.\tA new two stage training strategy is proposed, but the paper is vague about how to determine when the first stage is completed and the second stage could be started\n4.\tExperimental results show only a small improvement ~ 1% over baseline. Its not clear if these are statistically significant.", "questions_to_authors": "1.\tHow is the $Extract$ layer/ method implemented in Equation 4?\n2.\tHow do you determine when the first stage training is completed?\n3.\tIn Table 5, is the hyper-parameter analysis conducted on the test set or the validation set?", "paper_topic_and_main_contributions": "This paper studies the problem of character understanding from show scripts. They show that large language models like ChatGPT perform poorly on this complex task in a zero/few shot setting. To address this, authors propose a new multi-level contrastive learning method to train pre-trained language models (e.g. SpanBERT, $C^2$) in a supervised fashion. On two datasets and three character understanding tasks, the proposed method is shown to outperform relevant baselines.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Page 3, line 174: given a certain of text -> given a certain collection/ corpus of text?\n\nPage 4, line 288: post-train -> fine-tune"}, "extra_text": {}}, {"review_id": "orshF9Wyoi", "reviewer": "Reviewer_ydTT", "summary": "", "questions": "No questions.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for acknowledging our efforts in your feedback. \n\n__Expand our work in multi-modal scenarios__\nWe also noticed there are many well-built benchmarks and datasets[1][2] in multi-modal narratives. With the recent success of utilizing various contrastive learning methods in the multi-modal area, we are excited to expand our work into these multi-modal areas. Especially considering the key concept of our summary-conversation contrastive learning of leveraging multi-source information, we believe it's promising to apply our method in multi-modal scenarios. Thank you again for your time and effort in evaluating our work!\n\n[1] Kim, Joshua Y., et al. \"MONAH: Multi-Modal Narratives for Humans to analyze conversations.\" Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume. 2021.\n\n[2] Yang, Antoine, et al. \"Just ask: Learning to answer questions from millions of narrated videos.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The research is exhaustive, and results are convincing. The intuition of using a mix of summary and conversation data along with contrastive learning is interesting. The research can open avenues for enriching this with visual signals (in case the authors want to extend script analytics to videos etc.) and create more enriched character behavior models which could be of help in many fields. ", "reasons_to_reject": "I would have loved to see how the character understanding will change/enrich with visual signals. A lot of times textual content may not entirely convey the subtle sarcasm etc. Depending on the nature and details in the script some fine grained data elements can get missed. Could be choice of cloths or tone and pitch of conversations. If the authors could have extended the work on those aspects it would have made it more interesting to read. ", "questions_to_authors": "No questions.", "paper_topic_and_main_contributions": "A very interesting problem to solve. The paper tries to understand the characters in scripts which can span a few thousand pages too. The paper uses contrastive learning to solve the problem and achieves reasonable results. The comparison with LLMs is interesting and how the authors are able to beat the large models using summary and conversation encodings, is very interesting.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None"}, "extra_text": {}}, {"review_id": "rzoWiiBC2E", "reviewer": "Reviewer_Cuxc", "summary": "", "questions": "1. What is meant by \"fine-grained\" representation in L259? Do summaries and conversations convey \"fine-grained\" information? Summaries can also encompass general and global aspects, especially if \"fine-grained\" refers to local details.\n2. When compared to the baseline model, what are the additional computational costs associated with the two-stage training framework?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed evaluation of our work. We appreciate the time and effort you've dedicated to reviewing our paper. Here we would like to clarify our work by explaining the rationale behind our method in more detail.\n\n__Connection between the two challenges and our method__\nRegarding the first issue, we first analyze two challenges of script-based character understanding in Section 1, spanning lines 042-072. We also explain the relationship between the two contrastive losses and two challenges in Section 1, lines 073-096.\n\nSpecifically, the first challenge in character understanding is complex text type. To address it, we utilize summary because its straightforward and highly generalized narrative style can be a great supplement to the complex conversation. We use the summary as an auxiliary corpus to prompt the model to better capture fine-grained information (character features) from the conversation.\n\nThe second challenge in character understanding we analyze is the script length. We design a cross-sample contrastive learning to address this challenge. It aligns the same character's representation from different samples thus prompting PLM to comprehend character globally.\n\n__Positive pair construction issue__\nAs for your concern about the positive pair construction in the two contrastive losses, we recognize the need for further explanation and validation. Firstly, we need to emphasize that no method is perfect, and the construction of positive samples for contrastive learning will definitely introduce some noise. However, we believe the promising experiment results have proven the rationality of our positive sample construction.  \n\nBeyond that, samples of each character understanding task are built at the scene level, which means character representation from the conversation is learned in a relatively long conversational context. Therefore, the randomness in the conversation you mentioned can be largely avoided. Also, although topics and talks from different samples (scenes) can be different, character features and traits behind these conversations can be quite consistent. For instance, a character known for humor, like Sheldon, consistently exhibits his comedic nature irrespective of the context—whether on a date or among friends. It's important to note that we construct positive pairs using character representation extracted from the entire conversation, not limited to individual utterances.\n\n__Our contribution__\nConcerning the second rejection reason, we are the first to leverage cross-sample contrastive learning to address the text length issue mentioned in Section 1. We believe this breakthrough is even more critical than designing new contrastive learning pipelines. Additionally, we conduct fine-grained and in-depth analysis in Sections 6.4, 6.5 and 6.6 to validate our cross-sample contrastive learning comprehensively. All these analyses, designs and validation separate our work from only applying existing contrastive learning methods on the character understanding tasks.\n\n__Meaning of \"fine-grained\"__\nAs for the first question, we use fine-grained to refer to the rich and detailed information in conversations. However, as we state in Section 1, lines 47-59, the intricacy of conversation comprehension can hinder PLM from comprehending character well within such complex contexts. Therefore, we introduce summary as an additional and supplemental corpus to conversation, to facilitate PLM to better capture character information from fine-grained conversation. Although summaries may offer a more global and general perspective, they do not impede our utilization of this supplementary information to enhance our understanding of characters within fine-grained conversations.\n\n__Addtional costs__\nRegarding the second question, as we show in Table 8, our additional post-training stage (the first stage) uses less than half of the training epoch numbers of the second before the convergence of the two contrastive losses. In our experiments, the additional time and computational cost are about half of the normal training process.\n\nIn conclusion, we sincerely appreciate your meticulous evaluation and kindly request your reconsideration of our paper based on the above response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation behind the paper is clear and justified. The authors acknowledge the significance of different types of text and the importance of long-context descriptions in understanding a figure. The experimental design aligns well with these two aspects.\n2. The experiments conducted in the paper are extensive and demonstrate the effectiveness of the proposed method.\n3. The structure and organization of the paper are well-designed, resulting in a coherent and easily comprehensible flow of information.", "reasons_to_reject": "1. While it is acknowledged that text type and longer context are important factors in understanding characters, the connection between these two aspects and contrastive learning seems unclear and somewhat forced in this paper. For text type, the authors choose to focus on summary and conversation, without explicitly justifying why these sources were chosen over others. Additionally, while a summary provides a condensed description of a character, conversations can be random and distributed, which may not always align with the character's summary. This raises concerns about the validity of contrastive learning for positive pairs. Regarding global context, it remains unclear why similar utterance segments from the same person need to be considered positive pairs within the framework. For example, totally different talks may come from the same person.\n2. The implementation of cross-sample contrastive learning in this paper appears to be a conventional approach within the field of contrastive learning, lacking novel elements or significant advancements.", "questions_to_authors": "1. What is meant by \"fine-grained\" representation in L259? Do summaries and conversations convey \"fine-grained\" information? Summaries can also encompass general and global aspects, especially if \"fine-grained\" refers to local details.\n2. When compared to the baseline model, what are the additional computational costs associated with the two-stage training framework?", "paper_topic_and_main_contributions": "The paper proposes a multi-level contrastive learning framework for script-based character understanding. In the in-sample level, they utilize summary of the script and build a summary-conversation contrastive loss. They also provide a cross-sample contrastive loss to capture global feature. Extensive experiments show its effectiveness.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "L175: \"consists\" -> \"consist\""}, "extra_text": {}}, {"review_id": "23OldK6wbg", "reviewer": "Reviewer_metd", "summary": "", "questions": "1. In pracitcal scripts or stories, there are many entities/characters which have alias, e.g. a person could be called Mr. X besides his full name. Do you consider the alias alignment issue (aligning the alias/other mentions to an entity/character) in your work?\n2. Why did the summary-conversation loss work better than the cross-sample loss based on the results in Table 4? Please explain the potential resons.\n3. In table 4, we can see that without cross-sample loss some results become better than that with it, what are the reasons? \n4. How do we know the effectiveness/impact of the two-stage training? Maybe authors can explain something about this in the ablation study.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for dedicating your time and effort to the evaluation of our work. Here we address some rejection reasons and questions you raised with additional experiment results and analysis.\n\n__Our contribution__\nAs for the rejection reason, we acknowledge that adopting the contrastive learning method in character understanding and representation learning is quite straightforward. However, we are the first to propose this method in script-based character understanding and evaluate its effectiveness comprehensively. We believe our work will stimulate more focus on adopting contrastive methods to learn character representation. \n\n__Our method with automatically generated summary__\nRegarding your concern about the summary availability, large language models these days demonstrate comparable performance to humans in summarization, according to some recent works[1][2]. To validate our method's compatability with automatically generated summaries, we prompt ChatGPT to automatically generate summary for coreference resolution and character linking. Due to time reasons, we conduct experiment on C2-base and provide the experiment results in the table below. It's significant to highlight that the model trained on automatically generated summaries achieves comparable performance with that trained in human-written summarization. This outcome underscores the potential efficacy of integrating automatically generated summaries into our methodology.\n\n|          | B3        | CEAFφ4    | BLANC     | MICRO    | MACRO    |\n|----------|-----------|-----------|-----------|----------|----------|\n| Baseline | 82.71     | 74.28     | 89.80     | 85.6     | 80.4     |\n| ChatGPT  | 84.14     | **76.06** | 90.89     | 86.1     | 80.9     |\n| Human    | **84.23** | 75.91     | **91.15** | **86.4** | **81.1** |\n\n__Charcter entity extraction__\nRegarding the first question, in our preliminary experiment, we tried using some NER-based methods to extract coreference in summaries and align them to specific characters based on certain rules. However, we found this method would cause quite a lot of errors and influence the subsequent contrastive learning stage. Also, we find this method can’t significantly improve the number of distinct characters extracted in each summary. This is probably due to the fact that each character’s first appearance in the summary is always with his/her name, but not alias. Therefore, we only use character names to extract character representation from summaries.\n\n__Explanation of Table 4__\nTo answer the second question, one possible reason for summary-conversation contrastive learning’s better performance is the introduction of additional and different modal information. In our summary-conversation contrastive learning, we adopt summary as auxiliary information to facilitate PLM's understanding of the complex conversation. According to the previous works[3][4], PLM can benefit a lot from such multi-modal information. Compared with it, cross-sample conversation learning only utilizes information within a singular domain (conversation) in a global way.\n\nAs for the third question, it is noticeable that although the model without cross-sample contrastive learning performs better in some metrics, our method with the two contrastive losses together achieves the overall best performance considering all the metrics.\n\n__Validation of our two-stage learning__\nConcerning the last question, we tried to train the model using only the first stage with three losses (one supervised and two contrastive losses) together in our preliminary experiment. We found such one-stage learning would lead to a sub-optimal performance and when we add the second stage with solely-supervised training, the performance on the specific task can be further improved. This led us to design a two-stage training and we added some intuitive explanations of it in Section 4.4, lines 296-299. Below we provide the experiment results of baseline/one-stage/two-stage training in coreference resolution and character linking. The model used here is C2-base.\n\n|                  | B3        | CEAFφ4    | BLANC     | MICRO    | MACRO    |\n|------------------|-----------|-----------|-----------|----------|----------|\n| Baseline         | 82.71     | 74.28     | 89.80     | 85.6     | 80.4     |\n| One-stage        | 83.58     | 74.28     | 90.63     | 86.1     | 80.8     |\n| two-stage (Ours) | **84.23** | **75.91** | **91.15** | **86.4** | **81.1** |\n\nWe appreciate other suggestions and corrections and kindly request your reconsideration of our paper in light of the aforementioned points.\n\n[1] Gao, Mingqi, et al. \"Human-like summarization evaluation with chatgpt.\" arXiv preprint arXiv:2304.02554 (2023).\n\n[2] Zhang, Tianyi, et al. \"Benchmarking large language models for news summarization.\" arXiv preprint arXiv:2301.13848 (2023).\n\n[3] Radford, Alec, et al. \"Learning transferable visual models from natural language supervision.\" International conference on machine learning. PMLR, 2021.\n\n[4] Zhang, Hengyuan, et al. \"Fine-grained Contrastive Learning for Definition Generation.\" Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The main contribution of this paper is the two contrastive losses, where we can see their effectiveness in the ablation study. The two-stage training strategy could be a contribution as well.", "reasons_to_reject": "Contrastive loss is a straightforward way to enhance the character representation, so the overall novelty of this paper is limited. This work is limited by the resources such as the summary so it is not easy to be generalised to other languages or domains. Using automatically generated texts as summaries is not convincing.", "questions_to_authors": "1. In pracitcal scripts or stories, there are many entities/characters which have alias, e.g. a person could be called Mr. X besides his full name. Do you consider the alias alignment issue (aligning the alias/other mentions to an entity/character) in your work?\n2. Why did the summary-conversation loss work better than the cross-sample loss based on the results in Table 4? Please explain the potential resons.\n3. In table 4, we can see that without cross-sample loss some results become better than that with it, what are the reasons? \n4. How do we know the effectiveness/impact of the two-stage training? Maybe authors can explain something about this in the ablation study.", "paper_topic_and_main_contributions": "This paper proposes a framework for character understanding in scripts which consists of two different constrastive losses and a two-stage training strategy. The two constrative losses are summary-conversation loss and cross-sample contrastive loss, respectively, with the purposes to capture fine-grained and global character information at in-sample and cross-sample levels. Experiments on three different tasks (i.e. coreference resolution, character linking and character guessing) show significant improvements compared to baselines and ChatGPT 3.5 model.\n\nThe main contribution of this paper is the two contrastive losses, where we can see their effectiveness in the ablation study. However, as claimed in the limitation section, the summary-conversation loss is limited by the resources. Although authors said summaries could be obtained by text generation, the quality is a concerned issue which might propagate the errors to the task of character understanding. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "To name a few:\n1. Line 201: please don't use \"get\" in the paper because it is a spoken/informal word.\n2. Table 2 didn't correctly show the best results in bold, e.g. the best one is 79.83 in Column 4, 88.2 in Column 10 etc.\n3. Line 429: separate 'effectivenessin' into two words."}}]

Author rebuttal:

Meta-review: The paper proposes a multi-level contrastive learning framework for script-based character understanding. They utilized summary of the script and built a summary-conversation contrastive loss. Experiments are exhaustive and the results are interesting. There are some aspects that need further attention: novelty of the proposed contrastive loss based model; and detailed qualitative explanation.
Decision: Accept-Main