Symbolization, Prompt, and Classification: A Framework for Implicit Speaker Identification in Novels
Authors: Yue Chen
Tianwei He
Hongbin Zhou
Jia-Chen Gu
Heng Lu
Zhen-Hua Ling
Venue: EMNLP 2023 Findings (2023)
Keywords: Speaker Identification
Audiobook Production
Prompting
TL;DR: We propose a framework named SPC, which identifies implicit speakers in novels via symbolization, prompt, and classification.

Abstract: Speaker identification in novel dialogues can be widely applied to various downstream tasks, such as producing multi-speaker audiobooks and converting novels into scripts.
However, existing state-of-the-art methods are limited to handling explicit narrative patterns like "Tom said, '...'", unable to thoroughly understand long-range contexts and to deal with complex cases.
To this end, we propose a framework named SPC, which identifies implicit speakers in novels via symbolization, prompt, and classification.
First, SPC symbolizes the mentions of candidate speakers to construct a unified label set.
Then, by inserting a prompt we re-formulate speaker identification as a classification task to minimize the gap between the training objectives of speaker identification and the pre-training task.
Two auxiliary tasks are also introduced in SPC to enhance long-range context understanding.
Experimental results show that SPC outperforms previous methods by a large margin of 4.8% accuracy on the web novel collection, which reduces 47% of speaker identification errors, and also outperforms the emerging ChatGPT.
In addition, SPC is more accurate in implicit speaker identification cases that require long-range context semantic understanding.

Symbolization, Prompt, and Classification: A Framework for Implicit Speaker Identification in Novels
Yue Chen'; Tean-Wei He", Hong-Bin Zhou', Jia-Chen Gu', Heng Lu', Zhen-Hua Ling'
"National Engineering Research Center of Speech and Language Information Processing, University of Science and Technology of China, Hefei, China "Ximalaya Inc., Shanghai, China
cy321@mail www.sstcccoo edu. cn, {gujc, whwiingeesstteeeccceeeeee edu. cn,
(tianwei. he, hengbinnginnn. (ethhou@xxmmllyaaa com, bearlu607@gmail.com
Abstract
Type
Speaker
Sentence
Speaker identification novel dialogues can be widely applied various downstream tasks, such as producing multi-speaker audiobooks and converting novels into scripts. However, existing offte-of-the-are methods are limited to handling explicit narrative patterns like "Tom said, ".."", unable thoroughly understand long-range contexts and to deal with complex cases. To this end, we propose a framework named SPC, which identifies implicit speakers in novels via symbolization, prompt, and clas- sification. First, SPC symbolizes the mentions of candidate speakers to construct unified label set. Then, by inserting prompt we re- formulate speaker identification as classifi- cation task to minimize the gap between the training objectives of speaker identification and the pre-training task. Two auxiliary tasks are also introduced in SPC to enhance long-range context understanding. Experimental results show that SPC outperforms previous methods by large margin of 4.8% accuracy on the web novel collection, which reduces 47% of speaker identification errors, and also outperforms the emerging ChatGPT. In addition, SPC is more accurate in implicit speaker identification cases that require long-range context semantic understanding.
N1
Jin Xiu shouted with enthusiasm,
U1
Jin Xiu
"Let's sing song together!"
U2
Wu Zhongping
"Old songs or new songs?"
N2
Said Wu Zhongping.
U3
Lanxiang
"You should say the songs of the present or the songs of the past.
N3
Lanxiang corrected her friend with a laugh.
U4
Wu Zhongping
"Well, you' re right. For the songs of the past I only know Let's Sway Twin Oars."
US
Yangmin
"It's just the time for it."
N4
Said Gu Yangmin.
Table 1: A translated example from the Chinese novel World of Plainness. U1-U5 are utterances and NI-N4 narratives.
the major interaction between characters in novels, automatic identification of speakers can also be useful for novel-based knowledge mining tasks such as social network extraction (Jia et al., 2020) and personality profiling of the characters (Sang et al., 2022).
Table shows an example randomly sampled from the Chinese novel World of Plainness. For utterances U1, U2 U3, and U5, their speakers be easily determined by recognizing the explicit narrative patterns like "Said Wu Zhongping" in the previous or following sentence. U4 is an exception that does not fall into this explicit pattern. To correctly predict the speaker of U4 requires an understanding of the conversation flow. Although many speaker identification cases can be solved by recognizing narrative patterns, many complex examples still call for deep understanding of the surrounding context. We refer these complex examples as implicit speaker identification cases. They pose difficulties for existing speaker identification algorithms.
1 Introduction
Speaker identification in novel dialogues, also known as dialogue attribution (Muzny al., 2017: Cuesta-Lázaro et al., 2022), aims at identifying the speaking characters of utterances in fiction texts (Glass and Bangay, 2007: Elson and McKeown, 2010). It is an important task for various down- stream applications like automatically assigning appropriate voices to utterances in audiobook production (Pan al., 2021) and novel-to-scrip conversion (Soo et al., 2019). As dialogues serve as
"Work done during the internship at Ximalaya. *Corresponding author.

---

Most recent approaches for speaker identifi- cation (Chen al., 2021; Cuesta-Lázaro et al., 2022: Yu et al., 2022) are based on pre-trained language models (PLMs) like BERT (Devlin et al., 2019). PLM-based methods enjoy the advantage of the PLM's internal linguistic and commonsense knowledge obtained from pre-training. However, two main downsides should be highlighted for these methods. On the one hand, some methods truncate a context into shorter textual segments before feeding them to PLMs (Chen et al., 2021; Cuesta-Lázaro et al., 2022). Intuitively, this approach inevitably introduces bias to focus on short and local texts, and to identify speakers by recognizing explicit narrative patterns usually in the local context of the utterance. It may fail in implicit speaker identification cases when such patterns are not available and long-range semantic understanding is indispensable. On the other hand, some methods adopt an end-to-end setting (Yu et al., 2022) that sometimes extracts uninformative speakers like personal pronouns. Besides, they only perform mention-level speaker identification in which two extracted aliases of the same character won't 't be taken as the same speaker. In recent months, large language models (LLMs) have become the most exciting progress the NLP community. Although L.MMs show impressive zero-shot/few-sho capabilities in many benchmarks (Patel et al., 2023; Ouyang et al., 2022; Chung et al., 2022), how well they work for speaker identification remains unknown.
lary character classification tasks to incorporate supervision signals from speaker identification of neighborhood utterances. In this way, SPC can learn to capture the implicit speaker indication in a long-range context.
To measure the effectiveness of the proposed method and to test its speaker identification ability, SPC evaluated on four benchmarks for speaker identification in novels, specifically, the web novel collection, World of Plainness, Jin- Yong novels, and CSI dataset. Compared to the previous studies that conduct experiments on merely 1-18 labeled books (Yu et al., 2022), the web novel collection dataset contains 230 labeled web novels. This dataset enables neural models and to analyze their performance at different training data scales. Experimental results show that the proposed method outperforms the best-performing baseline model by a large margin of 4.8% accuracy on the web novel collection. Besides, this paper presents comparison with the most popular ChatGPT' and results indicate that SPC outperforms it in two benchmarks. To facilitate others reproduce our results, we have released our source code 2
to fully supervise the
To sum up, our contributions in this paper are three-fold: (1) We propose SPC, novel framework for implicit speaker identification in novels via symbolization, prompt, and classification. (2) The proposed method outperforms existing methods on four benchmarks for speaker identification in novels, and shows superior cross-domain perfor- mance after being trained on sufficient labeled data. (3) We evaluate ChatGPT on two benchmarks and present a comparison with the proposed method.
Drawing inspiration from the successful applica- train-
tion of prompt learning and ing in various tasks like sentiment analysis (Patel et al., 2023; Schick and Schütze, 2021), we propose a framework to identify speakers in novels via symbolization, prompt, and classification (SPC). SPC first symbolizes the mentions of candidate speakers construct a unified label for speaker classification. Then it inserts a prompt to introduce a placeholder for generating a feature for the speaker classifier. This approach minimizes the gap between the training objectives of speaker identification and the pre-training task of masked language modeling, and helps leverage the internal knowledge of PLMs. In previous studies, the inter- utterance correlation in conversations was shown to be useful for speaker identification in sequential utterances (He et al., 2013; Muzny et al., 2017; Chen et al., 2021). SPC also introduces auxil-
2 Related Work
In recent years, deep neural networks have shown great superiority in all kinds of NLP tasks (Kim, 2014; Chen and Manning. 2014), including text- based speaker identification. Candidate Scoring Network (CSN) (Chen et al., 2021) the first deep learning approach developed for speaker identification and outperforms the manual feature- based methods by significant margin. For each candidate speaker of an utterance, CSN first encodes the shortest text fragment which covers both the utterance and mention of the
'nups://chaa.opeeaai.oom/ https//gggha.ccomm/eeectmn/sssed
Identification

---

candidate speaker with a BERT. Then the features for the speaker classifier are extracted from the output of BERT. In this way, the model learns to identify speakers by recognizing superficial narrative patterns instead of understanding the context. Cuesta-Lázaro et al. (2022) migrates a dialogue state tracking style architecture to speaker identification. It encodes each sentence separately with Distill-BERT (Sanh et al., 2019) before modeling cross-sentence interaction with a Gated Recurrent Unit (Chung al., 2014). Then a matching score is calculated between each character and each utterance. However, this method still just utilizes the PLM to model local texts and results in poor performance. Yu et al. (2022) adopts end-to-end setting that directly locates the span of the speaker in the context. It feeds the concatenation of the utterance and its context RoBERTa (Liu al., 2019; Cui et al., 2021), after which the start and end tokens are predicted on the output hidden states. Yet under end-to-ent setting, only mention-level speaker identification is performed, in which two extracted aliases of the same character are taken as two different speakers. Previous studies have shown the great advantage of applying deep-learning methods to speaker identification in novel dialogues, but these methods either are limited to recognizing superficial patterns or only identify speakers at mention level. Our study proposes a method to identify speakers based on context understanding and improves the performance on implicit speaker identification problems when long-range semantic information is needed. The proposed method outperforms other competitors given sufficient labeled data, and is more efficient in data utilization.
clr {x Ny...,, 1, 1, "1, denotes the target utterance and --N 21 N2 denote the N1 + N₂ sentences surrounding u. The speaker of the target utterance assumed to be mentioned in the selected context, while the exceptions (should be rare) are discarded from the dataset. Assume that m candidate speakers are located in ctx by matching the text in ctr to the speakers' names.
3.2 Framework Introduction
Figure shows the framework of the proposed SPC. SPC takes the selected context as input and generates the likelihood of character classification as its output. First, the mentioned characters in the input context are replaced with symbols. Then, a prompt with a placeholder (the [MASK]) is inserted to the right of the target utterance. After that, placeholders of auxiliary tasks are introduced into the context. At last, the PLM encodes the processed context and classifies the characters each placeholder.
3.3 Character Symbolization
Character symbolization unifies the candidate speaker sets in diffcrent contexts, after which speaker identification can be formulated as a classification task. We assign a unique symbol CýÙ = m.... to each candidate character mentioned in ctx, and replace the mentions of each character with its corresponding symbol in ctr. Note that this mapping from characters symbols is only used for the specific selected context rather than for the whole novel, so as to reduce the number of classification labels. The character symbols form a local candidate set CS = (C1,..., Cm.. Let M be the maximum number of candidate characters. C1,.., Ca have been added to the special token vocabulary of the PLM in advance. The embeddings of these special tokens are randomly initialized and will be jointly optimized with other model parameters.
3 Methodology
3.1 Task Definition
Before we dive into the details of our proposed approaches, it would be necessary to declare the basic settings for the task of novel-based speaker identification. The sentences in the novel have been split and each sentence is identified as either an utterance or a narrative. The name and aliases of the speaking characters in the novel have been collected in advance. The occurrences of the speaking characters in the novel are referred as mentions. For the target utterance whose speaker we intend to identify, a selected context that covers the target utterance is extracted and denoted as
3.4 Prompt Insertion
Next, we insert into the selected context a prompt y= p="[prefix]
[postfix]". right after the target utterance. In the inserted prompt, "_ is the placeholder we aim to classify, while [prefix] and [postfix] are manually crafted strings on both sides of the placeholder. In practice, we choose
'In practice, we use a 21 21-sentence window. 97.2% of utterances in the web novel collection are included.

---

peaker diassiifcaiio for the la larget utterence
NUSC
NMC
NUSC MLMiHead
MLMHead
MLMHlead
MLMHHead
heai...
hot haz...
host...
hy ham... he1
Pre-trained Language Model
22:0 82..
DMAS 56
C-38.
songs
laughed...
you...
tht
Auxiliary Masking
Prompt Tempate
12:48:38
1993, B
38f3131
Old songs
rughed...
wondss
rompt Insertion
1938. you...
"B uotche
C:008
"O1S sorgg
C.2.
buuhhd....
Character Symbolization
P revious Utterance
Target Utterance Following Utte Unterance
"S3:8 iB...
"SFRF "8t.."
congs
Zhongp no.
buughd
Selected Context
Figure The framework of SPC. The block in green represents the target utterance, while the blocks in blue represent the inserted prompt templates. The uncolored blocks are other sentences the selected context. The mentions of candidate speakers are also colored.
and "KJ for [prefix] and [postfix] respectively, which combine mean "(_ said these words)" in English. The [MASK] token in the PLMM's vocabulary is used for the placeholder.
bias of the decoder layer. WT and by are pre- trained along with the PLM, while W D and bn are randomly initialized. s = [s1. SM] the output score vector which the first m scores correspond to the m candidate in C.S.
3.5 Speaker Classification based on a PLM
At the training stage, our goal is to assign the correct speaker higher score than other candidates, choose margin ranking loss to instruet the optimization. For example, assume the correct speaker the target utterance is Cj. then Cj is paired with each candidate in CSYCC. The speaker identification loss of u is calculater deuulatedd on the scores of each candidate pair, as:
Then we feed the processed context to a PLM which has been pre-trained on masked language modeling (MLM) to classify the missing character at the placeholder. Specifically, we utilize the pre-trained MIM head of the language model to produce scalar score أز = 1,..., m) for each candidate character Cy:
h' = LayerNorm CoyyeNorm((EEL(VR GELU Th br)),
(1)
C(u, ctz.
(3)
s W ph bp.
maz 88 mgn, 0};
h Rd is the output hidden state of the PLM corresponding the placeholder, where d is the hidden size of the PLM. Wy Rdxd and by R are the weight and bias of the linear transform layer. Wn D E IR M xx and by E IR M are the weight and
k=1.
where mgn is the ideal margin between the scores of the two candidates. At the inference stage, the candidate assigned the highest score is identified as the speaker.

---

3.6 Auxiliary Character Classification
the inference stage, do not mask the character mentions in 21 and x-1 retain the explicit narrative patterns. The prompt inserted after w;-1 and ui+1 at the inference stage to keep training- inference consistency.
Based on the classification task form, we de- signed two auxiliary character classification tasks: 1) neighborhood utterances speaker classification (NUSC) to utilize the speaker alternation pattern between adjacent utterances, and 2) masked men- tion classification (MMC) alleviate the exces- sive reliance on neighborhood explicit narrative patterns. NUSC teaches the model to classify the speakers of the neighborhood utterances of the target utterance. In this way, the model learns to utilize the dependency between the speakers of neighborhood utterances. MMC randomly masks the character mentions in the neighborhood sentences of the target utterance and quizzes the model classifying the masked characters. It corrupts the explicit narrative patterns like "Tom said" which usually exists in the neighborhood sentence of the utterance and guides the model to utilize long-range semantic information.
Experiments
4.1 Datasets
We conducted experiments on four Chinese speaker identification datasets which the web novel col- lection (WN), World of Plainness (WP) (Chen al., 2019) Jin-Yong novels (JY) (Jia al., 2020) and end-to-end Chinese Speaker Identification dataset (CSI) (Yu et al., 2022) 6 respectively. WN is a large internal speaker identification dataset. Its annotation details can be referred to in Appendix D. WN and CSI both consist of web novels of various genres and writing styles, while WP and JY printed literature with more normative writings. WP and JY can serve cross-domain evaluation datasets for WN. As no test data was provided in CSI, we use the development set of CSI as the test data and randomly sampled 10% instances from the training set for validation. The number of novels and utterances in each subset of each dataset is shown in Table 2. There are no overlapped novels in the different subsets of WN.
We relabel the target utterance as u; and its previous and following utterance as 14-1 and 1;+1- To perform NUSC, the prompt p is also inserted after u4-1 and 14+1, long these two utterances and their speakers are covered in ctr. Then, the model classifies the speakers for u;-1 and u4+1 as described in Section 3.5. The loss introduced by NUSC is the average speaker classification loss of the two neighborhood utterances:
For clarity, each subset of a dataset named dtataset-subset e.g., "WN-train" for the training set of WN. To evaluate the impact of a smaller training set, we also sampled 5 novels from WN- train to make smaller training set with about k utterances. To distinguish between the whole training set and the sampled one, referred to them WN-large and WN-small respectively.
CNUSC- = = (uz 1, ctx) +
2 C(ua ctx)). (4)
For MMC, we randomly mask the character mentions by a probability in the previous and following sentences of the target utterance (i.e. and æ1), provided the masked characters are mentioned somewhere other than these two sentences. The model is required to predict the masked characters. Let the indexes of the masked characters be MC,, then we have the average classification loss of each masked mention as:
4.2 Baselines
We compared SPC the following baselines. CSN (Chen et al., 2021) feeds a candidate-specific textual segment to the PLM and then outputs the likelihood of the corresponding candidate. A revision algorithm based on speaker alternation pat- tern is adopted to revise the speaker identification results in two-party dialogues. DST SI each sentence separately with the PLM before
LMMC- = |MC|
Σ Emgg(()) JEMC
(5)
in which Lmnn is the same margin ranking loss function as shown in E3.(3). After incorporating NUSC and MMC into the training objective, the loss function turns into:
al., 2022) encodes
'htps://gituu..com// http://twwoooomeeeeeeeeettthheee http:///////tattoo//////lllllaaa Liintieiddenttiicatiion
Datase--
LACC L(u,, ctr) af + BLMMC, (6)
www/huuyi-do//The-seeeeerr identification-comps--o Yong-novels
where a and 3 control the strength of supervision signals from NUSC and MMC respectively. At

---

Dataset Subset of books
of instances
between utterances to 1. We further included the previous and following 10 sentences of each conversation, and limited the maximum number of involved sentences to 30. More details can be referred to Appendix A.
train
118
477.989
WN
val
42
214,854
test
70
398.287
train
2,000
WP
val
1
298
test
298
4.4 Overall Results
train
17,159
JY
val
5,719
We tuned and evaluated the models on the same dataset (in-domain), or tuned the models on WN and evaluated them on WP and JY (cross-domaing Note that although we compared zero-shot GPT- 3.5-turbo to other cross-domain results, it hadn' 't been tuned on any data. The released CSI dataset masked 10% of tokens due copyright issues, so we collected E2F_ 22F_ST's performance on the masked CSI from the GitHub page of CSL. Validation/ Test accuracies are shown in Table We will mainly discuss the test results and leave the validation results for reference.
test
5,719
train
43,233
CSI
val
4,804
test
17,503
Table 2: Statistics of WN, WP, JY, and CSI datasets.
modeling cross-sentence interaction with GRU. The embeddings of the character mention and the utterance are dotted to obtain the likelihood. A CRF is employed to model the dependency between speakers of neighborhood utterances. Due to limited GPU memory, we only use a base model for DST_SI.
First, we can conclude from the table that RoBERTa-large performed better than RoBERTa- base and BERT-base for the same method. Regard- less of the specific PLM, the comparative relation- ship between different methods remains constant. So mainly focus the performance of different methods based on RoBERTa- on RoBERTa-large consistently performed better than or comparable to all non-LLM baselines in both in-domain evaluation and cross-domain evaluation. In the in-domain evaluation of WN, SPC outperformed the best opponent CSN by 4.8% and 3.9% trained on WN-l and WN-small, achieving overall accuracies of 94.6% and 90.0%. These remarkable improvements the errors reduced by 47% and 28%. As WN-test includes 70 web novels of various genres, we believe it reflects general performance on web novels. the in-domain evaluation on WP which is the only dataset evaluated in CSN paper (Chen et al., 2021), SPC still outperformed CSN by 1.0%. We observed MMC might cause serious overfitting for very small datasets like WP-train, MMC for WP-train. In cross-domain evaluations, SPC also consistently outperformed all non-LLM baselines, which shows its better generalizability to novels of unseen domains.
E2E_ST SI (Yu et al.. 2022) feeds the concatenation of the utterance and its context to the PLM, after which the start and end tokens are predicted on the output hidden states.
GPT-3.5-turbo was based GPT-3 (Patel al., 2023) and aligned to human preference by rein- forcement learning with human feedback (Ouyang et al., 2022). We prompted the model with the format (ccontext)# # utterance #The speaker of this utterance is:". We used a tolerant metric that the response was considered correct if the true speaker's name is sub-string of the response.? We only evaluated this baseline WP and JY due the expensive API costs.
SPC based
#Thh
4.3 Implementation Details
As context selection plays a critical part in speaker identification, we detail the context selection proce- dures for the methods we implemented. For SPC, we selected 21 21-sentence context window sur- rounding the target utterance, which corresponds to Ni N22 10 in Section 3.1. If the context win- dow exceeds the PLM's length limit (512 tokens for RoBERTa), we would truneate the context window fit the input length requirement. Since DST_S, is not open source, we implemented ourselves. We followed their paper and segment conversations by restricting the number of intervening narratives
didn't adopt
Although GPT-3.5-turbe underperformed WN- large tuned SPC, its zero-shot performance is still remarkable.
comparison, GPT-3.5-turbo has a much large number of parameters and benefits from its vast and general pre-training corpus, while
WWo also tried prompting with multiple choices of candidate names but the performance degraded drastically.

---

PLM
In-domain
Cross-Jomain
Training data
WN-large WN-small
WF-trann
TY-main
CSI-trant
WNN-aage
WN-small
WN-large WN-small
Test data
WN-jest
WN-tea
WP-test
TY-iest
CSI-test
WP-iest
WP-test
JY-iest
JY-test
E2R,ST (Ya et ol., 2022)
RoRERTa-larg
+
+
-80.9'
98.11/9831
.85.9
+-
+
d
+
al., 2021)
BERT-hase
-/82.51
-/-
CSN
RoBERTa-har hase
90.4/88.6
86.8/84.8
80.5/85.2
98.4/98.4
90.08888
85.6986.9
80.5182.2
96.69699
95.9/96.2
RoBERTa-large
91.8/89.8
88.6/86.1
85.2/88.9
98.8/98.7
91.3/913 91.1
97.6/7.9
96.7/96.8
DST_St SS (Cuustt- Lázuro et 2022)
RoBERTa-buse
86.0/84.1
71.6/69.2
63,4/61.7
96.7/96.9
91,1/87.2
7K.5/78.9
68.8/67.8
93.4/44.5
79.1/79.3
SPC (ours)
RoRFRTn-hase
94.3/93.2
88.7/87.2
83.2/86.2
98.5/98.6
90.3/9000
89.9/91.4
80.9/83.9
98.5488.5
97.09669
SPC (ours)
RoRRRTa-large
95.6/946
91.699000
90.3/89.9
98.8/98.9
90.5/90.6
93.0V91.9
88.9/8866
98. 98.9/98.8
98.3/98.1
G-T--,5-ttrro (zcro-sbol)
GPT-3.5-iurbo
+
+/-
+/-
90.3/89.6
90.3889.6
92.7/93.0
92.7/93.0
Table 3:
Validation/Test accuracies (%) of SPC and the baselines (t: numbers reported (Yu et al., 2022); 3:
number reported in (Chen al., 2021)). The results of GPT-3.5-turbo were obtained under zero-shot setting without using any training data. The cross-domain validation accuracies were obtained on WP-val//Y-val. The highest test accuracy in each column is emphasized in bold.
Training data
WN-large WN-test
WN-small WN-test
WP-train WP-test
JY-train JY-test
usually obey the speaker alternation pattern, which can be easily learned and utilized.
Test data
SPC
94.6
90.0
89.9
98.9
We further ablated prompting. To this end, we did not insert the prompt but extracted the CSN- style features from the output PLM to produce the likelihood of each candidate speaker. After ablating the prompt-based architecture, the perfor- mance of models trained on WN-large decreased by 0.6%, whereas those on WN-small and WP-train decreased drastically by 4.8% and 11.9%. It shows that prompting is helpful for boosting performance in a low-resource setting and verifies our starting point for developing this approach. Prompting closes the gap between the training objectives of speaker identification and the pre-training task, which can help the PLM understand the task and leverage its internal knowledge. JY is the exception in which performance did not degrade after this ablation, although its training set only contains 15k samples. We believe this is because JY is too easy and lacks challenging cases to discriminate between different ablations.
-CCC
93.8
89.4
86.2
98.7
-rrompt
93.2
84.6
74.3
98.7
Table 4: Evaluation accuracies (%) of SPC and its ablated methods. The indentation indicates each ablation is based on the previous ones. ACC stands for auxiliary character classification.
SPC excelled by effectively tuning on an adequate Nomain-speciffe corpus. It's worth mentioning that the response of GPT-3.5-turbo may contain more than one name, c.g., "Jin-bo bin- sister, Xiu" and "Run- ye's husband (Xiang-qian)". These responses may fool our evaluation criterion, as the response is only required to cover the true speaker.
4.5 Ablation study
We conducted ablation studies based on SPC to investigate the effect of each module, with results shown in Table 4.
We first removed ACC from SPC. As can be seen in the table, removing ACC dropped the evaluation accuracies by 0.8% and 0.6% on WN, by 0.2% on JY, and by 3.7% on WP. This indicates that the auxiliary character classification tasks are beneficial for improving speaker identification performance. Only the NUSC task was applied to training on WP-train, and it contributed a lot to the performance on WP-test. We think it's because the writing of WP is more normative than the writing of novels in WN. The sequential utterances in WP
To gain insight into how the performance of SPC and its ablated methods varies with the scale of training data, we trained them on varying numbers of novels sampled from WN-large and evaluated their performance on WN-test. To facilitate com- parison, we performed a similar evaluation on CSN. As is shown in Figure 2, every method achieved better accuracy with more training utterances. SPC consistently outperformed the other methods on all evaluated training data scales. It's observed that ACC brought almost constant improvements as the training data grew. While the prompt- based architecture was more effective in low-
We also conducted pilot experiments to use more neigh- borhood utterances for NUSC hut gained no improvement. See details in Appendix B.2.

---

96
100
Speaker identification accuracy (%)
SPC
94
90
Test accuracy (96
120)
70
60
88
SPC
SPC ACC
86
SPC - ACC Prompt
CSN
8
10
(81.11 (7.7) (4.5) (2.4)
(1.5)
(0.9)
[0.6]
[0.5]
[0.4] (0.3)
100 200
300
400
500
Sentence distance between the utterance and its speaker
Number of training utterances in thousands
Figure 2: The speaker identification accuracies (%) of SPC, CSN, and the ablated methods of SPC on WN- test, trained on increasing utterances sampled from WN- large. The number in parentheses denotes the number of novels used for training.
Figure 3: The speaker identification accuracies (%)
of utterance groups divided by the sentence distance between the utterance and its speaker. The numbers in parentheses along the x-axis are the proportions (%) of utterances groups.
data scenarios and the improvement became less significant as training utterances increased.
tive patterns like "Tom said". On the contrary, utter- ances with distant speaker mentions are typically implicit speaker identification cases and require context understanding, We calculated the speaker identification accuracies of the utterances WN- test, categorizing them based on the proximity of each utterance to the closest mention of its speaker. The comparison between SPC and CSN is shown in Figure 3. The term "sentence distance" refers to the absolute value of the utterance's sentence index minus the index of the sentence where the speaker's closest mention is found.
The deeper ablated SPC (SPC w/o. ACC prompt) initially under-performed CSN, but overtook CSN when the training data reached about 150k utterances. In comparison, CSN didn't benefit much from the increment of training data. As possible explanation, the deeper ablated method using longer context window (445.5 tokens avg.) gradually learned to handle implicit speaker identification cases that require long-range context understanding, after being trained on more data. However, CSN using a short context window (131.3 tokens avg.) still failed in these cases. It's also revealed that generally more data is needed to train models that take longer context window as input. However, with prompting and ACC, SPC overcomes this difficulty and learns to identify speakers in long context window with a much smaller data requirement. As is shown in Figure 2. SPC took merely 31k utterances to reach the accuracy of 90%, while about 180k utterances were needed by its deeper ablated method to gain the same performance.
It can be seen from the figure that, the sentence distance increased, both SPC and CSN identified speakers more inaccurately. Initially, at sentence distance = 1, both models performed comparably and achieved accuracies above 95% However, when sentence distance came to 2, the identification accuracy of CSN drastically dropped to 74%. Utterances with sentence distance can be regarded as implicit speaker identification cases, so CSN is not good identifying the speakers of these utterances. While SPC still maintained over 80% until sentence distance reached 5 and consistently outperformed CSN by 8.4% to 13.4% for sentence distance greater than 1. Thus it's verified that SPC has better understanding of the context compared to CSN, and thus can better deal with implicit speakers. The proportion of test utterances at each sentence distance is also shown in Figure 81.1 % of the utterances are situated at sentence distance 1, clucidating the reason behind CSN's commendable overall performance
4.6 Implicit Speaker Identification Analysis
this section, we're going to find out whether our proposed method shows a stronger ability to identify implicit speakers than other methods using shorter context window, like CSN.
Intuitively, if the speaker's mention is close to the utterance, then the speaker this utterance can probably be identified by recognizing explicit narra-

---

despite its incapacity to handle implicit speakers. We also conducted case study on the implicit speaker identification cases in WN-test, with a few translated examples provided in Appendix C. SPC did perform better than CSN in many challenging speaker identification cases.
in novels. In Interspeech 2019, 20th Annual Con- ference of the International Speech Communication Association, Graz, Austria, 15-19 September 2019, pages 1561-1565, ISCA.
Yue Chen, Zhen-Hua Ling, and Qing-Feng Liu. 2021. A neural-netwwork-basel approach identifying speakers in novels. In Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August 3 September 2021. pages 4114 4118. ISCA.
5 Conclusions
In this paper, we propose SPC, effective frame- work for implicit speaker identification in novels via symbolization, prompt, and classification. Ex- perimental results show SPC's superiority on four speaker identification benchmarks and its remark- able cross-domain capability. Furthermore, SPC significantly outperforms the strongest competitor CSN in implicit speaker identification cases that require deeper context understanding. We also evaluate ChatGPT on two speaker identification benchmarks and present a comparison with SPC. In the future, we hope to harness LLMS with longer input length limits to further improve speaker identification performance.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgum, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022. Scaling instruction- finetuned language models. CoRR, abs/2210. 11411.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho. and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep
Learning, December 2014.
Limitations
Carolina Cucsta-Lázaro, Animesh Prasad, and Trevor Wood. 2022. What does the sea say to the shore? A BERT based DST style approach for speaker to dialogue attribution in novels. In ACI. (7), pages 5820-5829. Association for Computational Linguistics.
Although SPC has proved its effectiveness in novel-based speaker identification, we consider two aspects that can be further improved. First, we only implemented SPC on small base models containing less than billion parameters. In light of the swift progress L.Ms,, investigating the full potential of these advanced LLMs holds significant value and promise for future advancements. Second, in real-world applications, our approach operates on the output of a character name extraction module which might produce incorrect results. Thus, it's worth studying how to improve the robustness of the speaker identification approach and make it less vulnerable to errors in the character names.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqqng Yang. 2021. Pre-training with whole word masking for chinese BERT. IEEE ACM Trans. Audio Speech Lang, Process., 29:3504-3514.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HILT (7), pages 4171- 4186. Association for Computational Linguistics.
David K. Elson and Kathleen R. McKeown. 2010. Automatic attribution of quoted speech in literary narrative. In AAAI, pages 1013-1019. AAAI Press.
Kevin Glass and Shaun Bangay. 2007. A naive salience- fiction
References
based method for speaker identification books. In Proceedings of the I 18th Annual Symposiun of the Pattern Recognition Association of South Africa (PRASA '07), pages 1-6.
Wankiang Che, Yunlong Feng, Libo Qin, and Ting Liu. 2021. N-LTP: an open-source neural language technology platform for chinese. In EMNLP (Demos), pages 42-49. Association for Computational Linguistics.
Hua He, Denilson Barbosa, and Grzegorz Kondrak.
2013. Identification of speakers novels. In ACI. (1), pages 1312-1320. The Association for Computer Linguistics.
Danqi Chen and Christopher D. Manning. 2014. fast and accurate dependency parser using neural networks. In EMNLP, pages 740-750. ACI.
Yuxiang Jia, Huayi Dou, Shuai Cao, and Hongying Zan. 2020. Speaker identification and its application to social network construction for chinese novels. Int. Asian Lang. Process., 30(4):2050018: 1: 120500018: 18.
Jia Miann Chen, Zhen-Hua Ling, and Li-Rong Dai. 2019. A chinese dataset for identifying speakers

---

Yoon Kim. 2014. Convolutional neural networks for sentence classification. In EMNLP, pages 1746- 1751. ACI.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. ICLR (Poster).
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1900.11 1692.
Grace Muzny, Michael Fang, Angel X. Chang, and Dan Jurafsky. 2017. A two-stage sieve approach for quote attribution. EACL (1), pages 460-470. Association for Computational Linguistics.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback. In NewrIPS.
Junjie Pan, Lin Wu, Xiang Yin, Pengfei Wu, Chenchang Xu, and Zejun Ma. 2021. understanding system for text-to-speech in chinese novels. In ICASSP, pages
chapter-wise
IEEE.
Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, Noah Constant, Colin Raffel, and Chris Callison- Burch. 2023. Bidirectional language models are also few-shot learners. In ICLR. OpenReview.net.
Yisi Sang, Xiangyang Mou, Mo Yu, Dakuo Wang, Jing Li, and Jeffrey M. Stanton. 2022. MBTI personality prediction for fictional characters using movie scripts. EMNLP (Findings), pages 6715- 6724. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019, Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01108.
Timo Schick and Hinrich Schützze 2021. Exploiting cloze-qeestions for few-shot text classification and natural language inference. In EACL. pages 255-269. Association for Computational Linguistics.
Matteo Soo, Ya-Chi Yang, and Von-Wun Soo. 2019. Automatic conversion of chinese fairy story into script- A preliminary report and proposal. TAAI, pages 1-6. IEEE.
Dian Yu. Ben Zhou, and Dong Yu. 2022. End-to- end chinese speaker identification. In NATCL-HLT. pages 2274-2285. Association for Computational Linguistics.

---

A Implementation Details
CSN
DST_SI SI
RoBERTi-large
39
RoBERTa- Ta-Baaee
7.6
14
3.6
SPC
CSN
DST_SI
Table 6: GPU hours consumed in training one epoch on
Batch size
128/8/32/128
192/12/32/32
64/4/16/16
WN-large.
Leamning 1/1/4/4 e-5
le-5
4e-5
Max epoch
Early stop
[prefix]
[postfix]
Acc. (%)
Wt T X fjAA)
Table 5: Hyper-parametenn for each method. Some parameters took different values for WN/WP/JY/CSI.
91.6
said these words)
(8
91.6
said)
We adopted Chinese RoBERTa-wwm-ex- and RoBERTa-wwm--et--be 9 for CSN and SPC, while only the base model was adopted for DST due to memory limitation. The large and base models contain 355M and 125M parameters re- spectively. We used BertTokenizer in Transformers toolkit 10 to tokenize the texts.
91.6
R -
91.5
friendly person)
18 7 K said these words
91.4
90.6
For SPC, we selected " (_ L (T) as the prompt, which means "( said these words)" in English. The maximum number of candidate speakers M was set 50. The training criterion of SPC has been demonstrated in Section with the ideal margin mgn set as 1.0. For SPC the auxiliary character classification loss factor a and in (6) were both set as 0.3. The mask probability for the masked mention classification task was 0.5. While the optimization criterion for DST_SI was minimizing the log-likelihood of the true speaker sequence. Adam optimizer (Kingma and Ba, 2015) employed for optimization and the learning rate decayed every epoch by multiplying 0.98. The best models were selected by validation accuracy. Other dyper-parameter: are shown in Table 5. Some of the parameters took different values for WN, WP, JY, and CSI respectively.
Table 7: Validation accuracies (%) on WN-val of different prompt templates. The English translations of meaningful prompts are provided. "_" represents an empty string.
Experiment Settings Discussion
B.1 Using Different Prompts
We tried different prompt templates how they affect speaker identification performance. As can be seen from Table 7, different prompt templates don't affect the performance much, but using a empty prompt would hurt the performance. This indicates SPC's insensitivity to prompt selection.
B.2 Using N-neighborhood Utterances
We conducted pilot experiments on using different numbers of neighborhood utterances for neigh- borhood utterance speaker classification (NUSC), We trained the models WN-small, showing their validation performance in Figure 4. It's clear that using 1-neighborhood utterances for NUSC (our setting for SPC) brings some improve- ments, compared to O-neighborhood (not applying NUSC). However, extending to more neighborhood utterances does not bring further improvements. A possible explanation is that the dependency between neighborhood utterances mostly lies in adjacent utterances, instead of distant ones.
As the CSI dataset does not provide candidate speakers, we utilized the open-source NLP toolkit LTP (Che et al., 2021) 11 a character name extractor to generate candidates.
We used 4 NVIDIA RTX 3090 GPUs for periments. The GPU hours consumed train each model for one epoch on WN-large shown Table 6. On WN-large the training each investigated model converges in 30 to 50 epochs.
"The pre-trained model checkpoints be obtained from and
C Case Study
https://hugginggfce... cwftfl/chineee-robee www.rese--bbbrrwwwwaawee
http://hwwgggggaroo.ooocccceeees coltlllchines--roeetttwwww wwm-exx.. https///ugginggace.
10 http:://gthub... httpps///iihuu.ooo/ugggggee
11 hwwhhuggingfaccccnnnssmeee
Figure shows three translated examples from WN- test, with speaker identification results different

---

90.1
Validation accuracy (%)
89.9
89.8
89.7
89.6
2
N-neighborhood utterances for NUSC
Figure The validation accuracies (%) of using different numbers of neighborhood utterances for NUSC (w/o. MMC).
methods listed at the bottom of each example. They are all implicit speaker identification examples without explicit narrative patterns. We observed SPC did much better than CSN in solving these challenging speaker identification cases that require deeper context understanding. Example 3 shows that the auxiliary character classification tasks (ACC) help the model to better capture the inter- utterance dependency.
D
Web Novel Collection Details
The web novel collection was annotated on 230 web novels by group of Chinese native annotators. The utterances in the novels were first identified based on quotes. Then, the annotators were instructed to mark the name of the speaker in the neighborhood context of each utterance, and the names of other characters were also marked if found.
Copyrights of the web novels belong to their respective proprietors. The authors are allowed to use the data for research purposes and should follow the principle of fair use. The data annotators are group of employed professional data annota- tors cach at least with bachelor's 's degree. Their wages in line with local regulations. Before the novels were handed to the annotators, they had been reviewed by group of content reviewers to filter any offensive information, including violence, terror, abuse, etc. The annotation instruction informed the annotators of the potential use of the annotated data for speaker identification research.

---

Example
stops, as looked at hair, sometring quictly revied her heart, that deepy commant neart fist
she met him, the hin softly".
eyes
warling speak, long pest,
thought sucderly something
came across mind, stops
gendy:
"How your family treat you?"
Su suddenly stopped, shockid raised he expected
Who the speakerr of Sertence No. 47
CSN Su Yuu
SPC w/o.. ACC, Xiao Xiao V
SPCC: Xino Xino /
Example 2
Milleee just want ask you now, else you about? thatyour mother hooplaall opened flightly shook slowl.
just sald
heard by
my motheris seriously notgoing Malea.
loered her eyes
want
Mnnain, sonr. I'm sorry, admitt that some things hid from like hap-leng, for six years, have control
"Butyou have already hurt me."
She spake through gritied toeth.
10- For the first time, she leels betrayed by her friend.
11. tmm somy, mally sony, we can't be friends the future, but we can also be parthers. Please don't we have to meet in the future."
12. Weeer"
13. YYesh""
14. suddenty hec she not
say.
goint meet
Finally saylng anything. picked up notebook
Who the Sentence No.
CSA Lue Amazin X
SPC w/a. ACC: Milie
SPC: Metee
Example
Novalane,
hai for
Pan-zhi whikp her delicate face and this time, Hongeiiu pushed the door with teanful full
pale, showing her weakness, but of nuthless came across her eyes, Mo-alang dacilely stepped forward help combo the Pan-zhi hed alresdy she hastly asked with concem:
Niss, you
well you shouldn' up.
Hong-aiu siu, the prince entered the palace, is it*
5- Hong-siu was startid, staring at Mi ooslann, this serve senvant girl must be goss ioing in Miss's ears. then she thought of the prince's purpose, her eyes suppressing the
sonoe, she squeezed aut smile:
Miss, the prince came to the palace only to moet the bad priminister."
"Hlngg all, you con't have to hide anything from me, he has come here break off the ongagement, definialy have see him, how can he breaic off without paying any price?*
There was sense of chill the words of Panahhi, Hong-xlu could not help but think the scone where she farced the Servant Li to ditnk the polsoned wine, shudser pessed through.
Who is the speaker of Sentence No.
CSN: Mo-siang X
w/o. Moxig X
V
Figure Translated examples of implicit speaker identification cases from WN-test.


Reviews:
[{"review_id": "8LYUEeSIMy", "reviewer": "Reviewer_M73c", "summary": "", "questions": "line 104: many readers may not understand the \"minimizes the gap\" argument.\n\nIn section 4.5, the authors draw some conclusions from differences observed on the WP test data (versus the WN test data). But according to Table 2, WP is tiny. There may be many novels in WN which are just like the single novel in WP. Why is this  special? Perhaps the authors should show a histogram over scores for individual novels in WN, and place the WP novel score in that context?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Re: ***The definition of \"speaker\", \"mention\", and \"occurrence\"***\n\nWe genuinely appreciate your thorough review of our manuscript, and we sincerely apologize for any confusion our work might have caused. In our task definition section (Sec 3.1), we highlighted that the \"occurrences\" are referred to as \"mentions\". Thus in your Fred example, the number of mentions/occurrences is 3. In an excerpt, each potential speaker has one or more mentions/occurrences. The \"unique mention\" should be another word--\"alias\". The reason why we refer to \"occurrence\" as \"mentions\" is to be consistent with the terminology of other papers [1][2]. We apologize that \"mention-level\" in line 084 is indeed misleading and we will replace it with \"alias-level\". In addition, we will replace the term \"mention\" with \"occurrence\" in Section 3 for clarity. Specifically, the verb \"mention\" in line 328 can be replaced by \"occur\".\n\n2. Re: ***The definition of \"instance\"***\n\nAn \"instance\" in Table 2 and line 363 refers to an utterance whose speaker we aim to identify, not an \"occurrence\". We will add the definition of \"instance\" in the revision.\n\n3. Re: ***The definition of \"distance\" in Section 4.6***\n\nIn Section 4.6, specifically at line 552, we introduce the term \"sentence distance\". This metric is calculated as the absolute difference between the index of the utterance's sentence and the index of the sentence containing the nearest occurrence of the speaker. The concept of \"sentence distance\" serves as an indicator of the proximity between the speaker and the utterance, and it plays a pivotal role in quantifying the level of implicitness within speaker identification cases. This critical analysis experiment serves as validation for our key contribution. We are committed to improving the clarity of this section.\n\n4. Re: ***Fail to show what features of written dialogues are better handled***\n\nIn this paper, our main focus is to deal with the implicit speaker identification cases. We introduce the term \"implicit speaker identification\" in the paragraph starting at line 046, referring to utterances lacking explicit narrative patterns such as \"... Said Tom\". Section 4.6 presents a quantitative analysis of our proposed SPC's improvement in implicit speaker identification cases. We also show some implicit speaker identification examples in Appendix C, demonstrating how our proposed approaches can better solve these cases. We will include more examples to show more features of the implicit speaker identification cases. Thanks for the advice!\n\n5. Re: ***Minimize the gap***\n\nIn the context of line 104, we wish to illustrate that in our introduced framework, we formulate speaker identification as a masked language modeling (MLM) task which is exactly the pre-training task of some PLMs. We will enhance its clarity to ensure easier comprehension for other readers.\n\n6. Re: ***WP vs. WN***\n\nIn Section 4.1, we introduce the difference between WP and WN. WP is a published literature work, while WN is a bunch of online novels with less formal writing styles. Besides, as WP is a widely used data resource in previous papers [2][3], we have to report results on WP for comparison. We will show a histogram over scores for individual novels in our revision. Thanks for the advice!\n\n7. Re: ***Typos***\n\nThank you for your kind and valuable advice for improving our writing. We really appreciate your advice and will adopt them in our revision.\n\nReferences:\n\n[1] He H, Barbosa D, Kondrak G. Identification of speakers in novels. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) 2013 Aug (pp. 1312-1320).\n\n[2] Chen Y, Ling ZH, Liu QF. A Neural-Network-Based Approach to Identifying Speakers in Novels. InInterspeech 2021 (pp. 4114-4118).\n\n[3] Yu D, Zhou B, Yu D. End-to-End Chinese Speaker Identification. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies 2022 Jul (pp. 2274-2285).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed system appears to be evaluated on a larger variety of data than have been used for the same purpose in the past. The authors do a suitable job of comparing their system to the work of multiple previous authors.", "reasons_to_reject": "The article is written insufficiently clearly, and understanding difficulties due to lack of clarity are likely to accumulate in the minds of readers, probably overwhelming many by the time the end of the article is reached. It seems that a large part of this problem is due to missing and crisp definitions of the terms \"speaker\", \"mention\", and \"occurrence\". (I think I understand that in a novel excerpt in which a dialogue between Fred and Nancy contains one reference to Fred as \"Fred\" and two references to him as \"the boy\", the number of speakers is 2, the number of mentions corresponding to Fred is 2, and the number of occurrences corresponding to Fred is 3. The text states on line 207 that mentions are occurrences, but on line 084 \"mention\" must be understood to mean \"(unique) mention\" rather than occurrence.) In addition, Table 2 and line 63 refer to \"instances\", which is not defined but could mean \"occurrences\". Finally, the authors use the phrase \"characters are mentioned\" (line 328) with \"mention\" as a verb, and it is not clear whether they mean \"(unique) characters are mentioned\" or \"character mentions are mentioned\". Section 3.3 is likely to be misunderstood (or just not understood) by most readers, and it is the first stage of the authors' pipeline. It is also not possible to understand the authors' definition of distance in the analysis part of the article.\n\nThe article also fails to provide compelling evidence of precisely which features of written dialogue/multilogue are handled better by their system than by existing systems. The ablation experiments -- although very welcome -- indicate which components of their proposed system are responsible for how much improvement, but not which features of written dialogue correspond to those components. I think the paper would be much stronger if the authors gave an example of dialogue/multilogue where \"short-range semantic understanding\" is necessary, where \"long-range pragmatic (presumably this is what the authors mean when the say \"patterns\") understanding\" is necessary, etc.", "questions_to_authors": "line 104: many readers may not understand the \"minimizes the gap\" argument.\n\nIn section 4.5, the authors draw some conclusions from differences observed on the WP test data (versus the WN test data). But according to Table 2, WP is tiny. There may be many novels in WN which are just like the single novel in WP. Why is this  special? Perhaps the authors should show a histogram over scores for individual novels in WN, and place the WP novel score in that context?", "paper_topic_and_main_contributions": "The article presents a novel system for attributing written utterances to characters. The system relies on three stages: symbolization, prompt, classification. Its performance is argued to be superior to that of earlier systems, as measured on larger data than has been used in the past.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The acronyms in Figure 1 should be resolved in its caption, since the figure is mentioned on line 220 on page 3; some (but not all) of those acronyms are resolved only o line 299 on page 5.\n\nline 388: \"x is dotted with y\" could be replaced by the more commonly used phrase \"to take the dot product between/of x and y\".\n\nline 406: \"expensive API costs\" -> \"high API costs\"\n\nlie 431: \"non-LLM baselines\": which ones are those? the authors could make this more onbious to all readers\n\nThe authors use of the label \"ChatGPT\" is limited to the abstract, introduction, and conclusion; everywhere else the label \"GPT-3.5-turbo\" is used, presumably to mean the same thing. It may be safer to not assume equality between these two labels."}}, {"review_id": "oezk39qH3Q", "reviewer": "Reviewer_rtML", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Re: ***The auxiliary tasks are not novel***\n\nAs demonstrated in Section 3.6, we introduce two auxiliary tasks: 1) neighborhood utterances speaker classification (NUSC) to utilize the speaker alternation pattern between adjacent utterances, and 2) masked mention classification (MMC) to alleviate the excessive reliance on neighborhood explicit narrative patterns. The two auxiliary tasks are introduced to tackle the specific challenges in training speaker identification models, and they enhance the performance. Unlike traditional dialogue tasks, novel-based speaker identification relies predominantly on understanding narrative context rather than solely on comprehending dialogues. We will address your concerns by conducting further investigations and comparing with existing works. Thanks for your reminder.\n\n2. Re: ***SPC handles the similar task of E2E_SI and BookNLP***\n\nIndeed, E2E_SI and the open-source toolkit BookNLP can also serve for speaker identification. Our unique contribution lies in introducing a novel speaker identification framework--SPC--which surpasses the performance of the top baseline, particularly in implicit speaker identification scenarios.\n\n3. Re: ***The add of prompting is a simple engineering***\n\nWhile we acknowledge that the addition of prompting alone may seem minor, it plays a pivotal role in our novel and robust framework for speaker identification. By incorporating the prompt, we not only introduce a placeholder for speaker symbol classification but also distinguish the target utterance from other utterances. We will highlight the significance of the prompt in our revision.\n\n4. Re: ***The improvements are not significant***\n\nAs shown in Table 3, our proposed SPC shows significant improvements in WN-test which is the largest test dataset. In cross-domain evaluation, you mentioned that SPC only absolutely surpasses CSN by 0.9% (indeed <1%) on JY-test. However, the accuracy of SPC and CSN is 98.8% compared to 97.9%, resulting in a 43% reduction in error rate. We will emphasize the improvements in terms of error rate.\n\n5. Re: ***SPC is meaningless under the LLM era***\n\nAt present, SPC exhibits greater robustness than LLMs and has the potential to be integrated with them to achieve even higher performance. \n\nWe observed that the persistence of hallucination remains evident within LLMs. This is evident in the large performance gap observed between GPT-3.5-turbo and SPC on JY (see Table 3), a relatively easier benchmark compared to WP. We experimented with various prompting strategies in Section 4.2 and selected the best one. However, despite showing great potential in the speaker identification task, LLMs currently fall short of SPC's performance when provided with sufficient training data. Within our research budget, we will also continue exploring alternative prompts and reporting meaningful results. \n\nFurthermore, SPC is characterized by its data efficiency (as discussed in Section 4.5), suggesting its capability to reduce the resources needed (in terms of labeled data and computation) for effectively adapting LLMs to the speaker identification task.\n\n6. Re: ***Missing evaluation on English datasets***\n\nOur paper focuses on the robustness and cross-domain adaptability of our proposed speaker identification framework in a single language, while multilingual adaptability is not one of our contributions. We also noticed there are English datasets for speaker identification, but these fall beyond the scope of this paper. We will consider adding these evaluations in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The task, experiments and analysis are described clearly; the proposed approach is able to surpass baselines most of the time, especially with improvement on implicit cases, which fulfills the motivation.", "reasons_to_reject": "Although the proposed approach does appear better than baselines on the experimented datasets, there are three aspects that I found limiting its significance:\n\n1. The proposed approach largely echos well-established insights from previous works, rather than presenting a new perspective. Including:\n    1. The proposed auxiliary tasks (or some variants) are common in many dialogue-understanding tasks, such as dialogue-act prediction or dialogue language model pretraining.\n    2. The prediction of speaker symbols does not have core difference from previous works such as E2E_SI, as they are all designed to predict the best speaker mention from the input, with varying model engineering or formality changes. Another similar example would be BookNLP that also predicts the best speaker mention.\n    3. The add of prompting in RoBERTa is merely a simple engineering work.\n2. The improvement of the proposed approach does not surpass baseline significantly, especially for the cross-domain setting, where the proposed approach has <1% improvement on 2 out of 4 evaluation sets. The most improvement is shown on WP-test; however, it then fails to offer better predictions than the zero-shot ChatGPT.\n3. ChatGPT shows impressive zero-shot performance, which is essentially on a par with supervised models, and even achieves SOTA on WP-test directly. Although there appears a gap on JY, it is unclear that whether ChatGPT can be further improved through some simple tuning such as adding CoT. The paper does not provide details on how ChatGPT is prompted. Nevertheless, the proposed approach does not offer meaningful advantages under the era of LLMs.\n\nIn addition, there are other similar datasets in English focusing on speaker attribution. See Missing References.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes an approach to resolve speaker identification on novel dialogues, with a particular focus on the implicit case, where an utterance is not accompanied with explicit speaker cues, requiring strong understanding of the dialogue flow.. The approach first replaces speaker mentions of different context with special symbols to have a unified speaker label set, followed by RoBERTa encoding and speaker classification upon the hidden state. Two auxiliary tasks are also introduced in training to facilitate the model learning. Experiments are conducted on three datasets that compare against three baselines, and the proposed approach shows improvement most of the time. In addition, ChatGPT is also evaluated, which shows on-par performance with zero-shot prompting.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[The Project Dialogism Novel Corpus: A Dataset for Quotation Attribution in Literary Texts](https://aclanthology.org/2022.lrec-1.628/)\n\n[Improving Automatic Quotation Attribution in Literary Novels](https://aclanthology.org/2023.acl-short.64.pdf)\n\n[Measuring Information Propagation in Literary Social Networks](https://aclanthology.org/2020.emnlp-main.47)"}}, {"review_id": "rWjtM55vAn", "reviewer": "Reviewer_eb8A", "summary": "", "questions": "- It is not clear how the model can predict the speakers in the cross-domain dataset, since the framework first symbolize the speakers then predict the speaker by classification. Are the characters in the cross-domain dataset also in included in the symbol set?\n- It is not clear how to run model inference on unknown data. If the character has aliases, how the model output will be normalized to a single speaker?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Re: ***The labels of auxiliary tasks***\n\nWe didn't annotate additional data and the performance gain is not from the increase of annotated data. For NUSC, we used the previous and the following utterances of the target utterance for auxiliary speaker classification. NUSC is only performed if the speakers of the previous and the following utterances are already labeled in the original dataset. For MMC, we just masked the characters from the original text and used the model to predict them, which also doesn't require any annotation. We will clarify that in the revision. Thanks for your reminder.\n\n2. Re: ***How can the model predict the speakers in unseen novels***\n\nFor an excerpt from an unseen novel, as discussed in Section 3.1, we require the aliases of the characters in the novel to be collected in advance. Then, we only need to replace the characters in the excerpt with the symbols ($C_1$, $C_2$,...) and our model will work smoothly. The symbol $C_i$ represents the $i^{th}$ character appearing in the excerpt, rather than mapping to a specific character. We will enhance the clarity of Sections 3.1-3.2.\n\n3. Re: ***How can the model output be normalized to a single speaker***\n\nAlias normalization occurs during character symbolization, through heuristic rules and manual correction. After the symbolization process, the different aliases are linked to the specific speaker and are replaced by the same symbol. We will clarify this process. Fully automated alias clustering is an interesting topic worth further exploration.\n\n4. Re: ***Typos***\n\nThank you for carefully reading our paper! We will revise the grammar error. As for Figure 1, the bars represent the prediction scores of the candidate speakers (see Section 3.5).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method is able to identify the speakers of the utterances appearing in novel corpus.\n- The model is able to generalize to cross-domain dataset with different writing styles.", "reasons_to_reject": "- Whether the labels of NUSC and MMC tasks are from the original dataset or annotated by the authors is not clear. If they are annotated by the authors, the performance gain may be from the increase of the training data. It makes more sense to use those labels for other baselines for a fair comparison.", "questions_to_authors": "- It is not clear how the model can predict the speakers in the cross-domain dataset, since the framework first symbolize the speakers then predict the speaker by classification. Are the characters in the cross-domain dataset also in included in the symbol set?\n- It is not clear how to run model inference on unknown data. If the character has aliases, how the model output will be normalized to a single speaker?", "paper_topic_and_main_contributions": "The authors describe a framework for identifying speakers in novels by symbolizing the speakers entities into a set of symbols, generating a prompt template for masked pre-training, then predict the speakers via a classification task. Within the framework, the authors add two auxiliary tasks to improve the identification performance: neighborhood utterances speaker classification (NUSC) and masked mention classification (MMC). The experiments show that the proposed framework outperforms other baselines including the GPT-3.5-turbo LLM. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "page 2 col 2 line 128: neural models and to analyze their performance*s*\nThe bar charts in Figure 1 should be C1/C2/C3 instead of S1/S2/S3.\n\n"}, "extra_text": {}}, {"review_id": "hquWg3feL7", "reviewer": "Reviewer_2pX6", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are truly grateful for your appreciation!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The manuscript proposes an interesting framework named SPC for identifying implicit speakers in novels via symbolization, prompt, and classification.\n\n2. Sufficient details on experimental setup was provided. \n\n3. The manuscript was clear and concise. ", "reasons_to_reject": "Null", "questions_to_authors": "", "paper_topic_and_main_contributions": "The manuscript introduces a framework named SPC for identifying implicit speakers in novels via symbolization, prompt, and classification.\nThe key research problem addressed in this manuscript is implicit speaker identification in long range contexts such as narratives, for example. Existing state-of-the-art focus on handling explicit identification and hence do not do well in handling long-range contexts and complex cases. The proposed framework in this study uses symbolization, prompt and classification(SPC) to overcome and address this challenge. Experimental results presented shows that the method outperforms previous state-of-the-art by a large margin of 4.8% accuracy on the web novel collection and is more accurate in implicit speaker identification cases, where long-range context semantic understanding may also be required. The authors also showed that the proposed method outperforms ChatGPT in two benchmarks. The manuscript concludes with sufficient ablation studies showing the effects of each of the components of the framework as well as possible limitations of the approach.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a model for attributing dialogue utterances in a novel to the characters in that novel, in particular handling cases without explicit speaker cues.  Overall, the reviewers agree this is a decent contribution with appropriate experimental support, but with somewhat limited innovative quality.  The comparison with ChatGPT is interesting, especially in cases where it is outperformed by the proposed SPC model.|meta
Decision: Accept-Findings