Plot Retrieval as an Assessment of Abstract Semantic Association (2024)

Abstract: Retrieving relevant plots from the book for a query is a critical task, which can improve the reading experience and efficiency of readers. Readers usually only give an abstract and vague description as the query based on their own understanding, summaries, or speculations of the plot, which requires the retrieval model to have a strong ability to estimate the abstract semantic associations between the query and candidate plots. However, existing information retrieval (IR) datasets cannot reflect this ability well. In this paper, we propose PLOTRETRIEVAL , a labeled dataset to train and evaluate the performance of IR models on the novel task Plot Retrieval. Text pairs in PLOTRETRIEVAL have less word overlap and more abstract semantic association, which can reflect the ability of the IR models to estimate the abstract semantic association, rather than just traditional lexical or semantic matching. Extensive experiments across various lexical retrieval, sparse retrieval, dense retrieval, and cross-encoder methods compared with human studies on PLOTRETRIEVAL show current IR models still struggle in capturing abstract semantic association between texts. PLOTRETRIEVAL can be the benchmark for further research on the semantic association modeling ability of IR models.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 146–161 August 11-16, 2024 ©2024 Association for Computational Linguistics
Plot Retrieval as an Assessment of Abstract Semantic Association
Shicheng Xu∗1,2 Liang Pang1† Jiangnan Li2 Mo Yu2† Fandong Meng2 Huawei Shen1 Xueqi Cheng1 Jie Zhou2
1CAS Key Laboratory of AI Security, Institute of Computing Technology, CAS 2Pattern Recognition Center, WeChat AI
xushicheng21s@ict.ac.cn pangliang@ict.ac.cn moyumyu@global.tencent.com
Abstract
Retrieving relevant plots from the book for a query is a critical task, which can improve the reading experience and efficiency of readers. Readers usually only give an abstract and vague description as the query based on their own understanding, summaries, or speculations of the plot, which requires the retrieval model to have a strong ability to estimate the abstract semantic associations between the query and candidate plots. However, existing information retrieval (IR) datasets cannot reflect this ability well. In this paper, we propose PLOTRETRIEVAL , a labeled dataset to train and evaluate the performance of IR models on the novel task Plot Retrieval. Text pairs in PLOTRETRIEVAL have less word overlap and more abstract semantic association, which can reflect the ability of the IR models to estimate the abstract semantic association, rather than just traditional lexical or semantic matching. Extensive experiments across various lexical retrieval, sparse retrieval, dense retrieval, and cross-encoder methods compared with human studies on PLOTRETRIEVAL show current IR models still struggle in capturing abstract semantic association between texts. PLOTRETRIEVAL can be the benchmark for further research on the semantic association modeling ability of IR models.

heading: 1 Introduction
text: We propose a new task, Plot Retrieval, which retrieves the relevant plots from the book for a query. The task is a spontaneous process in humans’ daily lives. When reading a book or coming across other life events that remind a plot, humans naturally require to find the target plot. As a result, Plot Retrieval is a common and natural scenario but has not been well-studied in NLP.
Although Plot Retrieval can be formalized as an information retrieval (IR) task, the key challenge
∗Work done during the Tencent Rhino-bird Research Elite Program at WeChat.
†Corresponding authors.
ODQA: MS MARCO Query: How bits in a pixel? Passage: The bit depth is the number of bits used to encode each color channel. Today, most digital images use a bit depth of 8, meaning that each color channel in a pixel is represented by 8 bits ...Clear and direct Q&A relationship Duplicate Retrieval: Quora Text 1: What can make Physics easy to learn? Semantic consistency
Text 2: How can you make physics easy to learn?
Ad-hoc Retrieval: Robust Query: Hydroponics
Term-based matching
Article: … Police discovered anything from a hundred to a thousand plants being grown in basements and garages, using hydroponics and market gardening techniques such as high intensity lighting …
Abstract Semantic Association in Plot Retrieval
Query: There is a person saving the Baron.
Plot: The stranger said ``Monsieur Baron, there is a thief and murderer in your house.'' Baron shook and said ``In my house?'' The stranger, calm and collected, dropped his cap as he stretched his arms, and continued: ``Monsieur Baron, watch out for murderers and thieves''
Traditional Lexical and Semantic Matching
Query is associated with Plot：In plot, a stranger reminds Baron to watch out for murderers and thieves, this is associated with saving Baron in the query.
Gap
Human Cognition
Figure 1: The gap between existing IR datasets and Plot Retrieval in estimating the relationship between texts.
in Plot Retrieval is estimating the abstract semantic association between two texts that cannot be simply measured by lexical or semantic matching. Specifically, we analyze the logs of online reading apps such as Kindle, iReader, Douban1 and find that the semantic association between the description of the plot given by the reader (i.e., query) and the actual plot in the book is very abstract. This abstract association is mainly because users integrate their own understanding, summaries, or speculations of the plot when writing the query, which makes it hard to directly associate plots to the query like traditional lexical matching, semantic similarity, or relevance. For example, for a plot: The stranger said “Monsieur Baron, there is a thief and murderer in your house.” Baron shook and said “In my house?” The stranger, calm and collected, dropped his cap as he stretched his arms, and continued: “Monsieur Baron, watch out for murderers and thieves”, and the query for this plot given by reader is There is a person saving the Baron. This association is generated by human
1 https://www.ireader.com.cn, https://book.douban.com.
146
cognition and is more difficult to estimate than just lexical or semantic matching because it requires IR models to understand that the stranger reminds Baron to watch out for murderers and thieves is actually associated with saving the Baron, even though their literal meanings are different. However, as shown in Figure 1, existing IR datasets do not reflect this abstract semantic association well. For example, in Open-domain Question-Answering such as MS MARCO (Nguyen et al., 2016), Natural Questions (Kwiatkowski et al., 2019), and SQuAD (Rajpurkar et al., 2016), the query and its corresponding passage form a clear and direct question-and-answer relationship. In Duplicate Retrieval such as Quora and MRPC (Dolan et al., 2004), the annotation is based on whether the semantics of the two texts are consistent. In Ad-hoc Retrieval such as Robust04 (Voorhees, 2004), lexical matching still accounts for the main part and semantic association is less (Xu et al., 2022).
A dataset that can reflect abstract semantic associations between texts generated by human cognition is important for the entire IR community to study the upper limit of the IR models’ ability to model semantic association. However, it is very difficult to obtain the annotated query-passage pairs with sufficient abstract semantic association. Annotating abstract semantic association pairs requires annotators to pay the high reading cost for passage, and have sufficient comprehension ability to write a query that looks very different from the passage but has abstract semantic association with it.
In this paper, for Plot Retrieval, a novel and challenging IR task, we propose a labeled dataset called PLOTRETRIEVAL with 430K query-plot pairs. Compared with existing IR datasets, text pairs in PLOTRETRIEVAL have the following obvious characteristics: (1) more abstract semantic association generated by human cognition and (2) less word overlap. These two characteristics enable PLOTRETRIEVAL not only to be used to perform training on Plot Retrieval task but also become the benchmark for evaluating the ability of IR models to estimate abstract semantic association between texts. In the construction of PLOTRETRIEVAL, we collect publicly available raw data from the Internet, which shares the idea with (Wan et al., 2019; Yu et al., 2023). To address the difficulty in annotation mentioned above, instead of directly asking the annotators to write a query that has abstract semantic association with the plot, we first use weakly
supervised information to collect query-plot pairs that may have semantic association, and let the annotator select the pairs that really contain abstract semantic association, regularize these pairs, and get the final query-plot pairs.
In experiments, first, we evaluate various lexical retrieval, sparse retrieval, dense retrieval, and crossencoder methods trained on mainstream IR datasets such as MS MARCO on PLOTRETRIEVAL, and find that these methods do not perform well, which shows the difference between PLOTRETRIEVAL and the current IR datasets. A noteworthy finding is that BM25, the strong zero-shot IR baseline based on lexical-matching (Thakur et al., 2021; Izacard et al., 2022), achieves better performance on BEIR (Thakur et al., 2021) than many neural IR models, but has worse performance on PLOTRETRIEVAL. This indicates that PLOTRETRIEVAL has the higher challenge for semantic understanding rather than simple literal matching. Second, we train IR models on our weakly supervised data and achieve better performance than the models trained on MS MARCO, which indicates the effectiveness of our annotation strategy. Third, human studies show that the current IR models are far behind human in capturing abstract semantic association, and there is a lot of room for improvement in future research. Our contributions are: • We propose a novel, critical and challenging task called Plot Retrieval, design a novel evaluation metric called N-RODCG and construct a dataset called PLOTRETRIEVAL for this task. • Extensive experiments across various IR models and the comparison with human studies on PLOTRETRIEVAL show that the current IR models still struggle in capturing abstract semantic association between texts and there is a lot of room for improvement in the future research. • We broaden the research field of Information Retrieval from lexical or semantic matching to more ambiguous abstract semantic association between texts, and PLOTRETRIEVAL can be used as an effective benchmark for evaluating this ability of IR models. We will release both English and Chinese versions of PLOTRETRIEVAL at https:// github.com/xsc1234/Plot-Retrieval for further research.

heading: 2 Related Work
text: Information Retrieval Datasets According to specific task, existing mainstream IR datasets
can be divided into: Open Domain Questionanswering (MS MARCO (Nguyen et al., 2016), Natural Questions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017), SQuAD (Rajpurkar et al., 2016), WebQuestions (Berant et al., 2013), FiQA (Maia et al., 2018), HotPotQA (Yang et al., 2018a) and CuratedTREC (Baudis and Sedivý, 2015), etc.), Ad-hoc Retrieval (Robust (Voorhees, 2004), ClueWeb (Yang et al., 2018b), MQ2007 (Qin et al., 2010)), Duplicate Retrieval (Quora, CQADupStack (Hoogeveen et al., 2015), MRPC (Dolan et al., 2004)), Entity Retrieval (DBPedia-Entity (Hasibi et al., 2017)), Argument Retrieval (ArguAna (Wachsmuth et al., 2018) and Touchè-2020 (Bondarenko et al., 2020)), Citation Prediction (SCIDOC (Cohan et al., 2020)) and Fact Checking (FEVER (Thorne et al., 2018) and Climate-FEVER (Diggelmann et al., 2020)). Existing datasets also cover a range of different domains of target documents like Bio-Medical articles (Tsatsaronis et al.), Tweets (Suarez et al., 2018), News (Soboroff et al., 2019).
In all the above datasets, the matching between texts can be summarized as a combination of lexical and semantic matching. The relationship of query-passage pairs in these datasets can usually be judged only by the literal meaning, without the need to deeply understand the semantics and judge the abstract association between semantics. Direct evidence is that BM25 (Robertson et al., 1995) can significantly defeat many neural IR models that have been trained on large-scale supervised datasets only through lexical matching on these datasets in the zero-shot setting (Thakur et al., 2021). PLOTRETRIEVAL has more abstract semantic association and less word overlap between texts, which is a more challenging dataset for IR models.
IR Datasets for Books Our dataset also extends into the significant domain of narrative literature for IR applications. While there exists an extensive list of datasets on story understanding (for more details, please refer to the survey (Sang et al., 2022)), there has been limited work addressing the IR aspect within the context of stories. In relation to our work, two other datasets are noteworthy. The first is RELiC (Thai et al., 2022), which frames the task as utilizing literary analysis paragraphs to retrieve quoted text. This task essentially falls within the realm of IR, although it lacks a standard format of IR queries. The second is NarrativeQA (Kočiskỳ et al., 2018), primarily designed as a book QA
dataset but adaptable for an IR task (Frermann, 2019; Mou et al., 2021). However, it comes with a limitation that it does not provide groundtruth for the retrieval purposes.

heading: 3 Task Description

heading: 3.1 Abstract Semantic Association
text: In the analysis of public data of online reading apps, we conclude five main manifestations of abstract semantic association between the query and the plot. (1) Query abstractly summarizes the plot (Summarization). (2) Query expresses feelings, analysis or comments about the characters or events in the plot (Expression). (3) Query depicts the characters in the plot (Description). (4) Query describes the overall visual information formed by the environment, characters, and events in the plot (Vision). (5) Query is motivated by the event in the plot to reminisce another related event (Reminiscence). Their statistics are shown in Figure 2.

heading: 3.2 Task Definition
text: Plot Retrieval aims to retrieve the relevant plots from the book for a query. Specifically, given a query q, a candidate set of plots P = {p1, p2, ..., pn} for a book and each plot pi consists of m sentences (m is a hyperparameter and we set it as 3). The model needs to give the ranking score for each pi ∈ P based on the association between plot pi and query q, rank the plots in P according to the score, and return a list R with Top-K plots. The challenge of this task is mainly in two aspects: (1) The semantic association between the query and the plot is very abstract. This is mainly because users integrate their own understanding, summaries, or speculations of the plot when writing descriptions. IR models struggle in identifying this abstract association. (2) Plots in the candidate set P come from the same book, they have semantic and entity relatedness to each other. It makes IR models hard to distinguish the semantic difference.

heading: 3.3 Evaluation Metric: N-RODCG
text: As for the evaluation metrics for Plot Retrieval, in addition to the common information retrieval metrics, such as MRR (Mean Reciprocal Rank) and Recall, we propose N-RODCG (Normalized Relative Offset Distance Discounted Cumulative Gain), a novel metric that is more in line with the actual reading scene. The motivation of this metric is that each plot of the candidate set is actually the segment of continuous texts in the original book, even if the retrieved plot is not exactly the ground-truth plot, as long as it is close enough to the groundtruth plot in the original book, the ground-truth plot will appear in the reader’s field of vision and be noticed by the reader. In addition, there is the strong semantic association between plots with small distances. N-RODCG comprehensively measures the ranking of the plots in R and their distance from the ground-truth plots. For a query q, given a retrieved list of plots R = [p′1, p ′ 2, ..., p ′ k] obtained from the model. Because each plot p′i consists of m sentences, we can get the position of p′i in the original text of the book, which is the average value of each sentence index in p′i and we call it si. Then the positions of the plots in R are S = [s1, s2, ..., sk]. And the positions (ti) of the ground-truth plots for q is T = [t1, t2, ..., tg], g is the number of groundtruth plots. The relative offset distance di between p′i and ground-truth plots of q can be computed as:
di = min(|si − t1|, |si − t2|, ..., |si − tg|). (1)
Then, we define the Discounted Cumulative Gain (Järvelin and Kekäläinen, 2002) between ROD and the ranking of the retrieved plots:
RODCG@k = k∑
i=1
f(di)
log(i+ 1) , (2)
where i is the ranking of plot p′i, f is the piecewise function (α is the window of the reader’s field of vision and we set it to 5 based on statistical data):
f(di) =
{ 1
di+1 , di < α; 0, otherwise. (3)
N-RODCG can be computed as:
N-RODCG@k = RODCG@k
I-RODCG@k , (4)
I-RODCG is the value when the plots in retrieval list R for q are optimally ranked, that is, the theoretical maximum value of N-RODCG.

heading: 4 PLOTRETRIEVAL
text: We introduce collection, filtering, translation, annotation, and statistics for PLOTRETRIEVAL in this section. More details are introduced in Appendix.

heading: 4.1 Overview of Dataset Construction
text: The row data of PLOTRETRIEVAL is collected from an online reading app on the Internet. Specifically, we notice recent reading apps allow readers to write publicly available comments on the texts in the book. Many of these comments include abstract descriptions of the plots in the corresponding texts. They are written by the readers based on their own understanding during book reading. While they are semantically associated with the plots, they require sufficient comprehension ability to discover and are challenging for IR models to identify. These comment-plot pairs constitute the weakly supervised signal for query-plot pairs in PLOTRETRIEVAL. We first filter these pairs to remove the comments that have obvious word overlap with plots or have little practical meaning. However, the filtered comment-plot pairs still cannot be directly used as PLOTRETRIEVAL, because the comments written by readers are free-style and have a lot of noise. We let the annotators do more identification and rewriting on them. After the human annotation, we exploit the labeled datasets to construct an automatic annotation model for fast, low-cost acquisition of large datasets. Last but not least, we ensure the complete independence of the training set and the test set during the construction of PLOTRETRIEVAL, which makes that there are enough differences in the domain between the training set and the test set to more reasonably evaluate the ability of the IR models to estimate abstract semantic association.

heading: 4.2 Dataset Construction
text: Step 1: Data Collection. We collect data for training set and test set separately. Specifically, for test set, we use 33 publicly available English books that are collected from Gutenberg project and processed by (Yu et al., 2023). We find 84 Chinese versions of these 33 English books that we have licenses of usage. We sample 52,924 public comments written by readers for various plots in these 84 books. For the training set, we collect 105 books from the same reading app and sample 1,005,480 comments. There is no overlap between books in the training set and the test set.
Step 2: Data Filtering. Before human annotation, we perform a preliminary filter on the collected data. Specifically, first, in order to make the description of the comment for the plot abstract enough, we remove the comments that have a lot of word overlap with the original texts in the book. Given a comment c and the original text t in the book marked by the comment c, we use NLTK2 to perform word tokenization on them and remove the stop words. Then we get the sets of words for them (C and T). We remove the comments that:
|C ∩ T| |C| > 0.5. (5)
Second, we remove the comments that have little practical meaning. That is, the comments that do not describe the plot but express the reader’s emotions such as “This is so funny!” or “I can’t understand this”. We use ChatGPT3 via prompting it to judge whether the comment is describing a specific plot rather than simply expressing emotion to complete this task. Considering that a large amount of data will bring high ChatGPT usage cost, we perform this filtering operation on the full test set and 50,000 samples of training set. For the other samples in the training set, we use the automatic annotation model for fast and low-cost filtering, which will be introduced in Step 5. After this, we get 7,661 samples in test set and 7,432 samples in training set for human annotation. Step 3: Human Annotation. For the sample with a comment c and the original text t in the book marked by the comment c, annotators have two tasks to finish. (1) Judge whether c contains the abstract description of the plot in t. (2) If so, mark the texts describing the plot from c and use the texts as the query q. After this step, we can get the queryplot pairs where there is the abstract semantic association between query and plot. Specifically, we first select nine annotators who have at least a high school education level, because our task requires the annotators to have a certain ability to understand literary works. We write the guidelines to help the annotators better understand the details of the annotation task. Before the formal annotation start, we conduct three rounds of pre-annotation and verify the pass rate of each annotator’s work. We select the annotator whose pass rate of work reaches 90% in the pre-annotation for formal annotation. In the formal annotation, for the results of
2https://www.nltk.org/ 3https://openai.com/blog/chatgpt
each annotator, we introduce another annotator to sample and validate the results and give the pass rate, which can measure whether two annotators agree with the results. We continue to screen and guide the annotators until the pass rate of each annotator reaches 95%. We select the samples that c are judged to contain abstract descriptions of t as the final samples. After this, we get 4,572 queryplot samples in the test set and 4,402 samples in the training set. Step 4: Translation and Corpus Construction. Since the majority of our collected data is in Chinese, we translate the collected data into English. For test set, all books have their public English versions (Step 1). So we (1) translate the comment c to English and (2) project the original text t in the Chinese book marked by the comment c to its content in the English version of the book. For the first task, we finish it by ChatGPT. For the second task, we use Spacy to sentencize the texts of books, use multilingual embedding LASER4 to embed sentences and use vecalign (Thompson and Koehn, 2019) to align the sentences between books based on sentence embeddings. For training set, because some books do not have the corresponding English versions, we directly translate c and t to English by Helsinki5, a neural machine translation model.
We use the collection of plots of books in the test set as the retrieval corpus, which means that when we test the retrieval performance of the IR models on PLOTRETRIEVAL, the samples in the training set do not appear in any test data. For the book, we divide every m sentences into a chunk (the basic unit of the corpus). We mark the chunks containing the sentences in t as ground truth for c. To ensure the semantic integrity of t, we also make t as a chunk and mark it as ground truth. Details of the corpus are shown in Appendix B.2. Step 5: Auto Annotation Model. For the large amount of data in the training set that has not been
4https://github.com/facebookresearch/LASER. 5https://huggingface.co/Helsinki-NLP
manually annotated, we construct a text-pair binary classifier to complete automatic annotation. Specifically, we train BERT6 (Devlin et al., 2019) on 50,000 samples of training set in Step 2 in which 4,402 are annotated as positives in Step 3 and the other are negatives. We use the trained classifier to automatically annotate the data in the training set. Although most of the data in the training set is constructed under the weak supervision of the automatic annotation model, experiments in Section 5.3 show that compared with large-scale supervised IR datasets, our training data is better for IR models to estimate the abstract semantic association.

heading: 4.3 Data Statistics
text: Table 1 shows the statistics of the training set and test set in PLOTRETRIEVAL. Most of the train and validation pairs are obtained from the auto annotation model in Step 5. Table 2 shows the word overlap between the query and candidate documents (calculated by Equ (5)). PLOTRETRIEVAL has the lowest overlap, especially compared to mainstream IR datasets such as MS MARCO. Therefore, compared to the existing IR datasets, the query-plot pairs in PLOTRETRIEVAL pose a higher challenge to the IR models. The pairs look very different but have abstract semantic association, rather than simple lexical or semantic matching.

heading: 5 Experiments
text: In this section, we evaluate various IR models on PLOTRETRIEVAL and perform human studies.
6https://huggingface.co/bert-base-uncased

heading: 5.1 Baselines
text: Lexical Retrieval. We use (1) BM25 (Robertson et al., 1995), a a bag-of-words retrieval method based on word-to-word exact matching. Sparse Retrieval. Following BEIR (Thakur et al., 2021), we select three mainstream sparse retrieval models including (1) DeepCT (learning dynamic term weights) (Dai and Callan, 2020), (2) SPARTA (learning a sparse representation that can be efficiently implemented as an inverted index) (Zhao et al., 2021) and (3) DocT5query (generating queries added to documents) (Nogueira and Lin, 2019). All of them are fine-tuned on MS MARCO. Dense Retrieval. (1) DPR (Karpukhin et al., 2020), a classical dense retrieval model based on bi-encoder and trained with BM25 hard negatives and in-batch contrastive loss. (2) ANCE (Xiong et al., 2021), it dynamically updates negatives during training. (3) TAS-B (Hofstätter et al., 2021) is trained with supervision from cross-encoder. (4) BERM (Xu et al., 2023a,b), a plug-and-play method to enable dense retrieval models to learn representations that are more suitable for matching. (5) Ernie-Search (Lu et al., 2022) trains dense retrieval model by cascade distillation from ColBERT (Khattab and Zaharia, 2020) and crossencoder. All of the above baselines are fine-tuned on MS MARCO. There are also some methods first pre-train models on large-scale datasets by self-supervised IR signal. (6) COCO-DR (Yu et al., 2022) is pre-trained on BEIR (Thakur et al., 2021). (7) coCondenser (Gao and Callan, 2022) and (8) RetroMAE (Xiao et al., 2022) are pretrained on English Wikipedia and BookCorpus. (8) Contriever (Izacard et al., 2022) is pre-trained on English Wikipedia and CCNet. All of these models are fine-tuned on MS MARCO after pre-training for IR. Late-Interaction. ColBERT (Khattab and Zaharia, 2020) performs late interaction on embeddings of each token to achieve finer-grained interaction than dense retrieval. This model is fine-tuned on MS MARCO. Re-Ranking. We use Cross-Encoder (Wang et al., 2020) that exploits self-attention for interaction between tokens as re-ranker, which has shown power in Book QA tasks (Mou et al., 2021). Before reranking, we first use Contriever to retrieve Top-100 documents for each query as its candidate list. This model is fine-tuned on MS MARCO. ChatGPT-Assisted. ChatGPT performs well on
various NLP tasks, we also explore its performance on Plot Retrieval. It is expensive to directly let ChatGPT inference on a large-scale corpus, so we prompt ChatGPT to generate the plot in the corresponding book for the query (query expansion (Carpineto and Romano, 2012)), and then use the generated plot as query and use Contriever to retrieve related plots from the corpus.

heading: 5.2 Experimental Settings
text: First, to explore the ability of the SOTA IR models trained on MS MARCO to estimate abstract semantic associations between texts, we evaluate the performance of them in zero-shot setting on the English version of PLOTRETRIEVAL. Second, to show the effectiveness of our weakly supervised training data, we compare the performance of IR models trained on weakly supervised training data in PLOTRETRIEVAL with existing IR datasets in the same training method and settings. We use bertbase-uncased and bert-base-chinese as pre-trained models for English and Chinese respectively. In training, we set the learning rate to 10−5. We train the model with 64 batch size on a single A100 GPU for 5 epochs and use Pytorch (Paszke et al., 2019) as the training framework. Third, the difficulty of PLOTRETRIEVAL for IR models can be reflected by the performance gap between IR models and humans on different datasets. We compare this gap on different IR datasets via human studies.

heading: 5.3 Experimental Results
text: Performance on PLOTRETRIEVAL. Table 3 shows the zero-shot performance of IR models trained on MS MARCO on test set of PLOTRETRIEVAL. We can draw the following four conclusions. (1) PLOTRETRIEVAL has more abstract semantic association and less word overlap between texts than existing IR datasets, which is more challenging for current SOTA IR models. This can be supported by the phenomenon that BM25, the strong zero-shot IR baseline based on termmatching (Thakur et al., 2021; Izacard et al., 2022), achieves better performance on BEIR (Thakur et al., 2021) than many neural IR models such as DPR, ANCE, and TAS-B, but has worse performance on PLOTRETRIEVAL than all neural IR baselines that can capture the semantic matching information. (2) More training data facilitates the estimation of abstract semantic association, even if the data is self-supervised. This can be supported by the phenomenon that models pre-trained on large-scale
datasets such as coCondenser, Contriever, COCODR, and RetroMAE have better performance than the models fine-tuned directly on MS MARCO. (3) More interactions between texts are conducive to the estimation of abstract semantic association. Cross-Encoder that exploits self-attention for finegrained interaction between tokens shows the best performance. (4) ChatGPT is not good at associating plots with their abstract corresponding queries. Using ChatGPT to generate the plot associated with the query, and using the generated content as the new query for retrieval by Contriever achieves worse performance. It is because we find that ChatGPT cannot accurately generate the plots associated with the query but generates the common content for the book such as the summary and background of the book. This makes the query ambiguous and indiscriminate.
Discussion on N-RODCG. In this paper, we propose a new evaluation metric named N-RODCG, which is more in line with the actual book reading scene. Specifically, traditional IR metrics such as MRR, Recall and NDCG can only reflect the difference in relevance between the texts in the returned rankted list and the ground-truth. However, in the book reading scene, a more reasonable metric is to reflect the distance between the retrieved texts and the ground-truth in the book. Because this can better reflect the retrieval models’ ability to help readers find the content they want from the book. The greater the value of the metric, the closer the retrieved texts is to the ground-truth in the book, and the easier for readers to find what they want to read.
Effect of Weakly Supervised Training Data. The weakly supervised training data we construct has positive significance for improving the performance of the IR models on the task Plot Retrieval. Specifically, we compare the performance of models trained on mainstream supervised datasets (human annotation) with the models trained on weakly supervised training data in PLOTRETRIEVAL. In English setting, we use two datasets as baselines. The one is MS MARCO, the large-scale labeled IR dataset. The other is RELiC (Thai et al., 2022), the large-scale labeled IR dataset that aims to retrieve evidence for literary claims, whose domain also involves book reading. In Chinese setting, we use DuReader (Qiu et al., 2022), a large-scale Chinese labeled IR dataset. These models are finetuned with the same method (DPR) and settings
and perform early stopping on validation pairs. Table 4 shows that weakly supervised training data in PLOTRETRIEVAL significantly improves the performance of the IR models on Plot Retrieval than mainstream supervised IR datasets with much more human annotations. We maintain the independence of the training set and test set in the process of data construction so that there is enough domain gap between them. Besides, although RELiC also belongs to the book domain, its performance is not significantly improved compared with MS MARCO. This further shows the effectiveness of our weakly supervised training data for IR models to learn the abstract semantic association between texts instead of just overfitting the domain.
Human Studies. We perform human studies to compare the performance gap of IR models and humans on MS MARCO, ODQA (consisting of Natural Questions, TriviaQA, SQuAD, WebQuestions), and PLOTRETRIEVAL. Specifically, we sample 500 queries from the test sets of these three datasets respectively, for each query, we construct a candidate list containing 1 ground truth and 19 negatives. We let the IR model and humans select the ground truth for the query from its candidate list and count the
accuracy. We use DPR (trained on MS MARCO) for MS MARCO, DPR (trained on ODQA) for ODQA, and Cross-Encoder (the best model in Table 3 and trained on MS MARCO and PLOTRETRIEVAL ) for PLOTRETRIEVAL as the IR models. We select three humans with college degrees for this study and count the average accuracy. Results in Figure 3 show that although the performance of the IR models on MS MARCO and ODQA is close to human, they still struggle in capturing abstract semantic association on PLOTRETRIEVAL.

heading: 6 Conclusion
text: In this paper, we propose a novel task called Plot Retrieval that retrieves relevant plots from the book for a query. Compared with the existing IR datasets, Plot Retrieval requires the IR models to have the strong ability to capture the abstract semantic association between texts rather than the simple lexical and semantic matching. It is meanly because readers integrate their own understanding, summaries, or speculations of the plot when writing the query. For the Plot Retrieval task, we propose PLOTRETRIEVAL, a large labeled dataset with more abstract semantic association and less word overlap between texts, which can be used as a benchmark to train and evaluate the ability of IR models to capture abstract semantic associations between texts. Extensive experiments across various lexical retrieval, sparse retrieval, dense retrieval, and crossencoder methods compared with human studies on PLOTRETRIEVAL show that the current IR models still struggle in capturing abstract semantic association between texts and there is a lot of room for improvement in future research.
Limitations
In this paper, we propose a novel task called Plot Retrieval. Plot Retrieval aims to retrieve the relevant plots for the query and has higher requirement for the ability of the information retrieval models to estimate the abstract semantic association between texts while existing information retrieval datasets are not satisfied. To achieve it, we collect and release PLOTRETRIEVAL, a large-scale information retrieval dataset with more abstract semantic association and less word overlap. However, although comparison with humans shows that current SOTA
IR models cannot perform well at this task, we do not propose an efficient solution such as novel model architecture and training method to solve this problem. Our contributions focus on proposing a more challenging retrieval task and dataset. Further research on the task will be carried out in future work.
Ethics Statement
In the construction of datasets, we prioritize the ethical use of data and are committed to upholding the highest standards when it comes to protecting user privacy and ensuring data integrity. Specifically, all the data within our dataset is collected exclusively from publicly available information from online applications (apps). We strictly adhere to the legal guidelines and terms of service of these apps during the data collection process. Our data collection practices prioritize user privacy. All personally identifiable information (PII) has been thoroughly masked or removed from the dataset. We declare that our work complies with the ACL Ethics Policy.

heading: Acknowledgements
text: This work was supported by the National Key R&D Program of China (2022YFB3103700, 2022YFB3103704), the National Natural Science Foundation of China (NSFC) under Grants No. 62276248 and U21B2046, and the Youth Innovation Promotion Association CAS under Grants No. 2023111.

heading: B Details of PLOTRETRIEVAL
text: B.1 Examples in PLOTRETRIEVAL Figure 5 shows some examples in PLOTRETRIEVAL.
B.2 Books in PLOTRETRIEVAL Table 5 shows the book name in the corpus of test set and the number of queries and plot chunks for each book.

heading: C Case Study
text: Table 6 shows the comparison of ground truth with Top-1 results retrieved by Contriever and BM25 respectively. The results of BM25 show that BM25 are limited to word overlap but cannot capture semantic level information. For the results of Contriever, they are limited to literal semantic matching, Contriever cannot deeply understand the meaning that the query really wants to express to find the most suitable plot.

heading: Future Work
text: Extensive experiments across various lexical retrieval, sparse retrieval, dense retrieval, and crossencoder methods compared with human studies on PLOTRETRIEVAL show that the current IR models still struggle in capturing abstract semantic association between texts and there is a lot of room for improvement in future research. Limitations
In this paper, we propose a novel task called Plot Retrieval. Plot Retrieval aims to retrieve the relevant plots for the query and has higher requirement for the ability of the information retrieval models to estimate the abstract semantic association between texts while existing information retrieval datasets are not satisfied. To achieve it, we collect and release PLOTRETRIEVAL, a large-scale information retrieval dataset with more abstract semantic association and less word overlap. However, although comparison with humans shows that current SOTA
IR models cannot perform well at this task, we do not propose an efficient solution such as novel model architecture and training method to solve this problem. Our contributions focus on proposing a more challenging retrieval task and dataset. Further research on the task will be carried out in future work