KnowFi: Knowledge Extraction from Long Fictional Texts
Year: 2021

Abstract:
Knowledge base construction has recently been extended to fictional domains like multivolume novels and TV/movie series, aiming to support explorative queries for fans and sub-culture studies by humanities researchers. This task involves the extraction of relations between entities. State-of-the-art methods are geared for short input texts and basic relations, but fictional domains require tapping very long texts and need to cope with nonstandard relations where distant supervision becomes sparse. This work addresses these challenges by a novel method, called KnowFi, that combines BERT-enhanced neural learning with judicious selection and aggregation of text passages. Experiments with several fictional domains demonstrate the gains that KnowFi achieves over the best prior methods for neural relation extraction.

heading: 1. Introduction
text: Motivation and Problem: Relation extraction (RE) from web contents is a key task for the automatic construction of knowledge bases (KB). It involves detecting a pair of entities in a text document and inferring if a certain relation (predicate) holds between them. Extracted triples of the form (subject, predicate, object) are used for populating and growing the KB. Besides this major use case, RE also serves other applications like text annotation and summarization, semantic search, and more.
Work on KB construction has mostly focused on general-purpose encyclopedic knowledge, about prominent people, places, products etc. and basic relations of wide interest such as birthplaces, spouses, writing of books, acting in movies etc. Vertical domains have received some attention, including health, food, and consumer products. Yet another case are KBs about fictional works [Hertling and Paulheim, 2020, Labatut and Bost, 2019], such as Game of Thrones (GoT), the Marvel Comics (MC) universe, Greek Mythology or epic books such as War and Peace by Leo Tolstoy or the Cartel novels by Don Winslow. For KBs about fictional domains, the focus is less on basic relations like birthplaces or spouses, but more on relations that capture traits of characters and key elements of the narration. Relations of interest are allies, enemies, membership in clans, betrayed, killed etc.
Applications of fiction KBs foremost including supporting fans in entity-centric search. Some of the fictional domains have huge fan communities, and search engines frequently receive queries such as “Who killed Catelyn Stark?” (in GoT). Entity summarization is a related task, for example, a user asking for the most salient traits of Ygritte (in GoT). Although fiction serves to entertain, some of the more complex domains reflect sub-cultural trends and the zeitgeist of certain epochs. Analyzing their narrative structures and networks
of entities is of interest to humanities scholars. For example, superhero comics originated in the 1940s and boomed in post-war years, reflecting that era’s zeitgeist (revived now). War and Peace has the backdrop of the Napoleonic wars in Russia, and the Cartel trilogy blends facts and fiction about drug trafficking. KBs enable deeper analyses of such complex texts for historians, social scientists, media psychologists and cultural-studies scholars.
State of the Art and its Limitations: RE with pre-specified relations for canonicalized entities is based on distant supervision via pre-compiled seed triples [Mintz et al., 2009, Suchanek et al., 2009]. Typically, these training seeds come from initial KBs, which in turn draw on Wikipedia infoboxes. The best RE methods are based on this paradigm of distant supervision, leveraging it for neural learning (e.g., [Soares et al., 2019, Wang et al., 2020, Yao et al., 2019, Zhang et al., 2017, Han et al., 2020]). They work well for basic relations, as there is no shortage of training samples (e.g., for birthplace or spouse). One of their key limitations is the bounded size of input text passages, typically a few hundred tokens only. This is not a bottleneck for basic relations where single sentences (or short paragraphs) with all three SPO components are frequent enough (e.g., in the full text of Wikipedia articles). However, for RE with non-standard relations over long fictional texts such as entire books, these limitations are major bottlenecks, if not showstoppers. This paper addresses the resulting challenges (also included among the open challenges in the overview by [Han et al., 2020]):
• How to go about distant supervision for RE targeting non-standard relations that have only few seed triples?
• How to cope with very long input texts, such as entire books, where relevant cues for RE are spread across passages?
Approach and Contributions: This paper presents a complete methodology and system for relation extraction from long fictional texts, called KnowFi (Knowledge extraction from Fictional texts). Our method leverages semi-structured content in wikis of fan communities on fandom.com (aka wikia.com). We extract an initial KB of background knowledge for 142 popular domains (TV series, movies, games). This serves to identify interesting relations and to collect distant supervision samples. Yet for many relations this results in very few seeds. To overcome this sparseness challenge and to generalize the training across the wide variety of relations, we devise a similarity-based ranking technique for matching seeds in text passages. Given a long input text, KnowFi judiciously selects a number of context passages containing seed pairs of entities. To infer if a certain relation holds between two entities, KnowFi’s neural network is trained jointly for all relations as a multi-label classifier.
Extensive experiments with long books on five different fictional domains show that KnowFi clearly outperforms state-of-the-art RE methods. Even on conventional short-text benchmarks with standard relations, KnowFi is competitive with the best baselines. As an extrinsic use case, we demonstrate the value of KnowFi’s KB for the task of entity summarization. The paper’s novel contributions are:
• a system architecture for the new problem of relation extraction from long fictional texts, like entire novels and text contents by fan communities (Section 3);
• a method to overcome the challenge of sparse samples for distant supervision for nonstandard relations (Section 4).
• a method to overcome the challenge of limited input size for neural learners, by judiciously selecting relevant contexts and aggregating results (Section 5);
• a comprehensive experimental evaluation with a novel benchmark for relation extraction from very long documents (Section 6), with code and data release upon publication.

heading: 2. Related Work
text: Relation Extraction (RE): Early work on RE from text sources has used rules and patterns, (e.g., [Agichtein and Gravano, 2000, Craven et al., 1998, Etzioni et al., 2004, Reiss et al., 2008]), with pattern learning based on the principle of relation-pattern duality [Brin, 1998]. Open IE [Banko et al., 2007, Mausam, 2016, Stanovsky et al., 2018] uses linguistic cues to jointly infer patterns and triples, but lacks proper normalization of SPO arguments. RE with pre-specified relations, on the other hand, is usually based on distant supervision via pre-compiled seed triples [Mintz et al., 2009, Suchanek et al., 2009]. A variety of methods have been developed on this paradigm, from probabilistic graphical models (e.g., [Pujara et al., 2015, Sa et al., 2017]) to deep neural networks (e.g., [Soares et al., 2019, Wang et al., 2020, Yao et al., 2019, Zhang et al., 2017, Han et al., 2020]). Distantly supervised neural learning has become the method of choice, with different granularities.
Sentence-level RE: Most neural methods operate on a per-sentence level. Distantsupervision samples of SPO triples serve to identify sentences that contain an entity pair (S and O) which stand in a certain relation. The sentence is then treated as a positive training sample for the neural learner. At test-time, the trained model can tag entity mentions and predict if the sentence expresses a given relation or not. This basic architecture has been advanced with bi-LSTMs, attention mechanisms and other techniques (e.g., [Zhang et al., 2017, Cui et al., 2018, Trisedya et al., 2019]). A widely used benchmark for sentence-level RE is TacRed [Zhang et al., 2017].
With recent advances on pre-trained language models like BERT [Devlin et al., 2019a] (or ElMo, GPT-3, T-5 and the like), the currently best RE methods leverage this asset for representation learning [Shi and Lin, 2019, Soares et al., 2019, Wadden et al., 2019, Yu et al., 2020].
Document-level RE: To expand the scope of inputs, Wang et al. [2019] proposed RE from documents, introducing the DocRed benchmark. However, the notion of documents is still very limited in size, given the restrictions in neural network inputs, typically around 10 sentences (e.g., excerpts from Wikipedia articles). Wang et al. [2020] is a state-of-theart method for this document-level RE task, utilizing BERT and graph convolutions for representation learning. Zhou et al. [2021] further enhanced this approach. None of these methods can handle input documents that are larger than a few tens of sentences. KnowFi is the first method that is geared for book-length input.
Fiction Knowledge Bases: Unterstanding characters in literary texts and constructing networks of their relationships and interactions has become a small topic in NLP (e.g., [Chaturvedi et al., 2016, Labatut and Bost, 2019, Srivastava et al., 2016]). The work of [Chu et al., 2019, 2020] has advanced this theme for entity typing and type taxonomies for
fictional domains. However, this work does not address learning relations between entities for KB population.
The DBkWik project [Hertling and Paulheim, 2020] has leveraged structured infoboxes of fan communities at wikia (now renamed to fandom.com), to construct a large KB of fictional characters and their salient properties. However, this is strictly limited to relations and respective instances that are present in infoboxes. Our work leverages wikia infoboxes for distant supervision, but our method can extract more knowledge from a variety of text sources, including storylines and synopses by fans and, most demandingly, the full text of entire books.

heading: 3. System Overview
text: The architecture of the KnowFi system is illustrated in Figure 1. There are two major components:
• Distant supervision involves pre-processing infoboxes from Wikia-hosted fan communities, to obtain seed pairs of entities. These are used to retrieve relevant passages from the underlying text corpora: either synopses of storylines in Wikia or full-fledged content of original books. As the number of passages per entity pair can be very large in books, we devise a judicious ranking of passages and feed only the top-k passages into the next stage of training the neural network. Details are in Section 4.
• Multi-context neural learning feeds the top-k passages, with entity markup, jointly into a BERT-based encoder [Devlin et al., 2019b]. On top of this representation learning, a multi-label classifier predicts the relations that hold for the input entity pair. Details are in Section 5.
Note that a passage can vary from a single sentence to a long paragraph. The two seed entities would ideally occur in the same sentence, but there are many cases where they are one or two sentences apart. Figure 2 shows example texts from a GoT synopses in Wikia and from one of the original books.
The pre-processing of Wikia infoboxes resulted in 2.37M SPO triples for ca. 8,000 different relation names between a total of 461.4k entities, obtained from 142 domains (movie/TV series, games etc.). This forms our background knowledge for distant supervision. For obtaining matching passages, we focused on the 64 most frequent relations, including friend, ally, enemy and family relationships. Note that this stage is not domain-specific. Later we apply the learned model to specific domains such as GoT or Marvel Comics.

heading: 4. Distant Supervision with Passage Ranking
text: The KnowFi approach to distant supervision differs from prior works in two ways:
• Passage ranking: Identifying the best passages that contain seed triples, by judicious ranking, and using only the top-k passages as positive training samples.
• Passages with gaps: Including passages where the entities of a seed triple merely occur in separate sentences with other sentences in between.
Passage ranking: Seed pairs of entities are matched by many sentences or passages in the input corpora. For example, the pair (Herminone, Harry) appears in 1539 sentences in the the seven volumes of the Harry Potter series together. Many of these contain cues that they stand in the friends relation, but there are also many sentences where the co-occurrence is merely accidental. This is a standard dilemma in distant supervision for multi-instance learning [Riedel et al., 2010, Li et al., 2020]. Our approach is to identify the best passages among the numerous matches, by judicious ranking on a per-relation basis.
For each relation, we build a prototype representation by selecting sentences that contain lexical matches of all three SPO arguments, where the predicate is matched by its label in the background knowledge or a short list (average length of three) of synonyms and close hyponyms or hypernyms (e.g., “allegiance” or “loyalty” matching ally), manually collected from WordNet. Newly seen passages for entity pairs can then be scored against the per-relation prototypes by casting both into tf-idf-weighted bag-of-word models (or alternatively, word2vec-style embeddings) and computing their cosine distance. This way, we rank candidate passage for each seed pair and target relation.
Passages with gaps: Unlike encyclopedic articles, long texts on fictional domains have a narrative style where single sentences are unlikely to give the full information in the most compact way. Therefore, we consider multi-sentence contexts where entity mentions across different sentences. In addition to simple paragraphs, we consider passages with gaps where we include sentences that are not necessarily contiguous but leave out uninformative sentences. This way, we maximize the value of limited-size text spans fed into the neural
learner. This is in contrast to earlier techniques that consume whole paragraphs and rely on attention mechanism for giving higher weight to informative parts.
KnowFi has two configuration parameters: the maximum number of sentences allowed between sentences that contain seed entities, and the number of sentences directly preceding or following the occurrence of a seed entity. In our experiments, we include text where the two entities appear at most 2 sentences apart and 1 preceding and 1 following sentence for each of the entity mentions, up to 512 tokens which is the current limit of BERT-based networks.
Negative training samples: In addition to the positive training samples by the above procedure, we generate negative samples by the following random process. For each relation r, we pick random entities e1 and e2 for each of the S and O roles such that there are other entities x and y for which the background knowledge asserts (e1, r, x) and (y, r, e2) with x 6= e2 and y 6= e1. This improves on the standard technique of simply choosing any pair e1, e2 for which (e1, r, e2) does not hold, by selecting more difficult cases and thus strengthening the learner. For example, both Herminone and Malfoy have some friends, but they are not friends of each other. The training of KnowFi uses a 1:1 ratio of positive to negative samples.

heading: 5. Multi-Context Neural Extraction
text: KnowFi is trained with and applicable to multiple passages as input to an end-to-end Transformer-based network with full backpropagation of cross-entropy loss. Our neural architecture has two specific components: a per-passage layer to learn BERT-based representations for each passage, and an aggregation layer that combines the signals from all input passages. In the experiments in this paper, the aggregation layer is configured to concatenate the representations of all passages, but other options are feasible, too.
Each input passage is encoded with markup of entity mentions. In addition, we determine semantic types for the entities, using the SpaCy tool (https://spacy.io/) that provides one type for each mention, chosen from a set of 18 coarse-grained types (person,
nationality/religion, event, etc.). The type of each entity mention in a passage is appended to the input vector. Figure 3 illustrates the neural network for multi-context RE.

heading: 6. LoFiDo Benchmark
text: To evaluate RE from long documents, we introduce the LoFiDo corpus (Long Fiction Documents). We compile SPO triples from infoboxes of 142 Wikia fan communities. After cleaning extractions and clustering synonyms, we obtain a total of 64 relations such as enemy, friend, ally, religion, weapon, ruler-of, etc.
For evaluating KnowFi and various baselines, we focus on 5 especially rich and diverse domains (i) Lord of the Rings (a series of three epic novels by J.R.R Tolkien), (ii) A Song of Ice and Fire (a series of five fantasy novels by George R.R. Martin, well-known for the Game of Thrones TV series based on it), (iii) Harry Potter (a series of seven books, written by J.K Rowling), (iv) Dune (a science-fiction novel by Frank Herbert), and (v) War and Peace (a classic novel by Leo Tolstoy). For the first four, Wikia infoboxes provide ground truth; for War and Peace, we manually crafted a small ground-truth KB. 20% of the triples from each of these universes are withheld for testing. For the first four domains, we consider both original novels as well as narrative synopses from Wikia as input sources. War and Peace is not covered by Wikia.
LoFiDo Statistics. Our LoFiDo corpus contains 81,025 instances for training and 20,257 instances for validation. For testing, we use five specific universes, which take input from both books and Wikia texts. The total number of instances in the test data from Wikia texts is 14,610, while in the case of books, it is 64,120. Ground-truth data for five test universes are provided for evaluation. Table 1 shows statistics on the training and validation data, while Table 2 shows statistics on the ground-truth of five domains in the test data. Further details on this dataset are in Appendix A and B. Code and data of KnowFi are available at https://www.mpi-inf.mpg.de/yago-naga/knowfi.

heading: 7. Experiments

heading: 7.1 Setup
text: Baselines. We compare KnowFi to three state-of-the-art baselines on RE:
• BERT-Type [Shi and Lin, 2019] which uses BERT-based encodings augmented with entity type information, also based on SpaCy output in our experiments for fair comparison.
• BERT-EM [Soares et al., 2019] which include entity markers in input sequences; • GLRE [Wang et al., 2020] which additionally computes global entity representations
and uses them to augment the text sequence encodings.
The first two baselines run on a per-sentence basis, whereas GLRE is a state-of-the art method for extractions from short documents, which we train on paragraph-level inputs. The inputs for these models (i.e. sentences or paragraphs) are randomly selected.
KnowFi Parameters. For passage selection, we rely on TF-IDF-based bag-of-words similarity between the passages and relation contexts, where each relation context contains the top-100 tokens, based on their tfidf scores. For selecting passages as multi-context input, we compute the cosine between tf-idf-based vectors of each passage against the relation-specific prototype vector; we select all passages with cosine above 0.5 as positive training samples. For the neural network, we use BERTLARGE (https://huggingface.co/transformers/model_ doc/bert.html) with 24 layers, 1024 hidden size and 16 heads. The learning rate is 5e − 5 with Adam, the batch size is 8, and the number of training epochs is 10.
Evaluation Metrics. The evaluation uses standard metrics like precision, recall and F1, averaged over all extracted triples. We report micro-averaged numbers for all relations together, and drill down on selected relations of interest. In addition, we report numbers for HITS@k and MRR. As ground-truth, we perform two different modes of evaluation:
• Automated evaluation is based on ground-truth from Wikia infoboxes. This is demanding on precision, but penalizes recall because of its limited coverage.
• Manual evaluation is based on obtaining assessments of extracted triples via crowdsourcing. This way, we include correct triples that are not in Wikia infoboxes, and thus achieve higher recall.

heading: 7.2 Results
text: Automated Evaluation. Table 3 shows average precision, recall and F1 score. We can see that sentence-level baselines achieve comparatively high coverage, due to considering every sentence. Yet their precision is extremely low. GLRE and KnowFi achieve much higher precision, though GLRE fails to achieve competitive recall, presumably because its training on all paragraphs lowers its predictive power. As an illustration, GLRE produces only 173 assertions from all Harry Potter books, while KnowFi produces 600.
We also observe that for all methods, extraction from books is considerably harder than from the more concise synopses in Wikia.
In addition to the P/R/F1 scores, in Table 4 we also take an entity-centric view and evaluate how well correct extractions rank. The HITs@k metric reports how often a correct result appears among the top extractions per entity-relation pair (e.g., among top-5
extracted enemies of Harry Potter), while MRR reports the mean reciprocal rank of the first extraction. We can observe that KnowFi outperforms all baselines on both metrics.
Manual Evaluation. The low absolute scores in the above evaluation largely stem from incomplete automated ground truth. We therefore conducted an additional manual evaluation. For each domain, we select top 100 extractions from the results and used crowdsourcing to manually label their correctness. The annotators were Amazon master workers with all time approval rate > 90%, and additional test questions were used to filter responses. We observed high inter-annotator agreement, on average of 0.81.
Table 5 shows results of our manual evaluation on four domains (Dune was left out due to complexity). As one can see, KnowFi outperforms the baselines on most input texts, and achieves a remarkable precision on both books and wikia texts (average of 0.57 on books and 0.73 on wikia texts).
We repeat the entity-centric evaluation with manual labels for three relations of special interest in fiction, friend, enemy and ally. We select 10 popular entities each from LoTR, GoT and Harry Potter. The resulting precision scores are shown in Table 6. As one can see, KnowFi is achieves high precision among its top extractions, e.g., 78% and 73% precision at rank 1 for friend assertions from books/Wikia texts.
Evaluation on Short-Text Datasets. To evaluate the robustness of KnowFi, we also evaluate its performance on the existing sentence-level RE dataset TACRED, and the short document-level RE dataset DocRED. The results are shown in Tbl. 7. We find KnowFi’s
performance on TACRED is on par with BERT-Type and BERT-EM (0.66 test-F1, versus 0.63 and 0.62 for the baselines), the modest gain indicating that the combination of entity types and markers is beneficial. On DocRED, KnowFi achieved 0.51 F1-score, slightly below the GLRE model at 0.57 F1-score. We hypothesize that the modest losses stem from the fact that GLRE is specifically tailored for the short documents of TACRED, where multicontext aggregation is not relevant. At the same time, the single contexts GLRE considers have no inherent size limitation, unlike the 2-sentence distance threshold used in KnowFi.
Ablation Study. To evaluate the impact of passage ranking, we ran KnowFi without passage ranking for both training and prediction. Instead, passages were randomly selected. In automated evaluation, without passage ranking, KnowFi achieves comparable recall but lower precision: 0.07 vs. 0.14 on books and 0.12 vs. 0.17 on Wikia texts. This pattern is also observed in manual evaluation, where KnowFi, without passage ranking, achieves a precision of 0.43 vs. 0.57 on books and 0.55 vs. 0.73 on Wikia texts. Further experiments and ablation studies are in Appendix C and D.
Error Analysis. The precision gain from automated to manual evaluation (Table 3 vs. Table 5) indicates that ground-truth incompleteness is a confounding factor. By inspecting a sample of 50 false positives, we found that 20% originated from incomplete ground truth, while 54% were indeed not inferrable from the given contexts, pointing to limitations of the distant-supervision-based training (e.g., extracting friendship from the sentence “Thorin came to Bilbo’s door”). Another 15% were errors in determining the subject or object in complex sentences with many entity mentions, e.g., extracting friend(Boromir, Gimli) from the sentence “Boromir went ahead, Legolas and Gimli, who by now had become friends, followed.” Finally, 7% of the false positives captured semantically related relations but missed the correct ones.
By sampling false negatives, we found that in 52% of the cases the retrieved contexts did not allow the proper inference, indicating limitations in the context retrieval and ranking. In 33% of the cases, a human reader could spot the relation in the top-ranked contexts (e.g., hasCulture (Legolas, Elf) in “He saw Legolas seated with three other Elves”). Appendix E shows some anecdotal examples for the output of KnowFi and baselines.

heading: 8. Extrinsic Use Case: Entity Summarization
text: To assess the salience in the extractions produced by KnowFi, we pursued a user study to compare entity summaries, one from KnowFi and one by randomly drawing from one of the three baselines. Each entity summary includes at most 5 best extractions (distinct relations) from the book series Lord of the Rings, Game of Thrones and Harry Potter. For each domain, we generate summaries for 5 popular entities. We give pairs of summaries, with randomized order, to Amazon master workers for selecting the more informative one. Table 8 shows an example of this crowdsourcing task. The annotators preferred KnowFibased summaries over BERT-Type, BERT-EM and GLRE in 93%, 64% and 81% of the cases, respectively.

heading: 9. Conclusion
text: To the best of our knowledge, this paper is the first attempt at relation extraction (RE) from long fictional texts, such as entire books. The presented method, KnowFi, is specifically geared for this task by its judicious selection and ranking of passages. KnowFi outperforms strong baselines on RE by a substantial margin, and it performs competitively even on the short-text benchmarks TacRed and DocRed. The absolute numbers for precision and recall show that there is still a lot of room for improvement. This underlines our hypothesis that long fictional texts are a great challenge for RE. Our LoFiDo corpus of Wikia texts, book contents, and ground-truth labels will be made available to foster further research. All information can be found at https://www.mpi-inf.mpg.de/yago-naga/knowfi.

heading: Acknowledgments
text: We thank Matteo Cannaviccio and Filipe Mesquita from Diffbot Inc. for their advice and support on labeling data, and Diffbot Inc. for free access to their RE API. We also thank our anonymous reviewers for their valuable comments.

heading: Appendix A. Training Data Extraction
text: Many current KBs like Yago, DBpedia or Freebase have been built by extracting the information from infoboxes, category network and leveraging the markup language of Wikipedia. The relations of these KBs are then used as schema for many later supervised relation extractors. However, for fiction, Wikipedia has too low coverage of entities and relevant relations.
Wikia. Wikia (or Fandom) is the largest web platform for fiction and fantasy. It contains over 385k communities with total of over 50M pages. Each community (usually discusses about one fictional universe) is organized as a single Wiki. With a wide range of coverage on fiction and fantasy, Wikia is one of the 10 most visited websites in the US (as of 2020)1.
Crawling. We download all universes which contain over 1000 content pages and have available dump files from Wikia, and get total of 142 universes in the end. From these universes, we extract all information from their category networks and infoboxes, and build a background knowledge base for each universe.
Definition A.1. Background KB of an universe is a collection of entities, entity mentions, simple facts that describe relations between entities and a type system of the universe.
Background knowledge extraction. To extract the background KBs, we follow a simple procedure:
• Type system construction: The type system is extracted from Wikia category network. We adapt the technique from the TiFi system [Chu et al., 2019] to structure and clean the type system.
• Entity detection: Entities and entity mentions can be easily extracted from the dump file. We consider page titles as the entities in the universe (except administration and category pages). On the other hand, entity mentions only appear in texts. By using Wiki markup, each mention can be extracted and linked to the entity with a confident score which is computed based on its frequency.
• Infobox extraction: Facts about each entity are extracted from its infobox. Infobox is presented in table format with the entity’s attributes and their values. Each extracted fact is presented in a triple with subject, predicate (relations) and object. In particular, we consider the main entity as subject, the attributes as predicates, and the values as objects. We manually check if there is any misspelling in the relations and merge them if necessary.
This results an average of 158k entities and 13.5k facts in each universe. The information from these background KBs is then used for all three later steps.
Relation Filtering. After extracting the background KBs, we get all relations from the facts of all universes and consider them as relation candidates that can be extracted in fictional domains. However, beside meta relations which are not really related to the content of universes, such as season, page, episode,..., there is much noise in the relations since they are manually created by fans. To remove noise and keep popular relations, we do relation filtering as follows:
1. https://ahrefs.com/blog/most-visited-websites/
• Pre-processing: a combination of stemming and keeping relations with length at least 3 (except for some relations like job, age, son, etc.).
• Infrequent-relation removing: we only keep relations which are in at least 5 universes and appear in over 20 facts.
• Meta-relation removing: we manually check if the relation is a meta-relation. In total, there are 247 relations considered as meta-relations.
• Misspelling detection: Misspelling relations are manually detected and grouped with the correct relations, for example, affilation and affiliation.
• Grouping: Synonym relations are manually grouped together, for example, leader and commander.
After relation filtering, we reduce the number of relations from over 8,000 to 64 relations. These relations are considered as popular relations in fictional domains and used as targets for the relation extraction step. We realize that in fictional domains, the relations expressing the friendly or hostile relationship between two entities are interesting, hence, we keep friend and enemy as two relations which are always extracted. Figure 4 shows statistics on training data. We publish the training data as supplementary material.

heading: Appendix B. Background KB Statistics
text: One of our contribution is the background KB dataset on popular universes in fictional domains. To have an overview about the dataset, Table 9 shows some statistics on our background KBs database, which include information about universes, entities, type systems, relations and facts.
From the 5 domains used for testing, the number of relations varies from 10 to 20, and the number of ground-truth triples varies from 1,100 to 4,700 for the first three domains, and was between 100 and 200 the last two.

heading: Appendix C. Additional Experiments
text: Similarity Threshold. In our experiments, we consider all passages with cosine above 0.5 as positive training samples (section 7.1). To assess the effect of the similarity threshold, we conduct an ablation study on it. Table 10 reports the automated results of KnowFi on both books and Wikia texts, where the threshold varies. For the author response we completed two other runs (threshold 0.4 and 0.6) that indicate modest influence, for the final version we would provide insights for all threshold values from 0 to 1 (in 0.1 step size). The results show that, with a similarity threshold around 0.5, the model achieves the best F1. By increasing the similarity threshold, the model is able to achieve higher precision but lower recall and vice versa.
Embedding-based Passage Ranking. KnowFi uses a simple TF-IDF-based schema for passage ranking. To assess the effectiveness of this method, we conduct an ablation study on the ranking step. Instead of using TF-IDF, we compute the embeddings of the passages and the relation contexts using Sentence-BERT [Reimers and Gurevych, 2019]. The cosine similarity between the passage embedding and the relation context embedding is then computed using the sklearn library. We select all passages with cosine above 0.0 (range [-1,1]), as positive training samples, with maximum of 5 passages per each training instance. Table 11 shows the automated results of KnowFi on both books and Wikia texts. The higher scores on recall shows that the embeddings can help the model capture the semantic relationships between passages and relation contexts better, especially when handling the cases of synonymy, while TF-IDF only handles the cases of lexical matching. However, in general, both techniques are on par, and embeddings do not improve the results, in terms of F1-score.
GLRE with Passage Ranking. In our experimental setup, the inputs of GLRE [Wang et al., 2020] (for both train and test) are randomly selected. To assess the effect of passage ranking on GLRE, we conduct a study where the inputs of GLRE are selected by using our method for passage ranking. Table 12 shows that, by using passage ranking to filter the
inputs, GLRE is able to achieve higher precision and recall, compared to GLRE without passage ranking. However, this enhanced variant is still inferior to KnowFi by a substantial margin.

heading: Appendix D. Impact of Training Data Quality
text: Training data is one of the most important factors that impact the quality of the supervised models, therefore, it is essential to maintain the quality of the training data, especially when working on specific domains where the training data is usually not available. To evaluate the quality of our training data, we compare KnowFi and a variant (i.e., without using passage ranking on training data collection) with two other methods which are trained using manual training datasets:
• TACRED [Zhang et al., 2017], a popular dataset for relation extraction on the sentence level. We train our relation extraction model using TACRED and use the model to extract the relations from the test data.
• Diffbot [Mesquita et al., 2019], a commercial API for relation extractions. We run the Diffbot API on our test data to extract the relations.
Note that these comparisons are not systematic, as there are confounding other differences between these two methods and KnowFi. In particular, the comparison with TACRED also comes with a reduction to sentence-level training examples only, while the Diffbot API is a pre-trained general-purpose extractor with necessary limitations in more specific use cases. We automatically evaluate the extractions on three popular relations, spouse, sibling, child, since they are contained in all datasets.
Results. Table 13 reports the results on two universes, Lord of the Rings and Game of Thrones. The results show that, our training data achieves comparable results with other datasets and even higher F1-scores, in both books and Wikia texts.

heading: Appendix E. Anecdotal Examples
text: Table 14 gives examples for the output of the various methods on sample contexts. The red color texts denote subjects and the blue color texts denote objects.

Review summaries:
This paper proposed a novel benchmark for relation extraction from very long documents and a new method -- KnowFi to deal with the problem. The dataset is constructed from Wikia containing long fictional texts. The method, KnowFi, generally contains two modules -- passage ranking and multi-context neural extraction. They first rank the source passages via a similarity function and then feed the top-$k$ passages to a BERT-based multi-context neural extraction module to get its relation. Experiments showed that KnowFi achieved strong performance on this new benchmark and two conventional short-text benchmarks.

The paper focuses on the relation extraction task from long fictional texts (i.e., book-level RE), which is an important step towards KB construction from books. The authors present a neural architecture to address the challenges such as sparsity, distant-supervision, size limitation, etc. Although the proposed method is not particularly novel, the paper presents a nice first work in this direction. The authors also provide a new benchmark LoFiDo for evaluating different methods, which is also a great contribution.