KnowFi: Knowledge Extraction from Long Fictional Texts

1 . Introduction . Motivation and Problem : Relation extraction ( RE ) from web contents is a key task for the automatic construction of knowledge bases ( KB ) . It involves detecting a pair of entities in a text document and inferring if a certain relation ( predicate ) holds between them . Extracted triples of the form ( subject , predicate , object ) are used for populating and growing the KB . Besides this major use case , RE also serves other applications like text annotation and summarization , semantic search , and more . Work on KB construction has mostly focused on general-purpose encyclopedic knowledge , about prominent people , places , products etc . and basic relations of wide interest such as birthplaces , spouses , writing of books , acting in movies etc . Vertical domains have received some attention , including health , food , and consumer products . Yet another case are KBs about fictional works [ Hertling and Paulheim , 2020 , Labatut and Bost , 2019 ] , such as Game of Thrones ( GoT ) , the Marvel Comics ( MC ) universe , Greek Mythology or epic books such as War and Peace by Leo Tolstoy or the Cartel novels by Don Winslow . For KBs about fictional domains , the focus is less on basic relations like birthplaces or spouses , but more on relations that capture traits of characters and key elements of the narration . Relations of interest are allies , enemies , membership in clans , betrayed , killed etc . Applications of fiction KBs foremost including supporting fans in entity-centric search . Some of the fictional domains have huge fan communities , and search engines frequently receive queries such as “ Who killed Catelyn Stark ? ” ( in GoT ) . Entity summarization is a related task , for example , a user asking for the most salient traits of Ygritte ( in GoT ) . Although fiction serves to entertain , some of the more complex domains reflect sub-cultural trends and the zeitgeist of certain epochs . Analyzing their narrative structures and networks of entities is of interest to humanities scholars . For example , superhero comics originated in the 1940s and boomed in post-war years , reflecting that era ’ s zeitgeist ( revived now ) . War and Peace has the backdrop of the Napoleonic wars in Russia , and the Cartel trilogy blends facts and fiction about drug trafficking . KBs enable deeper analyses of such complex texts for historians , social scientists , media psychologists and cultural-studies scholars . State of the Art and its Limitations : RE with pre-specified relations for canonicalized entities is based on distant supervision via pre-compiled seed triples [ Mintz et al. , 2009 , Suchanek et al. , 2009 ] . Typically , these training seeds come from initial KBs , which in turn draw on Wikipedia infoboxes . The best RE methods are based on this paradigm of distant supervision , leveraging it for neural learning ( e.g. , [ Soares et al. , 2019 , Wang et al. , 2020 , Yao et al. , 2019 , Zhang et al. , 2017 , Han et al. , 2020 ] ) . They work well for basic relations , as there is no shortage of training samples ( e.g. , for birthplace or spouse ) . One of their key limitations is the bounded size of input text passages , typically a few hundred tokens only . This is not a bottleneck for basic relations where single sentences ( or short paragraphs ) with all three SPO components are frequent enough ( e.g. , in the full text of Wikipedia articles ) . However , for RE with non-standard relations over long fictional texts such as entire books , these limitations are major bottlenecks , if not showstoppers . This paper addresses the resulting challenges ( also included among the open challenges in the overview by [ Han et al. , 2020 ] ) : • How to go about distant supervision for RE targeting non-standard relations that have only few seed triples ? • How to cope with very long input texts , such as entire books , where relevant cues for RE are spread across passages ? Approach and Contributions : This paper presents a complete methodology and system for relation extraction from long fictional texts , called KnowFi ( Knowledge extraction from Fictional texts ) . Our method leverages semi-structured content in wikis of fan communities on fandom.com ( aka wikia.com ) . We extract an initial KB of background knowledge for 142 popular domains ( TV series , movies , games ) . This serves to identify interesting relations and to collect distant supervision samples . Yet for many relations this results in very few seeds . To overcome this sparseness challenge and to generalize the training across the wide variety of relations , we devise a similarity-based ranking technique for matching seeds in text passages . Given a long input text , KnowFi judiciously selects a number of context passages containing seed pairs of entities . To infer if a certain relation holds between two entities , KnowFi ’ s neural network is trained jointly for all relations as a multi-label classifier . Extensive experiments with long books on five different fictional domains show that KnowFi clearly outperforms state-of-the-art RE methods . Even on conventional short-text benchmarks with standard relations , KnowFi is competitive with the best baselines . As an extrinsic use case , we demonstrate the value of KnowFi ’ s KB for the task of entity summarization . The paper ’ s novel contributions are : • a system architecture for the new problem of relation extraction from long fictional texts , like entire novels and text contents by fan communities ( Section 3 ) ; • a method to overcome the challenge of sparse samples for distant supervision for nonstandard relations ( Section 4 ) . • a method to overcome the challenge of limited input size for neural learners , by judiciously selecting relevant contexts and aggregating results ( Section 5 ) ; • a comprehensive experimental evaluation with a novel benchmark for relation extraction from very long documents ( Section 6 ) , with code and data release upon publication . 2 . Related Work . Relation Extraction ( RE ) : Early work on RE from text sources has used rules and patterns , ( e.g. , [ Agichtein and Gravano , 2000 , Craven et al. , 1998 , Etzioni et al. , 2004 , Reiss et al. , 2008 ] ) , with pattern learning based on the principle of relation-pattern duality [ Brin , 1998 ] . Open IE [ Banko et al. , 2007 , Mausam , 2016 , Stanovsky et al. , 2018 ] uses linguistic cues to jointly infer patterns and triples , but lacks proper normalization of SPO arguments . RE with pre-specified relations , on the other hand , is usually based on distant supervision via pre-compiled seed triples [ Mintz et al. , 2009 , Suchanek et al. , 2009 ] . A variety of methods have been developed on this paradigm , from probabilistic graphical models ( e.g. , [ Pujara et al. , 2015 , Sa et al. , 2017 ] ) to deep neural networks ( e.g. , [ Soares et al. , 2019 , Wang et al. , 2020 , Yao et al. , 2019 , Zhang et al. , 2017 , Han et al. , 2020 ] ) . Distantly supervised neural learning has become the method of choice , with different granularities . Sentence-level RE : Most neural methods operate on a per-sentence level . Distantsupervision samples of SPO triples serve to identify sentences that contain an entity pair ( S and O ) which stand in a certain relation . The sentence is then treated as a positive training sample for the neural learner . At test-time , the trained model can tag entity mentions and predict if the sentence expresses a given relation or not . This basic architecture has been advanced with bi-LSTMs , attention mechanisms and other techniques ( e.g. , [ Zhang et al. , 2017 , Cui et al. , 2018 , Trisedya et al. , 2019 ] ) . A widely used benchmark for sentence-level RE is TacRed [ Zhang et al. , 2017 ] . With recent advances on pre-trained language models like BERT [ Devlin et al. , 2019a ] ( or ElMo , GPT-3 , T-5 and the like ) , the currently best RE methods leverage this asset for representation learning [ Shi and Lin , 2019 , Soares et al. , 2019 , Wadden et al. , 2019 , Yu et al. , 2020 ] . Document-level RE : To expand the scope of inputs , Wang et al . [ 2019 ] proposed RE from documents , introducing the DocRed benchmark . However , the notion of documents is still very limited in size , given the restrictions in neural network inputs , typically around 10 sentences ( e.g. , excerpts from Wikipedia articles ) . Wang et al . [ 2020 ] is a state-of-theart method for this document-level RE task , utilizing BERT and graph convolutions for representation learning . Zhou et al . [ 2021 ] further enhanced this approach . None of these methods can handle input documents that are larger than a few tens of sentences . KnowFi is the first method that is geared for book-length input . Fiction Knowledge Bases : Unterstanding characters in literary texts and constructing networks of their relationships and interactions has become a small topic in NLP ( e.g. , [ Chaturvedi et al. , 2016 , Labatut and Bost , 2019 , Srivastava et al. , 2016 ] ) . The work of [ Chu et al. , 2019 , 2020 ] has advanced this theme for entity typing and type taxonomies for fictional domains . However , this work does not address learning relations between entities for KB population . The DBkWik project [ Hertling and Paulheim , 2020 ] has leveraged structured infoboxes of fan communities at wikia ( now renamed to fandom.com ) , to construct a large KB of fictional characters and their salient properties . However , this is strictly limited to relations and respective instances that are present in infoboxes . Our work leverages wikia infoboxes for distant supervision , but our method can extract more knowledge from a variety of text sources , including storylines and synopses by fans and , most demandingly , the full text of entire books . 3 . System Overview . The architecture of the KnowFi system is illustrated in Figure 1 . There are two major components : • Distant supervision involves pre-processing infoboxes from Wikia-hosted fan communities , to obtain seed pairs of entities . These are used to retrieve relevant passages from the underlying text corpora : either synopses of storylines in Wikia or full-fledged content of original books . As the number of passages per entity pair can be very large in books , we devise a judicious ranking of passages and feed only the top-k passages into the next stage of training the neural network . Details are in Section 4 . • Multi-context neural learning feeds the top-k passages , with entity markup , jointly into a BERT-based encoder [ Devlin et al. , 2019b ] . On top of this representation learning , a multi-label classifier predicts the relations that hold for the input entity pair . Details are in Section 5 . Note that a passage can vary from a single sentence to a long paragraph . The two seed entities would ideally occur in the same sentence , but there are many cases where they are one or two sentences apart . Figure 2 shows example texts from a GoT synopses in Wikia and from one of the original books . The pre-processing of Wikia infoboxes resulted in 2.37M SPO triples for ca . 8,000 different relation names between a total of 461.4k entities , obtained from 142 domains ( movie/TV series , games etc. ) . This forms our background knowledge for distant supervision . For obtaining matching passages , we focused on the 64 most frequent relations , including friend , ally , enemy and family relationships . Note that this stage is not domain-specific . Later we apply the learned model to specific domains such as GoT or Marvel Comics .

Summary: This paper proposed a novel benchmark for relation extraction from very long documents and a new method -- KnowFi to deal with the problem. The dataset is constructed from Wikia containing long fictional texts. The method, KnowFi, generally contains two modules -- passage ranking and multi-context neural extraction. They first rank the source passages via a similarity function and then feed the top-$k$ passages to a BERT-based multi-context neural extraction module to get its relation. Experiments showed that KnowFi achieved strong performance on this new benchmark and two conventional short-text benchmarks.