Abstractified Multi-instance Learning (AMIL) for Biomedical Relation Extraction

data and a long-tail distribution of fact triples . Many works leverage distant supervision which automatically generates labeled data by pairing a knowledge graph with raw textual data . Distant supervision produces noisy labels and requires additional techniques , such as multi-instance learning ( MIL ) , to denoise the training signal . However , MIL requires multiple instances of data and struggles with very long-tail datasets such as those found in the biomedical domain . In this work , we propose a novel reformulation of MIL for biomedical relation extraction that abstractifies biomedical entities into their corresponding semantic types . By grouping entities by types , we are better able to take advantage of the benefits of MIL and further denoise the training signal . We show this reformulation , which we refer to as abstractified multi-instance learning ( AMIL ) , improves performance in biomedical relationship extraction . We also propose a novel relationship embedding architecture that further improves model performance . 1 . Introduction . Relation extraction ( RE ) is a key facet of information extraction in large bodies of unstructured textual data . RE is particularly important in the biomedical domain where extracting relationships between pairs of biomedical entities , also known as “ fact triples ” , can produce new insights into complicated biological interactions . For instance , with the near-exponential growth of microbiome research [ Sa ’ ed et al. , 2019 ] , advanced RE methods may help discover important links between gut microbiota and diseases . It is in this context that we motivate our work . RE within the biomedical domain comes with two inherent challenges : there are more than 30 million scientific articles , with hundreds of thousands of articles published every year , and there is a corresponding lack of labeled data . To resolve these challenges , many have leveraged distant supervision techniques which pair knowledge graphs with raw textual data to automatically generate labels to train deep-learning models [ Gu et al. , 2019 , Su et al. , 2019 , Junge and Jensen , 2019 ] . We seek to improve distantly supervised biomedical RE methods in this work . We use the Unified Medical Language System ( UMLS ) Metathesaurus [ Bodenreider , 2004 ] for our knowledge graph and pair it with raw textual data from PubMed [ Canese and Weis , 2013 ] . To automatically generate labels , distantly supervised RE methods rely on a simple yet powerful assumption : any singular sentence that contains a pair of entities also expresses a relationship , as determined by the accompanying knowledge graph , between those entities [ Mintz et al. , 2009 ] . However , this assumption leads to a noisy training signal with many false positives as not all sentences express a relationship between an entity pair . To combat this , many works have leveraged multiinstance learning ( MIL ) [ Riedel et al. , 2010 , Hoffmann et al. , 2011 , Zeng et al. , 2015 ] where , instead of assessing single sentences , MIL assesses positive and negative bags of sentences that contain the same entity pair . Grouping sentences into bags greatly reduces noise in the training signal since a bag of sentences is more likely to express a relationship than a single sentence . This enables the model to better classify relationships between unseen entity pairs . However , similar to many NLP tasks , biomedical RE suffers from a long-tail distribution of fact triples , where many entity pairs are only supported by a few sentences of evidence . After processing the PubMed corpus , we observe that a majority ( ∼ 52 % ) of extracted triples are supported by fewer than three sentences . Creating bags of sentences for such entity pairs requires heavy up-sampling . For example , if a pair of entities is only supported by one sentence and a bag size is equal to 16 sentences , the single sentence is duplicated 15 times to fill the bag . This erases the benefit of MIL . To counter this issue , we introduce abstractified multi-instance learning ( AMIL ) where , instead of grouping entity pairs by name , we group entities by the corresponding semantic type as determined by UMLS . UMLS categorizes each entity with a semantic type within the UMLS semantic network . The UMLS semantic network is curated by human experts for decades and provides a rich ontology of biomedical concepts which we leverage to group multiple different entity pairs within a single MIL bag , reducing the need to up-sample sentences . For example , consider two sentences : ( 1 ) a sentence containing the entity pair ( fibula , tibia ) and ( 2 ) a second sentence containing the entity pair ( humerus , ulna ) . With distant supervision , we assume each sentence expresses the relationship linking both pairs , namely articulates with . Despite expressing the same relationship , without abstraction , these sentences are placed into separate MIL bags since bags are grouped by distinct entity pairs . By introducing abstractified multi-instance learning , the entities fibula , tibia , humerus , and ulna are grouped by their corresponding UMLS semantic type— “ Body Part , Organ , or Organ Component. ” This allows us to place the aforementioned sentences into the same MIL bag based on their entity type , creating a heterogeneous bag of entity pairs that express the same relationship . With this reformulation , bags containing a single duplicated sentence are reduced by half . AMIL produces better overall performance for biomedical RE with significant performance gains for “ rare ” triples . Here , we define “ rare ” triples as triples that are supported by fewer than eight sentences . These triples make up roughly 80 % of the long-tail distribution of triples . We also take inspiration from Soares et al . ( 2019 ) and conduct a suite of experiments with variations of relationship embedding architectures . Such experiments are underexplored in the biomedical domain and many are novel to the general task of relationship classification . Soares et al . report the best RE performance using a relationship representation consisting of embedded entity start markers—special span tokens that denote the beginning of an entity . We test this RE architecture in the biomedical domain and also test the performance of entity end markers . Moreover , we introduce a novel relationship representation , namely the middle mention pool , which pools word pieces between head and tail entities . This embedding architecture is inspired by the observation that context between two biomedical entities in a sentence often contains the information-rich and relationship-relevant signal . Our best performing relationship embedding architecture results from the combination of both entity end markers and the middle mention pool . We observe that this architecture further increases the performance of our relation classification model . In this paper , we make the following contributions : • We introduce abstractified multiple-instance learning ( AMIL ) , which achieves new state-of-theart performance for biomedical relationship extraction . We also report significant performance gains for rare fact triples . • We propose an improved relationship representation for biomedical relation extraction . We show that concatenating embedding tokens from entity end markers with the middle mention pool produces the best performing model . • We make all our code , saved models , and pre-processing scripts publicly available1 to facilitate future biomedical RE efforts . Pre-processing scripts can impact model performance and are important to prepare an up-to-date , ready-for-RE dataset from ever-growing PubMed and UMLS . Our results in Section 5 show that using updated pre-processing tools can improve model performance by ∼ 10 % . 2 . Related Work . Early works combining distant supervision with relation extraction [ Bunescu and Mooney , 2007 , Craven and Kumlien , 1999 ] relied on the strong assumption claiming that , if a relationship between two entities exists , then all sentences containing those entities express the corresponding relationship . This assumption was relaxed by Riedel et al . ( 2010 ) with the introduction of multi-instance learning ( MIL ) which claimed that if a relationship exists between two entities , at least one sentence that contains the two entities may express the corresponding relation . Hoffmann et al . ( 2011 ) build on the work of Riedel et al . by allowing for overlapping relations . Zeng et al . ( 2015 ) extend distantly supervised RE by combining MIL with a novel piecewise convolutional neural network ( PCNN ) . Lin et al . ( 2016 ) made further improvements by introducing an attention mechanism that attends to relevant information in a bag of sentences . This sentence-level attention mechanism for MIL inspired numerous subsequent works [ Luo et al. , 2017 , Han et al. , 2018a , Alt et al. , 2019 ] . Han et al . ( 2018a ) propose a joint training RE model that combines a knowledge graph with an attention mechanism and MIL . Dai et al . ( 2019 ) extend the work by Han et al . into the biomedical domain and use a PCNN for sentence encoding . Amin et al . ( 2020 ) propose an RE model that uses BioBERT [ Lee et al. , 2019 ] , a pretrained transformer based on bert [ Devlin et al. , 2019 ] , for sentence encoding . They leverage MIL with entity-marking methods following R-BERT [ Wu and He , 2019 ] and achieve the best performance when the directionality of extracted triples is matched to the directionality from the UMLS knowledge graph . Notably , the model proposed by Amin et al . ( 2020 ) does not benefit from sentence-level attention . We choose the model proposed by Amin et al . ( 2020 ) for our baseline model as it achieves the current state-of-the-art ( SOTA ) in biomedical RE . We also conduct a suite of experiments with variations of relationship embedding architectures . These experiments are inspired by Soares et al . ( 2019 ) who conduct experiments with six different embedding architectures and report performance on general-domain RE datasets . They show that constructing a relationship embedding with special entity start markers outperforms other architectures . We build on this work by ( 1 ) conducting similar experiments in the biomedical domain and ( 2 ) by proposing numerous novel architectures for a total of seventeen alternatives for a comprehensive comparison on the biomedical RE task . 3 . Datasets . UMLS Metathesaurus and Semantic Network : The UMLS Metathesaurus and Semantic Network is a knowledge graph of biomedical entities and their corresponding relationships . Following numerous previous works in biomedical relation extraction [ Zhang and Wang , 2015 , Dai et al. , 2019 , Amin et al. , 2020 ] , we only extract fact triples that contain a relationship other than “ synonymous ” , “ narrower ” , or “ broader ” . These general relationships make up the majority of relationships in the UMLS Semantic Network and we exclude them to focus on more substantive relationships . Using this filter , we extract 7,025,733 triples from the 2019AB UMLS release . 1. https : //github.com/IBM/aihn-ucsd/tree/master/amil PubMed : For our textual data , we use abstracts from the 2019 PubMed corpus2 . The corpus contains 34.4M abstracts which we segment into 158,848,048 unique sentences using Sci-Spacy [ Neumann et al. , 2019 ] . 4 . Method . Problem statement : given knowledge graph G and text corpus C , sentences s from C that contain exactly two distinct entities , ( ei1 , e i 2 ) , that are linked via relationship r i as determined by G , are grouped into bags Bi = { si1 , . . . , s i m } based on their corresponding entity types ( ei1−Type , e i 2−Type ) where ei1−Type , e i 2−Type ∈ ET and ET represents the set of all UMLS semantic types . Our goal is to predict the relationship ri that is expressed in each bag Bi , forming a fact triple T = { ( ei1 , ri , ei2 ) } . For simplicity , indices for bags , sentences , and entities are omitted if not required for clarity .

Summary: This paper proposes a variant of multi-instance learning (MIL) for biomedical entity extraction called abstractified multi-instance learning (AMIL), which uses entity type information to construct bags of instances. In the paper, the authors first point out that a long-tail distribution of fact triples (i.e., (entity1, relation, entity2)) diminishes the advantage of the standard MIL since many bags might end up with duplicated instances (i.e., heavy upsampling is required to fill all the bags). This is particularly a problem in the biomedical data given that 50+% of fact triples appear in less than three sentences in PubMed. Instead, AMIL groups training instances by UMLS semantic types, which are shared across different entities. This approach results in more diverse instances in the bags, maximizing the benefit of MIL. For evaluation, the authors construct their own training/dev/test split from PubMed and UMLS. The BioBERT-base RE models are trained with AMIL and compare with the prior work (Amin et al., 2020) retrained on the same data. The main results show that AMIL models outperform the baseline by large margin. The authors further investigate performance on rare/common triples and 17 different relationship representations.