Semi-Automating Knowledge Base Construction for Cancer Genetics

We propose two challenging tasks that are critical for characterizing the findings reported in penetrance studies : ( i ) Extracting snippets of text that describe ascertainment mechanisms , which in turn inform whether the population studied may introduce bias owing to deviations from the target population ; ( ii ) Extracting reported risk estimates ( e.g. , odds or hazard ratios ) associated with specific germline mutations . The latter task may be viewed as a joint entity tagging and relation extraction problem . To train models for these tasks , we induce distant supervision over tokens and snippets in full-text articles using the manually constructed knowledge base . We propose and evaluate several model variants , including a transformer-based joint entity-relation extraction model to extract < germline mutation , risk-estimate > pairs for different cancer types . We observe strong empirical performance , highlighting the practical potential for such models to aid KB construction in this space . We ablate components of our model , observing , e.g. , that a joint model for 1. https : //ask2me.org/index.php < germline mutation , risk-estimate > fares substantially better than a pipelined approach . 1 . Introduction . The published evidence base concerning genetic factors in cancer is vast and growing rapidly . A simple search for “ gene cancer study ” on PubMed2 yields over 238,000 research articles ( including clinical trial reports , meta-analyses , and other types of research ) ; about 16,500 of these were published in 2019 alone . This torrent of unstructured findings makes it difficult for domain experts to navigate and make sense of the evidence . Consequently , there is a critical need for tools designed to help clinicians monitor and synthesize the literature on genetics in cancer [ Bao et al. , 2019 ] . Indeed , the medical domain relies on centralization of knowledge [ Collins and Varmus , 2015 , Landrum et al. , 2017 , Forbes et al. , 2017 ] , because sorting through the vast published literature is too onerous for individual clinicians . Therefore , structured knowledge bases ( KBs ) derived from the literature have an important role to play in medicine generally , and in tracking progress and disseminating findings relevant to cancer genetics specifically . Biomedical Natural Language Processing ( NLP ) methods provide a means of extracting key information from the scientific and biomedical literature in general [ Kim , 2017 , Luan et al. , 2017 ] . And there has been some work specifically aiming to aid medical evidence synthesis via NLP techniques [ Lehman et al. , 2019 , Marshall et al. , 2018 , Cohen et al. , 2006 , Brockmeier et al. , 2019 , Schmidt et al. , 2020 ] . More specific to this work , a few efforts have focused on models for identifying literature relevant to cancer susceptibility [ Wallace et al. , 2012 , Bao et al. , 2019 ] . This paper extends state-of-the-art NLP technologies to aid identification and extraction of relevant information from biomedical research articles on gene-cancer associations . We envision a semi-automated process in which these models aid clinicians to facilitate efficient maintenance of a genetics in cancer knowledge base . The main contributions of this work are as follows . 1 . To our knowledge , this is the first effort to ( semi- ) automate extraction of key evidence from full-text cancer genetics papers . We introduce the tasks of extracting ascertainment mechanisms and reported risk metrics for particular mutations from the genetics in cancer literature . We propose distantly supervised strategies for learning models for one of these tasks . In contrast to most prior work , we operate over full-text articles rather than abstracts only ; this is critical as key information will not always be found in abstracts . We envision these models aiding domain experts ( with whom this is a collaborative effort ) in maintaining a KB , rather than fully automating this process . 2 . We propose and evaluate a joint model for extraction of relevant reported ( numerical ) measures of cancer risk and the germline mutations to which they correspond . We realize strong empirical performance using this approach , and perform ablations to investigate which components drive this . 2 . A repository of biomedical literature : https : //www.ncbi.nlm.nih.gov/pubmed/ . This is joint work with specialists at the Massachusetts General Hospital ( MGH ) who have up to now been manually synthesizing the medical literature associated with genecancer associations into a KB ( see Table 1 for an illustrative , condensed fragment ) . This is a valuable yet tedious and time-consuming endeavor . We identify targets which , if successfully extracted automatically , could greatly reduce the human labor required for upkeep of this KB . We show that by extending state-of-the-art NLP models and training these via distant supervision induced from the existing KB , we can achieve strong performance , which in turn suggests that such models may significantly reduce manual workload without compromising the ability to extract relevant information ( i.e. , without sacrificing the accuracy and comprehensiveness of the resultant KB ) . Our effort here naturally extends prior work that used NLP to identify literature relevant to genetics in cancer [ Bao et al. , 2019 ] . However , we focus on extraction of key information from the full-texts of such articles , once identified . Following discussions with the specialist team and a data exploration phase , we identified two tasks corresponding to elements that were both highly relevant to the medical researchers ( i.e. , key elements of their KB ) and potentially extractable from full-texts using natural language technologies . 1 . Identify snippets that correspond to ascertainment of populations studied in the underlying research described in articles . Ascertainment is a complex concept ; here we adopt a working definition such that snippets are considered relevant to ascertainment if they answer any of the following key questions . • What was the source of the study population , including geographical locations , ethnicities , and source cohorts ( e.g. , cancer registries , hospital-based retrospective cohorts , and community-based prospective cohorts ) ? • How many patients ( cases ) and/or controls were being enrolled ? • What were the inclusion and exclusion criteria when enrolling in the study population ? For example , a study may only enroll early-onset breast cancer patients , or patients with a strong family history of cancer . We aim to identify snippets of text that convey this information . 2 . Perhaps the most important data elements in cancer in genetics studies are the risk estimates ( i.e . penetrance ) associated with a particular pathogenetic germline mutation variant and a specific type cancer ( e.g. , BRCA2 mutation ) type . This information will be reported numerically using a standard metric that quantifies comparative risk , typically one of : Odds Ratio ( OR ) , Relative Risk ( RR ) , or Hazard Ratio ( HR ) . These are often reported in free text and only implicitly reference the corresponding genetic variant , for example : “ odds ratio for basal cell carcinoma was higher ( OR=3.8 ; 95 % CI , P = 0.002 ) ” . We provide more details in Section 2 . The remainder of this paper is structured as follows : We start by providing an overview of our approach in Section 2 , explaining how the data was curated ( 2.2 ) and ( distantly ) labeled ( 2.3 ) . Section 3 and Section 4 describe our experimental setup and our results , respectively . We summarize prior related work in Section 5 . Finally , we conclude in Section 6 with a discussion of the implications of this work and possible future directions in this area . 2 . Methods . 2.1 Overview . Given a Portable Document Format ( PDF ) full-text research article on gene-cancer associations , our system aims to extract sentences that describe the ascertainment mechanism for the underlying study , and spans that report the risk metrics for the specific gene-cancer association . For these tasks we define two pipelines that both operate over sentences comprising full-texts . 2.2 Data and Targets . Our clinical collaborators provided a risk object database ( ROD ) D comprising data that they manually extracted from 597 penetrance papers reporting gene-cancer associations , i.e. , D contains the sought-after KB elements for these papers . The semi-structured elements in D were manually extracted from full-text articles , to which we also have access . The majority of these ( 588 ) are PDFs , the remaining ( 9 ) are in HTML format . The sentences pertaining to ascertainment ( as described in Section 1 ) within D were identified as key targets for extraction . We provide examples of these below . In addition to sentences describing what demographic population was studied , ascertainment sentences included those indicating “ adjustments ” for ascertainment . For example , the following sentences were distantly labeled as positive for ascertainment . • A hospital-based study or a panel testing analysis with well-matched cases and controls , such as : 1 . A control population was defined from the National Danish Civil Registration System , with five population controls , matched on sex and year of birth , for mutation carriers as well as first-degree relatives . 2 . For age-adjusted analysis , the projected U.S. population ( year 2000 ) was used ; 84 % of the 3,399 individuals were white . • A proband-based/family-based study with appropriate ascertainment adjustment : such as GRL , modified segregation analysis , for example : 1 . To adjust for ascertainment , we used an ascertainment assumption–free approach in which we evaluated each family separately 2 . Once we verified that the SIR estimates were not influenced by such cohort-effects , our final analyses were based on population rates specific for each country , sex , and 5-year age group averaged from 1950 to 2009 which were applied to all followup , regardless of calendar year . The second key elements that we aim to extract are the reported risks associated with particular germline mutations . These are typically provided as Odds , Risk , or Hazard Ratios ( ORs , RRs , HRs ) , i.e. , floating point numeric values . These values are arguably the main results being conveyed by the article . There can be multiple metrics reported throughout the entire article , corresponding to different gene-cancer associations ( see Table 2 ) . Models must therefore exploit context to disambiguate to which gene a given metric corresponds . We derive data about these risk estimates through direct supervision as they appear in the risk object database provided to us by the medical experts . 2.3 Deriving Distant Supervision for Ascertainment Classification . Rather than direct supervision ( explicit labels on snippets of text extracted from PDFs ) , we have access only to the database manually compiled by physicians thus far ( D ) , which comprises semi-structured elements , pertaining to ascertainment , extracted from a set of articles . We aim to induce distant supervision over article snippets by heuristically matching these to entries in D. To this end , we retrieved the 597 full-texts ( mostly PDFs ) associated with all entries in D. We processed PDFs using Grobid3 to generate structured XML documents . These XML encodings are reasonably structured , and we were able to compose simple rules to extract all article text ( but not tables and figures ) . Next , we extracted sentences from all article sections ( save for the abstract ) using sciSpacy.4 We then constructed three different representations of these sentences : 1 . Simple Bag-of-Words . Average of vector representations obtained via sciSpacy , for every token in a given sentence . We fix the embedding size to 300 . 2 . Weighted TF-IDF . Weighted average of token vectors with their respective TF-IDF scores [ Schmidt , 2019 ] . 3 . Contextual Representation . We pass sentence tokens through SciBERT [ Beltagy et al. , 2019 ] ( this is a variant of BERT [ Devlin et al. , 2018 ] pre-trained on scientific corpora ) and take as a sentence representation the 768-dimensional [ CLS ] token embedding . Once these sentence representations are obtained we generate ‘ positive ’ labels for sentences corresponding to descriptions of ascertainment mechanisms . To derive these pseudolabels we compare all sentence representations from a document Di , to its true ( manually extracted ) ascertainment mechanism snippet , Ai , using cosine distance : cos ( s , Ai ) = sAi ‖t‖‖Ai‖ = ∑n j=1 sjAij√∑n j=1 ( sj ) 2 √∑n j=1 ( Aij ) 2 ( 1 ) Extracted ascertainment mechanisms often correspond to more than one sentence in a document ( on average , there are 2.6 of these per document ) . Therefore , we retain the top three sentences with the highest similarity scores , and we mark these as ‘ positives ’ . One issue with this approach is that using a similarity measure often yields ‘ false positives ’ that are more general than what we are after . We discus this further in the Appendix .

Summary: This work address KB construction in biomedical domain. Specifically, it proposes two tasks in cancer genetics domain: (1) extracting text snippets about ascertainment; (2) extract reported risk estimates for different germline mutations. They first created distant supervision based on existing manually extracted KB. For (1), they showed that classifier using BERT (especially SciBERT) based sentence representation significantly outperforms baseline models; for (2), they used a simple combination of BERT token emebeddings and a dense layer to jointly learn to classify spans into entities and their relation type, which performs better than SVM baselines, and disjoint learning baselines.