Semi-Automating Knowledge Base Construction for Cancer Genetics
Year: 2020

Abstract:
The vast and rapidly expanding volume of biomedical literature makes it difficult for domain experts to keep up with the evidence. In this work, we specifically consider the exponentially growing subarea of genetics in cancer. The need to synthesize and centralize this evidence for dissemination has motivated a team of physicians to manually construct and maintain a knowledge base that distills key results reported in the literature. This is a laborious process that entails reading through full-text articles to understand the study design, assess study quality, and extract the reported risk estimates associated with particular hereditary cancer genes (i.e., penetrance, defined as the risk of cancer with a pathogenic variant in a germline cancer susceptibility gene). In this work, we propose models to automatically surface key elements from full-text cancer genetics articles, with the ultimate aim of expediting the manual workflow currently in place. We propose two challenging tasks that are critical for characterizing the findings reported in penetrance studies: (i) Extracting snippets of text that describe ascertainment mechanisms, which in turn inform whether the population studied may introduce bias owing to deviations from the target population; (ii) Extracting reported risk estimates (e.g., odds or hazard ratios) associated with specific germline mutations. The latter task may be viewed as a joint entity tagging and relation extraction problem. To train models for these tasks, we induce distant supervision over tokens and snippets in full-text articles using the manually constructed knowledge base. We propose and evaluate several model variants, including a transformer-based joint entity-relation extraction model to extract <germline mutation, risk-estimate> pairs for different cancer types. We observe strong empirical performance, highlighting the practical potential for such models to aid KB construction in this space. We ablate components of our model, observing, e.g., that a joint model for 1. https://ask2me.org/index.php Wadhwa, Yin, Hughes & Wallace <germline mutation, risk-estimate> fares substantially better than a pipelined approach.

text: We propose two challenging tasks that are critical for characterizing the findings reported in penetrance studies: (i) Extracting snippets of text that describe ascertainment mechanisms, which in turn inform whether the population studied may introduce bias owing to deviations from the target population; (ii) Extracting reported risk estimates (e.g., odds or hazard ratios) associated with specific germline mutations. The latter task may be viewed as a joint entity tagging and relation extraction problem. To train models for these tasks, we induce distant supervision over tokens and snippets in full-text articles using the manually constructed knowledge base. We propose and evaluate several model variants, including a transformer-based joint entity-relation extraction model to extract <germline mutation, risk-estimate> pairs for different cancer types. We observe strong empirical performance, highlighting the practical potential for such models to aid KB construction in this space. We ablate components of our model, observing, e.g., that a joint model for
1. https://ask2me.org/index.php
<germline mutation, risk-estimate> fares substantially better than a pipelined approach.

heading: 1. Introduction
text: The published evidence base concerning genetic factors in cancer is vast and growing rapidly. A simple search for “gene cancer study” on PubMed2 yields over 238,000 research articles (including clinical trial reports, meta-analyses, and other types of research); about 16,500 of these were published in 2019 alone. This torrent of unstructured findings makes it difficult for domain experts to navigate and make sense of the evidence. Consequently, there is a critical need for tools designed to help clinicians monitor and synthesize the literature on genetics in cancer [Bao et al., 2019].
Indeed, the medical domain relies on centralization of knowledge [Collins and Varmus, 2015, Landrum et al., 2017, Forbes et al., 2017], because sorting through the vast published literature is too onerous for individual clinicians. Therefore, structured knowledge bases (KBs) derived from the literature have an important role to play in medicine generally, and in tracking progress and disseminating findings relevant to cancer genetics specifically.
Biomedical Natural Language Processing (NLP) methods provide a means of extracting key information from the scientific and biomedical literature in general [Kim, 2017, Luan et al., 2017]. And there has been some work specifically aiming to aid medical evidence synthesis via NLP techniques [Lehman et al., 2019, Marshall et al., 2018, Cohen et al., 2006, Brockmeier et al., 2019, Schmidt et al., 2020]. More specific to this work, a few efforts have focused on models for identifying literature relevant to cancer susceptibility [Wallace et al., 2012, Bao et al., 2019]. This paper extends state-of-the-art NLP technologies to aid identification and extraction of relevant information from biomedical research articles on gene-cancer associations. We envision a semi-automated process in which these models aid clinicians to facilitate efficient maintenance of a genetics in cancer knowledge base.
The main contributions of this work are as follows.
1. To our knowledge, this is the first effort to (semi-)automate extraction of key evidence from full-text cancer genetics papers. We introduce the tasks of extracting ascertainment mechanisms and reported risk metrics for particular mutations from the genetics in cancer literature. We propose distantly supervised strategies for learning models for one of these tasks. In contrast to most prior work, we operate over full-text articles rather than abstracts only; this is critical as key information will not always be found in abstracts. We envision these models aiding domain experts (with whom this is a collaborative effort) in maintaining a KB, rather than fully automating this process.
2. We propose and evaluate a joint model for extraction of relevant reported (numerical) measures of cancer risk and the germline mutations to which they correspond. We realize strong empirical performance using this approach, and perform ablations to investigate which components drive this.
2. A repository of biomedical literature: https://www.ncbi.nlm.nih.gov/pubmed/.
This is joint work with specialists at the Massachusetts General Hospital (MGH) who have up to now been manually synthesizing the medical literature associated with genecancer associations into a KB (see Table 1 for an illustrative, condensed fragment). This is a valuable yet tedious and time-consuming endeavor. We identify targets which, if successfully extracted automatically, could greatly reduce the human labor required for upkeep of this KB. We show that by extending state-of-the-art NLP models and training these via distant supervision induced from the existing KB, we can achieve strong performance, which in turn suggests that such models may significantly reduce manual workload without compromising the ability to extract relevant information (i.e., without sacrificing the accuracy and comprehensiveness of the resultant KB).
Our effort here naturally extends prior work that used NLP to identify literature relevant to genetics in cancer [Bao et al., 2019]. However, we focus on extraction of key information from the full-texts of such articles, once identified. Following discussions with the specialist team and a data exploration phase, we identified two tasks corresponding to elements that were both highly relevant to the medical researchers (i.e., key elements of their KB) and potentially extractable from full-texts using natural language technologies.
1. Identify snippets that correspond to ascertainment of populations studied in the underlying research described in articles. Ascertainment is a complex concept; here we adopt a working definition such that snippets are considered relevant to ascertainment if they answer any of the following key questions.
• What was the source of the study population, including geographical locations, ethnicities, and source cohorts (e.g., cancer registries, hospital-based retrospective cohorts, and community-based prospective cohorts)?
• How many patients (cases) and/or controls were being enrolled?
• What were the inclusion and exclusion criteria when enrolling in the study population? For example, a study may only enroll early-onset breast cancer patients, or patients with a strong family history of cancer.
We aim to identify snippets of text that convey this information.
2. Perhaps the most important data elements in cancer in genetics studies are the risk estimates (i.e. penetrance) associated with a particular pathogenetic germline mutation variant and a specific type cancer (e.g., BRCA2 mutation) type. This information will be reported numerically using a standard metric that quantifies comparative risk, typically one of: Odds Ratio (OR), Relative Risk (RR), or Hazard Ratio (HR). These are often reported in free text and only implicitly reference the corresponding genetic variant, for example: “odds ratio for basal cell carcinoma was higher (OR=3.8; 95% CI, P = 0.002)”. We provide more details in Section 2.
The remainder of this paper is structured as follows: We start by providing an overview of our approach in Section 2, explaining how the data was curated (2.2) and (distantly) labeled (2.3). Section 3 and Section 4 describe our experimental setup and our results, respectively. We summarize prior related work in Section 5. Finally, we conclude in Section 6 with a discussion of the implications of this work and possible future directions in this area.

heading: 2. Methods

heading: 2.1 Overview
text: Given a Portable Document Format (PDF) full-text research article on gene-cancer associations, our system aims to extract sentences that describe the ascertainment mechanism for the underlying study, and spans that report the risk metrics for the specific gene-cancer association. For these tasks we define two pipelines that both operate over sentences comprising full-texts.

heading: 2.2 Data and Targets
text: Our clinical collaborators provided a risk object database (ROD) D comprising data that they manually extracted from 597 penetrance papers reporting gene-cancer associations, i.e., D contains the sought-after KB elements for these papers. The semi-structured elements in D were manually extracted from full-text articles, to which we also have access. The majority of these (588) are PDFs, the remaining (9) are in HTML format.
The sentences pertaining to ascertainment (as described in Section 1) within D were identified as key targets for extraction. We provide examples of these below. In addition to sentences describing what demographic population was studied, ascertainment sentences included those indicating “adjustments” for ascertainment. For example, the following sentences were distantly labeled as positive for ascertainment.
• A hospital-based study or a panel testing analysis with well-matched cases and controls, such as:
1. A control population was defined from the National Danish Civil Registration System, with five population controls, matched on sex and year of birth, for mutation carriers as well as first-degree relatives.
2. For age-adjusted analysis, the projected U.S. population (year 2000) was used; 84% of the 3,399 individuals were white.
• A proband-based/family-based study with appropriate ascertainment adjustment: such as GRL, modified segregation analysis, for example:
1. To adjust for ascertainment, we used an ascertainment assumption–free approach in which we evaluated each family separately
2. Once we verified that the SIR estimates were not influenced by such cohort-effects, our final analyses were based on population rates specific for each country, sex, and 5-year age group averaged from 1950 to 2009 which were applied to all followup, regardless of calendar year.
The second key elements that we aim to extract are the reported risks associated with particular germline mutations. These are typically provided as Odds, Risk, or Hazard Ratios (ORs, RRs, HRs), i.e., floating point numeric values. These values are arguably the main results being conveyed by the article. There can be multiple metrics reported throughout the entire article, corresponding to different gene-cancer associations (see Table 2). Models must therefore exploit context to disambiguate to which gene a given metric corresponds. We derive data about these risk estimates through direct supervision as they appear in the risk object database provided to us by the medical experts.

heading: 2.3 Deriving Distant Supervision for Ascertainment Classification
text: Rather than direct supervision (explicit labels on snippets of text extracted from PDFs), we have access only to the database manually compiled by physicians thus far (D), which comprises semi-structured elements, pertaining to ascertainment, extracted from a set of articles. We aim to induce distant supervision over article snippets by heuristically matching these to entries in D.
To this end, we retrieved the 597 full-texts (mostly PDFs) associated with all entries in D. We processed PDFs using Grobid3 to generate structured XML documents. These XML encodings are reasonably structured, and we were able to compose simple rules to extract all article text (but not tables and figures).
Next, we extracted sentences from all article sections (save for the abstract) using sciSpacy.4 We then constructed three different representations of these sentences:
1. Simple Bag-of-Words. Average of vector representations obtained via sciSpacy, for every token in a given sentence. We fix the embedding size to 300.
2. Weighted TF-IDF. Weighted average of token vectors with their respective TF-IDF scores [Schmidt, 2019].
3. Contextual Representation. We pass sentence tokens through SciBERT [Beltagy et al., 2019] (this is a variant of BERT [Devlin et al., 2018] pre-trained on scientific corpora) and take as a sentence representation the 768-dimensional [CLS] token embedding.
Once these sentence representations are obtained we generate ‘positive’ labels for sentences corresponding to descriptions of ascertainment mechanisms. To derive these pseudolabels we compare all sentence representations from a document Di, to its true (manually extracted) ascertainment mechanism snippet, Ai, using cosine distance:
cos(s,Ai) = sAi ‖t‖‖Ai‖ = ∑n j=1 sjAij√∑n j=1 (sj) 2 √∑n j=1 (Aij) 2
(1)
Extracted ascertainment mechanisms often correspond to more than one sentence in a document (on average, there are 2.6 of these per document). Therefore, we retain the top three sentences with the highest similarity scores, and we mark these as ‘positives’. One issue with this approach is that using a similarity measure often yields ‘false positives’ that are more general than what we are after. We discus this further in the Appendix.

heading: 2.4 Joint Entity-Relation Extraction to Determine Risk Estimates
text: A key piece of information reported in genetics in medicine studies is the reported risk ratio for particular germline mutations. In the risk object database (D), a given row corresponds to one cancer risk estimate for a specific <germline mutation, risk-estimate> pair and one penetrance paper may correspond to multiple such rows inD. Therefore, our second task is to extract these metrics (Odds/Risk/Hazard Ratio) and identify the germline mutation to which they correspond. We perform this extraction over individual sentences independently.
Figure 1 provides a schematic of our approach. For this task we propose a transformerbased model that jointly extracts the relevant entities (risk-estimates, genes) and predicts their relations (positive, negative) directly from spans within individual sentences.
3. Grobid is a tool for extracting and parsing PDFs into structured XML/TEI encoded documents with a particular focus on technical and scientific publications: https://github.com/kermitt2/ grobid-client-python. 4. A sister NLP library to SpaCy (https://spacy.io/) built for biomedical text: https://allenai.github. io/scispacy/
Concretely, an input document X is treated as a set of sentences X =s1, s2, s3. . . .s|X|. The task of entity recognition entails predicting a label ei (here: germline mutation, risk-estimate, or none) for each token within a sentence. Relation extraction involves predicting whether a <germline mutation, risk-estimate> pair is a positive (true) relation or not.
Our model considers fixed length (10-15) spans within sentences and classifies each token into possible entity types. The model then classifies these entity-types into relations (positive or negative).
We use SciBERT [Beltagy et al., 2019] to generate token level representations. We sum these and the [CLS] token embedding to incorporate overall context, yielding the final input for entity classification:
ei = softmax(We · ([CLS] + ti)) + be) (2)
For every pair of entity candidates (ex, ey), where neither is of type none, our next task involves predicting the best relation type rx,y. To form a combined localized representation of (ex, ey), we take the sum of the embeddings from ex through ey and then take the elementwise product of this representation with the [CLS] token representing the span. We pass this output through a fully connected layer followed by a sigmoid, yielding a binary probability prediction. Therefore, if i and j correspond to the token indices of entities ex and ey respectively, then the predicted probability corresponding to rx,y is:
rx,y = σ(Wr · ([CLS] · j∑ k=i (ek) + br) (3)
Much like [Luan et al., 2019], our loss function is the (unweighted) sum of the loglikelihood of the two tasks: ∑
(D,R∗,E∗) X
{logP(R∗|E,D) + logP(E∗|D)} (4)
Where E∗ and R∗ are the true entities and relations, respectively. X is the collection of all training documents.
Simple, disjoint model for entity-relation extraction: Does joint training yield improved performance? To investigate this we also train two simple disjoint models, based on [Shi and Lin, 2019], for the same task. In this approach, entity tokens are classified using a dense layer on top of representations induced by SciBERT [Beltagy et al., 2019]. Then, for the relation extraction task, the model takes input as [[CLS] sentence-span [SEP] entity [SEP] entity [SEP]]. We then discard the sentence after the first [SEP] token, preserving only the contextualized representation which is then concatenated and passed into a fully connected network to obtain the output.

heading: 3. Experimental Setup
text: We use sciSpacy [Neumann et al., 2019] to carry out all text preprocessing, including tokenization and sentence splitting. To train our transformer based models, we use the HuggingFace [Wolf et al., 2019] transformer library in PyTorch [Paszke et al., 2019]. We’re providing the model code along with the raw annotations of penetrance papers, indexed by their PubMed IDs5.
Data: We obtained annotated data (Table 3) from clinicians as described in Section 2.2. Table 3 reports basic data statistics and information about train, development, and test set splits that we have created. Importantly, for both of our tasks, these are disjoint at the document level, meaning that sentences in the respective splits are drawn from corresponding unique sets of documents.
Training: For the ascertainment classification task, our best performing model on dev data (fine-tuned, SciBERT) uses the Adam [Kingma and Ba, 2014] optimization scheme, a batch size of 32 and a learning rate of 2e-5 over 4 epochs.
For the joint entity tagging and relation extraction task, we again use the Adam optimizer, with a batch size of 16, and a linear decay learning rate schedule. The learning rate here is 5e-5. We observe that the model peaks after 8-10 epochs. We also observe that the same hyperparameters suffice with our disjoint entity-relation classification models.
All of our hyperparameter tuning was carried out on development sets; the results we report are from the held out test set, unseen during model development.

heading: 4. Results
text: We report results on test sets in Tables 4 and 5. For all models, we report F1 score, precision, recall, accuracy, and matthews correlation coefficient (MCC).
For the first task of ascertainment classification, we observe that a transformer based approach (i.e., a simple classification layer on top of SciBERT) outperforms baselines, as one would expect. However, we were somewhat surprised by the performance difference observed between variants initialized with BERT and SciBERT weights; the latter achieves a 9 point absolute gain in F1, apparently demonstrating the considerable advantage afforded by domain-specific pre-training.
For the second task of extracting <germline mutation, risk-estimate> pairs, we observe similar performance (using SciBERT) on the entity classification task with both joint and the pipelined approach. However, the joint model outperforms the pipelined approach in the subsequent relation inference task, yielding a substantial difference of 13 points in F1 score. This demonstrates the promise of joint — as opposed to pipelined — approaches for this task.
Qualitatively, we observe that our models produce high quality results, in the sense that the ostensibly misclassified examples are ambiguous to a clinical evaluator. We provide
5. https://github.com/sominwadhwa/kbcCanGen.
evidence for this along with additional ablation experiments (e.g., model sensitivity to specific numerical tokens for risk ratio extraction) in the Appendix.

heading: 5. Related Work
text: Work related to this effort includes general methods for scientific information extraction [Liu et al., 2016, Luan et al., 2019], evidence mining [Rinott et al., 2015], and the use of distant supervision to create weakly (and noisily) labelled data at scale [Mintz et al., 2009, Augenstein et al., 2015].
Recent work on exploiting distant supervision has focused largely on the analysis of free text on social media [Purver and Battersby, 2012, Marchetti-Bowick and Chambers, 2012] and on relation extraction [Riedel et al., 2010, Mintz et al., 2009, Nguyen and Moschitti, 2011]. Our work relates more to the former approaches of using heuristic and distant supervision to induce weak, sometimes noisy, labels for a downstream task. Despite this inherent noise, we find that we are able to exploit such weak labels to train layers on top of modern contextualized encoder models [Devlin et al., 2018, Peters et al., 2018] pretrained on scientific text [Beltagy et al., 2019], yielding extractors that can capture the desired information with high fidelity. This extends past work in which weak or distant forms of supervision have specifically been used to train models for information extraction from scientific and biomedical texts [Quirk and Poon, 2016, Jain et al., 2016, Wallace et al., 2016, Norman et al., 2019].
Designing language technologies for (semi-)automated information extraction in the biomedical domain remains an active area of research [Liu et al., 2016, Patel et al., 2018]. For example, there is a line of related work designing and evaluating extraction methods to aid biomedical literature synthesis [Jonnalagadda and Petitti, 2013, Kiritchenko et al., 2010, Nye et al., 2018]. Much of the recent work in this space has involved using neural models to extract entities [Greenberg et al., 2018], relations between these entities [Song et al., 2018, Krasakis et al., Verga et al., 2018, Wadden et al., 2019] and reported findings [Lehman et al., 2019]. We build on these models in the present work.
In oncology research specifically, classification models have been applied to designate staging and assess cancer recurrence [Deng et al., 2019, Hughes et al., 2020, Friedlin et al., 2010]. Fiszman et al. [2010] used semantic analysis of text to identify cardiovascular risk factors in medical literature. Elsewhere, Lossio-Ventura et al. [2016] explicitly considered the long-term aim of constructing an obesity-cancer KB, though their work was limited to studies in obesity and titles/abstracts, rather than full-texts.
None of the aforementioned works have attempted to infer population characteristics or risk estimates from the cancer in genetics literature, despite the importance of establishing a centralized KB for the cancer in genetics literature, and the burden that manually curating and maintaining this resource currently imposes. This effort is a first step toward building models to mitigate manual workload, thereby allowing KB construction to scale as the literature continues to rapidly grow.

heading: 6. Conclusions
text: We have proposed the practically important task of extracting relevant information from cancer genetics literature, with the aim of helping domain experts (clinicians) maintain an up-to-date knowledge base of genetics in cancer results.
We acquired full-text PDFs of gene-cancer studies from which clinicians previously extracted key structured information from full-texts into a database, and we used these entries to induce distant supervision over spans within the corresponding articles. Using these derived (weak) labels, we trained a classification model to identify key snippets that relate to ascertainment bias, a key consideration in such evidence.
We then considered the more challenging task of extracting <germline-mutation, risk> pairs as an entity-relation problem. For this we proposed a BERT-based joint entity tagging and relation extraction model. Through ablation we observe: (1) This joint approach fares substantially better than a two-step pipelined approach, and, (ii) Initialization to BERT parameters learned “in-domain” also provides a considerable performance increase [Beltagy et al., 2019].
Going forward, we hope to evaluate the utility of these models in practice to assess the degree to which they actually help expedite knowledge-base construction. This work will assist the domain experts in: assessing study quality quickly and accurately, identify the most representative population-level risk for a specific gene-cancer association. Ultimately, risks identified in the primary literature may serve as a foundation for providing individualized cancer prevention practice and clinical care. More generally, we hope this first effort spurs additional work on models for automatically making sense of the genetics in cancer literature.

heading: 7. Acknowledgements
text: This work was funded in part by the National Institutes of Health (NIH) under the National Library of Medicine (NLM) grant 2R01LM012086, and by the National Science Foundation (NSF) CAREER award 1750978.

Review summaries:
This work address KB construction in biomedical domain. Specifically, it proposes two tasks in cancer genetics domain: (1) extracting text snippets about ascertainment; (2) extract reported risk estimates for different germline mutations. They first created distant supervision based on existing manually extracted KB. For (1), they showed that classifier using BERT (especially SciBERT) based sentence representation significantly outperforms baseline models; for (2), they used a simple combination of BERT token emebeddings and a dense layer to jointly learn to classify spans into entities and their relation type, which performs better than SVM baselines, and disjoint learning baselines.