Venue: NAACL 2019

# Document-Level \(N\)-ary Relation Extraction with Multiscale Representation Learning

Robin Jia\({}^{1}\) Cliff Wong\({}^{2}\) Hoifung Poon\({}^{2}\)

\({}^{1}\) Stanford University, Stanford, California, USA

\({}^{2}\) Microsoft Research, Redmond, Washington, USA robinjia@cs.stanford.edu, {cliff.wong,hoifung}@microsoft.com

Work done as an intern at Microsoft Research.

###### Abstract

Most information extraction methods focus on binary relations expressed within single sentences. In high-value domains, however, \(n\)-ary relations are of great demand (e.g., drug-gene-mutation interactions in precision oncology). Such relations often involve entity mentions that are far apart in the document, yet existing work on cross-sentence relation extraction is generally confined to small text spans (e.g., three consecutive sentences), which severely limits recall. In this paper, we propose a novel multiscale neural architecture for document-level \(n\)-ary relation extraction. Our system combines representations learned over various text spans throughout the document and across the subrelation hierarchy. Widening the system's purview to the entire document maximizes potential recall. Moreover, by integrating weak signals across the document, multi-scale modeling increases precision, even in the presence of noisy labels from distant supervision. Experiments on biomedical machine reading show that our approach substantially outperforms previous \(n\)-ary relation extraction methods.

## 1 Introduction

Knowledge acquisition is a perennial challenge in AI. In high-value domains, it has acquired new urgency in recent years due to the advent of big data. For example, the dramatic drop in genome sequencing cost has created unprecedented opportunities for tailoring cancer treatment to a tumor's genetic composition Bahcall (2015). Despite this potential, operationalizing personalized medicine is difficult, in part because it requires painstaking curation of precision oncology knowledge from biomedical literature. With tens of millions of papers on PubMed, and thousands more added every day,1 we are sorely in need of automated methods to accelerate manual curation.

Footnote 1: ncbi.nlm.nih.gov/pubmed

Prior work in machine reading has made great strides in sentence-level binary relation extraction. However, generalizing extraction to \(n\)-ary relations poses new challenges. Higher-order relations often involve entity mentions that are far away in the document. Recent work on \(n\)-ary relation extraction has begun to explore cross-sentence extraction Peng et al. (2017); Wang and Poon (2018), but the scope is still confined to short text spans (e.g., three consecutive sentences), even though a document may contain hundreds of sentences and tens of thousands of words. While this already increases the yield compared to sentence-level extraction, it still misses many relations. For example, in Figure 1, the drug-gene-mutation relations between PF02341066, ALK, G123S(D) (PF02341066 can treat cancers with mutation G1123S(D) in gene ALK) can only be extracted by substantially expanding the scope. High-value information, such as latest medical findings, might only be mentioned once in the corpus. Maximiz

Figure 1: Two examples of drug-gene-mutation relations from a biomedical journal paper. The relations are expressed across multiple paragraphs, requiring document-level extraction.



ing recall is thus of paramount importance.

In this paper, we propose a novel multiscale neural architecture for document-level \(n\)-ary relation extraction. By expanding extraction scope to the entire document, rather than restricting relation candidates to co-occurring entities in a short text span, we ensure maximum potential recall. To combat the ensuing difficulties in document-level extraction, such as low precision, we introduce multiscale learning, which combines representations learned over text spans of varying scales and for various subrelations (Figure 2). This approach deviates from past methods in several key regards.

First, we adopt an entity-centric formulation by making a single prediction for each entity tuple occurring in a document. Previous \(n\)-ary relation extraction methods typically classify individual mention tuples, but this approach scales poorly to whole documents. Since each entity can be mentioned many times in the same document, applying mention-level methods leads to a combinatorial explosion of mention tuples. This creates not only computational challenges but also learning challenges, as the vast majority of these tuples do not express the relation. Our entity-centric formulation alleviates both of these problems.

Second, for each candidate tuple, prior methods typically take as input the contiguous text span encompassing the mentions. For document-level extraction, the resulting text span could become untenably large, even though most of it is unrelated to the relation of interest. Instead, we allow discontiguous input formed by multiple discourse units (e.g., sentence or paragraph) containing the given entity mentions.

Finally, while an \(n\)-ary relation might not reside within a discourse unit, its subrelations might. In Figure 1, the paper first mentions a gene-mutation subrelation, then discusses a drug-mutation subrelation in a later paragraph. By including subregions in our modeling, we can predict \(n\)-ary relations even when all \(n\) entities never co-occur in the same discourse unit.

With multiscale learning, we turn the document view from a challenge into an advantage by combining weak signals across text spans and subrelations. Following recent work in cross-sentence relation extraction, we conduct thorough evaluation in biomedical machine reading. Our approach substantially outperforms prior \(n\)-ary relation extraction methods, attaining state-of-the-art results on a large benchmark dataset recently released by a major cancer center. Ablation studies show that multiscale modeling is the key to these gains.2

Footnote 2: Our code and data will be available at hanover. azurewebsites.net

## 2 Document-Level \(N\)-ary Relation Extraction

Prior work on relation extraction typically formulates it as a mention-level classification problem. Let \(e_{1},\ldots,e_{n}\) be entity mentions that co-occur in a text span \(T\). Relation extraction amounts to classifying whether a relation \(R\) holds for \(e_{1},\ldots,e_{n}\) in \(T\). For the well-studied case of binary relations within single sentences, \(n=2\) and \(T\) is a sentence.

In high-value domains, however, there is increasing demand for document-level \(n\)-ary relation extraction, where \(n>2\) and \(T\) is a full document that may contain hundreds of sentences. For example, a molecular tumor board needs to know if a drug is relevant for treating cancer patients with a certain mutation in a given gene. We can help the tumor board by extracting such ternary interactions from biomedical articles. The mention-centric view of relation extraction does not scale well to this general setting. Each of the \(n\) entities may be mentioned many times in a document, resulting in a large number of candidate mention tuples, even though the vast majority of them are irrelevant to the extraction task.

In this paper, we adopt an _entity-centric_ formulation for document-level \(n\)-ary relation extraction. We use upper case for entities (\(E_{1},\cdots,E_{n}\)) and lower case for mentions (\(e_{1},\cdots,e_{n}\)). We define an \(n\)-ary relation candidate to be an \((n+1)\)-tuple \((E_{1},\ldots,E_{n},T)\), where each entity \(E_{i}\) is mentioned at least once in the text span \(T\). The relation extraction model is given a candidate \((E_{1},\ldots,E_{n},T)\) and outputs whether or not the tuple expresses the relation \(R\).3 Deciding what information to use from the various entity mentions within \(T\) is now a modeling question, which we address in the next section.

Footnote 3: It is easy to extend our approach to situations where \(k\) mutually exclusive relations \(R_{1},\ldots,R_{k}\) must be distinguished, resulting in a \((k+1)\)-way classification problem.

## 3 Our Approach: Multiscale Representation Learning

We present a general framework for document-level \(n\)-ary relation extraction using multiscale 

representation learning. Given a document with text \(T\) and entities \(E_{1},\ldots,E_{n}\), we first build mention-level representations for groups of these entities whenever they co-occur within the same discourse unit. We then aggregate these representations across the whole document, yielding entity-level representations for each subset of entities. Finally, we predict whether \(E_{1},\ldots,E_{n}\) participate in the relation based on the concatenation of these entity-level representations. These steps are depicted in Figure 2.

### Mention-level Representation

Let the full document \(T\) be composed of discourse units \(T_{1},\ldots,T_{m}\) (e.g., different paragraphs). Let \(T_{j}\) be one such discourse unit, and suppose \(e_{1},\ldots,e_{n}\) are entity mentions of \(E_{1},\ldots,E_{n}\) that co-occur in \(T_{j}\). We construct a contextualized representation for mention tuple \((e_{1},\ldots,e_{n})\) in \(T_{j}\). In this paper, we use a standard approach by applying a bi-directional LSTM (BiLSTM) to \(T_{j}\), concatenating the hidden states for each mention, and feeding this through a single-layer neural network. We denote the resulting vector as \(r(R,e_{1},\ldots,e_{n},T_{j})\) for the relation \(R\).

### Entity-level Representation

Let \(M(R,E_{1},\ldots,E_{n},T)\) denote the set of all mention tuples \((e_{1},\ldots,e_{n})\) and discourse units \(T_{j}\) within \(T\) such that each \(e_{i}\) appears in \(T_{j}\). We can create an entity-level representation \(r(R,E_{1},\ldots,E_{n},T)\) of the \(n\) entities by combining mention-level representations using an aggregation operator \(\mathcal{C}\):

A standard choice for \(\mathcal{C}\) is max pooling, which works well if it is pretty clear-cut whether a mention tuple expresses a relation. In practice, however, the mention tuples could be ambiguous and less than certain individually, yet collectively express a relation in the document. This motivates us to experiment with \(\mathrm{logsumexp}\), the smooth version of max, where

\[\mathrm{logsumexp}(x_{1},\ldots,x_{k})=\log\sum_{i=1}^{k}\exp(x_{i}).\]

This facilitates accumulating weak signals from individual mention tuples, and our experiments show that it substantially improves extraction accuracy compared to max pooling.

Figure 2: Multiscale representation learning for document-level \(n\)-ary relation extraction, an entity-centric approach that combines mention-level representations learned across text spans and subrelation hierarchy. (1) Entity mentions (e.g., gefitinib, a drug; EGFR, a gene; T790M, a variant) are identified from text, and mentions that co-occur within a discourse unit (e.g., paragraph) are isolated. (2) Within each discourse unit, mention-level representations are computed for each tuple of entity mentions. These representations may correspond to the entire \(n\)-ary relation or subregions over subsets of entities (drug-variant, drug-gene, gene-variant). (3) At the document scale, mention-level representations for both the \(n\)-ary relation and its subrelations are combined into entity-level representations. (4) Entity-level representations are used to predict the relation.

 



### Subrelations

For higher-order relations (i.e., larger \(n\)), it is less likely that they will be completely contained within a discourse unit. Often, the relation can be decomposed into subrelations over subsets of entities, each of which is more likely to be expressed in a single discourse unit. This motivates us to construct entity-level representations for subrelations as well. The process is straightforward. Let \(R_{S}\) be the \(|S|\)-ary subrelation over entities \(E_{S_{1}},\cdots,E_{S_{|S|}}\), where \(S\subseteq\{1,\ldots,n\}\) and \(|S|\) denotes its size. We first construct mention-level representations \(r(R_{S},e_{S_{1}},\cdots,e_{S_{|S|}},T)\) for \(R_{S}\) and its relevant entity mentions, then combine them into an entity-level representation \(r(R_{S},E_{S_{1}},\cdots,E_{S_{|S|}},D)\) using the chosen aggregation operator \(\mathcal{C}\). We do this for every \(S\subseteq\{1,\dots,n\}\) with \(|S|\geq 2\) (including the whole set, which corresponds to the full relation \(R\)). This gives us an entity-level representation for each subrelation of arity at least \(2\), or equivalently, each subset of entities of size at least \(2\).

### Relation Prediction

To make a final prediction, we first concatenate all of the entity-level representations \(r(R_{S},E_{S_{1}},\ldots,E_{S_{|S|}},D)\) for all \(S\subseteq\{1,\dots,n\}\) with \(|S|\geq 2\). The concatenated representation is fed through a two-layer feedforward neural network followed by a softmax function to predict the relation type.

It is possible that for some subrelations \(R_{S}\), all \(|S|\) entities do not co-occur in any discourse unit. When this happens, we set \(r(R_{S},E_{S_{1}},\ldots,E_{S_{|S|}})\) to a bias vector which is learned separately for each \(R_{S}\). This ensures that the concatenation is done over a fixed number of vectors, e.g., 4 for a tenary relation (three binary subrelations and the main relation). Importantly, this strategy enables us to make meaningful predictions for relation candidates even if all \(n\) entities never co-occur in the same discourse unit; such candidates would never be generated by a system that only looks at single discourse units in isolation.

### Document Model

Our document model is actually a family of representation learning methods, conditioned on the choice of discourse units, subtelations, and aggregation operators. In this paper, we consider sentences and paragraphs as possible discourse units. We explore \(\max\) and \(\operatorname*{logsumexp}\) as aggregation operators. Moreover, we explore ensemble prediction as an additional aggregation method. Specifically, we learn a restricted multiscale model by limiting the text span to a single discourse unit (e.g., a paragraph); the model still combines representations across mentions and subrelations. At test time, given a full document with \(m\) discourse units, we obtain independent predictions \(p_{1},\ldots,p_{m}\) for each discourse unit. We then combine these probabilities using an ensemble operator \(\mathcal{P}\). A natural choice for \(\mathcal{P}\) is max, though we also experiment with noisy-or:

\[\mathcal{P}(p_{1},\cdots,p_{k})=1-\prod_{i=1}^{k}~{}(1-p_{i}).\]

It is also possible to ensemble multiple models that operate on different discourse units, using this same operator.

Our model can be trained using standard supervised or indirectly supervised methods. In this paper, we focus on distant supervision, as it is a particularly potent learning paradigm for high-value domains. Our entity-centric formulation is particularly well aligned with distant supervision, as distant supervision at the entity level is significantly less noisy compared to the mention level, so we don't need to deploy sophisticated denoising strategies such as multi-instance learning (Hoffmann et al., 2011).

\begin{table}
\begin{tabular}{|l|c c c|} \hline  & Sentence & Paragraph & Document \\  & level & level & level \\ \hline Text Units & \(2,326\) & \(3,687\) & \(3,362\) \\ \hline Pos. Examples & \(2,222\) & \(4,906\) & \(8,514\) \\ \hline Neg. Examples & \(2,849\) & \(13,371\) & \(323,584\) \\ \hline \end{tabular}
\end{table}
Table 1: Statistics of our training corpus using PMC-OA articles and distant supervision from CIVIC, GDKD, and OncoKB. “Text Units” refers to the number of distinct sentences, paragraphs, and documents that contain a candidate triple of drug, gene, mutation.

\begin{table}
\begin{tabular}{|l|c c|} \hline  & Development & Test \\ \hline Documents & \(118\) & \(225\) \\ Annotated facts & \(701\) & \(1,324\) \\ \hline Paragraphs per document & \(101\) & \(105\) \\ Sentences per document & \(314\) & \(320\) \\ Words per document & \(6,871\) & \(7,010\) \\ \hline \end{tabular}
\end{table}
Table 2: Statistics of the CKB evaluation corpus.



Experiments

### Biomedical Machine Reading

We validate our approach on a standard biomedical machine reading task: extracting drug-gene-mutation interactions from biomedical articles Peng et al. (2017); Wang and Poon (2018). We cast this task as binary classification: given a drug, gene, mutation, and document in which they are mentioned, determine whether the document asserts that the mutation in the gene affects response to the drug. For training, we use documents from the PubMed Central Open Access Subset (PMCOA)4. For distant supervision, we use three existing knowledgebases (KBs) with hand-curated drug-gene-mutation facts: CIVIC,5 GDKD Dienstmann et al. (2015), and OncoKB Chakravarty et al. (2017). Table 1 shows basic statistics of this training data. Past methods using distant supervision often need to up-weight positive examples, due to the large proportion of negative candidates. Interestingly, we found that our document model was robust to this imbalance, as re-weighting had little effect and we didn't use it in our final results.

Footnote 4: www.ncbi.nlm.nih.gov/pmc

Evaluating distant supervision methods is challenging, as there is often no gold-standard test set, especially at the mention level. Prior work thus resorts to reporting sample precision (estimated proportion of correct system extractions) and absolute recall (estimated number of correct system extractions). This requires subsampling extraction results and manually annotating them. Subsampling variance also introduces noise in the estimate.

Instead, we used CKB CORETM, a public subset of the Clinical Knowledgebase (CKB)6 Patterson et al. (2016), as our gold-standard test set. CKB COREtm contains document-level annotation of drug-gene-mutation interactions manually curated by The Jackson Laboratory (JAX), an NCI-designated cancer center. It is a high-quality KB containing facts from a few hundred PubMed articles for 86 genes, with minimal overlap with the three KBs we used for distant supervision. To avoid contamination, we removed CKB entries whose documents were used in our training data, and split the rest into a development and test set. See Table 2 for statistics. We tuned hyperparameters and thresholds on the development set, and report results on the test set.

Footnote 5: civic.genome.wustl.edu

Footnote 6: ckbhome.jax.org

### Implementation Details

We conducted standard preprocessing and entity linking, similar to Wang and Poon (2018) (see Section A.1). Following standard practice, we masked all entities of the same type with a dummy token, to prevent the classifier from simply memorizing the facts in distant supervision. Wang and Poon (2018) observed that many errors stemmed from incorrect gene-mutation association. We therefore developed a simple rule-based system that predicts which gene-mutation pairs are valid (see Section A.2). We removed candidates that contained a gene-mutation pair that was not predicted by the rule-based system.

### Main Results

We evaluate primarily on area under the precision recall curve (AUC).7 We also report maximum recall, which is the fraction of true facts for which a candidate was generated. Finally, we report precision, recall, and F1, using a threshold tuned to maximize F1 on the CKB development set.

Footnote 7: We compute area using average precision, which is similar to a right Riemann sum. This avoids errors introduced by the trapezoidal rule, which may overestimate area.

We compared our multiscale system (MultiScale) with three restricted variants (SentLevel, ParaLevel, DocLevel). SentLevel and ParaLevel restricted training and prediction to single discourse units (i.e., sentences and paragraphs), and produced a document-level prediction by applying the ensemble operator over individual discourse units. DocLevel takes the whole document as input, with each paragraph as a discourse unit. MultiScale further combined SentLevel, ParaLevel, and DocLevel using the ensemble operator. For additional details about the models, see Section A.3. We also compared MultiScale with DPL Wang and Poon (2018), the prior state of the art in cross-sentence \(n\)-ary relation extraction. DPL classifies drug-genemutation interactions within three consecutive sentences using the same model architecture as Peng et al. (2017), but incorporates additional indirect supervision such as data programming and joint inference. We used the DPL code from the authors and produced a document-level prediction similarly using the ensemble operator. In the base version, we used max as the ensemble operator. We also evaluated the effect when we 

used noisy-or as the ensemble operator, as well as when we applied the gene-mutation filter during postprocessing.

Table 3 shows the results on the CKB test set. In all scenarios, our full model MultiScale substantially outperforms the prior state-of-the-art system (DPL). For example, in the best setting, using both noisy-or and the gene-mutation filter, the full model improves over DPL by \(8.4\) AUC points. Multiscale learning is the key to this performance gain, with MultiScale substantially outperforming more restricted variants. Not surprisingly, expanding extraction scope from sentences to paragraphs resulted in the biggest gain, already surpassing DPL. Conducting end-to-end learning over a document-level representation, as in DocLevel, is beneficial compared to ensembling over predictions for individual discourse units SentLevel, ParaLevel), especially in the base version. Interestingly, MultiScale still attained significant gain over DocLevel with an ensemble over SentLevel and ParaLevel, suggesting that the document-level representation can still be improved. In addition to prediction accuracy, the document-level models also have much more room to grow, as maximum recall is about 20 absolute points higher in MultiScale and DocLevel, compared to ParaLevel or DPL.8

Footnote 8: The difference in actual recall is less pronounced, as we chose thresholds to maximize F1 score. We expect actual recall to increase significantly as document-level models improve, whereas the other models are closer to their ceiling.

The ensemble operator had a surprisingly large effect, as shown by the gain when it was changed from max (base version) to noisy-or. This suggests that combining weak signals across multiple scales can be quite beneficial. Our handcrafted gene-mutation filter also improved all systems substantially, corroborating the analysis of Wang and Poon (2018). In particular, without the filter, it is hard for the document-level models to achieve high precision, so they sacrifice a lot of recall to get good F1 scores. Using the filter helps them attain significantly higher recall while maintaining respectable precision.

Figure 3 shows the precision-recall curves for the four models (with noisy-or and gene-mutation filter). DocLevel has higher maximum recall than ParaLevel, but generally lower precision at the same recall level. By ensembling all three variants, MultiScale achieves the best combi

Figure 3: Precision-recall curves on CKB (with noisy-or and gene-mutation filter). MultiScale attained generally better precision than ParaLevel, and higher maximum recall like DocLevel.

 

nation: it generally improves precision while capturing more cross-paragraph relations. This can also be seen in Table 4, where we ablate each of the three variants used by MultiScale. All three variants in the ensemble contributed to overall performance.

We use \({\rm logsumexp}\) as the aggregation operator to combine mention-level representations into an entity-level one. If we replace it with max pooling, the performance drops substantially across the board, as shown in Table 5. For example, MultiScale lost 3.8 absolute points in AUC. Such difference is also observed in Verga et al. (2018). As in comparing ensemble operators, this demonstrates the benefit of combining weak signals using a multiscale representation.

### Cross-sentence and Cross-paragraph Extractions

Compared to standard sentence-level extraction, our method can extract relations among entities that never co-occur in the same sentence or even paragraph. Figure 4 shows the proportion of correctly predicted facts by MultiScale that are expressed across paragraph or sentence boundaries. MultiScale can substantially improve the recall by making additional cross-sentence and cross-paragraph extractions. We manually inspected twenty correct cross-paragraph extractions (with the chosen threshold for the precision/recall numbers in Table 3) and found that our model was able to handle some interesting linguistic phenomena. Often, a paper would first describe the mutations present in a patient cohort, and later describe the effects of drug treatment. There are also instances of bridging anaphora, for example via cell lines. One paper first stated the gene and mutation for a cell line "_The FLT3-inhibitor resistant cells Ba/F3-ITD+691_, _Ba/F3-ITD+842_, ..., which harbored FLT-ITD plus F69IL_, **Y842C**, ...mutations..."_, and later stated the drug effect on the cell line "**E6201** also demonstrated strong anti-proliferative effects in **FLT3**-inhibitor resistant cells....such as Ba/F3-ITD+691_, _Ba/F3-ITD+842_..."_.

### Subrelation Decomposition

As a baseline, we also consider a different document-level strategy where we decompose the \(n\)-ary relation into subrelations of lower arity, train independent classifiers for them, then join the subrelation predictions into one for the \(n\)-ary relation. We found that with distant supervision, the

Figure 4: Breakdown of MultiScale recall based on whether entities in a correctly extracted fact occurred within a single sentence, cross-sentence but within a single paragraph, or only cross-paragraph. Adding cross-sentence and cross-paragraph extractions is important for high recall.

\begin{table}
\begin{tabular}{|l|c|c|c c c|} \hline System & AUC & MR & P & R & F1 \\ \hline
**Base version** & & & & & \\ SentDuoMut & 31.0 & 40.8 & 60.0 & 40.7 & 48.5 \\ SentDrugGene & 17.9 & 64.2 & 31.4 & 27.9 & 29.5 \\ ParadrugMut & 39.9 & 57.7 & 49.3 & 50.3 & 49.8 \\ ParadrugGene & 19.9 & 68.9 & 32.1 & 18.9 & 23.8 \\ \hline + **Noisy-Or** & & & & & \\ SentDrugMut & 32.6 & 40.8 & 61.1 & 39.4 & 47.9 \\ SentDrugGene & 23.5 & 64.2 & 36.3 & 34.2 & 35.2 \\ ParadrugMut & 42.0 & 57.7 & 49.9 & 51.5 & 50.7 \\ ParadrugGene & 26.1 & 68.9 & 46.1 & 29.5 & 36.0 \\ \hline \end{tabular}
\end{table}
Table 6: Results of subrelation decomposition baselines on CKB, with the gene-mutation filter. MR=max recall, P=precision, R=recall.

 

gene-mutation subrelation classifier was too noisy. Therefore, we focused on training drug-gene and drug-mutation classifiers, and joined each with the rule-based gene-mutation predictions to make ternary predictions. Table 6 shows the results on CKB. The paragraph-level drug-mutation model is quite competitive, which benefits from the fact that the gene-mutation associations in a document are unique. This is not true in general \(n\)-ary relations. Still, it trails MultiScale by a large margin in predictive accuracy, and with an even larger gap in the potential upside (i.e., maximum recall). The drug-gene model has higher maximum recall, but much worse precision. This low precision is expected, as it is usually not valid to assume that if a drug and gene interact, then all possible mutations in the gene will have an effect on the drug response.

### Error Analysis

While much higher compared to other systems, the maximum recall for MultiScale is still far from 100%. For over 20% of the relations, we can't find all three entities in the document. In many cases, the missing entities are in figures or supplements, beyond the scope of our extraction. Some mutations are indirectly referenced by well-known cell lines. There are also remaining entity linking errors (e.g., due to missing drug synonyms).

We next manually analyzed some sample prediction errors. Among 50 false positive errors, we found a significant portion of them were actually true mentions in the paper but were excluded by curators due to additional curation criteria. For example, CKB does not curate a fact referenced in related work, or if they deem the empirical evidence as insufficient. This suggests the need for even higher-order relation extraction to cover these aspects. We also inspected 50 sample false negative errors. In 40% of the cases, the textual evidence is vague and requires corroboration from a table or figure. In most of the remaining cases, there is direct textual evidence, though they require cross-paragraph reasoning (e.g., bridging anaphora). While MultiScale was able to process such phenomena sometimes, there is clearly much room to improve.

## 5 Related Work

\(N\)-ary relation extractionPrior work on \(n\)-ary relation extraction generally follows Davidson semantics by reducing the \(n\)-ary relation to \(n\) binary relations between the reified relation and its arguments, a.k.a. _slot filling_. For example, early work on the Message Understanding Conference (MUC) dataset aims to identify event participants in news articles Chinchor (1998). More recently, there has been much work in extracting semantic roles for verbs, as in semantic role labeling Palmer et al. (2010), as well as properties for popular entities, as in Wikipedia Infobox Wu and Weld (2007) and TAC KBP9. In biomedicine, the BioNLP Event Extraction Shared Task aims to extract genetic events such as expression and regulation Kim et al. (2009). These approaches typically assume that the whole document refers to a single coherent event, or require an event anchor (e.g., verb in semantic role labeling and trigger word in event extraction). We instead follow recent work in cross-sentence \(n\)-ary relation extraction Peng et al. (2017); Wang and Poon (2018); Song et al. (2018), which does not have these restrictions.

Footnote 9: [http://www.nist.gov/tac/2016/KBP/ColdStart/index.html](http://www.nist.gov/tac/2016/KBP/ColdStart/index.html)

Document-level relation extractionMost information extraction work focuses on modeling and prediction within sentences Surdeanu and Ji (2014). Duan et al. (2017) introduces a pre-trained document embedding to aid event detection, but their extraction is still at the sentence level. Past work on cross-sentence extraction often relies on explicit coreference annotations or the assumption of a single event in the document Wick et al. (2006); Gerber and Chai (2010); Swampilali and Stevenson (2011); Yoshikawa et al. (2011); Koch et al. (2014); Yang and Mitchell (2016). Recently, there has been increasing interest in general cross-sentence relation extraction Quirk and Poon (2017); Peng et al. (2017); Wang and Poon (2018), but their scope is still limited to short text spans of a few consecutive sentences. These methods all extract relations at the mention level, which does not scale to whole documents due to the combinatorial explosion of relation candidates. Wu et al. (2018) applies manually crafted rules to heavily filter the candidates. We instead adopt an entity-centric approach and combine mention-level representations to create an entity-level representation for extraction. Mintz et al. (2009) aggregates mention-level features into entity-level ones within a docu 

binary relations within single sentences. Kilicoglu (2016) used hand-crafted features to improve cross-sentence extraction, but they focus on binary relations, and their documents are limited to abstracts, which are substantially shorter than the full-text articles we consider. Verga et al. (2018) applies self-attention to combine the representations of all mention pairs into an entity pair representation, which can be viewed a special case of our framework. Their work is also limited to binary relations and abstracts, rather than full documents.

Multiscale modelingDeep learning on long sequences can benefit from multiscale modeling that accounts for varying scales in the discourse structure. Prior work focuses on generative learning such as language modeling Chung et al. (2017). We instead apply multiscale modeling to discriminative learning for relation extraction. In addition to modeling various scales of discourse units (sentence, paragraph, document), we also combine mention-level representations into an entity-level one, as well as sub-relations of the \(n\)-ary relation. McDonald et al. (2005) learn \({n\choose 2}\) pairwise relation classifiers, then construct maximal cliques of related entities, which also bears resemblance to our subrelation modeling. However, our approach incorporates the entire subrelation hierarchy, provides a principled end-to-end learning framework, and extracts relations from the whole document rather than within single sentences.

Distant supervisionDistant supervision has emerged as a powerful paradigm to generate large but potentially noisy labeled datasets Craven et al. (1999); Mintz et al. (2009). A common denoising strategy applies multi-instance learning by treating mention-level labels as latent variables Hoffmann et al. (2011). Noise from distant supervision increases as extraction scope expands beyond single sentences, motivating a variety of indirect supervision approaches Quirk and Poon (2017); Peng et al. (2017); Wang and Poon (2018). Our entity-centric representation and multiscale modeling provide an orthogonal approach to combat noise by combining weak signals spanning various text spans and subtelations.

## 6 Conclusion

We propose a multiscale, entity-centric approach for document-level \(n\)-ary relation extraction. We vastly increase maximum recall by scoring document-level candidates. Meanwhile, we preserve precision with a multiscale approach that combines representations learned across the subrelation hierarchy and text spans of various scales. Our method substantially outperforms prior cross-sentence \(n\)-ary relation extraction approaches in the high-value domain of precision oncology.

Our document-level view opens opportunities for multimodal learning by integrating information from tables and figures Wu et al. (2018). We used the ternary drug-gene-mutation relation as a running example in this paper, but knowledge bases often store additional fields such as effect (sensitive or resistance), cancer type (solid tumor or leukemia), and evidence (human trial or cell line experiment). It is straightforward to apply our method to such higher-order relations. Finally, it will be interesting to validate our approach in a real-world assisted-curation setting, where a machine reading system proposes candidate facts to be verified by human curators.

## Acknowledgements

We thank Sara Patterson and Susan Mockus for guidance on precision oncology knowledge curation and CKB data, Hai Wang for help in running experiments with deep probabilistic logic, and Tristan Naumann, Rajesh Rao, Peng Qi, John Hewitt, and the anonymous reviewers for their helpful comments. R.J. is supported in part by an NSF Graduate Research Fellowship under Grant No. DGE-114747.

## References

* Bahcall (2015) Orli Bahcall. 2015. Precision medicine. _Nature_, 526:335.
* Chakravarty et al. (2018) Debyani Chakravarty, Jianjiong Gao, Sarah Phillips, Ritika Kundra, Hongxin Zhang, Jiaojiao Wang, Julia E. Rudolph, Rona Yaeger, Tara Soumerai, Moriah H. Nissan, Matthew T. Chang, Sarat Chandarlapaty, Tiffany A. Traina, Paul K. Paik, Alan L. Ho, Feras M. Hannash, Andrew Grupe, Shrujal S. Baxi, Margaret K. Callahan, Alexandra Snyder, Ping Chi, Daniel C. Danila, Mrinal Gonundez, James J. Harding, Matthew D. Hellmann, Gopa Iyer, Yelena Y. Janijgian, Thomas Kaley, Douglas A. Levine, Maeye Lowery, Antonio Omuro, Michael A. Postow, Dana Rathkopf, Alexander N. Shoushtari, Neerav Shukla, Martin H. Voss, Gerlinda Paraiso, Ahmet Zehir, Michael F. Berger, Barry S. Taylor, Leonard B. Saltz, Gregory J. Riely, Marc Ladanyi, David M. Hyman, Jos Baselga, Paul Sabbatini, David B. Solit, 

and Nikolaus Schultz. 2017. Oncokb: A precision oncology knowledge base. _JCO Precision Oncology_.
* Chinchor (1998) Nancy Chinchor. 1998. Overview of MUC-7/MET-2. Technical report, Science Applications International Corporation, San Diego, CA.
* Chung et al. (2017) Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2017. Hierarchical multiscale recurrent neural networks. In _ICLR_.
* Craven et al. (1999) M. Craven, J. Kumlien, et al. 1999. Constructing biological knowledge bases by extracting information from text sources. In _ISMB_, pages 77-86.
* Dienstmann et al. (2015) Rodrigo Dienstmann, In Sock Jang, Brian Bot, Stephen Friend, and Justin Guinney. 2015. Database of genomic biomarkers for cancer drugs and clinical targetability in solid tumors. _Cancer Discovery_, 5.
* Duan et al. (2017) Shaoyang Duan, Ruifang He, and Wenli Zhao. 2017. Exploiting document level information to improve event detection via recurrent neural networks. In _IJCNLP_.
* Gerber and Chai (2010) Matthew Gerber and Joyce Y. Chai. 2010. Beyond NomBank: A study of implicit arguments for nominal predicates. In _Proceedings of the Forty-Eighth Annual Meeting of the Association for Computational Linguistics_.
* Hoffmann et al. (2011) R. Hoffmann, C. Zhang, X. Ling, L. S. Zettlemoyer, and D. S. Weld. 2011. Knowledge-based weak supervision for information extraction of overlapping relations. In _Association for Computational Linguistics (ACL)_, pages 541-550.
* Kilicoglu (2016) Halil Kilicoglu. 2016. Inferring implicit causal relationships in biomedical literature. In _Proceedings of the 15th Workshop on Biomedical Natural Language Processing_.
* Kim et al. (2009) Jin-Dong Kim, Tomoko Ohta, Sampo Pyysalo, Yoshinobu Kano, and Junichi Tsujii. 2009. Overview of bionlp-09 shared task on event extraction. In _Proceedings of the BioNLP Workshop 2009_.
* Kingma and Ba (2014) D. Kingma and J. Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Koch et al. (2014) Mitchell Koch, John Gilmer, Stephen Soderland, and S. Daniel Weld. 2014. Type-aware distantly supervised relation extraction with linked arguments. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1891-1901. Association for Computational Linguistics.
* McDonald et al. (2005) Ryan McDonald, Fernando Pereira, Seth Kulick, Scott Winters, Yang Jin, and Pete White. 2005. Simple algorithms for complex relation extraction with applications to biomedical IE. In _Proceedings of the Forty-Third Annual Meeting on Association for Computational Linguistics_.
* Mintz et al. (2009) M. Mintz, S. Bills, R. Snow, and D. Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In _Association for Computational Linguistics (ACL)_, pages 1003-1011.
* Palmer et al. (2010) Martha Palmer, Daniel Gildea, and Nianwen Xue. 2010. Semantic role labeling. _Synthesis Lectures on Human Language Technologies_, 3(1).
* Patterson et al. (2016) Sarah E. Patterson, Rangjiao Liu, Cara M. Statz, Daniel Durkin, Anuradha Lakshminarayana, and Susan M. Mockus. 2016. The clinical trial landscape in oncology and connectivity of somatic mutational profiles to targeted therapies. _Human Genomics_, 10.
* Peng et al. (2017) Nanyun Peng, Hoifung Poon, Chris Quirk, and Kristina Toutanova Wen tau Yih. 2017. Cross-sentence \(N\)-ary relation extraction with graph LSTMs. _Transactions of the Association for Computational Linguistics_, 5:101-115.
* Pyssalo et al. (2013) Sampo Pyssalo, F Gunter, Hans Moen, T Salakoski, and Sophia Ananiadou. 2013. Distributional semantics resources for biomedical text processing. In _Proceedings of Languages in Biology and Medicine_.
* Quirk and Poon (2017) Chris Quirk and Hoifung Poon. 2017. Distant supervision for relation extraction beyond the sentence boundary. In _Proceedings of the Fifteenth Conference on European chapter of the Association for Computational Linguistics_.
* Song et al. (2018) Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. \(N\)-ary relation extraction using graph state LSTM. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Surdaanu and Ji (2014) Mihai Surdaanu and Heng Ji. 2014. Overview of the English Slot Folling Track at the TAC2014 Knowledge Base Population evaluation. In _Proceedings of the TAC-KBP 2014 Workshop_, pages 1-15, Gaithersburg, Maryland, USA.
* Swampillali and Stevenson (2011) Kumutha Swampillali and Mark Stevenson. 2011. Extracting relations within and across sentences. In _Proceedings of the Conference on Recent Advances in Natural Language Processing_.
* Vaswani et al. (2017) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. 2017. Attention is all you need. _arXiv preprint arXiv:1706.03762_.
* Verga et al. (2018) Patrick Verga, Emma Strubell, and Andrew McCallum. 2018. Simultaneously self-attending to all mentions for full-abstract biological relation extraction. In _NAACL_.
* Wang and Poon (2018) Hai Wang and Hoifung Poon. 2018. Deep probabilistic logic: A unifying framework for indirect supervision. In _EMNLP_.
* Wick et al. (2006) Michael Wick, Aron Culotta, and Andrew McCallum. 2006. Learning field compatibilities to extract 

database records from unstructured text. In _Proceedings of the Conference on Empirical Methods in Natural Language Processing_.
* Wu and Weld (2007) Fei Wu and Daniel S. Weld. 2007. Autonomously seam-mentifying wikipedia. In _Proceedings of the Sixteenth ACM Conference on Conference on Information and Knowledge Management_.
* Wu et al. (2018a) S. Wu, L. Hsiao, X. Cheng, B. Hancock, T. Rekatsinas, P. Levis, and C. R'e. 2018a. Fonduer: Knowledge base construction from richly formatted data. In _Proceedings of SIGMOD 2018_.
* Wu et al. (2018b) Sen Wu, Luke Hsiao, Xiao Cheng, Braden Hancock, Theodoros Rekatsinas, Philip Levis, and Christopher R. 2018b. Fonduer: Knowledge base construction from richly formatted data. In _SIGMOD_.
* Yang and Mitchell (2016) Bishan Yang and Tom Mitchell. 2016. Joint extraction of events and entities within a document context. In _NAACL_.
* Yoshikawa et al. (2011) Katsumasa Yoshikawa, Sebastian Riedel, Tsutomu Hirao, Masayuki Asahara, and Yuji Matsumoto. 2011. Coreference based event-argument relation extraction on biomedical text. _Journal of Biomedical Semantics_, 2(5).

## Appendix A Appendices

### Preprocessing

Full-text documents in this study were obtained from PMC. The text was first tokenized using NLTK10, then entities were extracted using a combination of regular expressions and dictionary lookups. To identify mutation mentions, we applied a regular expression rule for missense mutations. To identify gene mentions, we used dictionary lookup from the HUGO Gene Nomenclature Committee (HGNC)11 dataset. To identify drug mentions, we used dictionary lookup from a curated list of drugs and their synonyms. For our training set, our list of drugs consists of all the drugs present in the distant supervision knowledge bases and selected cancer-related drugs from DrugBank12 (770 drugs total). For our test set, our drug dictionary consists of all the drugs in CKB (1119 drugs).

Footnote 10: [https://www.nltk.org/](https://www.nltk.org/)

Footnote 11: [https://www.genemanes.org/](https://www.genemanes.org/)

Footnote 12: [https://www.drugbank.ca/](https://www.drugbank.ca/)

### Gene-mutation Rule-based System

Here we describe our rule-based system for linking mutations and genes within a document. We first generate a global mapping of mutations to sets of genes by combining publicly-available mutation-gene datasets (COSMIC13, COSMIC Cell Lines Project14, CIViC15, and OncoKB16). We then augment this mapping by finding the gene that most frequently co-occurs with each mutation in all of PubMed Central (PMC) full-text articles based on three high-precision rules:

Footnote 13: [https://cancer.sanger.ac.uk/cosmic](https://cancer.sanger.ac.uk/cosmic)

Footnote 14: [https://cancer.sanger.ac.uk/cell_lines](https://cancer.sanger.ac.uk/cell_lines)

Footnote 15: [https://civicdb.org](https://civicdb.org)

Footnote 16: [http://oncockb.org/](http://oncockb.org/)

1. Gene and mutation are in the same token (e.g., "EGFR-T790M")
2. Gene token is followed by mutation token (e.g., "EGFR T790M")
3. Gene token is followed by a token of any single character and then followed by mutation token (e.g., "EGFR - T790M")

For each mutation, we start with the first rule, and find all text matches for a gene with that mutation and rule. If we found at least one match, we add the gene that occurred in the most matches to the global map. Otherwise, we repeat with the next rule.

Each mutation in the global mutation-gene map is mapped to more than \(20\) genes on average. However, within the context of a document, each mutation is (usually) associated with just a single gene. Given a document containing a mutation, we associate that mutation with the gene that (1) is in the global mutation-gene map for that mutation, and (2) appears closest to any mutation mention in the document.

To associate genes for the remaining mutations, we apply two recall-friendly regular expression rules within that document:

1. Mutation is in same sentence as "_GENE_ mut"
2. Mutation is in same paragraph as "_GENE_ mutation"

We choose the first gene in the document that satisfies one of the two rules, in the above order. If there is still no matching gene at this point, the most frequent gene in the document is selected for that mutation.

### Model Details

We used \(200\)-dimensional word vectors, initialized with word2vec vectors trained on a biomedical text corpus Pyysalo et al. (2013). We updated 

these vectors during training. At each step, our BiLSTM received as input a concatenation of the word vectors and a \(100\)-dimensional embedding of the index of the current discourse unit within the document. Following Vaswani et al. (2017), we used sinusoidal embeddings to represent these indices. We used a single-layer bidirectional LSTM with a \(200\)-dimensional hidden state. Mention-level representations were \(400\)-dimensional and computed from BiLSTM hidden states using a single linear layer followed by the \(\tanh\) activation function. For the final prediction layer, we used a two-layer feedforward network with \(400\) hidden units and ReLU activation function. We train using the Adam optimizer Kingma and Ba (2014) with learning rate of \(1\times 10^{-5}\). During training, we consider each document to be a single batch, which allows us to reuse computation for different relation candidates in the same document.