SlowGuess/NAACL_2022_OCR huggingface dataset:
We further improve RE after training the pipeline  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$  by augmenting the intermediate steps in  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$  with the retrieved evidence from FER.
4.1 When to Augment Intermediate Steps
The evidence predicted by FER is unique to each triplet  $(e_h, e_t, r)$ . However, consider the total number of all possible triplets (around 40 million in the develop set of DocRED), it is computationally prohibitive to augment the inference result of each triplet with individually predicted evidence. Instead, following the idea of selective prediction (El-Yaniv et al., 2010), we identify the triplet subset  $\mathcal{U}$  for which the model is uncertain about its relation predictions with the original pipeline SAIS $_{\text{All}}^{\text{O}}$ . More specifically, we set the model's confidence for  $(e_h, e_t, r)$  as  $\mathbb{L}_{h,t,r}^{\text{O}} = \mathbb{L}_{h,t,r}^{\text{RE}} - \mathbb{L}_{h,t,\text{TH}}^{\text{RE}}$ . Then, the uncertain set  $\mathcal{U}$  consists of triplets with the lowest  $\theta\%$  absolute confidence  $|\mathbb{L}_{h,t,r}^{\text{O}}|$ . Consequently, we reject the original relation predictions for  $(e_h, e_t, r) \in \mathcal{U}$  and apply evidence-based data augmentation to enhance the performance (more details in Section 4.2).
To determine the rejection rate  $\theta \%$  (note that  $\theta \%$  is NOT a hyperparameter), we first sort all the triplets in the develop set based on their absolute confidence  $|\mathbb{L}_{h,t,r}^{\mathrm{O}}|$ . When  $\theta \%$  increases, the risk (i.e., inaccuracy rate) of the remaining triplets that are not in  $\mathcal{U}$  is expected to decrease, and vice versa. On the one hand, we wish to reduce the risk for more accurate relation predictions; on the other hand, we want a low rejection rate so that data augmentation on a small rejected set incurs little computational cost. To balance this trade-off, we set  $\theta \%$  as the rate that achieves the minimum of  $\mathrm{risk}^2 +\mathrm{rejection rate}^2$ . As shown in Figure 2, we find  $\theta \% \approx 4.6\%$  in the develop set of DocRED. In practice, we can further limit the maximum number of rejected triplets per entity pair. By setting it as
Figure 2: Trade-off between risk and rejection rate on the develop set of DocRED.
10 in experiments, we reduce the size of  $\mathcal{U}$  to only  $1.5\%$  of all the triplets in the DocRED develop set.
4.2 How to Augment Intermediate Steps
Consider a triplet  $(e_h, e_t, r) \in \mathcal{U}$ . We first assume its validity and calculate the probability  $\mathbb{P}_{h,t,r,s}^{\mathrm{FER}}$  of a sentence  $s$  being part of  $\mathcal{V}_{h,t,r}$  based on Section 3.5. Then in a similar way to how  $\mathbb{L}_{h,t,r}^{\mathrm{O}}$  is generated with  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$ , we design two types of evidence-based data augmentation as follows:
Pseudo Document-based  $(\mathbf{SAIS}_{\mathrm{All}}^{\mathrm{D}})$ : Construct a pseudo document using sentences with  $\mathbb{P}_{h,t,r,s}^{\mathrm{FER}} > \alpha^{\mathrm{FER}}$  and feed it into the original pipeline to get the confidence  $\mathbb{L}_{h,t,r}^{\mathrm{D}}$ .
Attention Mask-based (SAIS $_{\mathbf{All}}^{\mathbf{M}}$ ): Formulate a mask  $\mathbf{P}_{h,t,r}^{\mathrm{FER}} \in \mathbb{R}^{N_d}$  based on  $\mathbb{P}_{h,t,r,s}^{\mathrm{FER}}$  and modify the context embedding to  $\mathbf{c}_{h,t} = \mathbf{H}^\top \frac{\mathbf{A}_h \otimes \mathbf{A}_t \otimes \mathbf{P}_{h,t,r}^{\mathrm{FER}}}{\mathbf{1}^\top (\mathbf{A}_h \otimes \mathbf{A}_t \otimes \mathbf{P}_{h,t,r}^{\mathrm{FER}})}$ . Maintain the rest of the pipeline and get the confidence  $\mathbb{L}_{h,t,r}^{\mathrm{M}}$ .
Following Xie et al. (2022), we ensemble  $\mathbb{L}_{h,t,r}^{\mathrm{D}}$ ,  $\mathbb{L}_{h,t,r}^{\mathrm{M}}$ , and the original confidence  $\mathbb{L}_{h,t,r}^{\mathrm{O}}$  with a blending parameter  $\tau_r \in \mathbb{R}$  (Wolpert, 1992) for each relation  $r \in \mathcal{R}$  as
$$
\begin{array}{l} \mathbb {P} _ {h, t, r} ^ {\mathrm {B}} = \sigma (\mathbb {L} _ {h, t, r} ^ {\mathrm {B}}) \\ = \sigma \left(\mathbb {L} _ {h, t, r} ^ {\mathrm {O}} + \mathbb {L} _ {h, t, r} ^ {\mathrm {D}} + \mathbb {L} _ {h, t, r} ^ {\mathrm {M}} - \tau_ {r}\right). \tag {18} \\ \end{array}
$$
These parameters are trained by minimizing the binary cross-entropy loss on  $\mathcal{U}$  of the develop set:
$$
\begin{array}{l} \ell^ {B} = - \sum_ {\left(e _ {h}, e _ {t}, r\right) \in \mathcal {U}} \left(y _ {h, t, r} ^ {\mathrm {R E}} \log \mathbb {P} _ {h, t, r} ^ {\mathrm {B}} \right. \\ \left. + \left(1 - y _ {h, t, r} ^ {\mathrm {R E}}\right) \log \left(1 - \mathbb {P} _ {h, t, r} ^ {\mathrm {B}}\right)\right), \tag {19} \\ \end{array}
$$
where  $y_{h,t,r}^{\mathrm{RE}} = 1$  if  $(e_h,e_t,r)$  is valid, and 0 otherwise. When making relation predictions for
$(e_h, e_t, r) \in \mathcal{U}$ , we check whether its blended confidence is positive (i.e.,  $\mathbb{L}_{h,t,r}^{\mathrm{B}} > 0$ ).
In this way, we improve the RE performance when the model is uncertain about its original predictions and save the computational cost when the model is confident. The overall steps for evidence-based data augmentation and ensemble inference  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{B}}$  are summarized in Appendix B. These steps are executed only after the training of  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$  and, therefore, adds negligible computational cost.
We evaluate the proposed SAIS method on the following three document-level RE benchmarks. DocRED (Yao et al., 2019) is a large-scale crowdsourced dataset based on Wikipedia articles. It consists of 97 relation types, seven entity types, and 5,053 documents in total, where each document has 19.5 entities on average. CDR (Li et al., 2016) and GDA (Wu et al., 2019) are two biomedical datasets where CDR studies the binary interactions between disease and chemical concepts with 1,500 documents and GDA studies the binary relationships between gene and disease with 30,192 documents. We follow Christopoulou et al. (2019) for splitting the train and develop sets.
We run our experiments on one Tesla A6000 GPU and carry out five trials with different seeds to report the mean and one standard error. Based on Huggingface (Wolf et al., 2019), we apply cased BERT-base (Devlin et al., 2019) and RoBERTa-large (Liu et al., 2019) for DocRED and cased SciBERT (Beltagy et al., 2019) for CDR and GDA. The embedding dimension  $H$  of BERT or SciBERT is 768, and that of RoBERTa is 1,024. The number of groups  $K$  in all group bilinear layers is 64.
For the general hyperparameters of language models, we follow the setting in (Zhou et al., 2021). The learning rate for fine-tuning BERT is  $5\mathrm{e} - 5$  that for fine-tuning RoBERTa or SciBERT is  $2\mathrm{e} - 5$  and that for training the other parameters is  $1\mathrm{e} - 4$ . All the trials are optimized by AdamW (Loshchilov and Hutter, 2019) for 20 epochs with early stopping and a linearly decaying scheduler (Goyal et al., 2017) whose warm-up ratio  $= 6\%$ . Each batch contains 4 documents and the gradients of model parameters are clipped to a maximum norm of 1.
For the unique hyperparameters of our method, we choose 2 from  $\{1,1.5,2\}$  for the focal hyperparameters  $\gamma^{\mathrm{CR}}$  and  $\gamma^{\mathrm{PER}}$  based on the develop
Model	DocRED Dev	DocRED Test
Relation	Evidence	Relation	Evidence
Ign F1	F1	F1	Ign F1	F1	F1
HeterGSAN-BERTbase (Xu et al., 2021d)	58.13	60.18	-	57.12	59.45	-
GAIN-BERTbase (Zeng et al., 2020)	59.14	61.22	-	59.00	61.24	-
DRN-BERTbase (Xu et al., 2021c)	59.33	61.39	-	59.15	61.37	-
SIRE-BERTbase (Zeng et al., 2021)	59.82	61.60	-	60.18	62.05	-
BERTbase (Wang et al., 2019)	-	54.16	-	-	53.20	-
E2GRE-BERTbase (Huang et al., 2020)	55.22	58.72	47.14	-	-	-
SSAN-BERTbase (Xu et al., 2021a)	57.03	59.19	-	56.06	58.41	-
ATLOP-BERTbase (Zhou et al., 2021)	59.22	61.09	-	59.31	61.30	-
DocuNet-BERTbase (Zhang et al., 2021)	59.86	61.83	-	59.93	61.86	-
Eider-BERTbase (Xie et al., 2022)	60.51	62.48	50.71	60.42	62.47	51.27
SAISBAll-BERTbase (Ours)	59.98 ± 0.13	62.96 ± 0.11	53.70 ± 0.21	60.96	62.77	52.88
RoBERTlarge (Ye et al., 2020)	57.19	59.40	-	57.74	60.06	-
SSAN-RoBERTlarge (Xu et al., 2021a)	60.25	62.08	-	59.47	61.42	-
E2GRE-RoBERTlarge (Huang et al., 2020)	-	-	-	60.30	62.50	50.50
ATLOP-RoBERTlarge (Zhou et al., 2021)	61.32	63.18	-	61.39	63.40	-
DocuNet-RoBERTlarge (Zhang et al., 2021)	62.23	64.12	-	62.39	64.55	-
Eider-RoBERTlarge (Xie et al., 2022)	62.34	64.27	52.54	62.85	64.79	53.01
SAISBAll-RoBERTlarge (Ours)	62.23 ± 0.15	65.17 ± 0.08	55.84 ± 0.23	63.44	65.11	55.67
Table 1: RE and ER results (\%) on DocRED. Ign F1 refers to the F1 score excluding the relation instances mentioned in the train set. Baselines using  $\mathrm{BERT}_{\mathrm{base}}$  are separated into the graph-based (upper) and transformer-based (lower) groups. We report the test scores from the official scoreboard and the baseline scores from the corresponding papers.  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{B}}$  achieves state-of-the-art performance on both RE and ER. Full details in Appendix C.
set. We also follow Xie et al. (2022) for setting the FER prediction threshold  $\alpha^{\mathrm{FER}}$  as 0.5 and all the relative task weights  $\eta^{\mathrm{Task}}$  for Task  $\in$  {CR, ET, PER, FER} as 0.1.
Besides RE, DocRED also suggests to predict the supporting evidence for each relation instance. Therefore, we apply  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{B}}$  to both RE and ER. We report the results of  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{B}}$  as well as existing graph-based and transformer-based baselines in Table  $1^{2}$  (full details in Appendix C). Generally, thanks to PLMs' strength in modeling long-range dependencies, transformer-based methods perform better on RE than graph-based methods. Moreover, most earlier approaches are not capable of ER despite the interpretability ER adds to the predictions. In contrast, our  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{B}}$  method not only establishes a new state-of-the-art result on RE, but also outperforms the runner-up significantly on ER.
Since neither CDR nor GDA annotates evidence sentences, we apply  $\mathrm{SAIS}_{\mathrm{RE} + \mathrm{CR} + \mathrm{ET}}^{\mathrm{O}}$  here. It is
trained with RE, CR, and ET and infers without data augmentation. As shown in Table 2 (full details in Appendix C), our method improves the prior best RE F1 scores by  $2.7\%$  and  $1.8\%$  absolutely on CDR and GDA, respectively. It indicates that our proposed method can still improve upon the baselines even if only part of the four complementary tasks are annotated and operational.
To investigate the effectiveness of each of the four complementary tasks proposed in Section 3, we carry out an extensive ablation study on the DocRED develop set by training SAIS with all possible combinations of those tasks. As shown in Table 3, without any complementary tasks, the RE performance of SAIS is comparable to ATLOP (Zhou et al., 2021) due to similar neural architectures. When only one complementary task is allowed, PER is the most effective single task, followed by ET. Although FER is functionally analogous to PER, since FER only involves the small subset of entity pairs with valid relations, the performance gain brought by FER alone is limited. When
2For a fair comparison, we report the scores of SSAN (Xu et al., 2021a) without being pretrained on an extra dataset.
Model	CDR	GDA
LSR (Nan et al., 2020)	64.8	82.2
SciBERT (Beltagy et al., 2019)	65.1	82.5
DHG (Zhang et al., 2020)	65.9	83.1
SSAN-SciBERT (Xu et al., 2021a)	68.7	83.7
ATLOP-SciBERT (Zhou et al., 2021)	69.4	83.9
SIRE-BioBERT (Zeng et al., 2021)	70.8	84.7
DocuNet-SciBERT (Zhang et al., 2021)	76.3	85.3
SAIS0RE+CR+ET-SciBERT (Ours)	79.0 ± 0.8	87.1 ± 0.3
SAIS0RE+ET-SciBERT	75.9 ± 0.9	86.1 ± 0.5
SAIS0RE+CR-SciBERT	74.5 ± 0.4	85.4 ± 0.2
SAIS0RE-SciBERT	72.8 ± 0.6	84.5 ± 0.3
two tasks are used jointly, the pair of PER and ET, which combines textual contexts and entity type information, delivers the most significant improvement. The pair of PER and FER also performs well, which reflects the finding in (Peng et al., 2020) that context is the most important source of information. The version with all tasks except CR sees the least drop in F1, indicating that CR's supervision signals on capturing contexts can be covered in part by PER and FER. Last but not least, the SAIS pipeline with all four complementary tasks achieves the highest F1 score. Similar trends are also recognized on CDR and GDA in Table 2, where SAIS trained with both CR and ET (besides RE) scores higher than its single-task counterpart.
Moreover, as compared to the original pipeline  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$ , pseudo document-based data augmentation  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{D}}$  acts as a hard filter by directly removing predicted non-evidence sentences, while attention mask-based data augmentation  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{M}}$  distills the context more softly. Therefore, we observe in Table 4 that  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{D}}$  earns a higher precision, whereas  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{M}}$  attains a higher recall. By ensembling  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$ ,  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{D}}$ , and  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{M}}$ , we improve the RE F1 score by  $0.57\%$  absolutely on the DocRED develop set.
To obtain a more insightful understanding of how textual contexts and entity type information help with RE, we present a case study in Figure 3 (a). Here,  $\mathrm{SAIS}_{\mathrm{RE} + \mathrm{ET}}^{\mathrm{O}}$  is trained with the task (i.e., ET) related to entity type information while  $\mathrm{SAIS}_{\mathrm{RE} + \mathrm{CR} + \mathrm{PER} + \mathrm{FER}}^{\mathrm{O}}$  is trained with the tasks (i.e., CR, PER, and FER) related to textual contexts.
Table 2: RE F1 results  $(\%)$  on the CDR and GDA test sets. The baseline scores are from the corresponding papers.  $\mathrm{SAIS}_{\mathrm{RE} + \mathrm{CR} + \mathrm{ET}}^{\mathrm{O}}$  scores the highest on both datasets. Full details in Appendix C.
CR	ET	PER	FER	RE	F1
✓	61.18 ± 0.09
✓	✓	61.41 ± 0.11
✓	✓	61.52 ± 0.10
✓	✓	61.68 ± 0.04
✓	✓	61.44 ± 0.07
✓	✓	✓	61.65 ± 0.12
✓	✓	✓	61.79 ± 0.08
✓	✓	✓	61.64 ± 0.10
✓	✓	✓	61.88 ± 0.05
✓	✓	✓	61.81 ± 0.04
✓	✓	✓	61.85 ± 0.10
✓	✓	✓	✓	62.13 ± 0.04
✓	✓	✓	✓	62.06 ± 0.09
✓	✓	✓	✓	61.91 ± 0.06
✓	✓	✓	✓	61.98 ± 0.05
✓	✓	✓	✓	✓	62.39 ± 0.08
Table 3: Ablation study  $(\%)$  using SAIS $^0$ -BERT $_{\text{base}}$  to assess the effectiveness of the four complementary tasks (i.e., CR, ET, PER, and FER) for RE based on the DocRED develop set.
SAIS0All	SAISDAll	SAISMAll	Precision	Recall	F1
✓	66.58	58.70	62.39
✓	73.21	45.59	56.19
✓	53.14	67.49	59.46
✓	✓	71.14	54.35	61.62
✓	✓	61.61	62.90	62.25
✓	✓	✓	67.76	58.79	62.96
Table 4: Ablation study  $(\%)$  using BERTbase to assess the effectiveness of data augmentation (i.e., original  $\mathrm{(SAIS_{All}^{O})}$ , pseudo document-based  $\mathrm{(SAIS_{All}^{D})}$ , and attention mask-based  $\mathrm{(SAIS_{All}^{M})}$ ) for RE based on the DocRED develop set.
Compared to  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$ , which is trained with all four complementary tasks, they both exhibit drawbacks qualitatively. In particular,  $\mathrm{SAIS}_{\mathrm{RE} + \mathrm{ET}}^{\mathrm{O}}$  can easily infer the relation "country" between Entities E and C based on their respective types "ORG" and "LOC", whereas  $\mathrm{SAIS}_{\mathrm{RE} + \mathrm{CR} + \mathrm{PER} + \mathrm{FER}}^{\mathrm{O}}$  may misinterpret Entity E as of type "PER" and infer the relation "citizenship" wrongly. On the other hand,  $\mathrm{SAIS}_{\mathrm{RE} + \mathrm{CR} + \mathrm{PER} + \mathrm{FER}}^{\mathrm{O}}$  can directly predict the relation "place_of_birth" between Entities A and B by pattern matching, while overemphasizing the type "LOC" of Entity B may cause  $\mathrm{SAIS}_{\mathrm{RE} + \mathrm{ET}}^{\mathrm{O}}$  to deliver the wrong relation prediction "location". Last but
(a) Case Study on the Effectiveness of Textual Contexts and Entity Type Information:
Document: Eleazar Lipsky
[0] Eleazar Lipsky ... was a ... playwright born in Bronx, ..., United States.
[1] He wrote the novels that formed the basis of two ... films, Kiss of Death ... and The People Against O'Hara ...
[3] Lipsky, ..., was an assistant district attorney ... and served as legal counsel to the Mystery Writers of America.
Entity A (PER): Eleazar Lipsky, Lipsky Entity D (MISC): The People Against O
Entity B (LOC): Bronx Entity C (LOC): United States Entity E (ORG): Mystery Writers of America
Entity Pair:  $(\mathbb{E},\underline{\mathbb{C}})$
Relation: Truth : country SAIS<sub>0</sub> _ {All} : country
Entity Pair: (A, B)
Relation: Truth : place_of_birth SAIS<sub>All</sub>: place_of_birth
Entity Pair:  $(\mathbb{D},\mathbb{C})$
Relation: Truth : country_of_origin SAIS<sub>0</sub> All: country_of_origin
SAISRE+ET country SAISRE+CR+PER+FEER: citizens
SAISO: location  $\mathrm{SAISO}_{\mathrm{RE} + \mathrm{CR} + \mathrm{PER} + \mathrm{FER}}$  : place_of
SAIS $_{\text{RE+ET}}^0$  : no Relation  
SAIS $_{\text{PE:CR:PER:EEP}}^0$  : no Relation
(b) Case Study on the Difference between FER and PER:
Document: Carl Buchheimer
[0] Carl Buchheimer ... was a German
constructivist artist ...
[1] which he began in 1925.
[2] He was born in Hanover, Germany.
[6] He died in Hanover in 1964.
Entity A (PER): Carl Buchheimer
Entity B (LOC): Hanover, Hanover
Relation: place_of_birth
Evidence: Truth: [0, 2]
Entity Pair:  $(\underline{\Delta},\underline{\mathsf{B}})$
Relation: place_of_death
Evidence: Truth: [0, 6]
FER: [0, 2]
PER: [0, 1, 2, 6]
FER: [0, 6]
Figure 3: (a) Case study on the effectiveness of textual contexts and entity type information based on models' extracted relations from the DocRED develop set. By capturing contexts across sentences and regularizing them with entity type information,  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$  extracts relations of better quality. (b) Case study on the difference between FER and PER based on retrieved evidence from the DocRED develop set. FER considers evidence unique to each relation for better interpretability. Irrelevant sentences are omitted here.
not least,  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$  effectively models contexts spanning multiple sentences and regularizes them with entity type information. As a result, it is the only SAIS variant that correctly predicts the relation "country_of_origin" between Entities D and C.
Furthermore, to examine why SAIS (which uses FER for retrieving evidence) outperforms Eider (Xie et al., 2022) (which uses PER) significantly on ER in Table 1, we compare the performance of FER and PER based on a case study in Figure 3 (b). More specifically, PER identifies the same set of evidence for both relations between Entities A and B, among which Sentence 2 describes "place_of_birth" while Sentence 6 discusses "place_of_death". In contrast, FER considers an evidence set unique to each relation and outputs more interpretable results.
In this paper, we propose to explicitly teach the model to capture the major information sources of RE—textual contexts and entity types by Supervising and Augmenting Intermediate Steps (SAIS). Based on a broad spectrum of carefully designed tasks, SAIS extracts relations of enhanced quality due to more effective supervision and retrieves more accurate evidence for improved interpretability. SAIS further boosts the performance with evidence-based data augmentation and ensemble inference while preserving the computational cost by assessing model uncertainty. Experiments on three benchmarks demonstrate the state-of-the-art performance of SAIS on both RE and ER.
If given a plain document, we shall utilize existing tools (e.g., spaCy) to get noisy annotations and apply our method afterward. It is also interesting to investigate how other tasks (e.g., named entity recognition) could be incorporated into the multitask learning pipeline of our SAIS method. We plan to explore these extensions in future works.
Nguyen Bach and Sameer Badaskar. 2007. A review of relation extraction. *Literature review for Language and Statistics II*.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In EMNLP.
Rui Cai, Xiaodong Zhang, and Houfeng Wang. 2016. Bidirectional recurrent convolutional neural network for relation classification. In ACL.
Mary Elaine Califf and Raymond J. Mooney. 1999. Relational learning of pattern-match rules for information extraction. In AAAI.
Qiao Cheng, Juntao Liu, Xiaoye Qu, Jin Zhao, Jiaqing Liang, Zhefeng Wang, Baoxing Huai, Nicholas Jing Yuan, and Yanghua Xiao. 2021. Hacred: A large-scale relation extraction dataset toward hard cases in practical applications. In ACL.
Fenia Christopoulou, Makoto Miwa, and Sophia Ananiadou. 2019. Connecting the dots: Document-level neural relation extraction with edge-oriented graphs. In EMNLP.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep
bidirectional transformers for language understanding. In NAACL.
Ran El-Yaniv et al. 2010. On the foundations of noise-free selective classification. JMLR.
Jun Feng, Minlie Huang, Li Zhao, Yang Yang, and Xiaoyan Zhu. 2018. Reinforcement learning for relation classification from noisy data. In AAAI.
Priya Goyal, Piotr Dolkar, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. 2017. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677.
Zhijiang Guo, Yan Zhang, and Wei Lu. 2019. Attention guided graph convolutional networks for relation extraction. In ACL.
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid O Seaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2019. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. arXiv preprint arXiv:1911.10422.
Kevin Huang, Guangtao Wang, Tengyu Ma, and Jing Huang. 2020. Entity and evidence guided relation extraction for docred. arXiv preprint arXiv:2008.12283.
Robin Jia, Cliff Wong, and Hoifung Poon. 2019. Document-level  $n$ -ary relation extraction with multiscale representation learning. In *NAACL*.
Bo Li, Wei Ye, Zhonghao Sheng, Rui Xie, Xiangyu Xi, and Shikun Zhang. 2020. Graph enhanced dual attention network for document-level relation extraction. In COLING.
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database.
Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dólar. 2017. Focal loss for dense object detection. In ICCV.
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In EMNLP.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. 2020. Reasoning with latent structure refinement for document-level relation extraction. In ACL.