SlowGuess/NAACL_2022_OCR huggingface dataset:
Yuxin Xiao $^{1}$ , Zecheng Zhang $^{2}$ , Yuning Mao $^{3}$ , Carl Yang $^{4}$ , Jiawei Han $^{3}$
$^{1}$ Carnegie Mellon University,  $^{2}$ Stanford University,
<sup>3</sup>University of Illinois at Urbana-Champaign, <sup>4</sup>Emory University
<sup>1</sup>yuxinxia@cs.cmu.edu <sup>2</sup>zecheng@cs.stanford.edu
3{yuningm2,hanj}@illinois.edu 4j.carlyang@emory.edu
Stepping from sentence-level to document-level, the research on relation extraction (RE) confronts increasing text length and more complicated entity interactions. Consequently, it is more challenging to encode the key information sources—relevant contexts and entity types. However, existing methods only implicitly learn to model these critical information sources while being trained for RE. As a result, they suffer the problems of ineffective supervision and uninterpretable model predictions. In contrast, we propose to explicitly teach the model to capture relevant contexts and entity types by Supervising and Augmenting Intermediate Steps (SAIS) for RE. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately so as to enhance interpretability. By assessing model uncertainty, SAIS further boosts the performance via evidence-based data augmentation and ensemble inference while reducing the computational cost. Eventually, SAIS delivers state-of-the-art RE results on three benchmarks (DocRED, CDR, and GDA) and outperforms the runner-up by  $5.04\%$  relatively in F1 score in evidence retrieval on DocRED. $^{1}$
Playing a crucial role in the continuing effort of transforming unstructured text into structured knowledge, RE (Bach and Badaskar, 2007) seeks to identify relations between an entity pair based on a given piece of text. Earlier studies mostly pay attention to sentence-level RE (Zhang et al., 2017; Hendrickx et al., 2019) (i.e., the targeting entity pair co-occur within a sentence) and achieve promising results (Zhang et al., 2019; Zhou et al., 2020). Based on an extensive empirical analysis,
Peng et al. (2020) reveals that textual contexts and entity types are the major information sources that lead to the success of prior approaches.
Given that more complicated relations are often expressed by multiple sentences, recent focus of RE has been largely shifted to the document level (Yao et al., 2019; Cheng et al., 2021). Existing document-level RE methods (Zeng et al., 2020; Zhou et al., 2021) utilize advanced neural architectures such as heterogeneous graph neural networks (Yang et al., 2020) and pre-trained language models (Xu et al., 2021b). However, although documents typically include longer contexts and more intricate entity interactions, most prior methods only implicitly learn to encode contexts and entity types while being trained for RE. As a result, they deliver inferior and uninterpretable results.
On the other hand, it has been a trend that many recent datasets support the training of more powerful language models by providing multi-task annotations such as coreference and evidence (Yao et al., 2019; Li et al., 2016; Wu et al., 2019). Therefore, in contrast to existing methods, we advocate for explicitly guiding the model to capture textual contexts and entity type information by Supervising and Augmenting Intermediate Steps (SAIS) for RE. More specifically, we argue that, from the input document with annotated entity mentions to the ultimate output of RE, there are four intermediate steps involved in the reasoning process. Consider the motivating example in Figure 1:
(1) Coreference Resolution (CR): Although Sentence 0 describes the "citizenship" of "Carl Linnaeus the Younger" and Sentence 1 discusses the "father" of "Linnaeus filius", the two names essentially refer to the same person. Hence, given a document, we need to first resolve various contextual roles represented by different mentions of the same entity via CR.
(2) Entity Typing (ET): After gathering contextual information from entity mentions, ET reg-
1Our code is available at https://github.com/xiaoyuxin1002/SAIS.
Human Language Technologies, pages 2395 - 2409
Figure 1: Motivating example adapted from DocRED. From the input document with annotated entity mentions to the RE output, there are four intermediate steps involved in the reasoning process. These steps are complementary to RE, in the sense that CR, PER, and FER capture textual contexts while ET preserves entity type information.
ularizes entity representations with the corresponding type information (e.g., Entity A, "Linnaeus filius", is of type "PER" (person)). Within an entity pair, the type information of the head and tail entities can be used to filter out impossible relations, as the relation "year_of_birth" can never appear between two entities of type "PER", for instance.
(3) Pooled and (4) Fine-grained Evidence Retrieval (PER and FER): A unique task for locating the relevant contexts within a document for an entity pair with any valid relation is to retrieve the evidence sentences supporting the relation. Nonetheless, some entity pairs may not express valid relations within the given document (e.g., Entities D and B in the example). Meanwhile some entity pairs possess multiple relations (e.g., Entity A is both "edu-cated_at" and an "employee" of Entity D), each with a different evidence set. Therefore, we use PER to distinguish entity pairs with and without valid supporting sentences and FER to output more interpretable evidence unique to each valid relation of an entity pair.
In this way, the four intermediate steps are complementary to RE, in the sense that CR, PER, and FER capture textual contexts while ET preserves entity type information. Consequently, by explicitly supervising the model's outputs in these intermediate steps via carefully designed tasks, we extract relations of improved quality.
In addition, based on the predicted evidence, we filtrate relevant contexts by augmenting specific intermediate steps with pseudo documents or attention masks. By assessing model confidence, we apply these two kinds of evidence-based data augmentation together with ensemble inference, only when the model is uncertain about its original predictions. Eventually, we further boost the performance with negligible computational cost.
Altogether, our SAIS method achieves state-of-the-art RE performance on three benchmarks (DocRED (Yao et al., 2019), CDR (Li et al., 2016), and GDA (Wu et al., 2019)) due to more effective supervision and enhances interpretability by improving the evidence retrieval (ER) F1 score on DocRED by  $5.04\%$  relatively compared to the runner-up.
Consider a document  $d$  containing sentences  $\mathcal{S}_d = \{s_i\}_{i=1}^{|S_d|}$  and entities  $\mathcal{E}_d = \{e_i\}_{i=1}^{|E_d|}$  where each entity  $e$  is assigned an entity type  $c \in \mathcal{C}$  and appears at least once in  $d$  by its mentions  $\mathcal{M}_e = \{m_i\}_{i=1}^{|M_e|}$ . For a pair of head and tail entities  $(e_h, e_t)$ , document-level RE aims to predict if any relation  $r \in \mathcal{R}$  exists between them, based on whether  $r$  is expressed by some pair of  $e_h$ 's and  $e_t$ 's mentions in  $d$ . Here,  $\mathcal{C}$  and  $\mathcal{R}$  are pre-defined sets of entity and relation types, respectively. Moreover, for  $(e_h, e_t)$  and each of their valid relations  $r \in \mathcal{R}_{h,t}$ , ER aims to identify the subset  $\nu_{h,t,r}$  of  $\mathcal{S}_d$  that is sufficient to express the triplet  $(e_h, e_t, r)$ .
Early research efforts on RE (Bach and Badaskar, 2007; Pawar et al., 2017) center around predicting relations for entity pairs at the sentence level (Zhang et al., 2017; Hendrickx et al., 2019). Many pattern-based (Califf and Mooney, 1999; Qu et al., 2018; Zhou et al., 2020) and neural network-based (Cai et al., 2016; Feng et al., 2018; Zhang et al., 2019) models have shown impressive results. A recent study (Peng et al., 2020) attributes the success of these models to their ability to capture textual contexts and entity type information.
Nevertheless, since more complicated relations can only be expressed by multiple sentences, there has been a shift of focus lately towards document-level RE (Yao et al., 2019; Li et al., 2016; Cheng
et al., 2021; Wu et al., 2019). According to how an approach models contexts, there are two general trends within the domain. Graph-based approaches (Nan et al., 2020; Wang et al., 2020; Zeng et al., 2020; Li et al., 2020; Zeng et al., 2021; Xu et al., 2021c,d; Sahu et al., 2019; Guo et al., 2019) typically infuse contexts into heuristic-based document graphs and perform multi-hop reasoning via advanced neural techniques. Transformer-based approaches (Wang et al., 2019; Tang et al., 2020; Huang et al., 2020; Xu et al., 2021a; Zhou et al., 2021; Zhang et al., 2021; Xie et al., 2022; Ye et al., 2020) leverage the strength of pre-trained language models (Devlin et al., 2019; Liu et al., 2019) to encode long-range contextual dependencies. However, most prior methods only implicitly learn to capture contexts while being trained for RE. Consequently, they experience ineffective supervision and uninterpretable model predictions.
On the contrary, we propose to explicitly teach the model to capture textual contexts and entity type information via a broad spectrum of carefully designed tasks. Furthermore, we boost the RE performance by ensembling the results of evidence-augmented inputs. Compared to EIDER (Xie et al., 2022), we leverage the more precise and interpretable FER for retrieving evidence and present two different kinds of evidence-based data augmentation. We also save the computational cost by applying ensemble learning only to the uncertain subset of relation triplets. As a result, our SAIS method not only enhances the RE performance due to more effective supervision, but also retrieves more accurate evidence for better interpretability.
3 Supervising Intermediate Steps
This section describes the tasks that explicitly supervise the model's outputs in the four intermediate steps. Together they complement the quality of RE.
Given the promising performance of pre-trained language models (PLM) in various downstream tasks, we resort to PLM for encoding the document. More specifically, for a document  $d$ , we insert a classifier token "[CLS]" and a separator token "[SEP]" at the start and end of each sentence  $s \in S_d$ , respectively. Each mention  $m \in \mathcal{M}_d$  is wrapped with a pair of entity markers "\*" (Zhang et al., 2017) to indicate the position of entity mentions. Then we feed the document, with alternating
segment token indices for each sentence (Liu and Lapata, 2019), into a PLM:
$$
\mathbf {H}, \mathbf {A} = \operatorname {P L M} (d), \tag {1}
$$
to obtain the token embeddings  $\mathbf{H} \in \mathbb{R}^{N_d \times H}$  and the cross-token attention  $\mathbf{A} \in \mathbb{R}^{N_d \times N_d}$ .  $\mathbf{A}$  is the average of the attention heads in the last transformer layer (Vaswani et al., 2017) of the PLM.  $N_d$  is the number of tokens in  $d$ , and  $H$  is the embedding dimension of the PLM. We take the embedding of “*” or “[CLS)” before each mention or sentence as the corresponding mention or sentence embedding, respectively.
3.2 Coreference Resolution (CR)
As a case study, it is reported by Yao et al. (2019) that around  $17.6\%$  of relation instances in DocRED require coreference reasoning. Hence, after encoding the document, we resolve the repeated contextual mentions to an entity via CR. In particular, consider a pair of mentions  $(m_i, m_j)$ , we determine the probability of whether  $m_i$  and  $m_j$  refer to the same entity by passing their corresponding embeddings  $\mathbf{m}_i$  and  $\mathbf{m}_j$  through a group bilinear layer (Zheng et al., 2019). The layer splits the embeddings into  $K$  equal-sized groups  $([\mathbf{m}_i^1, \dots, \mathbf{m}_i^K] = \mathbf{m}_i$ , similar for  $\mathbf{m}_j$ ) and applies bilinear with parameter  $\mathbf{W}_m^k \in \mathbb{R}^{H / K \times H / K}$  within each group:
$$
\mathbb {P} _ {i, j} ^ {\mathrm {C R}} = \sigma \left(\sum_ {k = 1} ^ {K} \mathbf {m} _ {i} ^ {k \top} \mathbf {W} _ {m} ^ {k} \mathbf {m} _ {j} ^ {k} + b _ {m}\right), \tag {2}
$$
where  $b_{m}\in \mathbb{R}$  and  $\sigma$  is the sigmoid function.
Since most mention pairs refer to distinct entities (each entity has only 1.34 mentions on average in DocRED), we adopt the focal loss (Lin et al., 2017) on top of the binary cross-entropy to mitigate this extreme class imbalance:
$$
\begin{array}{l} \ell_ {d} ^ {\mathrm {C R}} = - \sum_ {m _ {i} \in \mathcal {M} _ {d}} \sum_ {m _ {j} \in \mathcal {M} _ {d}} \left(y _ {i, j} ^ {\mathrm {C R}} (1 - \mathbb {P} _ {i, j} ^ {\mathrm {C R}}) ^ {\gamma^ {\mathrm {C R}}} \log \mathbb {P} _ {i, j} ^ {\mathrm {C R}} \right. \\ \left. + \left(1 - y _ {i, j} ^ {\mathrm {C R}}\right) \left(\mathbb {P} _ {i, j} ^ {\mathrm {C R}}\right) ^ {\gamma^ {\mathrm {C R}}} \log \left(1 - \mathbb {P} _ {i, j} ^ {\mathrm {C R}}\right)\right) w _ {i, j} ^ {\mathrm {C R}}, \tag {3} \\ \end{array}
$$
where  $y_{i,j}^{\mathrm{CR}} = 1$  if  $m_i$  and  $m_j$  refer to the same entity, and 0 otherwise. Class weight  $w_{i,j}^{\mathrm{CR}}$  is inversely proportional to the frequency of  $y_{i,j}^{\mathrm{CR}}$ , and  $\gamma^{\mathrm{CR}}$  is a hyperparameter.
3.3 Entity Typing (ET)
In a pair of entities, the type information can be used to filter out impossible relations. Therefore,
we regularize entity embeddings via ET. More specifically, we first derive the embedding of an entity  $e$  by integrating the embeddings of its mentions  $\mathcal{M}_e$  via logsumexp pooling (Jia et al., 2019):  $\mathbf{e} = \log \sum_{m\in \mathcal{M}_e}\exp (\mathbf{m})$ . Since entity  $e$  could appear either at the head or tail in an entity pair, we distinguish between the head entity embedding  $\mathbf{e}_h^\prime$  and the tail entity embedding  $\mathbf{e}_t^\prime$  via two separate linear layers:
$$
\mathbf {e} _ {h} ^ {\prime} = \mathbf {W} _ {e _ {h}} \mathbf {e} + \mathbf {b} _ {e _ {h}}, \mathbf {e} _ {t} ^ {\prime} = \mathbf {W} _ {e _ {t}} \mathbf {e} + \mathbf {b} _ {e _ {t}}, (4)
$$
where  $\mathbf{W}_{e_h},\mathbf{W}_{e_t}\in \mathbb{R}^{H\times H}$  and  $\mathbf{b}_{e_h},\mathbf{b}_{e_t}\in \mathbb{R}^H$
However, no matter where  $e$  appears in an entity pair, its head and tail embeddings should always preserve  $e$ 's type information. Hence, we calculate the probability of which entity type  $e$  belongs to by passing  $\mathbf{e}_{\nu}^{\prime}$  for  $\nu \in \{h,t\}$  through a linear layer
$$
\mathbb {P} _ {e} ^ {\mathrm {E T}} = \varsigma \left(\mathbf {W} _ {c} \tanh  \left(\mathbf {e} _ {\nu} ^ {\prime}\right) + \mathbf {b} _ {c}\right), \tag {5}
$$
followed by the multi-class cross-entropy loss:
$$
\ell_ {d} ^ {\mathrm {E T}} = - \sum_ {e \in \mathcal {E} _ {d}} \sum_ {c \in \mathcal {C}} y _ {e, c} ^ {\mathrm {E T}} \log \mathbb {P} _ {e, c} ^ {\mathrm {E T}}, \tag {6}
$$
where  $\mathbf{W}_c\in \mathbb{R}^{|\mathcal{C}|\times H}$ ,  $\mathbf{b}_c\in \mathbb{R}^{|\mathcal{C}|}$ , and  $\varsigma$  is the softmax function.  $y_{e,c}^{\mathrm{ET}} = 1$  if  $e$  is of entity type  $c$ , and 0 otherwise.
3.4 Pooled Evidence Retrieval (PER)
To further capture textual contexts, we explicitly guide the attention in the PLM to the supporting sentences of each entity pair via PER. That is, we want to identify the pooled evidence set  $\mathcal{V}_{h,t} = \cup_{r\in \mathcal{R}_{h,t}}\mathcal{V}_{h,t,r}$  in  $d$  that is important to an entity pair  $(e_h,e_t)$ , regardless of the specific relation expressed by a particular sentence  $s\in \mathcal{V}_{h,t}$ . In this case, given  $(e_h,e_t)$ , we first compute a unique context embedding  $\mathbf{c}_{h,t}$  based on the cross-token attention from Equation 1:
$$
\mathbf {c} _ {h, t} = \mathbf {H} ^ {\top} \frac {\mathbf {A} _ {h} \otimes \mathbf {A} _ {t}}{\mathbf {1} ^ {\top} \left(\mathbf {A} _ {h} \otimes \mathbf {A} _ {t}\right)}. \tag {7}
$$
Here,  $\otimes$  is the element-wise product.  $\mathbf{A}_h$  is  $e_h$ 's attention to all the tokens in the document (i.e., the average of  $e_h$ 's mention-level attention). Similar for  $\mathbf{A}_t$ . Then we measure the probability of whether a sentence  $s \in S_d$  is part of the pooled supporting evidence  $\mathcal{V}_{h,t}$  by passing  $(e_h, e_t)$ 's context embedding  $\mathbf{c}_{h,t}$  and sentence  $s$ 's embedding  $s$  through a group bilinear layer:
$$
\mathbb {P} _ {h, t, s} ^ {\text {P E R}} = \sigma \left(\sum_ {k = 1} ^ {K} \mathbf {c} _ {h, t} ^ {k \top} \mathbf {W} _ {p} ^ {k} \mathbf {s} ^ {k} + b _ {p}\right), \tag {8}
$$
where  $\mathbf{W}_p^k\in \mathbb{R}^{H / K\times H / K}$  and  $b_{p}\in \mathbb{R}$
Again, we face a severe class imbalance here, since most entity pairs (97.1% in DocRED) do not have valid relations or supporting evidence. As a result, similar to Section 3.2, we also use the focal loss with the binary cross-entropy:
$$
\begin{array}{l} \ell_ {d} ^ {\mathrm {P E R}} = - \sum_ {e _ {h} \in \mathcal {E} _ {d}} \sum_ {e _ {t} \in \mathcal {E} _ {d}} \sum_ {s \in \mathcal {S} _ {d}} \left(y _ {h, t, s} ^ {\mathrm {P E R}} (1 - \mathbb {P} _ {h, t, s} ^ {\mathrm {P E R}}) ^ {\gamma^ {\mathrm {P E R}}} \right. \\ \log \mathbb {P} _ {h, t, s} ^ {\text {P E R}} + (1 - y _ {h, t, s} ^ {\text {P E R}}) (\mathbb {P} _ {h, t, s} ^ {\text {P E R}}) ^ {\gamma^ {\text {P E R}}} \\ \log \left(1 - \mathbb {P} _ {h, t, s} ^ {\text {P E R}}\right) w _ {h, t, s} ^ {\text {P E R}}, \tag {9} \\ \end{array}
$$
where  $y_{h,t,s}^{\mathrm{PER}} = \mathbb{1}\{s\in \mathcal{V}_{h,t}\}$  , class weight  $w_{h,t,s}^{\mathrm{PER}}$  is inversely proportional to the frequency of  $y_{h,t,s}^{\mathrm{PER}}$  and  $\gamma^{\mathrm{PER}}$  is a hyperparameter.
3.5 Fine-grained Evidence Retrieval (FER)
In addition to PER, we would like to further refine  $\nu_{h,t}$ , since an entity pair could have multiple valid relations and, correspondingly, multiple sets of evidence. As a result, we explicitly train the model to recover contextual evidence unique to a triplet  $(e_h,e_t,r)$  via FER for better interpretability. More specifically, given  $(e_h,e_t,r)$ , we first generate a triplet embedding  $\mathbf{l}_{h,t,r}$  by merging  $\mathbf{e}_h$ ,  $\mathbf{e}_t$ ,  $\mathbf{c}_{h,t}$ , and  $r$ 's relation embedding  $\mathbf{r}$  via a linear layer:
$$
\mathbf {l} _ {h, t, r} = \tanh  \left(\mathbf {W} _ {l} \left[ \mathbf {e} _ {h} \| \mathbf {e} _ {t} \| \mathbf {c} _ {h, t} \| \mathbf {r} \right] + \mathbf {b} _ {l}\right), \tag {10}
$$
where  $\mathbf{W}_l\in \mathbb{R}^{H\times 4H}$ ,  $\mathbf{b}_l\in \mathbb{R}^H$ ,  $\|$  represents concatenation, and  $\mathbf{r}$  is initialized from the embedding matrix of the PLM.
Similarly, we use a group bilinear layer to assess the probability of whether a sentence  $s \in S_d$  is included in the fine-grained evidence set  $\mathcal{V}_{h,t,r}$ :
$$
\mathbb {P} _ {h, t, r, s} ^ {\mathrm {F E R}} = \sigma \left(\sum_ {k = 1} ^ {K} \mathbf {l} _ {h, t, r} ^ {k \top} \mathbf {W} _ {f} ^ {k} \mathbf {s} ^ {k} + b _ {f}\right), \tag {11}
$$
where  $\mathbf{W}_f^k\in \mathbb{R}^{H / K\times H / K}$  and  $b_{f}\in \mathbb{R}$
Since FER only involves entity pairs with valid relations, the class imbalance is milder here than in PER. Hence, let  $y_{h,t,r,s}^{\mathrm{FER}} = \mathbb{1}\{s\in \mathcal{V}_{h,t,r}\}$ , we deploy the standard binary cross-entropy loss:
$$
\begin{array}{l} \ell_ {d} ^ {\mathrm {F E R}} = - \sum_ {e _ {i} \in \mathcal {E} _ {d}} \sum_ {e _ {j} \in \mathcal {E} _ {d}} \sum_ {r \in \mathcal {R} _ {h, t}} \sum_ {s \in \mathcal {S} _ {d}} \left(y _ {h, t, r, s} ^ {\mathrm {F E R}} \log \mathbb {P} _ {h, t, r, s} ^ {\mathrm {F E R}} \right. \\ \left. + \left(1 - y _ {h, t, r, s} ^ {\mathrm {F E R}}\right) \log \left(1 - \mathbb {P} _ {h, t, r, s} ^ {\mathrm {F E R}}\right)\right). \tag {12} \\ \end{array}
$$
3.6 Relation Extraction (RE)
Based on the four complementary tasks introduced above, for an entity pair  $(e_h, e_t)$ , we encode relevant contexts in  $\mathbf{c}_{h,t}$  and preserve entity type information in  $\mathbf{e}_h'$  and  $\mathbf{e}_t'$ . Ultimately, we acquire the contexts needed by the head and tail entities from  $\mathbf{c}_{h,t}$  via two separate linear layers:
$$
\mathbf {c} _ {h} ^ {\prime} = \mathbf {W} _ {c _ {h}} \mathbf {c} _ {h, t} + \mathbf {b} _ {c _ {h}}, \mathbf {c} _ {t} ^ {\prime} = \mathbf {W} _ {c _ {t}} \mathbf {c} _ {h, t} + \mathbf {b} _ {c _ {t}}, \tag {13}
$$
where  $\mathbf{W}_{c_h},\mathbf{W}_{c_t}\in \mathbb{R}^{H\times H}$  and  $\mathbf{b}_{c_h},\mathbf{b}_{c_t}\in \mathbb{R}^H$  and then combine them with the type information to generate the head and tail entity representations:
$$
\mathbf {e} _ {h} ^ {\prime \prime} = \tanh (\mathbf {e} _ {h} ^ {\prime} + \mathbf {c} _ {h} ^ {\prime}), \mathbf {e} _ {t} ^ {\prime \prime} = \tanh (\mathbf {e} _ {t} ^ {\prime} + \mathbf {c} _ {t} ^ {\prime}). (1 4)
$$
Next, a group bilinear layer is utilized to calculate the logit of how likely a relation  $r \in \mathcal{R}$  exists between  $e_h$  and  $e_t$ :
$$
\mathbb {L} _ {h, t, r} ^ {\mathrm {R E}} = \sum_ {k = 1} ^ {K} \mathbf {e} _ {h} ^ {\prime \prime k \top} \mathbf {W} _ {r} ^ {k} \mathbf {e} _ {t} ^ {\prime \prime k} + b _ {r}, \tag {15}
$$
where  $\mathbf{W}_r^k\in \mathbb{R}^{H / K\times H / K}$  and  $b_{r}\in \mathbb{R}$
As discussed earlier, only a small portion of entity pairs have valid relations, among which multiple relations could co-exist between a pair. Therefore, to deal with the problem of multi-label imbalanced classification, we follow Zhou et al. (2021) by introducing a threshold relation class TH and adopting an adaptive threshold loss:
$$
\begin{array}{l} \ell_ {d} ^ {\mathrm {R E}} = - \sum_ {e _ {h} \in \mathcal {E} _ {d}} \sum_ {e _ {t} \in \mathcal {E} _ {d}} \\ \left[ \sum_ {r \in \mathcal {P} _ {h, t}} \log \left(\frac {\exp \mathbb {L} _ {h , t , r} ^ {\mathrm {R E}}}{\sum_ {r ^ {\prime} \in \mathcal {P} _ {h , t} \cup \{\mathrm {T H} \}} \mathbb {L} _ {h , t , r ^ {\prime}} ^ {\mathrm {R E}}}\right) \right. \\ \left. + \log \left(\frac {\exp \mathbb {L} _ {h , t , \mathrm {T H}} ^ {\mathrm {R E}}}{\sum_ {r ^ {\prime} \in \mathcal {N} _ {h , t} \cup \{\mathrm {T H} \}} \mathbb {L} _ {h , t , r ^ {\prime}} ^ {\mathrm {R E}}}\right) \right]. \tag {16} \\ \end{array}
$$
In essence, we aim to increase the logits of valid relations  $\mathcal{P}_{h,t}$  and decrease the logits of invalid relations  $\mathcal{N}_{h,t}$ , both relative to TH.
Overall, with the goal of improving the model's RE performance by better capturing entity type information and textual contexts, we have designed four tasks to explicitly supervise the model's outputs in the corresponding intermediate steps. To this end, we visualize the entire pipeline  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$  in Appendix A and integrate all the tasks by minimizing the multi-task learning objective
$$
\ell = \sum_ {d \in \mathcal {D} _ {\text {t r a i n}}} \left(\ell_ {d} ^ {\mathrm {R E}} + \sum_ {\text {T a k}} \eta^ {\text {T a k}} \ell_ {d} ^ {\text {T a k}}\right), \tag {17}
$$
where  $\mathrm{Task} \in \{\mathrm{CR}, \mathrm{ET}, \mathrm{PER}, \mathrm{FER}\}$ .  $\eta^{\mathrm{Task}}$ 's are hyperparameters balancing the relative task weight.
During inference with the current pipeline  $\mathrm{SAIS}_{\mathrm{All}}^{\mathrm{O}}$ , we predict if a triplet  $(e_h, e_t, r)$  is valid (i.e., if relation  $r$  exists between entity pair  $(e_h, e_t)$ ) by checking if its logit is larger than the corresponding threshold logit (i.e.,  $\mathbb{L}_{h,t,r}^{\mathrm{RE}} > \mathbb{L}_{h,t,\mathrm{TH}}^{\mathrm{RE}}$ ). For each predicted triplet  $(e_h, e_t, r)$ , we assess if a sentence  $s$  belongs to the evidence set  $\mathcal{V}_{h,t,r}$  by checking if  $\mathbb{P}_{h,t,r,s}^{\mathrm{FER}} > \alpha^{\mathrm{FER}}$  where  $\alpha^{\mathrm{FER}}$  is a threshold.
4 Augmenting Intermediate Steps