SlowGuess/NAACL_2022_OCR huggingface dataset:
Yang Li $^{1}$ , Guodong Long $^{1}$ , Tao Shen $^{1}$  and Jing Jiang $^{1}$
$^{1}$ Australian AI Institute, School of Computer Science, FEIT, University of Technology Sydney
yang.li-17@student.uts.edu.au,
{guodong.long,tao.shen,jing.jiang}@uts.edu.au
Distant supervision uses triple facts in knowledge graphs to label a corpus for relation extraction, leading to wrong labeling and long-tail problems. Some works use the hierarchy of relations for knowledge transfer to long-tail relations. However, a coarse-grained relation often implies only an attribute (e.g., domain or topic) of the distant fact, making it hard to discriminate relations based solely on sentence semantics. One solution is resorting to entity types, but open questions remain about how to fully leverage the information of entity types and how to align multi-granular entity types with sentences. In this work, we propose a novel model to enrich distantly-supervised sentences with entity types. It consists of (1) a pairwise type-enriched sentence encoding module injecting both context-free and -related backgrounds to alleviate sentence-level wrong labeling, and (2) a hierarchical type-sentence alignment module enriching a sentence with the triple fact's basic attributes to support long-tail relations. Our model achieves new state-of-the-art results in overall and long-tail performance on benchmarks.
Human-curated knowledge graphs (KGs), play a critical role in many downstream tasks but suffer from the incompleteness (Xiong et al., 2018; Yao et al., 2019). As a remedy, relation extraction is to distinguish the relation between two entities according to their semantics in text, but a major obstacle is a lack of sufficient labeled corpus. Fortunately, distant supervision can be used to annotate a raw text corpus via KGs for relation extraction, a.k.a. distantly supervised relation extraction (DSRE). This is based on a strong assumption that a sentence containing two entities will express the semantics of their relation in a KG (Riedel et al., 2010).
The assumption cannot always hold, leading to the wrong labeling problem. For example, both
Figure 1: Two sentences with the same long-tail relation. For each sentence, multi-granular relations from top to bottom are pointed by its best pairwise types, which indicates not all pairwise types provide the same contribution. Blue is subject entity, and red is object entity. The 1st sentence relies on the direct pairwise types due to its relation-irrelevant semantics while the 2nd sentence integrates its relation-relevant semantics and pairwise types to enhance its representation.
"Jobs founded Apple" and "Jobs ate Apple" are labeled with "/BUSINESS/COMPANY/FOUNDERS" according to a KG triple fact (Steven Jobs, /BUSINESS/COMPANY/FOUNDERS, Apple Inc). A basic technique for this problem is selective attention (Zeng et al., 2015; Lin et al., 2016; Ji et al., 2017) under multi-instance learning framework (Riedel et al., 2010; Hoffmann et al., 2011). Given a bag of sentences with the same entity pair, it learns to select correct one(s) by an end-to-end attention. The other major challenge is known as the long-tail problem, caused by domain mismatching during distant supervision. That is, many relation labels correspond only to a limited number of training sentences in the corpus (Ye et al., 2019). For example, in a DSRE benchmark, the distant supervision is an encyclopedic KG (i.e., Freebase (Bollacker et al., 2008)) while the corpus is news articles from the New York Times (NYT), so relations, like "/PEOPLE/PERSON/RELIGION", scarcely appear. As illustrated by Li et al. (2020b) and Zhang et al. (2019), more than  $70\%$  of relation labels in NYT can be regarded as long-tail relations.
To mitigate the long-tail problem, some works
Findings of the Association for Computational Linguistics: NAACL 2022, pages 316 - 326
(Han et al., 2018; Zhang et al., 2019; Li et al., 2020b) resort to the hierarchy of relations for knowledge transfer from data-rich relations to the long-tail ones since the relations have coarse-grained overlap. They focus on interactive operations between hierarchical relations and intra-bag sentences, including relation-to-sentence attention (Han et al., 2018) as a hierarchical extension of selective attention, and sentence-to-relation attention (Li et al., 2020b) enriching sentences with multigranular relations. As such, they achieve knowledge transfer by learning to distinguish coarse-grained relations for sentences with sufficient data, which provides a latent constraint for the long-tail relations. However, a coarse-grained relation usually denotes the only basic attribute of the distant oracle triple fact in KG, so a sentence scarcely contains its semantics and we can only imply the relation via background information. Again, true-labeled "Jobs founded Apple", does not explicitly contain any semantics of its coarse-grained relation "/BUSINESS/COMPANY", but we can directly reason it from the predicate founded and type of Apple. Thus, it is a challenge for a hierarchical DSRE model to correctly imply coarse-grained relations based solely on sentences, not to mention the existence of the wrong labeling problem.
A direct yet promising way to overcome this challenge is to incorporate extra information for entities (Vashishth et al., 2018; Hu et al., 2019; Chu et al., 2020). One popular source is the entity types, i.e., an entity's "ISA" attributes in KG, which characterizes the entity from multiple perspectives (Chen et al., 2020). As Figure 1 shows, although the 1st sentence's semantics is irrelevant to relation, the pairwise types people.deceased_person and location.location directly align with the fine grained relation. However, existing works (Vashishth et al., 2018; Chu et al., 2020) ignore this potential of explicit structured types information.
In this work, we aim to improve DSRE by exploiting structured information in the entity types from both pairwise and hierarchical perspectives to alleviate the wrong labeling and the long-tail problems respectively. To this end, we first propose a context-free type-enriched embedding module to generate word embeddings with pairwise types associated with the entity pair in a bag. As shown in Figure 1, even without the corresponding semantic support, pairwise types can provide direct attributes of entities to align with the relation. Besides, we
develop a context-related type-sentence alignment module to generate robust sentence representation with pairwise types. Since entities have specific characteristic in certain semantics, we leverage semantics to select proper pairwise types and then enrich sentence representation, as the 2nd sentence in Figure 1 shows. Such an alignment is enhanced by a guidance from the relation to auto-seek for associations between pairwise types and sentences.
At the meantime, hierarchical information has been proven crucial in knowledge transfer for long-tail relations (Han et al., 2018; Zhang et al., 2019; Li et al., 2020b). Thereby, we naturally extend the base alignment module into a hierarchy by proposing a hierarchical type-sentence alignment module. An intuitive example in Figure 1 shows that different grained relations are pointed by various granular pairwise types. This indicates that these pairwise types contain hierarchical semantics, which makes it feasible to extend base alignment into hierarchy. Thus, the strong association between pairwise types and coarse-grained relations can improve knowledge transfer for long-tail relations.
We conduct extensive experiments on two popular benchmarks, NYT-520k and NYT-570k, showing that our model achieves new state-of-the-art overall and long-tail performance. Further analyses reveal insights into our model.
Task Definition. Given a bag of sentences  $\mathcal{B} = \{s_1, \ldots, s_N\}$  containing a pair of subject  $e^{(s)}$  and object  $e^{(o)}$  entities, the distant supervision (Mintz et al., 2009) assigns the sentence bag with a relation label  $r$  according to KG triple fact. The goal of relation extraction is to predict the relation label  $\hat{r}$  of an entity pair based on the corresponding sentences  $\mathcal{B}$ . Labels of coarse-grained relations,  $[r^{(1)}, \ldots, r^{(M)}]$ , can be derived from the mention of  $r$ . For instance, when  $r = /BUSINESS/COMPANY/FOUNDERS$ ,  $r^{(1)} = /BUSINESS/COMPANY$  and  $r^{(2)} = /BUSINESS$ . In the following, we will detail our approach, as illustrated in Figure 2.
2.1 Context-Free Type-Enriched Word Emb
Following most previous DSRE works, we first.   
tokenize each sentence  $s_j\in \mathcal{B}$  and employ a   
word2vec method (Mikolov et al., 2013) to derive a sequence of word embeddings by looking up a learnable matrix  $W^{(emb)}\in \mathbb{R}^{d_e\times |\mathbb{V}|}$  , i.e.,   
 $\tilde{\pmb{X}}^j = [\tilde{\pmb{x}}_1^j,\dots ,\tilde{\pmb{x}}_n^j ]\in \mathbb{R}^{d_e}$  , where  $\mathbb{V}$  denotes word
Figure 2: Our proposed model, called Hierarchical Relation-guided Type-Sentence Alignment Model (HiRAM), for DSRE.
vocabulary.  $j$  denotes the index of a sentence in the bag and  $n$  denotes the sentence length. In the sequel, we omit  $j$  if no confusion is caused. Then, as a common practice in DSRE (Zeng et al., 2014), a word's relative distances to both the subject and object entities (a.k.a relative positions) also play significant roles. The distances are first denoted as two integers  $(dist^{(s)}$  and  $dist^{(o)}\in \mathbb{Z})$  and then embedded into two learnable vectors  $(\tilde{\pmb{x}}_i^{(ds)})$  and  $\tilde{\pmb{x}}_i^{(do)}\in \mathbb{R}^{d_p})$ . Therefore, the updated sequence of word embeddings is  $\pmb {X}^j = [\pmb {x}_1,\dots ,\pmb {x}_n]$ , where  $\pmb {x}_i = [\tilde{\pmb{x}}_i;\tilde{\pmb{x}}_i^{(ds)};\tilde{\pmb{x}}_i^{(do)}]\in \mathbb{R}^{d_w}$ , [;] denotes vector concatenation, and  $d_{w}\coloneqq d_{e} + 2d_{p}$ .
Previous works (Li et al., 2020a,b) also found that explicitly enriching each word with both entity embeddings (i.e.,  $e^{(s)}$  and  $e^{(o)}$ ) in a context-free manner is important to DSRE's success. However, many entities scarcely appear in the raw corpus and have multi-characteristics (e.g., Apple could be a fruit or a company). Thus, the model is hard to distinguish the relations only via sentence semantics.
Therefore, we leverage entity types to characterize entities' attributes. That is, given an entity  $e$ , its types are defined as a set of type mentions, i.e.,  $\mathcal{T} = \{t_1, t_2, \ldots\}$ . However, previous works (Chu et al., 2020) directly concatenate the entity types of both  $e^{(s)}$  and  $e^{(o)}$ , completely regardless of potentials of explicit structured information of types. As demonstrated by Krompaß et al. (2015), a relation in KG is usually constrained by the entity types of  $e^{(s)}$  and  $e^{(o)}$  simultaneously (i.e., pairwise types), instead of their individuals. We thereby propose a pairwise type embedding module to enrich the word embedding  $X$  also in a context-free manner.
Type and Pairwise Type Embedding. First, given an entity type set  $\mathcal{T} = \{t_1,t_2,\ldots \}$  (either  $\mathcal{T}^{(s)}$  for subject or  $\mathcal{T}^{(o)}$  for object), we tokenize each type mention  $t_j$  into a sequence of words, then embed the words by looking up  $\pmb{W}^{(emb)}$ , and lastly derive the type embedding  $\pmb{t}_j$  by applying a mean-pooling to the word embeddings of the mention. The embedding of the entire type is
$$
\boldsymbol {T} = \left[ \boldsymbol {t} _ {1}, \boldsymbol {t} _ {2}, \dots \right] \in \mathbb {R} ^ {\left| \mathcal {T} \right| \times d _ {e}}. \tag {1}
$$
As such, we subsequently define the embedding of the pairwise type by considering a combination of every subject  $\forall t_{l}^{(s)}\in \mathcal{T}^{(s)}$  and object type  $\forall t_k^{(o)}\in \mathcal{T}^{(o)}$ . Instead of sole semantics via a vector concatenation, we take into account the prior structured information in each type pair by leveraging a translational scheme (Bordes et al., 2013). Hence, we represent each type pair  $(t_l^{(s)},t_k^{(o)})$  as
$$
\boldsymbol {c} _ {l, k} = \left[ \tilde {\boldsymbol {c}} _ {l, k} ^ {(s e m)}; \tilde {\boldsymbol {c}} _ {l, k} ^ {(s t r)} \right] \in \mathbb {R} ^ {4 d _ {e}}, \tag {2}
$$
$$
\text {w h e r e}, \tilde {\boldsymbol {c}} _ {l, k} ^ {(s e m)} = \boldsymbol {t} _ {l} ^ {(s)} \odot \boldsymbol {W} ^ {(s e m)} \boldsymbol {t} _ {k} ^ {(o)},
$$
$$
\text {a n d} \tilde {\boldsymbol {c}} _ {l, k} ^ {(s t r)} = \boldsymbol {t} _ {k} ^ {(o)} - \boldsymbol {t} _ {l} ^ {(s)}.
$$
Here, “ $\odot$ ” denotes Hadamard product, and  $W^{(sem)}$  denotes a learnable projection.  $\tilde{c}_{l,k}^{(sem)}$  aims to capture the prior semantic relation in the pair (Nickel et al., 2011) since not all types combinations are valid in the whole dataset.  $\tilde{c}_{l,k}^{(str)}$  aims to measure its structured relation. Lastly, we denote all the embeddings of pairwise types as
$$
\boldsymbol {C} = \left\{\boldsymbol {c} _ {l, k} \right\} _ {\forall l \in [ 1, | \mathcal {T} ^ {(s)} | ], \forall k \in [ 1, | \mathcal {T} ^ {(o)} | ]}, \tag {3}
$$
where  $C\in \mathbb{R}^{4d_e\times m}$  and  $m = |\mathcal{T}^{(s)}|\cdot |\mathcal{T}^{(o)}|$ .
Type-Enriched Word Embedding. However, an open question still remains about how to operate on variable-length embeddings of pairwise types,  $C$ , to enrich each word embedding,  $x_{j} \in X$ , in a context-free manner. Inspired by self-attentive sentence encoding (Lin et al., 2016), we present a bag-level type-attentive module, which compresses  $C$  into a single vector representation to facilitate type-enriching. Intuitively, such self-attentive module is focused on the prior knowledge of the type pair in the corpus. Formally, we first generate a global query (Lin et al., 2016) with structured information of both entities and types to retrieve possible prior pairwise types, i.e.,
$$
\tilde {\boldsymbol {q}} ^ {(f)} = \left[ \boldsymbol {e} ^ {(o)}; \operatorname {P o o l} \left(\boldsymbol {T} ^ {(o)}\right) \right] - \left[ \boldsymbol {e} ^ {(s)}; \operatorname {P o o l} \left(\boldsymbol {T} ^ {(s)}\right) \right], \tag {4}
$$
followed by a standard Bilinear-based attention,
$$
\boldsymbol {q} ^ {(f)} = \boldsymbol {C} \cdot \operatorname {s o f t m a x} \left(\boldsymbol {C} ^ {T} \boldsymbol {W} ^ {(s a)} \boldsymbol {q} ^ {(f)}\right) \in \mathbb {R} ^ {4 d _ {e}}, \tag {5}
$$
where “.” denotes matrix multiplication and  $\mathbf{W}^{(sa)}$  is a learnable weight matrix. Lastly, we use a gate as in (Li et al., 2020b) to derive the context-free type-enriched word embedding, i.e.,
$$
\boldsymbol {g} _ {i} ^ {(g f)} = \operatorname {S i g m o i d} (\operatorname {M L P} ([ \boldsymbol {x} _ {i}; \boldsymbol {q} ^ {(f)} ]; \theta^ {(g f 1)})), \tag {6}
$$
$$
\boldsymbol {x} _ {i} ^ {(g f)} = \operatorname {M L P} \left(\left[ \boldsymbol {x} _ {i}; \boldsymbol {q} ^ {(f)} \right]; \theta^ {(g f 2)}\right), \tag {7}
$$
$$
\boldsymbol {v} _ {i} = \boldsymbol {g} _ {i} ^ {(g f)} \odot \boldsymbol {x} _ {i} + \left(\mathbf {1} - \boldsymbol {g} _ {i} ^ {(g f)}\right) \odot \boldsymbol {x} _ {i} ^ {(g f)}, \tag {8}
$$
where MLP denotes a multi-layer perceptron (MLP) module. Hence, word embeddings for  $s$  are updated to  $\pmb {V} = [\pmb {v}_1,\dots ,\pmb {v}_n]\in \mathbb{R}^{d_w\times n}$ .
2.2 Context-Related Type-Sent Alignment
Sentence Encoding. In DSRE, piecewise convolutional neural network (PCNN) (Zeng et al., 2015) is used for sentence embedding. 1D-CNN (Kim, 2014) is first invoked over  $\mathbf{V}$  for contextualized representations. Then a piecewise max-pooling performs over the output sequence to obtain sentence-level embedding with highlighted entity positions:
$$
\boldsymbol {H} = \left[ \boldsymbol {h} _ {1}, \dots , \boldsymbol {h} _ {n} \right] = 1 \text {D - C N N} (\boldsymbol {V}; \theta^ {(c n n)}),
$$
$$
\boldsymbol {s} = \tanh  ([ \operatorname {P o o l} (\boldsymbol {H} ^ {(1)}); \operatorname {P o o l} (\boldsymbol {H} ^ {(2)}); \operatorname {P o o l} (\boldsymbol {H} ^ {(3)}) ]),
$$
where  $\pmb{H}^{(1)}, \pmb{H}^{(2)}$  and  $\pmb{H}^{(3)}$  are three consecutive parts of  $\pmb{H}$  by dividing  $\pmb{H}$  w.r.t. the indices of subject  $e^{(s)}$  and object  $e^{(o)}$  entities. Consequently,  $s \in \mathbb{R}^{d_h}$  is the resulting sentence-level embedding.
Type-Sentence Alignment. Consider that types are not comprehensive enough to align with multi-gramular relations, we leverage semantic context to select valid pairwise types for generating robust sentence representation. Hence, we first calculate alignment scores between a sentence  $s \in \mathbb{R}^{d_h}$  and the embeddings of pairwise types  $C \in \mathbb{R}^{4d_e \times m}$  by using a simple Bilinear layer, i.e.,
$$
\tilde {\boldsymbol {C}} = \operatorname {M L P} (\boldsymbol {C}; \theta^ {(p)}) \in \mathbb {R} ^ {d _ {h} \times m}, \tag {9}
$$
$$
\boldsymbol {a} = \operatorname {s o f t m a x} \left(\tilde {\boldsymbol {C}} ^ {T} \boldsymbol {W} ^ {(a l)} \boldsymbol {s}\right) \in \mathbb {R} ^ {m}. \tag {10}
$$
Then, we enrich the sentence embedding with the aligned type pairs via another gating mechanism:
$$
z = \tilde {C} \cdot a \tag {11}
$$
$$
\boldsymbol {g} = \operatorname {S i g m o i d} (\operatorname {M L P} ([ s; z ]; \theta^ {(g)})), \tag {12}
$$
$$
\tilde {\boldsymbol {u}} = \boldsymbol {g} \odot \boldsymbol {s} + (1 - \boldsymbol {g}) \odot \boldsymbol {z}. \tag {13}
$$
Lastly, following previous success (Li et al., 2020b; Devlin et al., 2019), we leverage a residual connection (He et al., 2016) with layer normalization (Ba et al., 2016) to derive the final context-related type-enriched sentence embedding, i.e.,
$$
\boldsymbol {u} = \operatorname {L a y e r N o r m} (\boldsymbol {s} + \tilde {\boldsymbol {u}}; \theta^ {(l m)}). \tag {14}
$$
Relation-Guided Alignment at the Sentence Level. Due to the severe wrong labeling problem at the sentence level, previous DSRE works usually skip over sentence-level relation supervisions. Fortunately, empowered by the proposed context-free type enrichment and context-related type-sentence alignment, we can utilize the sentence-level relation label even if the relation label is wrong. The reason for this is that, a sentence has already been equipped with structured background to support sentence-level relation even if the sentence semantics cannot deliver the relation. We applied an MLP-based neural classifier to the type-enriched sentence embedding,  $\mathbf{u}$ , to determine the relation at the sentence level, i.e.,
$$
P ^ {(s l)} (\hat {r} | \boldsymbol {u}) = \operatorname {s o f t m a x} \left(\operatorname {M L P} \left(\boldsymbol {u}; \theta^ {(s l)}\right)\right), \tag {15}
$$
where,  $P^{(sl)}(\hat{r} | \boldsymbol{u})$  is a categorical distribution over all possible relations. Hence, the training objective is to minimize the cross-entropy loss,
$$
\mathcal {L} ^ {(s l)} = - \sum_ {\mathcal {D}} \sum_ {\mathcal {B}} \log P ^ {(s l)} (\hat {r} = r | \boldsymbol {u}), \tag {16}
$$
where  $\mathcal{D}$  denotes a DSRE dataset consisting of sentence bags  $\mathcal{B}$ . The guidance from the sentence-level
relation leads to strong type-sentence alignment (as illustrated in §3.1 and §3.2). As a result, the sentence-level wrong labeling problem is alleviated. In contrast, previous works w/ sentence-level relation supervisions (Li and Roth, 2002) suffer from the confirmation bias problem (Chen et al., 2019) caused by the sentence-level wrong labeling.
2.3 Hierarchical Type-Sentence Alignment
Inspired by former works (Han et al., 2018; Zhang et al., 2019; Li et al., 2020b) for handling long-tail relations, we also extend our basic model into hierarchy. However, the basic attributes contained by coarse-grained relation are irrelevant to the semantics in sentences. Thus, instead of direct operating on the hierarchy of relations (i.e., from fine-grained  $r$  to coarse-grained  $[r^{(1)} \ldots r^{(M)}]$  relations), we leverage coarse-grained entity types describing the domain/type properties of the entities in the triple facts to enrich each sentence via the guidance from coarse-grained relation.
Formally, we adapt the relation-guided type-sentence alignment (§2.2) into hierarchy, which shares a high-level inspiration with multi-head attention (Vaswani et al., 2017). First, we reuse the architecture from Eq.(9-14) by defining
$$
\begin{array}{l} \boldsymbol {a} ^ {(l)}, \tilde {\boldsymbol {C}} ^ {(l)} = \operatorname {T S - A l i g n} ^ {(l)} (\boldsymbol {s}, \boldsymbol {C}), \forall l \in [ 1, M ], \\ \boldsymbol {u} ^ {(l)} = \operatorname {T S - I n t e g r a t e} ^ {(l)} (\boldsymbol {a} ^ {(l)}, \tilde {\boldsymbol {C}} ^ {(l)}, \boldsymbol {s}), \tag {17} \\ \end{array}
$$
where TS-Align() denotes Eq.(9-10) to obtain type-sentence alignment  $\pmb{a}^{(l)}$  and TS-Integrate() denotes Eq.(11-14) to generate enriched sentence representation  $\pmb{u}^{(l)}$  at level  $l$ . Note that, these modules are parameter-untied from each other. Then, we update the sentence-level relation-guided loss in Eq.(16) to its hierarchical version, i.e.,
$$
\mathcal {L} ^ {(s l)} = - \sum_ {\mathcal {D}, \mathcal {B}, l \in [ 1, M ]} \log P ^ {(s l)} \left(\hat {r} ^ {(l)} = r ^ {(l)} \mid \boldsymbol {u} ^ {(l)}\right) \tag {18}
$$
Again, learnable parameters of the sentence-level classifiers across  $l$  are also untied. Lastly, we obtain the hierarchical type-enriched representation, i.e.,
$$
\boldsymbol {u} ^ {(h)} = \left[ \boldsymbol {u}; \boldsymbol {u} ^ {(1)}; \dots ; \boldsymbol {u} ^ {(M)} \right] \in \mathbb {R} ^ {(1 + M) d _ {h}}. \tag {19}
$$
Different to previous works (Han et al., 2018; Zhang et al., 2019; Li et al., 2020b) focusing on hierarchical relation embeddings, our work explores the constraints by pairwise types for relations to mitigate sentence-level wrong labeling and uses the hierarchy of entity types on par with that of the relation to improve long-tail performance.
2.4 Relation Classification and Objectives
Lastly, we put the sentences back into the bag and derive bag-level embedding for the final relation classification. Hence, for a bag  $\mathcal{B} = [s_1,\dots s_N]$  we can obtain sentence embeddings of all the sentences  $U^{(h)} = [\pmb {u}_1^{(h)},\dots ,\pmb {u}_N^{(h)}]$ , where  $\pmb{u}_{j}^{(h)}$  is hierarchical type-enriched sentence encoding derived from Eq.(19). To preserve the hierarchical information learned in  $\pmb{u}_{j}^{(h)}$ , we proposed to apply multiple selective modules to its different parts, i.e.,
$$
\boldsymbol {b} = \operatorname {M u l - S e l - A t t n} (\boldsymbol {U} ^ {(h)}) = [ \boldsymbol {b} ^ {(0)}; \boldsymbol {b} ^ {(1)}; \dots ; \boldsymbol {b} ^ {(M)} ],
$$
$$
\boldsymbol {b} ^ {(0)} = \operatorname {S e l - A t t n} ([ \boldsymbol {u} _ {1}; \dots , \boldsymbol {u} _ {N} ]),
$$
$$
\pmb {b} ^ {(l)} = \mathrm {S e l - A t t n} ([ \pmb {u} _ {1} ^ {(l)}; \dots , \pmb {u} _ {N} ^ {(l)} ]), \forall l \in [ 1, M ].
$$
where,  $\operatorname{Sel}\text{-}\operatorname{Attn}()$  represents the selective attention among the sentences in each granular relation, and Mul-Sel-Attn() represents the selective attention among the multi-granular bag representations. For bag representation,  $b^{(0)}$  denotes the finest grained and  $b^{(l)}$  denotes coarser grained. Lastly, we use an MLP-based classifier upon  $b$  to derive a bag-level categorical distribution, i.e.,
$$
P ^ {(b l)} (\hat {r} | e ^ {(s)}, e ^ {(o)}, \mathcal {B}). \tag {20}
$$
Meanwhile, the corresponding training loss is
$$
\mathcal {L} ^ {(b l)} = - \sum_ {\mathcal {D}} P ^ {(b l)} (\hat {r} = r | e ^ {(s)}, e ^ {(o)}, \mathcal {B}). \tag {21}
$$
Therefore, the final training objective is to minimize a linear combination of both sentence-level in Eq.(16) and bag-level (in Eq.(21)) losses, i.e.,
$$
\mathcal {L} = \mathcal {L} ^ {(b l)} + \beta \mathcal {L} ^ {(s l)}. \tag {22}
$$