Hierarchical constrained attention for distantly supervised
relation extraction
Bao Liu

Guilin Qi∗

Yuxin Zhang

bliu187@foxmail.com
School of Computer Science andEngineering, Southeast University
Nanjing, China
Key Laboratory of New Generation
Artificial Intelligence Technology and
Its Interdisciplinary Applications(Southeast University), Ministry of
Education
China

gqi@seu.edu.cn
School of Computer Science andEngineering, Southeast University
Nanjing, China
Key Laboratory of New Generation
Artificial Intelligence Technology and
Its Interdisciplinary Applications(Southeast University), Ministry of
Education
China

zzyx_cs@seu.edu.cn
School of Computer Science andEngineering, Southeast University
Nanjing, China
Key Laboratory of New Generation
Artificial Intelligence Technology and
Its Interdisciplinary Applications(Southeast University), Ministry of
Education
China

ABSTRACTKEYWORDS

Distantly supervised relation extraction (DSRE) can automaticallycollect training data with existing knowledge bases. However, label
noise and long-tailed distributions severely affect the performanceof DSRE. Most previous studies alleviate these problems by themulti-instance learning to construct a sentence bag as the input of
the classifier, while the background information related to DSRE is
not fully utilized. In this paper, we design a hierarchical constrained
attention to exploit background information such as relation hierarchy and entity types. Furthermore, a hierarchical constrained
attention-based distantly supervised relation extraction framework(HCAT) is proposed. Specifically, HCAT employs a hierarchical relation extraction framework to propagate information from data-rich
top-layer relations to data-poor long-tailed relations. To further
facilitate the information sharing between different relations, graphattention networks are used to encode all the relations connectedby entity types. In addition, for the label noise problem, at eachlevel of the relation hierarchy, entity types are concatenated into
corresponding sentences and relations to better identify the valid
sentences for bag representations. Substantial experimental results
demonstrate that our model HCAT achieves significant improvement over the previous methods for both denoising and long-tailed
relation extraction1 .

Distant Supervision, Relation Extraction, Label Noise, Long-tailed
Distributions, Hierarchical Constrained Attention

CCS CONCEPTS• Computing methodologies → Natural language processing.

∗ Corresponding author.
1 https://github.com/bliu187/HCAT

Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.
IJCKG’23, December 08–09, 2023, Tokyo, Japan© 2023 Association for Computing Machinery.
ACM ISBN 978-1-4503-XXXX-X/23/12. . . $15.00
https://doi.org/XXXXXXX.XXXXXXXACM Reference Format:
Bao Liu, Guilin Qi, and Yuxin Zhang. 2023. Hierarchical constrained attention for distantly supervised relation extraction. In Proceedings of The 12th
International Joint Conference on Knowledge Graph (IJCKG’23). ACM, New
York, NY, USA, 9 pages. https://doi.org/XXXXXXX.XXXXXXX

1INTRODUCTION

Relation extraction (RE) aims to extract relation facts between two
given entities from unstructured text, which plays an importantrole in many natural language processing applications [14, 5, 15].
For example, during the knowledge graph construction process,there are still many missing facts. As a key technology for knowledge graph completion, relation extraction can effectively identify
semantic relationships between existing entity pairs [28, 7, 13].
Conventional supervised approaches have shown their capabilities
in tackling RE problems, but they require large-scale labeled training data which is time-consuming and labor-intensive. In this case,distantly supervised relation extraction (DSRE) [23] is proposed
to automatically generate large-scale labeled data by aligning an
existing knowledge base (KB) and plain text. It assumes that if one
entity pair holds a relation in the existing KB, then all sentencesthat mention the entity pair will express this relation. Although
DSRE can bring abundant training data, it still suffers from two
major challenges.
The first challenge is label noise caused by the aforementioned
strong assumption. As illustrated in Figure 1, there is a relation
triple < Kevin Durant, Birthplace, Washington > in the knowledge
base, and sentence S1 and S2 both contain the same entity pair <Kevin Durant, Washington >. According to the distant supervision
assumption, sentence S1 and S2 are both labeled with the relation
Birthplace and used as training data. However, sentence S2 does
not express the relation Birthplace and obtains a wrong label. To
mitigate the interference of label noise, the multi-instance learning(MIL) framework [27, 11, 31] was employed to identify the relationof a common entity pair for a bag of sentences. In addition, to takeadvantage of all sentences in the bag, the attention mechanism wasIJCKG’23, December 08–09, 2023, Tokyo, Japan

introduced into DSRE [20], which can dynamically assign different
weights to sentences to mitigate the influence of noisy sentences.
The second challenge is long-tailed relation extraction. Althoughdistant supervision methods can generate a large amount of training data, there is a large gap in these training data with different
labels. For example, if we treat relations with less than 100 training
sentences as long-tailed relations, then there are more than 60%
long-tail relations suffering from data deficiency in the popular New
York Times (NYT) [27] dataset. To accomplish this, the hierarchybased strategy [9, 40, 18] has been extensively studied in the last
few years, which developed a relation hierarchical tree and tried
to mine latent correlation between relations on different layers,
information can propagated from data-rich top-layer relations todata-poor long-tailed ones.
To alleviate the label noise and long-tailed distributions, someresearchers attempt to exploit the background information to improve the relation extraction performance of distant supervisionmodel, e.g., entity types [19], entity descriptions [12], and relation
hierarchy [9], which can mitigate the interference of noisy sentences in the bag, and connect different relations to facilitate thetransfer of information between data-rich relations and long-tailed
relations. However, previous works separately exploit backgroundinformation, focusing only on a specific aspect of the backgroundinformation, which fails to realize its full potential value. Therefore,
we suspect that employing appropriate methods to simultaneously
use multiple background information may better promote the performance of distantly supervised relation extraction.
To fully exploit the background information, we design a hierarchical constrained attention to employ both background information: relation hierarchy and entity types. Furthermore, a hierarchical constrained attention-based distantly supervised relation
extraction framework (HCAT) is proposed for both label noise andlong-tailed distributions. In detail, HCAT employs a hierarchicalrelation extraction framework [9] to propagate information fromdata-rich relations to long-tailed relations, and graph attention networks (GAT) [32] are used to encode all relations and entity types,
which can further promote the information sharing between different relations. Moreover, for the label noise problem, entity types areconnected to corresponding sentences and relation representations
at each level of the relation hierarchy as the external information
constraints, which can improve the selective attention mechanism’s
ability to identify valid sentences.
The main contributions are summarized as follows:• We design a hierarchical constrained attention network to
make full use of background information such as relation
hierarchy and entity types, which can facilitate valid sentence recognition in bags and information transfer between
different relations.
• We propose a hierarchical constrained attention-based distantly supervised relation extraction framework, which canalleviate both label noise and long-tailed relation problems.
• We conduct comprehensive experiments on the popular New
York Times (NYT) [27] dataset. Our model HCAT receives
state-of-the-art performance in terms of multiple metrics.
The remainder of our paper is organized as follows. An overview
of related work is introduced in Section 2. Section 3 details the

Bao Liu, Guilin Qi and Yuxin Zhang

proposed HCAT model. We present the extensive experiments for
performance comparison and ablation studies in Section 4. Section
5 concludes the paper and gives some important future directions.

2

RELATED WORK

Distant supervision can automatically generate a large amount oflabeled training data for relation extraction tasks. However, theaforementioned strong assumption also brings two important challenges: label noise and long-tailed distributions.
To alleviate the label noise problem, multi-instance learning(MIL) was employed to alleviate the strong assumption of DSRE
[27, 11, 31], which extracts relation from a sentence bag instead of asingle sentence for an entity pair. Then, various denoising strategies
under the MIL framework were proposed. Lin et al. [20] designed
sentence-level selective attention to get the bag representation by
giving sentences different weights. Qu et al. [26] proposed a wordlevel attention to increase the attention weights of those criticalwords. Yu et al. [35] elaborated segment-level attention to mine theinformation of continuous words in a sentence, which can capture
dependencies between different entity pairs and corresponding relations. In addition, Yuan et al. [37] developed bag-level selective
attention to pay more attention to entity pairs with higher quality.
Ye et al. [34] designed an intra-bag- and inter-bag-level attention
to alleviate the noise that existed in both sentences and bags. In
addition, many other denoising techniques have been introduced
to DSRE, such as consensus-enhanced training [21], cross-stitch
bi-encoders [4], deep clustering [33], interaction-and-response networks [29], etc.
To tackle the long-tailed distributions problem, existing methods
are mostly based on relation hierarchy. Although long-tail relations
contain less training data, sufficient training data likely exists intheir ancestor or sibling relations. Han et al. [9] first incorporated
the hierarchical information of relations to DSRE and proposeda hierarchical instance-level attention, which can transfer knowledge from data-rich relations to data-poor ones. Then, many otherstrategies based on relation hierarchy were proposed, such as topdown classification [36], global hierarchy embeddings [24], andrecursive hierarchy interactive attention [8]. In addition, Zhang
et al. [40] developed a coarse-to-fine knowledge-aware attention
network to learn relation information through knowledge graphembeddings. Li et al. [18] designed a relation-augmented attention
network to enrich the sentence representations, which incorporate relation information to sentence embeddings at each level ofthe relation hierarchy. Contrastive learning was also introduced to
DSRE to alleviate the long-tail problem [16]. Recently, a novel constraint graph-based relation extraction framework (CGRE) [19] wasproposed, which uses graph convolution networks to propagate information between different relations and designs constraint-aware
attention for bag representation.
All the aforementioned works try to alleviate the influence ofnoisy sentences and facilitate information transfer between different relations, but the background information is not fully utilized.
Therefore, we exploit both relation hierarchy and entity types information and propose a hierarchical constrained attention-based
distantly supervised relation extraction framework (HCAT) for bothlabel noise and long-tailed distributions.

Hierarchical constrained attention for distantly supervised relation extraction

IJCKG’23, December 08–09, 2023, Tokyo, Japan

Plain text

Knowledge base

S1: Kevin Durant was born on September 29,
1988, in Washington.

(Kevin Durant, Birthplace, Washington)(Bill Gates, Founded, Microsoft)

S2: Kevin Durant plans to go to Washington
for NBA game tomorrow night.

Training data
Sentence

Entity pair

S1: Kevin Durant was born on September 29, 1988, in
Washington.

(Kevin Durant, Washington)

Birthplace(Correct)

S2: Kevin Durant plans to go to Washington for NBA
game tomorrow night.

(Kevin Durant, Washington)

Birthplace(Wrong)Label

Figure 1: The illustration of DSRE process.

3 APPROACH
3.1 Task definition
Distantly supervised relation extraction (DSRE) can automaticallyobtain a large amount of training data, but it is also accompanied by
the problem of noisy label sentences and long-tailed distributions.
Therefore, DSRE employs the multi-instance learning (MIL) framework, all sentences are divided into multiple bags {B1, B2, ..., B𝑘 },each bag B = {𝑆 1, 𝑆 2, ..., 𝑆𝑛 } groups some sentences with the sametarget entity pair <𝑒 1 , 𝑒 2 >, and each sentence is defined as a word
sequence 𝑆 = {𝑤 1, 𝑤 2, ..., 𝑤𝑚 }. In addition, all relations are predefined as a set R = {𝑟 1, 𝑟 2, ..., 𝑟𝑙 }. The goal of distantly supervised
relation extraction is to predict the relations for all the given entity
pairs.

3.2

Framework

To alleviate the interference of label noise and long-tailed distributions, we employ hierarchical relation extraction framework [9]and graph attention networks [32] to facilitate the transfer of information between different relations, and long-tailed relations
and benefit from the data-rich relations. Furthermore, the encoded
entity types are used as additional constraints for the bag representation, which can better identify valid sentences to mitigate theimpact of label noise. As illustrated in Figure 2, our proposed model
HCAT consists of three key modules:• Sentence encoder. Given a bag of sentences with a target entity pair, the sentence encoder is employed to extract
features of each sentence in the bag.
• Graph encoder. Given all relations and corresponding entity types, we first conduct the hierarchy constraint graphand then adopt the graph attention networks (GAT) [32] toextract the interactive features of the relations and entity
types.

• Hierarchical constrained attention. This module is designed to get bag embeddings at each level of the relation
hierarchy, the entity types are used as the external information constraint to identify valid sentences.

3.3

Sentence encoder

In this module, we first employ an entity-aware embedding [17] toencode each word in a sentence, and then adopt the piecewise convolutional neural network (PCNN) [38] to get the entire sentence
representation.
3.3.1 Entity-aware embeding. Given a sentence 𝑆 = {𝑤 1, 𝑤 2, ..., 𝑤𝑚 },We firstly use a pre-trained word2vec model [22] to map each word
𝑤𝑖 into a 𝑑 𝑤 -dimensional vectors w𝑖 . Then following Li et al. [17],the position and entity information are incorporated into each word
vector w𝑖 to capture their semantic and syntactic information.
1
2
The entity information is represented as w𝑒 and w𝑒 , they are
the word vectors of target entity pair <𝑒 1 , 𝑒 2 >. The position information [39] is the relative distances between each word 𝑤𝑖 and thetarget entity pair. For example, in the sentence "Kevin Durant wasborn on September 29, 1988, in Washington.", the relative distancefrom 𝑏𝑜𝑟𝑛 to entity 𝑒 1 (𝐾𝑒𝑣𝑖𝑛𝐷𝑢𝑟𝑎𝑛𝑡) and entity 𝑒 2 (𝑊 𝑎𝑠ℎ𝑖𝑛𝑔𝑡𝑜𝑛)are 2 and -8, respectively. Then, these two distances are embedded
1
2
as two 𝑑𝑝 -dimensional vectors p𝑖𝑒 and p𝑖𝑒 . We concatenate theentity and position information to each word and get two types ofword representations as follows:
12

f𝑖𝑒 = [w𝑖 ; w𝑒 ; w𝑒 ] ∈ R3𝑑 𝑤 ,12

f𝑖 = [w𝑖 ; p𝑖𝑒 ; p𝑖𝑒 ] ∈ R𝑑 𝑤 +2𝑑𝑝 ,𝑝(1)

where the word dimension 𝑑 𝑤 and position dimension 𝑑𝑝 are bothpre-defined. And two types of word vectors compose the corre𝑒 } and F =sponding sentence representations F𝑒 = {f1𝑒 , ..., f𝑚𝑝𝑝𝑝{f1 , ..., f𝑚 }. Consequently, we employ the entity-aware embedding

IJCKG’23, December 08–09, 2023, Tokyo, Japan

Bao Liu, Guilin Qi and Yuxin Zhang

Hierarchical constrained attention module
A sentence bag
S1

...

Sm

Sentence encoder

...

U1

X1

...

Xm

Hier(1)

X1(1)

...

Xm(1)

Hier(2)

X1(2)

...

Xm(2)

...

Xm(k)

Um

B(1)

Concatenate

Entity typeembeddings

B(2)Augmenta2

Entity types

B(k)

...

...
Hier(k) X1(k)Concatenate

Relation hierarchy

Concatenate

...

R(1)

C(1)

R(2)

C(2)

Bag output

Graph
encoder

...

...

...

...

...
...

R(k)

C(k)

Classifier...

Figure 2: The overview of our proposed model, HCAT.
to get the sentence representation as follows:

Finally, all the pooling results are concatenated to get the sentence feature representation U as follows:

𝛼 = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 (𝜆 · (W𝑒 F𝑒 + b𝑒 )),F̃𝑝 = 𝑡𝑎𝑛ℎ(W𝑝 F𝑝 + b𝑝 ),S = 𝛼 ⊙ F𝑒 + (1 − 𝛼) ⊙ F̃𝑝 ,3.4

Where "⊙" indicates the element-wise product, 𝜆 is a trade-off
weight. W𝑒 , W𝑝 , b𝑒 and b𝑝 are all learnable parameters.
3.3.2 Piecewise convolutional neural network (PCNN). To effectively extract features of the given sentence S = {s1, ..., s𝑚 }, we
employ the piecewise convolutional neural network (PCNN) [38]as sentence encoder and get a high-dimensional representation.
Firstly, 𝑘 convolutional kernels W𝑐 = {w𝑐1, ..., w𝑐𝑘 }, w𝑐𝑖 ∈ R𝑤 areslide on the sentence sequence as follows:
h𝑖 𝑗 = w𝑐𝑖 s 𝑗 −𝑤+1:𝑗

1 ⩽ 𝑖 ⩽ 𝑘, 1 ⩽ 𝑗 ⩽ 𝑚,(3)

where s𝑖:𝑗 operates the concatenating from s𝑖 to s 𝑗 . Then the hidden
(1)

(2)

(3)

representation is divided into three segments {h𝑖 𝑗 , h𝑖 𝑗 , h𝑖 𝑗 }according to the positions of the target entity pair <𝑒 1 , 𝑒 2 >. In
addition, the piecewise max pooling operation is conducted asfollows:
(1)

q𝑖

(2)

q𝑖

(1)

= max (h𝑖 𝑗 )1⩽𝑗⩽𝑝 1=

(2)max (h𝑖 𝑗 )𝑝 1 < 𝑗⩽𝑝 2

(3)

q𝑖

1 ⩽ 𝑖 ⩽ 𝑘,

U = 𝑡𝑎𝑛ℎ(q1:𝑘 ) ∈ R3𝑘

(2)

(4)

(3)

= max (h𝑖 𝑗 )𝑝 2 < 𝑗⩽𝑚where 𝑝 1 and 𝑝 2 are the positions of 𝑒 1 and 𝑒 2 , respectively. The
piecewise max pooling layer can effectively capture structural information between two entities.

(5)

Graph encoder

To facilitate the transfer of information between different relations,
we exploit the background information such as relation hierarchyand entity types. we firstly combine the hierarchical relation extraction framework [9] and entity type constraints [19] to construct hierarchical constraint graph. Then, graph attention networks (GAT)[32] are employed to encode the relations and entity types for
information exchange between them.
3.4.1 Hierarchical constraint graph construction. Most relations inknowledge bases are composed of hierarchical structures. Given a
relation set R = {𝑟 1, 𝑟 2, ..., 𝑟𝑙 }, for each relation 𝑟 ∈ R, we can obtaina hierarchical chain of its ancestors {𝑟 (1) , 𝑟 (2) , ..., 𝑟 (ℎ) }. For example, given a relation /𝑠𝑝𝑜𝑟𝑡𝑠/𝑠𝑝𝑜𝑟𝑡𝑠_𝑡𝑒𝑎𝑚/𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛 in the knowledge base Freebase [2], the obtained hierarchy chain is denotedas {/𝑠𝑝𝑜𝑟𝑡𝑠, /𝑠𝑝𝑜𝑟𝑡𝑠/𝑠𝑝𝑜𝑟𝑡𝑠_𝑡𝑒𝑎𝑚, /𝑠𝑝𝑜𝑟𝑡𝑠/𝑠𝑝𝑜𝑟𝑡𝑠_𝑡𝑒𝑎𝑚/𝑙𝑜𝑐𝑎𝑡𝑖𝑜𝑛}.
In addition, a root relation is defined as the unique common ancestor of all relations, and all chains form a tree-like hierarchicalstructure as shown in Figure 3, each relation at a different level istreated as a separate node in the relation hierarchical tree.
Under the hierarchical relation extraction framework, long-tailed
relations can benefit from their ancestors or siblings. For example,
if /𝑝𝑒𝑜𝑝𝑙𝑒/𝑑𝑒𝑐𝑒𝑎𝑠𝑒𝑑_𝑝𝑒𝑟𝑠𝑜𝑛/𝑝𝑙𝑎𝑐𝑒_𝑜 𝑓 _𝑑𝑒𝑎𝑡ℎ is a long-tailed relation with a small number of training sentences, it is difficult totrain an effective model to identify this relation without sufficienttraining data. The hierarchical relation extraction framework computes scores for those sentences containing the same entity pair onHierarchical constrained attention for distantly supervised relation extraction

IJCKG’23, December 08–09, 2023, Tokyo, Japan

G. The node v𝑖 of the 𝑘-th layer in the GAT is computed as follows:
root

(𝑘 )

v𝑖

© ∑︁
(𝑘 −1) ª=𝜎­
𝛼𝑖 𝑗 Wv 𝑗®,« 𝑗 ∈𝑁𝑖
¬

(6)
(𝑘 )

in which 𝑁𝑖 is the set of neighbor nodes for v𝑖/people/sports

and W is the(𝑘 −1)(𝑘 −1)
weight matrix. 𝛼𝑖 𝑗 is the attention score of node v𝑖for v 𝑗,which is computed as:/people/deceased_person/people/deceased_person/place_of_death/people/deceased_person/place_of_burial/people/ethnicity/people/ethnicity/geographic_distribution
𝑒𝑥𝑝 𝐿𝑒𝑎𝑘𝑦𝑅𝑒𝐿𝑈 𝑒𝑖 𝑗
𝛼𝑖 𝑗 = Í,𝑘 ∈𝑁𝑖 𝑒𝑥𝑝 (𝐿𝑒𝑎𝑘𝑦𝑅𝑒𝐿𝑈 (𝑒𝑖𝑘 ))/sports/sports_team/sports/sports_team/location

where 𝐿𝑒𝑎𝑘𝑦𝑅𝑒𝐿𝑈 (·) is an activation function, 𝑒𝑖 𝑗 is the originalattention score before the above softmax operation, and is calculated
as follows:(𝑘 −1)

𝑒𝑖 𝑗 = a Wv𝑖each layer of the hierarchies. For instance, as the father of the longtailed relation /𝑝𝑒𝑜𝑝𝑙𝑒/𝑑𝑒𝑐𝑒𝑎𝑠𝑒𝑑_𝑝𝑒𝑟𝑠𝑜𝑛/𝑝𝑙𝑎𝑐𝑒_𝑜 𝑓 _𝑑𝑒𝑎𝑡ℎ, relation/𝑝𝑒𝑜𝑝𝑙𝑒/𝑑𝑒𝑐𝑒𝑎𝑠𝑒𝑑_𝑝𝑒𝑟𝑠𝑜𝑛 contains many other children relations
such as /𝑝𝑒𝑜𝑝𝑙𝑒/𝑑𝑒𝑐𝑒𝑎𝑠𝑒𝑑_𝑝𝑒𝑟𝑠𝑜𝑛/𝑝𝑙𝑎𝑐𝑒_𝑜 𝑓 _𝑏𝑢𝑟𝑖𝑎𝑙. All the training sentences of children relations construct the training data of/𝑝𝑒𝑜𝑝𝑙𝑒/𝑑𝑒𝑐𝑒𝑎𝑠𝑒𝑑_𝑝𝑒𝑟𝑠𝑜𝑛, which can effectively alleviate the problem of insufficient training data.
Furthermore, we employ entity types to connect different relations. Firstly, for these relations in the bottom level of the hierarchical tree, we can find the entity type constraint information from thecorresponding knowledge base. Due to the large number of entity
types contained in the original constraint information, we adopt
18 entity types T = {𝑡 1, ..., 𝑡 18 } defined in 𝑂𝑛𝑡𝑜𝑁𝑜𝑡𝑒𝑠 5.0 [25], anda unified type 𝑂𝑡ℎ𝑒𝑟𝑠 is defined to represent entities that do notbelong to the 18 types following [19]. Then, for other relations at
higher levels of the hierarchical tree, because the semantic information of these relations is too broad to be constrained by specifictypes, we use a special type 𝑁𝑜𝑛𝑒 to represent them. Finally, we
denote the raw hierarchical constraint graph as G = {T , R, C},
where T and R are the entity type set and relation set respectively, and they form the node set V = T ∪ R. For each constraint12
12(𝑡𝑟𝑒 , 𝑟, 𝑡𝑟𝑒 ) ∈ C, 𝑡𝑟𝑒 and 𝑡𝑟𝑒 denote as the head entity type and tail1
2
entity type of the relation 𝑟 , respectively. (𝑡𝑟𝑒 , 𝑟 ) and (𝑟, 𝑡𝑟𝑒 ) form
the edge set E.

(𝑘 −1); Wv 𝑗(8)(𝑘 −1)

features Wv𝑖; Wv 𝑗to the attention score.
Finally, for the raw hierarchical constraint graph G, we canget the output V (𝑘 ) = {v𝑘1 , ..., v𝑘𝑛 𝑣 } of GAT encoder, which can bedivided into relation feature representations R = {r1, ..., r𝑛𝑟 } andentity type feature representations T = {t1, ..., t𝑛𝑡 } according to thecategory of each node.

3.5

Hierarchical constrained attention

In this section, we design a hierarchical constrained attention network to construct the bag representation. To further exploit thebackground information, we concatenate the entity types to thecorresponding sentences and relations at each level of the relation
hierarchy, which can improve the ability of the selective attention
mechanism to identify valid sentences. For each relation ri ∈ R,
the corresponding entity types are obtained from the hierarchicalconstraint graph and concatenated to the relation ri as follows:12

c𝑖 = [r𝑖 ; t𝑟𝑒 ; t𝑟𝑒 ],(9)

For each sentence U in the sentence bag, the NER tool2 is em1
2
ployed to recognize the corresponding entity pair types (t𝑢𝑒 , t𝑢𝑒 ),which are concatenated to the sentence U as follows:1

2

X = [U; t𝑢𝑒 ; t𝑢𝑒 ],(10)Then, we use the relation-augmented mechanism [18] to embed
relations to sentence representations at different levels of the relation hierarchy, which can further facilitate the information transfer
among different relations. Specifically, at the 𝑘-level of relation
hierarchy, the sentence representation X is applied as a query to
(𝑘 )
(𝑘 )
relation set C (𝑘 ) = {c1 , ..., c𝑛𝑟 } with a dot product:𝛽 (𝑘 ) = 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (XT C (𝑘 ) ),3.4.2 Graph attention networks (GAT). We use the graph attention networks (GAT) [32] to encode the raw hierarchical constraintgraph. Compared with other graph neural networks, GAT can effectively learn graph structure information and assign different
weights to neighbor nodes. Note that all the node representationsof the raw hierarchical constraint graph G are randomly initialized
as a matrix V = {v1, ..., v𝑛 𝑣 }, each vector v𝑖 ∈ V denotes a node in,

where a is a weight vector to map the concatenated high-dimensional(𝑘 −1)

Figure 3: Relation hierarchy.

(7)g (𝑘 ) = C (𝑘 ) 𝛽 (𝑘 ) ,(11)

where 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (·) denotes a normalization function along the last
dimension. In addition, an element-wise gate with residual connection [10] and layer normalization [1] are used to merge the2 https://github.com/flairNLP/flair

IJCKG’23, December 08–09, 2023, Tokyo, Japan

Bao Liu, Guilin Qi and Yuxin Zhang

relation-aware representation g (𝑘 ) into the sentence representation X:
(𝑘 )

(𝑘 )

𝛾 (𝑘 ) = 𝑠𝑖𝑔𝑚𝑜𝑖𝑑 (W𝑔 [X; g (𝑘 ) ] + b𝑔 ),e (𝑘 ) = 𝛾 (𝑘 ) ◦ X + (1 − 𝛾 (𝑘 ) ) ◦ g (𝑘 ) ,
X
X
(𝑘 )

(𝑘 )

(12)

e (𝑘 ) )),= 𝐿𝑎𝑦𝑒𝑟 𝑁𝑜𝑟𝑚(X + 𝑀𝐿𝑃 ( X(𝑘 )where W𝑔 and b𝑔 are both learnable parameters, 𝑀𝐿𝑃 (·) denotes a multi-layer perceptron to increase nonlinearity.
Next, the attention score 𝜆𝑖 of each sentence X (𝑘 ) and the relation c can be obtained as(𝑘 )𝑒𝑖 = X𝑖c,𝑒𝑥𝑝 (𝑒𝑖 )𝜆𝑖 = Í𝑛,𝑗=1 𝑒𝑥𝑝 (𝑒 𝑗 )(13)Then, we obtain the bag representation at each level of the relation hierarchy as follows:z (𝑘 ) =

𝑛
∑︁

(𝑘 )

𝜆𝑖 X𝑖

,

4 EXPERIMENTS
4.1 Dataset and evaluation metrics
We adopt the widely used NYT [27] as the datasets to conduct ourexperiments. NYT is developed by aligning the corps of New YorkTimes with relational facts in Freebase [2], which has been mainly
released in the filtered version NYT-520K and non-filtered version
NYT-570K. They both contain 53 relations which include a specific
𝑁 𝐴 denoting no relation between the entity pair. The difference
between the two versions is that the training set and test set of
NYT-520K do not contain the same entity pairs. In the training set,
the sentence numbers of NYT-520K and NYT-570K are 523 312 and
570 088, respectively, while the sentence numbers of NYT-520K and
NYT-570K are both 172 448 in the test set.
We conduct the comparison experiments using four standard
metrics following [9, 19, 18], which are precision-recall (PR) curves,the area under a curve (AUC), Top-N precision (P@N) and Hits@K.
The first three metrics are adopted for the denoising evaluation,and the last one is used for the long-tailed evaluation. The detailed
definitions are displayed as follows:• PR curve exhibits the tradeoff between precision (y-axis)
and recall (x-axis) for different probability thresholds.

(14)

𝑖=1

Finally, we concatenate all the bag representations at each level ofthe relation hierarchy as the ultimate representation of sentence bag,and a softmax classifier is used to obtain a categorical distribution
over all relations as bag prediction:
𝑃 = (𝑟 |B; G; 𝜃 ) = 𝑠𝑜 𝑓 𝑡𝑚𝑎𝑥 (Wz + b)

𝑁𝑡𝑟𝑢𝑒_𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒,
𝑁𝑡𝑟𝑢𝑒_𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 + 𝑁 𝑓 𝑎𝑙𝑠𝑒_𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒
𝑁𝑡𝑟𝑢𝑒_𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒
𝑟𝑒𝑐𝑎𝑙𝑙 =,𝑁𝑡𝑟𝑢𝑒_𝑝𝑜𝑠𝑖𝑡𝑖𝑣𝑒 + 𝑁 𝑓 𝑎𝑙𝑠𝑒_𝑛𝑒𝑔𝑎𝑡𝑖𝑣𝑒
𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 =(15)• AUC indicates the area under the PR curve. A model with ahigher AUC score achieves better performance.
• P@N is designed to evaluate the relation extraction models with multi-instance learning and denotes the precision
values of the sentences with top-N prediction confidences.
• Hits@K measures the probability that the true relation isachieved in the top-K predictions of the model for the sentences with a long-tailed distribution.

where 𝜃 is the set of parameters, W and b are the weight and bias,
respectively.

3.6

(18)

Optimization

A main objective and an auxiliary objective are introduced to optimize our model. The main objective is a bag-level prediction and isdefined as minimizing cross-entropy:𝑚

𝐿 (𝑟𝑒 ) = −1 ∑︁𝑙𝑜𝑔𝑃 (𝑟𝑖 |B𝑖 ; G; 𝜃 ),
𝑚 𝑖=1

(16)

where 𝑚 is the number of bags and 𝑟𝑖 is the relation of B𝑖 .
The auxiliary objective [18] is used to guide each sentence augmenting with correct relation representations at each level of therelation hierarchy. That is,
𝐿 (ℎ𝑖𝑒𝑟 ) = −

𝑚 ∑︁
𝑛 ∑︁
𝑙
∑︁
1
(𝑘 )
𝑙𝑜𝑔𝛽 (𝑘 ) ,𝑚 · 𝑛 · 𝑙 𝑖=1 𝑗=1
[𝑟𝑖 ]

(17)

𝑘=1

where [·] is indexing operation, 𝑛 denotes the number of sentencesin each bag, and 𝑙 is the number of relation hierarchy levels.
Eventually, the two objectives above are integrated to optimize
the proposed model, i.e., 𝐿 = 𝐿 (𝑟𝑒 ) + 𝐿 (ℎ𝑖𝑒𝑟 ) .
In the test phase, for the input sentences, the ground-truth label
is unknown. Therefore, we calculate posterior probabilities of all
the relations and select the highest one as the prediction result [20].

4.2

Baselines

We compare our proposed HCAT model with the following fivecompetitive methods:• PCNN+ATT [20] designs selective attention over multiple
sentences to alleviate wrong labeling, which is the mostclassical model for the DSRE task.
• PCNN+HATT [9] uses hierarchical attention to exploit thecorrelations of relations.
• SeG [17] proposes an entity-aware embedding approachmerging the position features and entity embeddings to thesentence representation.
• CoRA [18] designs a collaborating relation-augmented attention network incorporating relational information into
sentence representation.
• CGRE [19] proposes a novel constrain graph-based DSRE
model by concatenating the entity type into the sentence
and relation representation.

Hierarchical constrained attention for distantly supervised relation extraction

Table 1: Parameter settings.
Component

Parameter

NYT-520K

NYT-570K

Sentence
encoderfilter number
window size
word size
position size
coefficient 𝜆

230
3
50
5
18

230
3
50
5
18

Graph
encoder

embedding size
hidden size
layer number
output size

100
400
2
1250

100
400
2
1250

Classifier

input size

900

900

Optimizationbatch size
learning rate
dropout rate

160
0.03
0.5

160
0.08
0.5

4.3

Experimental settings

All the experiments are conducted on the Ubuntu 18.04 platform.
We employ the pre-trained word2vec from OpenNRE3 to initial
the word embeddings, and adopt Xavier [6] to initialize all weightmatrixes and position vectors, all the bias vectors are initialized to
0. We use mini-batch SGD [3] for optimization and apply dropout[30] before the classifier layer to prevent overfitting. Table 1 shows
the detailed settings of our experiments.

4.4

Comparision results

The comparison results of different methods are shown for denoising and long-tail relation problems. All results of baseline CGRE4
are from the original paper, however, the experiments on the otherfour baselines (i.e., PCNN+ATT3 , PCNN+HATT5 , SeG6 and CoRA7 )were only conducted on NYT-570K, so we experiment with the official source code to obtain these results not reported in the original
papers.
To compare the denoising performance of different models, following the previous works [9, 19, 18], we conduct experiments
using three widely used metrics: P@N, AUC, and PR curve. As illustrated in Table 2, our proposed HCAT achieves optimal values inall metrics at the same time. Specifically, For P@N, HCAT improvesbaseline approaches by at least 2.2 percentage points. Our model
achieves the AUC of 0.422 and 0.546 on NYT-520K and NYT-570K,
which outperforms the strong baselines by 0.02 and 0.17, respectively. The PR curves corresponding to AUC are shown in Figure 4.
These comparison results indicate our proposed hierarchical constrained attention mechanism can significantly improve the model
performance in handling the problem of wrong-labeled relation
extraction.
For the long-tailed distribution, we conduct experiments on the
NYT-570 dataset following the previous works [9, 19, 18], Hits@K
3 https://github.com/thunlp/OpenNRE
4 https://github.com/tmliang/CGRE
5 https://github.com/thunlp/HNRE
6 https://github.com/YangLi1221/SeG
7 https://github.com/YangLi1221/CoRa

IJCKG’23, December 08–09, 2023, Tokyo, Japan[9] is employed as the evaluation metric. We extract the long tailrelations with less than 100/200 training instances from the test
set. Then the Hits@K metric is used to compute the probability
that the true label falls into the top-K recommendations of themodel. We also use macro average during the computing process.
As illustrated in Table 3, our proposed HCAT achieves the best
4 out of 6 settings and the second-best value in the remaining 2
settings. In addition, compared with the other models, our proposed
HCAT improves at least 7.8 percentage points on both top-15 andtop-20 recommendations with training instances fewer than 100
and 200. This shows that our proposed HCAT is better at extracting
relations with long-tailed distributions due to the efficient transfer
of information between different relations.
Compared with CGRE, our model has achieved improvements
in all metrics on the two widely used datasets NYT-520K and NYT570K in terms of denoising. In terms of long-tail evaluation, ourmodel HCAT achieves progress on Hits@15 and Hits@20 but regresses on Hits@10, a possible reason is that our model exploits therelation hierarchy information based on CGRE, which improves therelation extraction performance while increasing the model complexity, so more data is required for training. Therefore, adopting
other methods to use relation hierarchy information based on asmall increase in the model complexity is worth exploring in future
research.

4.5

Ablation study

Our proposed distantly supervised relation extraction model HCAT
exploits the background information such as relation hierarchy andentity types. To verify the effect of each element in improving theperformance of the model, we separately remove them from themodel as follows:• ∼ w/o Hier. The relation hierarchy is removed from the
proposed model HCAT.
• ∼ w/o Typ. The entity types are deleted from both the sentence and relation representations.
• ∼ w/o Gra. The graph encoder is removed, and the embeddings of all relations and entity types are randomly generated.
The results are shown in Table 4, it can be seen that comparedto HCAT, the performance of all models is degraded on multiple
metrics. Experimental results show that employing relation hierarchy and entity type can effectively improve the performance ofdistantly supervised relation extraction models for both denoisingand long-tailed relation extraction.

5

CONCLUSION AND FUTURE WORKIn this paper, we design a hierarchical constrained attention to
exploit the background information such as relation hierarchy andentity types. Furthermore, we propose a hierarchical constrained
attention-based distantly supervised relation extraction framework(HCAT), which adopts a hierarchical relation extraction framework
to facilitate the transfer of information between data-rich relations
and long-tailed ones. In addition, entity types are employed forconstructing hierarchical constraint graphs and concatenating to
the corresponding relations and sentences, which can further facilitate the information sharing between different relations andIJCKG’23, December 08–09, 2023, Tokyo, Japan

Bao Liu, Guilin Qi and Yuxin Zhang

Table 2: (%)P@N and AUC values of different models on NYT-520K and NYT-570K.

Model
PCNN+ATT [20]PCNN+HATT [9]SeG [17]
CoRA [18]CGRE [19]HCAT (Ours)

P@100
74.2
73.1
83.0
80.2
82.7
87.0

1.00

AUC
34.7
37.9
39.5
42.0
41.7
42.2

0.90

0.80
0.75

0.85
0.80
0.75
0.70

0.65

0.65
0.05

0.10

0.15
Recall

0.20

0.25

0.30

PCNN+ATT
PCNN+HATT
SeG
CoRA
CGRE
HCAT(Ours)

0.90

0.70

0.60
0.00

AUC
39.2
42.1
51.2
52.9
51.9
54.6

0.95

Precision

0.85

NYT-570K
P@200 P@300
74.0
71.5
80.9
79.2
85.7
80.2
85.9
81.5
86.4
81.8
93.0
87.3

P@100
73.1
85.3
86.5
90.1
88.9
95.0

1.00

PCNN+ATT
PCNN+HATT
SeG
CoRA
CGRE
HCAT(Ours)

0.95

Precision

NYT-520K
P@200 P@300
72.5
68.0
72.5
72.3
77.5
73.7
79.0
76.3
80.3
76.5
82.5
80.0

0.60
0.00

0.05

0.10

(a) NYT-520K

0.15
Recall

0.20

0.25

0.30

(b) NYT-570K

Figure 4: Precision-recall (PR) curves.
Table 3: Hits@K (Macro) on relations with training instances< 100/200.
Training instances

<100

<200

Table 4: (%)Hits@K (Macro) and AUC values of different models on NYT-570K.
<100

Model

Hits@K

10

15

20

10

15

20

PCNN+ATT [20]PCNN+HATT [9]SeG [17]
CoRA [18]CGRE [19]

<5.0
29.6
62.1
66.6
77.8

7.4
51.9
70.9
72.0
77.8

40.7
61.1
83.5
87.0
87.0

17.2
41.4
70.6
72.7
81.8

24.2
60.6
75.2
77.3
81.8

51.5
68.2
88.9
89.4
89.4

HCAT (Ours)

70.0

88.3

96.7

75.0

90.3

97.2

identify the valid sentences for bag representations. Substantial experiments show that our proposed HCAT achieves state-of-the-art
performance on multiple metrics, including denoising metrics (i.e.,
PR curve, AUC, P@N) and long tail metric (Hits@K).
In the future, we will explore some directions as follows:• Effective approach to exploit more background information.
There is also a lot of other background information related
to distantly supervised relation extraction, it is desirable toexploit them for further performance improvement.

<200

AUC

Hits@10

Hits@20

Hits@10

Hits@20∼ w/o Hie∼ w/o Typ∼ w/o Gra

67.5
68.9
65.7

94.1
93.9
90.3

74.1
74.3
73.6

95.1
94.9
91.5

53.2
53.5
52.9

HCAT

70.0

96.7

75.0

97.2

54.6

• Application in other RE scenarios. Our proposed hierarchicalconstrained attention is a universal module, which can beeasily applied to other RE tasks, i.e., document-level relation
extraction, and multimodal relation extraction.

ACKNOWLEDGMENTS
This work is partially supported by National Nature Science Foundation of China under No. U21A20488. We thank the Big Data
Computing Center of Southeast University for providing the facility support on the numerical calculations in this paper.

Hierarchical constrained attention for distantly supervised relation extraction

REFERENCES
[1]
[2]

[3]
[4]

[5]
[6]

[7]
[8]
[9]

[10]
[11]

[12]

[13]
[14]
[15]

[16]
[17]

[18]

[19]

[20]

[21]

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. CoRR, abs/1607.06450.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.
2008. Freebase: a collaboratively created graph database for structuring humanknowledge. In Proceedings of the 2008 ACM SIGMOD international conference
on Management of data, 1247–1250.
Andrew Cotter, Ohad Shamir, Nati Srebro, and Karthik Sridharan. 2011. Bettermini-batch algorithms via accelerated gradient methods. Advances in neural
information processing systems, 24.
Qin Dai, Benjamin Heinzerling, and Kentaro Inui. 2022. Cross-stitching text
and knowledge graph encoders for distantly supervised relation extraction. In
Proceedings of the 2022 Conference on Empirical Methods in Natural Language
Processing, 6947–6958.
Huifang Du, Zhongwen Le, Haofen Wang, Yunwen Chen, and Jing Yu. 2022.
Cokg-qa: multi-hop question answering over covid-19 knowledge graphs. DataIntelligence, 4, 3, 471–492.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training
deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. JMLR Workshop andConference Proceedings, 249–256.
Lingbing Guo, Qingheng Zhang, Wei Hu, Zequn Sun, and Yuzhong Qu. 2019.
Learning to complete knowledge graphs with deep sequential models. DataIntelligence, 1, 3, 289–308.
Ridong Han, Tao Peng, Jiayu Han, Hai Cui, and Lu Liu. 2022. Distantly supervised relation extraction via recursive hierarchy-interactive attention andentity-order perception. Neural Networks, 152, 191–200.
Xu Han, Pengfei Yu, Zhiyuan Liu, Maosong Sun, and Peng Li. 2018. Hierarchicalrelation extraction with coarse-to-fine grained attention. In Proceedings of the2018 Conference on Empirical Methods in Natural Language Processing, 2236–
2245.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual
learning for image recognition. In Proceedings of the IEEE conference on computer
vision and pattern recognition, 770–778.
Raphael Hoffmann, Congle Zhang, Xiao Ling, Luke Zettlemoyer, and Daniel S
Weld. 2011. Knowledge-based weak supervision for information extraction of
overlapping relations. In Proceedings of the 49th annual meeting of the association
for computational linguistics: human language technologies, 541–550.
Guoliang Ji, Kang Liu, Shizhu He, and Jun Zhao. 2017. Distant supervision
for relation extraction with sentence-level attention and entity descriptions.
In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence,
3060–3066.
Ting Jia, Yuxia Yang, Xi Lu, Qiang Zhu, Kuo Yang, and Xuezhong Zhou. 2022.
Link prediction based on tensor decomposition for the knowledge graph ofcovid-19 antiviral drug. Data Intelligence, 4, 1, 134–148.
Zhengbao Jiang, Zhicheng Dou, and Ji-Rong Wen. 2016. Generating query facets
using knowledge bases. IEEE transactions on knowledge and data engineering,29, 2, 315–329.
Junzhuo Li and Deyi Xiong. 2022. Kafsp: knowledge-aware fuzzy semantic parsing for conversational question answering over a large-scale knowledge base.
In Proceedings of the 60th Annual Meeting of the Association for Computational
Linguistics, 461–473.
Tingwei Li and Zhi Wang. 2023. Ldrc: long-tail distantly supervised relation
extraction via contrastive learning. In Proceedings of the 2023 7th International
Conference on Machine Learning and Soft Computing, 110–117.
Yang Li, Guodong Long, Tao Shen, Tianyi Zhou, Lina Yao, Huan Huo, andJing Jiang. 2020. Self-attention enhanced selective gate with entity-aware embedding for distantly supervised relation extraction. In Proceedings of the AAAI
conference on artificial intelligence number 05. Vol. 34, 8269–8276.
Yang Li, Tao Shen, Guodong Long, Jing Jiang, Tianyi Zhou, and Chengqi
Zhang. 2020. Improving long-tail relation extraction with collaborating relationaugmented attention. In Proceedings of the 28th International Conference on
Computational Linguistics, 1653–1664.
Tianming Liang, Yang Liu, Xiaoyan Liu, Hao Zhang, Gaurav Sharma, andMaozu Guo. 2023. Distantly-supervised long-tailed relation extraction using
constraint graphs. IEEE Transactions on Knowledge & Data Engineering, 35, 07,
6852–6865.
Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, and Maosong Sun. 2016.
Neural relation extraction with selective attention over instances. In Proceedingsof the 54th Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers), 2124–2133.
Ruri Liu, Shasha Mo, Jianwei Niu, and Shengda Fan. 2022. Ceta: a consensus
enhanced training approach for denoising in distantly supervised relation
extraction. In Proceedings of the 29th International Conference on Computational
Linguistics, 2247–2258.

IJCKG’23, December 08–09, 2023, Tokyo, Japan

[22]
[23]

[24]
[25]

[26]
[27]
[28]
[29]
[30]
[31]

[32]
[33]
[34]

[35]
[36]

[37]

[38]

[39]

[40]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.
Distributed representations of words and phrases and their compositionality.
Advances in neural information processing systems, 26.
Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint
Conference of the 47th Annual Meeting of the ACL and the 4th International Joint
Conference on Natural Language Processing of the AFNLP, 1003–1011.
Tao Peng, Ridong Han, Hai Cui, Lin Yue, Jiayu Han, and Lu Liu. 2022. Distantly
supervised relation extraction using global hierarchy embeddings and local
probability constraints. Knowledge-Based Systems, 235, 107637.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders
Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards
robust linguistic analysis using ontonotes. In Proceedings of the Seventeenth
Conference on Computational Natural Language Learning, 143–152.
Jianfeng Qu, Dantong Ouyang, Wen Hua, Yuxin Ye, and Ximing Li. 2018. Distantsupervision for neural relation extraction integrated with word attention andproperty features. Neural Networks, 100, 59–69.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations
and their mentions without labeled text. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 148–163.
Ying Shen, Ning Ding, Hai-Tao Zheng, Yaliang Li, and Min Yang. 2020. Modeling
relation paths for knowledge graph completion. IEEE Transactions on Knowledge
and Data Engineering, 33, 11, 3607–3617.
Wei Song, Weishuai Gu, Fuxin Zhu, and Soon Cheol Park. 2023. Interaction-andresponse network for distantly supervised relation extraction. IEEE Transactions
on Neural Networks and Learning Systems, 193–201.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan
Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks fromoverfitting. The journal of machine learning research, 15, 1, 1929–1958.
Mihai Surdeanu, Julie Tibshirani, Ramesh Nallapati, and Christopher D Manning. 2012. Multi-instance multi-label learning for relation extraction. In Proceedings of the 2012 joint conference on empirical methods in natural language
processing and computational natural language learning, 455–465.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, PietroLiò, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Suizhu Yang, Yanxia Liu, Yuantong Jiang, and Zhiqiang Liu. 2023. More refinedsuperbag: distantly supervised relation extraction with deep clustering. Neural
Networks, 157, 193–201.
Zhi-Xiu Ye and Zhen-Hua Ling. 2019. Distant supervision relation extraction
with intra-bag and inter-bag attentions. In Proceedings of the 2019 Conferenceof the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, Volume 1 (Long and Short Papers), 2810–2819.
Bowen Yu, Zhenyu Zhang, Tingwen Liu, Bin Wang, Sujian Li, and Quangang
Li. 2019. Beyond word attention: using segment attention in neural relation
extraction. In IJCAI, 5401–5407.
Erxin Yu, Wenjuan Han, Yuan Tian, and Yi Chang. 2020. Tohre: a top-downclassification strategy with hierarchical bag representation for distantly supervised relation extraction. In Proceedings of the 28th international conference on
computational linguistics, 1665–1676.
Yujin Yuan, Liyuan Liu, Siliang Tang, Zhongfei Zhang, Yueting Zhuang, Shiliang Pu, Fei Wu, and Xiang Ren. 2019. Cross-relation cross-bag attention for
distantly-supervised relation extraction. In Proceedings of the AAAI conference
on artificial intelligence number 01. Vol. 33, 419–426.
Daojian Zeng, Kang Liu, Yubo Chen, and Jun Zhao. 2015. Distant supervision for
relation extraction via piecewise convolutional neural networks. In Proceedingsof the 2015 conference on empirical methods in natural language processing,
1753–1762.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In Proceedings of
COLING 2014, the 25th international conference on computational linguistics:
technical papers, 2335–2344.
Ningyu Zhang, Shumin Deng, Zhanlin Sun, Guanying Wang, Xi Chen, Wei
Zhang, and Huajun Chen. 2019. Long-tail relation extraction via knowledge
graph embeddings and graph convolution networks. In Proceedings of the 2019
Conference of the North American Chapter of the Association for Computational
Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),
3016–3025.