Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks
Authors: Zhaohui Yan
Songlin Yang
Wei Liu
Kewei Tu
Venue: EMNLP 2023 Main (2023)
Keywords: Entity and Relation Extraction
high-order inference
hypergraph neural network
TL;DR: This article proposes a novel joint Entity and Relation Extraction model, which employs hypergraph neural network for higher-order modeling and outperforms the previous state-of-the-art.

Abstract: Entity and Relation Extraction (ERE) is an important task in information extraction. Recent marker-based pipeline models achieve state-of-the-art performance, but still suffer from the error propagation issue. Also, most of current ERE models do not take into account higher-order interactions between multiple entities and relations, while higher-order modeling could be beneficial.In this work, we propose HyperGraph neural network for ERE (HGERE), which is built upon the PL-marker (a state-of-the-art marker-based pipleline model). To alleviate error propagation, we use a high-recall pruner mechanism to transfer the burden of entity identification and labeling from the NER module to the joint module of our model. For higher-order modeling, we build a hypergraph, where nodes are entities (provided by the span pruner) and relations thereof, and hyperedges encode interactions between two different relations or between a relation and its associated subject and object entities. We then run a hypergraph neural network for higher-order inference by applying message passing over the built hypergraph. Experiments on three widely used benchmarks (ACE2004, ACE2005 and SciERC) for ERE task show significant improvements over the previous state-of-the-art PL-marker.

Joint Entity and Relation Extraction with Span Pruning and Hypergraph Neural Networks
Zhaohui Yan 1.2 Songlin Yang"; Wei Liu Kewei Tu
School of Information Science and Technology, ShanghaiTech University
Shanghai Engineering Research Center of Intelligent Vision and Imaging
MITT CSAIL
fyanzhh, liuwei4, tukw)eshanghai tech. edu.
yangs166emit. edu
Abstract
Recently, the seminal work of (Zhong and Chen, 2021) shows that pipeline decoding with a frus- tratingly simple marker-based encoding strategy i.e., inserting solid markers (Baldini Soares et al., 2019; Xiao et al., 2020) around predicted subject and object spans in the input text achieves state-of-the-art RE performance. Modified sen- tences (with markers) are fed into powerful pre- trained large language models (LLM) to obtain more subject- and object-aware representations for RE classification, which is the key to the perfor- mance improvement. However, current marker- based pipeline models (e.g., the recent state-of-the- art ERE model PL-marker (Ye et al., 2022)) only send predicted entities from the NER module to the RE module, therefore missing entities would never have the chance to be re-predicted, suffering from the error propagation issue. On the other hand, for joint decoding approaches (e.g. Table Filling methods (Miwa and Sasaki, 2014; Zhang et al., 2017; Wang and Lu, 2020)) suffer from the error propagation issue-it is hard to incorporate markers for leveraging LLMs, since entities are not predicted prior to relations. Our de- sire is to obtain the best of two worlds, being able to marker-based encoding mechanism for enhanc- ing RE performance and meanwhile alleviating the error propagation problem. We adopt PL-marker as the backbone of our proposed model and a span pruning strategy to mitigate error propagation. That is, instead of sending only predicted entity spans to the RE module, we over-predict candidate spans that the recall of gold entity spans is nearly perfect (but there also could be many non-entity spans), transferring the burden of entity classification and labeling from the NER module to the RE module of PL-marker. The number of over-predicted spans is upper- hoperrbbuuded, balancing the computational com- plexity of marker-based encoding and the recall of gold entity span. Empirically, we find this simple strategy by itself clearly improves PL-marker.
Entity and Relation Extraction (ERT)) is im- portant task in information extraction. Recent marker-based pipeline models achieve state-of- the-art performance, but still suffer from the егтоr propagation issue, Also, most current ERE models do not take into account higher- order interactions between multiple entities and relations, while higher-order modeling could be beneficial.In In this work, we propose Hyper- Graph neural network for ERE (HGERE), which is built upon the PL-marker (a state-of-the-ard marker-bee based pipleline model). To alleviate error propagation,ww we use high-recall pruner mechanism transfer the burden of entity iden- tification and labeling from the NER module to the joint module our model. For higher-order modeling, build hypergraph, where nodes are entities (provided by the span pruner) and relations thereof, and hyperedges encode inter- actions between two different relations be- tween a relation and its associated subject and object entities. We then run hypergraph neu- ral network for higher-order inference by apply- ing message passing over the built hypergraph. Experiments three widely used benchmarks (ACE2004, ACE2005 and SciERC) for ERE task show significant improvements over of-t--o--hh-rrr the--ar PL-marker.
they do not
previous state-of
1 Introduction
Entity and Relation Extraction (ERE) is a funda- mental task in information extraction (IE), compro- mising two sub-tasks: Named Entity Recognition (NER) and Relation Extraction (RE). There is long debate joint vs. pipeline methods for ERE. Pipeline decoding extracts entities first and pre- dicts relations solely on pairs of extracted entities, while joint decoding predicts entities and relations simultancously.
"This work was done when Songlin at ShanghaiTech. Corresponding Author 'Souree oode is availahel at httpps://gite http::///gtttuuu.e com/ yanzhh/HGERE

---

We further incorporate higher-order interac- tion module into our model. Most previous ERE models either implicitly model the interactions be- tween instances by shared parameters (Wang and Lu, 2020; Yan et al., 2021; ; Wang et al., 2021) or use traditional graph neural network that mod- els pairwise connections between relation and an entity (Sun et al., 2019). It is difficult for these ap- proaches to explicitly model higher-order relation- ships among mulli-instances, e.g. the dependency among relation and its corresponding subject and object entitics. Many recent works structured prediction tasks show that explicit higher-order modeling is still beneficial even with powerful large pretrained encoders (Zhang et al., 2020a; Li et al., 2020; Yang and Tu, 2022; Zhou et al., 2022, inter alia), motivating us to order module to enhance performance.
from the span pruner) and relations thereof, and hyperedges encode the interactions between either two relations with shared entities or a relation and its associated subject and object entity spans. In contrast, existing GNN models for IE (Sun et al., 2019; Nguyen et al., 2021) only model the pair- wise interactions between relation and one of its corrsponding entity. We empirically show the ad- vantages of our higher-order interaction module (i.e., hypergraph neural network) over MFVI and tranditional GNN models.
Our contribution is three-fold: i) We adopt a simple and effective span pruning method to mit- igate the eтror propagation issue, enforcing the power of marker-based encoding. ii) We pro- pose novel hypergraph neural network enhanced higher-order model, outperforming higher-order CRF-based models with MFVI. iii) We show great improvements over the prior state-of-the-ar PI.- marker on three commonly used benchmarks for ERE: ACE2004, ACE2005 and SciERC.
additional higher-
A common higher-order modeling approach is by means of probabilistic modeling (i.e., condi- tional random field (CRF)) with end-to-end Mean- Field Variational Inference (MFVI), which can be seamlessly integrated into neural networks as a re- current neural network layer (Zheng et al., 201 5a). and has been widely used various structured pre- diction tasks, such as dependency parsing (Wang et al., 2019), semantic role labeling (Li et al., 2020; Zhou et al., 2022), and information extraction (Jia et al., 2022). However, the limitations of CRF modeling with MFVI are i): CRF's potential func- tions are parameterized in log-linear forms with strong independence assumptions, suffering from low model capacitics (Qu et al., 2022), ii) MFVI uses
2 Background
2.1 Problem formulation
Given sentence X with n tokens: 21, Ø2, ... Zn- an entity span is sequence of tokens labeled with an entity type and a relation is an entity span pair labeled with a relation type. We denote the set of all entity spans of the sentence with a span length limit L by S(X) ST()) and ED( as the start and end token indices of the span 8;-
{s1, $2,, 8m) } and define
The joint ERE task is to simultancously solve the NER and RE tasks. Let C. be the set of entity types and C, be the set relation types. For cach span 8; E S(X), the NER task is to predict an entity type (e,) E Ce or (s:) = null if the span is not an entity. The RE task is to predict a relation type yr (rgz C, or yr (riz) null for each span rij (si, 8j), 81,85 S(X).
Bernoulli distributions to ap- proximate the otherwise multimodal true posterior distributions, oversimplifying the inference prob- lem and thus is sub-optimal. Therefore we need more expressive tools improve the quality of hideer-order inference. Fortunately, there are many recent works in the machine learning community showing that graph neural networks (GNN) can be used as an inference tool and outperform approxi- mate statistical inference algorithms (e.g., MFVI) (Yoon et al., 2018; Zhang et al., 2020b; Kuck et al., 2020; Satorras and Welling, 2021) (see (Hua, 2022) for survey). Inspired by these works, we employ hypergraph neural network (HyperGNN) instead of MFVI for high-order inference and propose our model HGERE (HyperGraph Neural Network for ERE). Concretely, we build a hypergraph where nodes are candidate subjects and objects (obtained
2.2 Packed levitated marker (PL-marker)
Zhong and Chen (2021) insert two pairs of solid markers (i.e., [S] and [\,S]) to highlight both the subject and object entity spans in given sentence, and this simple approach achieves state-of-the-ar RE performance. We posit that this is because LLM more aware of the subject and object spans (with markers) and thus can produce better span repre- sentations to improve RE. But this strategy needs to iterate over all possible entity span pairs and is

---

Pretrained pruner
Hyper- GNN
sentences
Encoder
Classifier
Predictions
K(K
1) relation representations
An example hypergraph "tersibcopgp"
K unique object
subject node
temary odge
representutions
sibling edge
object node
co-purent edge
Max-pooling
relation node
grundpurent edge
subject representations
representations
21
One output sequences
as an example
K input sequences
for a sentence with spans
Xeray Mrneenyssss..................
... Jole: Pol. ...o
Figure I: Illustration of our framework
thus slow. To tackle the efficiency problem, Zhong and Chen (2021) propose an approximated vari- ant wherein each possible entity span is associated with pair of levitated markers (i.e., [O] and ()) whose representations are initialized with the posi- tional embedding of the span's 's start and end tokens, and all such levitated markers are concatenated to the end of the sentence. As such, levitated-mart based encoding needs only one pass, significantly improving efficiency at the cost of slight perfor- mance drop. Zhong and Chen (2021) also propose a masked attention mechanism such that the origi- nal input text tokens are not able to attend to mark- ers, while markers can attend to paired markers (but not unpaired markers) and all input text tokens. As a consequence, the relative positions of levitated markers in the concatenated sentence do not matter at all, eliminating potential implausible inductive bias the concatenation order.
oriented packing scheme. Furthermore, to balance the efficiency and the model expressiveness, Ye et (2022) combine solid markers and levitated markers, proposing Subject-oriented Packing for Span Pair in the RE module. That is, if there are m entities, they copy the sentence for m times, and for each copy, they use solid markers to mark a different entity as the subject and concatenate the levitated markers of all other entities (as objects) at the end of the sentence.
Method
Our method built upon the state-of- the-art PL-marker. We employ a high-recall span pruner to obtain candidate entity spans, similar to the NER module in Pl.-marker. However, instead of aiming to accurately prediet all possible entity spans, our pruner focuses on removing unlikely candidates to achieve a much higher recall. Then we feed the candidate span set to the RE module to obtain entity and relation representations, which are used to initialize the node representations our hypergraph neural network for higher-order inference with a message passing scheme. Finally, perform NER and RE based on the refined entity and relation representations. Fig. depicts the neural architecture of our model.
Overview.
However, marker-base encoding is only used in RE, not in NER. To leverage marker- based encod- ing in the NER module for modeling span inter- relations, PL-marker (Ye al., 2022) associates each possible span with two levitated markers and concatenates all of them to the end of the input sen- tence. However, this strategy could make the input sentence extremely long since there are quadratic number of spans. To solve this issue, Pl--markr clusters the markers based the starting position of their corresponding spans, and divides them into N groups. Then the input sentence is duplicated N times and each group of levitated markers is concatenated to the end of one sentence copy. Ye et al. (2022) refers to this strategy as neighborhood-
3.1 Span Pruner
We adopt the neighborhood-orieneed packing scheme from PL-marker for span encoding, except that we simply predict entity existence (i.e., binary classification) instcad of predicting entity labels during the training phrase. See Appendix A.4 for

---

details.
(st, 85) are:
To produce a candidate span set, we rank all the spans by their scores and take top K as our prediction Sp(X). We assume that the number of entity spans of sentence is linear to its length n, so K is set to À n where À is a coefficient. For very long sentence, the number of entity spans is often sublinear to while for very short sentence, we wish to keep enough candidate spans, so additionally set an upper and lower bound: K max (mii: min(A- n. mmxx))
hg(s;)) = FFN, (Ih,( ((vO)))))
h,(rg) FFN, ((h, (s))))
Repeating K times, we get all K subject repre- sentations and K(K - relation representations. As the object representation of sj is not identical for different subject span 8;, there are K object representation sets h 1 S max max--nooing layer to obtain a unique object repre- sentation for each object span sj SX()::
K. We apply a
In practice, with our span pruner, more than 99% gold entity spans are included in the candidate set for all three datasets. If we predict entities as in PL- marker instead of pruning, only around 95% and 80% gold entities are kept in the predicted entities for ACE2005 and SciERC respectively, leading to severe error propagation (see $5.1 1 for an ablation study).
ho(s)) MMxxpooling (h)
3.3 Joint ERE Model: with Hypergraph Neural Networks
Higher-order Inference
Hypergraph Building So far, the representations of the entities and relations from the backbone mod- ule do not explicitly consider beneficial interactions among related instances. To model higher-order interactions among relation and its associated sub- ject and object entities as well between any two relations sharing an entity, we build a hypergraph = (V,E) E) to connect the related instances. The nodes set V is composed of candidate subjects, ob- jects (provided by the span pruner) and all possible pairwise relations thereof, and we denote them as V. [wr] E [1, K],, Vo {rEli E [1, K]} and V, {ur iij E [], + j}.
The span pruner trained independently from the joint ERE model introduced in the next section. This is because the joint ERE training loss will be defined based on candidate entity spans produced by the span pruner. When sharing parameters, the pruner would provide different candidate span set during training, leading moving targets and thereby destabilizing the whole training process.
3.2 Joint ERE Model: First-order Backhone
The backbone module is based on the RE mod- ule of PL-marker. Concretely, given an input sen- tence X = (21, ..., _n and a subject span 8; = (IST(i): ())()) E Sp(X) provided by the span pruner, every entity span 8y C (,))), < j < K,j * could be a candidate object span of 8, The module inserts pair of solid markers [S] and [VS]] before and after the subject span and assign every object span sj pair of levitated mark- ers [O]; and [[O],- As shown below, the levitated markers are packed together and inserted at the end of the input sequence a PLM:
Hyperedges & capture the interactions we are (sub-obj-rel)
concerned with, and they can be divided into two categories: the subject-ohject-relate hyperedges Esor and the relation-relation (rel-rel) hyperedges E,r, Each hyperedge ea E Egor con- nects a subject node vgg an object node vo and the corresponding relation node of and we refer to these hyperedges as ternary edges (ler for short). Each rel-rel edge 1ik Err connects two relation nodes with shared subject or object entity. We assume in relation, the subject the parent node and the object is the child node, and then we can refine rel-rel edges into three subtypes, sibling (sib, connecting v/ and ufk)). co-parent (cop, connect- ing 3f and u and grand- parent (gP, connecting of and with) following the common definitions in the dependency parsing literature.
x1,- .... [S]. 7ST(), ..... TED(): [S]...., In, OO),..., [0] [NO],, [10]K
Then we obtain the contextualized hidden represen- tation h of the modified input sequence and the final subject representation is:
If we incorporate all aforementioned hyperedges into the hypergraph, we obtain the tersibcopgp vari- ant which illustrated in Fig. By removing
h, (st) FFN, ([h): h (I))S))
FFN represents a single linear layer in this work.
The object representation of s; for the current sub-
some types of hyperedges we can get different vari- ants, but without loss of generality we describe
ject 8; and the representation of relation rig =

---

the message passing scheme in the following using tersibcopgp.
B(e,v) w (
exp,((e.v)
d'(e,u) = EvEN((O) exp (e',
As such, we can define CRF on the hyper- graph and leverage probabilistic inference algo- rithms such MFVI for higher-order inference. However, as discussed in $1, we can use a more ex- pressive method to improve inference quality and introduce a HyperGraph Neural Network (HGNN) as described next.
g'(u) (v) Σ d'(e eENe(
where a(-) is non-linear activator and w, W are two trainable parameters. An entity node would receive messages only from ter edges while a re- lation node would receive messages from both ter edges and rel-rel edges.
Initial node representation For relation node of with its associated subject node uf and object node vy. we use g' (2,,), g' g' (eb) to denote their respective representation outputs from the I- HGNN layer. Initial node representations (be- fore being fed to HGNN) are g" (vt) = (s.). g (u²) = (s3) and g (u²) = (rij). respec- tively (from the backhone module).
Training We obtain refined g' ('( from the final layer of HGNN. Give entity span 8j E (p))), concatenate the corresponding subject represen- tation g' (vtg and object representation (((")) to ob- tain the entity representation, and compute the prob- ability distribution over the types {C.) U(nul1}:
Pe (ūelss)) = Softmax(FFN g'
Message representation A hyperedge connect- ing to nodes serve as the bridge for message passing between nodes connected by it. Let.N((v) be the set of hyperedges connecting to a node v.
Given a relation ray (s,, 8, 8j (p()). we compute the probability distribution over the types {c.) U(null]:
For a ter hyperedge e Esgr connecting subject node vi. a object node v and a relation node of the message representation carries is:
(r(llra))) = Softmax(FFNÇ (g"(r)))
We use the cross-entropy loss for both entity and relation prediction:
hr 1j = FFN (g' (u+))
Le =
Σ
log(P((v((s;))
hs FFN (g'
()E)(()
ho: FFN (g'
L,
Σ
(L,8)ES(())
m' (c = FEN (hrl, hs ho!)
where y and y" are gold entity and relation types respectively. The total loss is I. = Le + Tar-
where o is the Hadamard product.
rel-rel edge jikk E Eyr. z E {sib, cop, gp) connects two relations sharing an entity. For sim- plicity, we denote them relation and b. If we fix a as a A off then as previously described, relation b is with for sib edge, o for cop edge, and oith for gp edge. The message ez ijk carries is given by,
Experiment
Datasets We experiment on SciERC (Luan et al.. 2018), 8), ACE2004 (Doddington et al., 2004) and ACE2005 (Walker et al.. 2006). We follow Ye et al. (2022) split ACE2004 into 5 folds and split ACE2005 and SciERC into train/dev/test sets. See Appendix A.I for detailed dataset statisties.
(a(a)) FFN (a))
lb((b) F me(gt = FFN (a) h)
Evaluation metrics We report micro labeled Fl measures for NER and RE. For RE, the difference between Rel and Rel+ is that the former requires correct prediction of subject and object entity spans and the relation type between them, while the latter additionally requires correct prediction of subject and object entity types.
Node representation update We aggregate mes- sages for each node Ne(w) with an attention mechanism by taking a learned weighted sum, and add the aggregated mes- sage to the prior node representation,
from adjacent edges
YYo et al. (2022) count symmetric relation twice for evaluation which is inconsistent with previous work.

---

ACE2005
ACE2004
SciERC
Models
Encoder
Ent
Rel
Rel+
Ent
Rel
Rel+
Ent
Rel
Rel+
(Wadden ct al., 2019)'
88.6
63.4
67.5
48.4
(Wang et al., 2021)'
88.8
64.3
87.7
60.0
68.4
36.9
(Zhong and Chen, 2021)*
90.1
67.7
64.8
89.2
63.9
60.1
68.9
50.1
36,8
(Yan et al., 2021)
66.8
38.4
(Shen et al., 2021)*
87.6
66.5
62.8
70.2
52,4
BERT SciBERT
(Nguyen al., 2021)
88.9
68.9
(Ye et al.. 2022),,
89.1
68.3
65.1
88.5
66.3
62.2
68.8
51.1
38.3
Backbone"
90.0
69.8
66.7
89.5
66.6
62.1
71.3
52.3
40.2
GCN"
90.2
69.6
66.5
90.0
67.6
63.5
74.1
54.8
42.9
MFVI*
90.2
69.7
67.1
89.7
67.4
63.4
73.3
54.7
42.5
HGERE" (our model)
90.2
70.7
67.5
89.9
68.2
64.2
74.9
55.7
43.6
(Liu al., 2022)
T5aæ
91.3
72.7
70.5
(Wang and Lu, 2020)
89.5
67.6
64.3
88.6
63.3
59.6
(Wang et al., 2021)"
90.2
66.0
89.5
63.0
(Zhong and Chen, 2021)*
90.9
69.4
67.0
90.3
66.1
62.2
(Yan et al., 2021)
89.0
66.8
89.3
62.5
(Ye ct al., 2022)
ALBERT
91.3
72.5
70.5
90.5
69.3
66.1
Backbone"
91.5
72.9
70.2
91.6
70.2
66.6
GCN"
91.7
73.1
69.9
92.0
71.5
67.9
MFVI"
91.6
72.7
70.1
89.9
68.5
65.1
HGERE" (our model)
91.9
73,5
70.8
91.9
71.9
68.3
Table I: Fl scores and standard deviations on ACE20044, ACT:2005 and SciERRC. The models marked with leverage
cross-sentence information. A model with subscript re means we re-evaluate the model with the evaluation method commonly used in other work?. Backbone, MFVI and GCN are baseline models.
Baseline Our baseline models include: i) Back- bone. It is described in Sect. 3.2 and does not contain the higher-order interaction module. ii) GCN. It has similar architecture to Sun et al. (2019); Nguyen et al. (2021) and does not contain higher-order hyperedges. See Appendix A.6 for detailed description. iii) MFVI. defines CRF on the same hypergraph as our model and uses MFVI instcad of hypergraph neural networks for higher-order inference. See Appendix A.5 for a detailed description.
on ACE2004. For MFVI we use the same variants as used in HGERE.
Table shows the main results. Surprisingly, Backbone outperforms prior approaches in almost all metrics by large margin (except ACE2004 with BERT, and ACE2005 with ALBERT), which we attribute the reduction of error propagation with span pruning mechanism. Our proposed model HGERE outperforms almost all basclines all metrics (except the entity metric on ACE2004), validating that using hyperedges to encode higher- order interactions is effective (compared with GCN) and that using hypergraph neural networks for higher-order modeling and inference is better than CRF-based probabilistic modeling with MFVL. Fi- nally, we remark that HGERE obtains state-of-the-art performances on all the three datasets.
Implementation details For fair compari- son with previous work, we use beri-base- uncased(Devlin et al., 2019) and albert-xxlarge- v/LLan al., 2020) the base encoders for ACE2004 and ACE2005, sciberr--scivocbb-nneee (Beltagy et al., 2019) as the base encoder for Sci- ERC. GCN and MFVI are also built upon Back- bone. The implementation details of experiments are in Appendix A.3.
Analysis
5.1 Effectiveness of the span pruner
To study the effectiveness of the span pruner, we replace it with entity identifier which is the orig- inal NER module from PL-marker and is trained only on entity existence. The performance of the span pruner and the entity identifier (denoted by Fid) on entity existence is shown in Table 2. We can observe that we replace the span pruner with the entity identifier, the recall of gold unlabeled
Main results For HGERE, we report the best results among the following variants of hyper- graphs with different types of hyperedges: ter, cop. sib, gp, tersib, tercop, tergp, tersibcop, tersibgp. tercopgp, and tersibcopgp. The best variants of HGERE are tersibcop on SciERC and ACE2005 (BERT,) :); tersib on ACE2005 (ALBERT); tercop

---

SciERC
ACE200S (BERT,) B)
SciERC
HGERE
P
R
F1
P
R
FI
Ent
Rel
Rel+
train
98.8
98.2
98.5
100.0
99.9
99.9
Backbone
71.3
52.3
40.2
Eid
dev
81.0
81.6
81.3
94.7
94.6
94.7
test
80.4
78.7
79.5
95.6
95.8
95.7
ter
74.2
55.1
42.6
sib
71.7
54.3
41.7
train
38.0
99.2
54,9
37.2
99.9
54.2
Pruner
dev
38.1
99.1
55.0
36.4
99.7
53.3
cop
71.7
52.9
40.8
test
38.7
99.2
55.7
37.0
99.8
54.0
8p
71.3
51.9
40.1
Table 2:
Evaluation results on entity existence of the span pruner vs, an entity identifier.
tersib
74.7
55.9
43.3
tercop
74.7
55.7
43.6
tergp
74.5
54.9
42.4
SciERC
Ent
Rel
Rel+
tersibcop
74.9
55.7
43.6
tersibgp
74.2
54.1
41.8
Backbone
69.4
50.3
39.0
Eid
tercopgp
74.7
54.0
42.3
HGERE
69.4
51.5
39.5
tersibcopgp
74.3
54.6
41.7
Backbone
71.3
52.3
40.2
Table 4: Fl scores of HGERE with different graph topolo- gies on the SciERC test set.
pruner
HGERE
74.9
55.7
43.6
ACE2005 (BERT,) B)
Ent
Rel
Rel+
Backbone
89.5
68.3
65.3
and cop variants, the NER performance improves slightly, which attribute the shared encoder of NER and RE tasks 3 On the other hand, in the ter variant, entity node representations are it- eratively refined, resulting in significantly better NER performance than Backbone (74.2 vs. 71.3). Combining ter edges with other rel-rel edges (e.g., sib) is generally better than using ter alone in terms of NER performance, suggesting that joint (and higher-order) modeling of NER and RE indeed has positive influence on NER, while prior pipeline approaches (e.g.. PL-marker) cannot enjoy the ben- efit of such joint modeling.
Fid
HGERE
89.5
68.5
66.0
Backbone
90.0
69.8
66.7
pruner
HGERE
90.2
70.7
67.5
Table 3: Fl scores of Backbone and HGERE with and without a pre-trained span pruner on the SciERC and ACE2005 (BERT'p B) test set.
entity spans drops from 99.2 to 78.7 on the Sci- ERC test set, and drops from 99.8 to 95.8 on the ACI20055 test set. We further investigate how the choice of the span pruner the entity identifier influences NER and RE performances. The results are shown in Table We can see that without span pruner, both NER and RE performances drop significantly, validating the usefulness of using a span pruner. Moreover, it has a consequent influ- ence on the higher-order inference module (i.c., HGNN). Without a span pruner, the improvement from using a HGNN over Backbone is marginal compared to that with span pruner. We posit that without pruner many gold entity spans could not exist in the hypergraph of HGNNs, making true en- tities and relations less connected in the hypergraph and thus diminishing the usefulness of HGNNs.
For RE, sib and cop have positive effects on the performance (despite gp having a negative ef- feet somehow), showing the advantage of modeling interactions between two different relations. Fur- ther combining them with ter improves RE perfor- mances in all cases, indicating that NER also has positive effect RE and confirming again the advantage of joint modeling of NER and RE.
5.3 Inference speed of higher-order module
To analyze the computing cost of our higher-order module, we present the inference speed of HGERE with three baseline models Backbone, GCN and MFVI on the test sets of SciERC and ACE2005. Inference speed is measured by the number of candidate entities processed per second. The sults are shown in Table 5. We can observe that
5.2 Effect of the choices of hyperedges
We compare different variants of HGNN with dif- ferent combinations of hyperedges. Note that if ter not used, entity nodes do not have any hyper- edges connecting to them, so their representations would not be refined. We can see that in the sib
'Though Zhong and Chen (2021) argue that using shared encoders would suffer from the feature confusion problem. later works show that shared encoders can still outperform separated encoders (Yan et al., 2021, 2022).

---

Predicted labels
when utilizing a relatively smaller PLM, HGERE. GON and MFVI were slightly slower than the first- order model Backbone. However, the difference in speed between HGERE and the other models was relatively small. When using ALBERT, which is much slower than BERTB, B, all four models demon- strated comparable inference speeds.
method
0
4
othersclantfiitermm
-2
16
0
2
-10
Thue labels
task
0
0
1
5
generic
10
0
-2
material
2
0
10
0
-8
metric
-2
0
0
33
1
null
-3
-42
-15
29
-14
-6
0
othersciertifitemm
method
task
genaric
material
matric
null
SciERC SciBERT
ACE2005
B B
ALBERT
Backbone
19.4
38.0
6.1
(a) Eiror correction matrix of HGERE vs. Backbone of entiitis
GCN
15.7
33.8
6.3
MFVI
16.5
36.9
6.1
Predicted abels
HGERE 15.7
30.7
6.0
part-of- 2
-2
0
-1
0
0
-6
Table Comparison of inference speed (#entities/sec) between HGERE and three baseline models on test sets of SciERC and ACE2005.
used-foi -1
20
2
0
0
0
-11
feature-of
-2
1
0
0
0
-1
True labels
evaluste for
0
0
0
-8
hypanym-of-
0
11
0
0
-10
conjunction
11
5.4 Error correction analysis
compare
-6
We provide quantitative сπrоr correction analy- sis between our oigeer-order approach HGERE and the first-order bascline Backbone on the SciERC dataset in Fig. 2. We can that most error cor- rections of entities and relations made by HGERE come from two categories. The first category is where Backbone incorrectly predicts a true entity or relation as null, and the second category is where Backbone incorrectly assigns a label to a null sam- plc.
part-ol
for
featura-af
evaluate- for hyponym-ot
conunction
compare
nuli
(h) Error correction matrix of HGERE vs. Backbone of rela- tions
Figure 2: Error correction of entity and relation types on the SciERC dataset. Red color indicates positive corrections and blue color indicates negative correc- tions. Specifically, positive numbers on the diagonal of the matrix (in red color) indicate that HGERE makes more correct predictions compare Backbone; nega- tive numbers on non-diagonal entries (in red color) indi- cate that HGERE makes fewer wrong predictions compare to Backbone. Numbers in blue indicate the opposite. We do not count the null-null case.
6 Related Work
Entity and relation extraction The entity and relation extraction task has been studied for a long time. The mainstream methods could be divided into pipeline and joint approaches. Pipeline meth- ods tackle the two subtasks, named entity recogni- tion and relation extraction, consecutively (Zelenko et al., 2003; Chan and Roth, 2011; Zhong and Chen, 2021; Ye et al., 2022). By utilizing new marker- based embedding method, Ye et al. (2022) becomes the new state-of-the-ar ERE model. However, pipeline models have the inherent error propagation problem and they could not fully leverage interac- tions across the two subtasks. Joint approaches, on the other hand, can alleviate the problem by simul- taneously tackling the two subtasks, empirically revealed by Yan et al. (2022). Various joint ap- proaches have been proposed to tackle ERE. Miwa and Bansal (2016); Katiyar and Cardie (2017) use a stacked model for joint learning through shared
parameters. Miwa and Sasaki (2014); Gupta et al. (2016); Wang and Lu (2020); Wang et al. (2021); Yan et al. (2021) tackle both the NER and RE tasks as tagging entries of table. Fu et al. (2019); Sun et al. (2019) leverage a graph convolutional net- work (GCN) on an instance dependency graph enhance instance representations. (Nguyen et al., 2021) propose framework to tackle multiple Infor- mation Extraction tasks jointly including the ERE task where a GCN is used to capture the interac- tions between related instances.
Another line of research is based on text-to-text models for structure prediction including ERE. Nor- mally they are not task-specialized and could solve

---

several structure prediction tasks in unified way (Paolini et al., 2021; Lu et al.. 2022: Liu et al., 2022).
based models which do not contain hyperedges connecting multiple instances and showed the im- provement, suggesting that modeling higher-order interactions between multiple instances is benefi- cial. Finally, we compared our model with the most popular higher-order CRF models with MFVI and showed the advantages of HGNN in higher-order modeling.
This work is similar to Sun et al. (2019); Nguyen et al. (2021) for we both use graph neural net- work to enhance the instance representations. The main difference is that the GCN they use cannot adequately model higher-order relationship among multiple instances, while our hypergraph neural network is designed for higher-order modeling.
Limitations
Our model achieves a significant improvement in most cases (on ACE2004, SciERC datasets and ACE2005 with Bert While on ACE2005 with stronger encoder (e.g.. ALBERT) we observe less siginificant improvements. We posit that, with powerful encoders, the recall of gold entity spans would increase, thereby mitigating the error propa- gation issue and diminishing the benefit of using a span pruning mechanism.
CRF-based higher-order model A commonly used higher-order model utilizes approximate infer- ence algorithms (mean-field variational inference or loopy belief propagation) on CRFs. Zheng et al. (2015b) formulate the mean-field variational infer- ence algorithm on CRFs as stack of recurrent neu- ral network layers, leading to an end-to-end model for training and inference. Many higher-order mod- els employ this technique for various NLP tasks, such as semantic parsing (Wang et al., 2019; Wang and Tu, 2020) and information extraction (Jia et al., 2022).
Another concern regarding our model is com- putational efficiency. The time complexity of the Subject-oriented Packing for Span Pair encoding scheme from PL-marker grows linearly with the size of candidate span size. Recall that we over- predict many spans using a span pruning mech- anism, which slows down the running time. In practice, our model's running time is around as three times that of PL-marker.
Hypergraph neural network Hypergraph neu- ral network (HyperGNN) is another way to con- struet an higher-order model. Traditional Graph Neural Networks employ pairwise connections among nodes, whereas HyperGNNs use hyper- graph structure for data modeling. Feng et al. (2019) and Bai et al. (2021) proposed spectral- based HyperGNNs utilizing the normalized hyper- graph Laplacian. Arya et al. (2020) is a spatial- based HyperGNN which aggregates messages in a two-stage procedure. Huang and Yang (2021) proposed UniGNN, a unified framework for inter- preting the message passing process in HyperGNN. Gao et al. (2023) introduced a general high-order multi-modal data correlation modeling framework learn an optimal representation in single hyper- graph based framework.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (61976139).
References
Devanshu Arya, Deepak K Gupta, Stevan Rudinac, and Marcel Worring. 2020. Hypersage: Generaliz- ing inductive representation learning on hypergraphs. arXiv preprint arXh::2010.0455:
Song Bai, Feihu Zhang, and Philip HS Torr. 2021. Hy- pergraph convolution and hypergraph attention. Pat- tern Recognition, 110:107637.
7 Conclusion
In this paper, we present HGERE, a joint entity and relation extraction model equipped with a span pruning mechanism and higher-order interaction module (i.c., HGNN). We found that simply using the span pruning mechanism by itself greatly im- prove the performance over prior state-of-the-art PL--marker, indicating the existence of the error propugation problem for pipeline methods. We compared our model with prior tranditional GNN-
Livio Baldini Soares, Nicholas FitzGerald. Jeffrey Ling. and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Pro- ceedings of the 57th Anmual Meeting of the Asso- ciation for Computational Linguisties, pages 2895- 2905, Florence, Italy. Association for Computational Linguistics.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical

---

Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNI.--ICCN)) pages 3615- 3620, Hong Kong, China, Association for Computa- tional Linguistics.
Yue Gao. Yifan Feng, Shuyi Ji, and Rongrong Ji. 2023, Hgnn+: General hypergraph neural networks. IEEE Transactions Pattern Analysis and Machine Intel- ligence, 45(3):318 3113319999999
Pankaj Gupta, Hinrich Schütze, and Bernt Andrassy. 2016. Table filling multi-task recurrent neural net- work for joint entity and relation extraction. In Pro- ceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2537-2547, Osaka, Japan. The COL- ING 2016 Organizing Committee.
Chen Cai and Yusu Wang. 2020. A note on over- smoothing for graph neural networks. ArXiv, abs/2006.1333 13318.
Yee Seng Chan and Dan Roth. 2011. Exploiting syntactico-semane structures for relation extraction. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 551-560, Portland, Oregon, USA. Association for Computational Lin- guistics.
Chenqing Hua. 2022. Graph neural networks intersect probabilistic graphical models: A survey. ArXiv, Abs/2206.06089.
Jing Huang and Jie Yang. 2021. Unignn: a unified framework for graph and hypergraph neural net- works. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21, pages 2563-2569. International Joint Conferences on Artificial Intelligence Organization. Main Track.
Jacob Devlin. Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Zixia Jia, Zhaohui Yan, Wenjuan Han, Zilong Zheng, and Kewei Tu. 2022. Joint information extraction with cross-task and cross-instance high-order model- ing, ArXiv, 0ts//2212.0922))
Arzoo Katiyar and Claire Cardie. 2017. Going out limb: Joint extraction of entity mentions and relations without dependency trees. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 917-928, Vancouver, Canada. Association for Computational Linguistics.
George Doddington, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extrac- tion (ACE) program tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04) Lishon, Portugal. European Language Resources As- sociation (ELRA).
Jonathan Kuck, Shuvam Chakraborty, Hao Tang, Rachel Luo, Jiaming Song, Ashish Sabharwal, and Stefano Ermon. 2020. Belief propagation neural networks. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Timothy Dozat and Christopher D. Manning. 2016. Deep biaffine attention for neural dependency pars- ing. ArXiv, abs/161
Markus Eberts and Adrian Ulges. 2020. Span-based joint entity and relation extraction with transformer pre-training. In ECAI 2020 24th European Confer- ence Artificial Intelligence, 29 August-8 Septem- ber 2020, Santiago de Compostela. Spain, August 29 September& 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), volume 325 of Frontiers in Artificial In- telligence and Applications, pages 2006-2013. IOS Press.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for sule-super learning of language representations. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Zuchao Li, Hai Zhao, Rui Wang, and Kevin Parnow. 2020. High-order semantic role labeling. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 134-1155,, Online, Association for Computational Linguistics.
Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. 2019. Hypergraph neural networks. In Proceedings of the AAAI conference on artificial intelligence, volume 33. pages 2558-3565.
Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell, and Mrinmaya Sachan. 2022. Autore- gressive structured prediction with language models. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 993-1005, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Tsu-Jui Fu, Peng-Hsuan Li, and Wei- Yun Ma. 2019. GraphRel: Modeling text as relational graphs for joint entity and relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 1409-1418, Florence, Italy. Association for Computational Linguistics.

---

Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu, 2022. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 9777, Dublin, Ireland. Association for Computational I Linguisties.
Changzhi Sun, Yeyun Gong, Yuanbin Wu, Ming Gong, Daxin Jiang, Man Lan, Shiliang Sun, and Nan Duan. 2019. Joint type inference on entities and relations via graph convolutional networks. In Proceedings of the 57th Annual Meeting of the Association for Com- putational Linguistics, pages 1361-1370, Florence, Italy. Association for Computational Linguisties.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 3219-3232, Brussels, Belgium. Association for Computational Linguistics.
David Wadden, Ulme Wennherg, Yi Luan, and Han- naneh Hajishirzi. 2019, Entity. relation, and event extraction with contextualized span representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP--JCCNIP) pages 5784- 5789, Hong Kong, China. Association for Computa- tional Linguistics.
Makoto Miwa and Mohit Bansal. 2016. End-to-end re- lation extraction using LSTMs on sequences and tree structures. In Proceedings of the 54th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages I 1105-1116, Berlin, Germany. Association for Computational Linguistics.
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. 2006. Ace 2005 multilingual training corpus. Linguistic Data Consortium.
Jue Wang and Wei Lu. 2020. Two are better than one: Joint entity and relation extraction with table- sequence encoders. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1706-1721, Online. As- sociation for Computational Linguistics.
Makoto Miwa and Yutaka Sasaki. 2014. Modeling joint entity and relation extraction with table represen- tation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1858-1869, Doha, Qatar. Associa- tion for Computational Linguisties.
Xinyu Wang. Jingxian Huang, and Kewei Tu. 2019. Second-order semantic dependency parsing with end- to-end neural networks. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 4609-4618, Florence, Italy, Asso- ciation for Computational Linguistics.
Minh Van Nguyen, Viet Dac Lai, and Thien Huu Nguyen. 2021. Cross-task instance representation interactions and label dependencies for joint infor- mation extraction with graph convolutional networks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional 1. Linguistics: Human Lnguuge Technologies. pages 27-38, Online. Association for Computational Linguistics.
Xinyu Wang and Kewei Tu. 2020. Second-order neu- ral dependency parsing with message passing and end-to-end training. In Proceedings of the 1st Con- ference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th Interna- tional Joint Conference on Natural Language Pro- cessing, pages 93-99, Suzhou, China. Association for Computational Linguistics.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAL, Ci- cero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation be- tween augmented natural languages. In International Conference on Learning Representations.
Yijun Wang. Changzhi Sun. Yuanbin Wu, Hao Zhou, Lei Li, and Junchi Yan. 2021. UniRE: A unified label space for entity relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pages 220 221, Online. Association for Computational Linguistics.
Meng Qu, Huiyu Cai, and Jian Tang. 2022. Neural struc- tured prediction for inductive node classification. In International Conference on Learning Representa- tions.
Victor Garcia Satorras and Max Welling. 2021. Neu-
ral enhanced belief propagation on factor graphs. In The 24th International Conference on Artificial In- telligence and Statistics, AISTATS 2021, April 13-15, 1021, Viruual Event, volume 130 of Proceedings of Machine Learning Research, pages 685-693. PMLR.
Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han. Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin. 2020. Denoising relation extraction from document- level distant supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 3683-3688, On- line. Association for Computational Linguistics.
Yongliang Shen, Xinyin Ma, Yechun Tang, and Weim- ing Lu. 2021. A trigger-sense memory flow frame- work for joint entity and relation extraction. In Pro- ceedings of the web conference 2021, pages 1704- 1715.
Zhaohui Yan, Zixia Jia, and Kewei Tu. 2022. An empir- ical study of pipeline vs. joint approaches to entity and relation extraction. In Proceedings of the 2nd

---

Conference of the Asia-Paciff Chapter of the Asso- ciation for Computational Linguistics and the 12th International Joint Conference on Natural Langwage Processing (Volume 2: Short Papers), pages 437-443, Online only. Association for Computational Linguis- tics.
2015 IEEE International Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 1529-1537. IEEE Computer Society.
Shuai Zheng. Sadeep Jayasumana, Bemardiio Romera- Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du Chang Huang, and Philip HS Torr. 2015b. Condi- tional random fields as recurrntt neural networks. In Proceedings of the IEEE international conference on computer vision, pages 1529-1537.
Zhiheng Yan, Chong Zhang, Jinlan Fu, Qi Zhang, and Zhongyu Wei. 2021. A partition filter network for joint entity and relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 185-197, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 50-61, Online. Association for Computational Linguistics.
Songlin Yang and Kewei Tu. 2022. Combining (second- order) graph-based and headed-span-haat projective dependency parsing. In Findings of the Association for Computational Linguistics: ACL 2022, pages 1428-1434, Dublin, Ireland. Association for Compu- tational Linguistics.
Shilin Zhou, Qingrong Xia, Zhenghua Li, Yu Zhang, Yu Hong, and Min Zhang. 2022. Fast and accu- rate end-to-end span-based semantic role laheling as word-based graph parsing. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 4160-4171, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Deming Ye. Yankai Lin, Peng Li, and Maosong Sun. 2022. Packed levitated marker for entity and relation extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1900-4917, Dublin, Ireland. Association for Computational Linguisties.
A
Appendix
KiJung Yoon, Renjie Liao, Yuwen Xiong, Lisa Zhang, Ethan Fetaya, Raquel Urtasun, Richard S. Zemel. and Xaq Pitkow. 2018. Inference in probabilistic graphical models by graph neural networks. In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Workshop Track Proceedings. OpenRe- view.net.
A.1 Datasets
We use ACE2004, ACE2005 and SciERC datasets in our experiments, the data statisties of each dataset is shown in Table 6.
#sent
#entity
#relation
Dmitry Zelenko, Chinatsu Aone, and Anthony Richardella. 2003. Kernel methods for relation extraction. Journal of machine learning research, 3(Feb): 1083-1 106.
ACE2004
8683
22735
4087
ACE2005
14525
38287
7070
SciERC
2687
8094
4648
Table 6: The statistic of datasets
Meishan Zhang, Yue Zhang, and Guohong Fu, 2017. End-to-end neural relation extraction with global op- timization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1730-1748 1740. Copenhagen, Denmark. Associa- tion for Computational Linguistics.
A.2 Bidirectional prediction of RE
Following previous work (Eberts and Ulges, 2020; Ye et al., 2022), we establish an inverse relation for each asymmetric relation for bidirectional predic- tion. The model can learn the inverse relations of asymmetric relations and improve the performance in this way.
Yu Zhang, Zhenghua Li, and Min Zhang. 2020a. Effi- cient second-order TreeCRF for neural dependency parsing. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 3295-3305, Online. Association for Computa- tional Linguistics.
A.3 Implementation details
Zhen Zhang, Fan Wu, and Wee Sun Lee. 2020b. Fac- tor graph neural networks. In Neural Information Processing Systems.
We adopt the same cross-sentence information incorporating method used in (Zhong and Chen, 2021; Ye al., 2022) which extend the original sentence to fixed window size W with its left and right context. We set W = 512 for SciERC, W 384 for ACE2004 and W = 256 for ACE2005. For
Shuai Zheng, Sadeep Jayasumana, Bernardino Romera- Paredes, Vibhav Vineet, Zhizhong Su, Dalong Du, Chang Huang. and Philip H. S. Torr. 2015a. Condi- tional random fields as recurrent neural networks. In

---

the pruner training and inference, consider the span length limitation L of 12 for SciERC and for ACE2004 and ACE2005. For pruners of any datasets and PLMs, the top-K ratio À 0.5, the boundaries of K are min = 3, max = 18. We use three hypergraph convolution layers for GCN, MFVI and HGERE. As the entity recall is high enough, pruners use ACE2004 and ACE2005 are only trained with BERT B- For all experiments, we run each configuration with 5 different seeds and report the average micro-F1 scores and standard devia- tion.
is dhiar-dimension vector,
(s)(s;) FFNsT (XSrt))
(s:)):) FFNED (XD()) xe(i))
(siar((:) [bsT(s;) (8:): W (s)n((():;;
the symbol; is the concatenation operation, FFNsT and FFNED are feed-forward layers with an out- put size dm and Wp E [(d==-1)shhard(tttt is a learn-able weight. The attentive pooling layer is weighted average over the contextualize token representations in the span,
For the pruner, the output sizes of FFNST; FFNED and FFN, are dm 768, the bi-affine embedding size doiaf 256, the output size of FFNamn is 256.
exp Wi
w/ = FFN (x)): =
Σ STT() exp
hart (s:)
Σ
wyXj
STO)S
For the backbone module, the output sizes of FFN,, FFN, and FFN, are tuned on [400, 512, 768] for all datasets.
and the final span representation is,
h(s)) FFN aut [Lsia((s):; bann(s:)
For the hypergraph neural network, the output sizes of FFNNer y FFNter, FFNN"r FFN,, FFN, are tumed among [256, 400, 512] and fixed on 400 for all experiments on SciERC. The output sizes of FFN FFN: tune on [256, 400, 512, 768) for all experiments. For GCN, MFVI and HGERE, we all use three layers to refine the node representations. We train our models with Adam optimizer and a liner scheduler with warmup ratio of 0.1. We tune the eps of Adam optimizer on [le - 8, 1e 1 9] for ACE2005, and eps=le- - 8 for other datasets. The batch size of all experiments are 18. The learning rate of PLM are 2e - 5, for other module the learn- ing rate is tune [5c 5, 1e 1). The epochs on SciERC for Backbone are 20, and 30 for other models. The epochs on ACE2004 and ACE2005 (BERTB) are 15, on ACE2004 and ACE2005 (AL- BERT) are 10. We do all experiments on a A40 GPU with apex fp16 training option on.
Training and Inference Given the gold binary tag y(s)) E {0,1} (indicating the existence of a candidate span in the gold span set), we train the span pruner with the binary cross-entropy (BCE) loss:
((s;) = Sigmoid(FFN(((s)))
L
Σ (u((s) log ((s))
1515m
+ (1- v-:)s)) log [s:)]
A.5 Mean-field Variant Inference
Here we introduce the method used in baseline MFVI. The hyperedges in our graph replaced by factors in MFVI, so there are also four kinds of factors: ter, sib, cop, gp.
first-order scores We use the node representa- tions score the entities and relations for each label (include the null).
A.4 Details of the span pruner
We obtain contextualized representations of the tokens and levitated marker representations (for [O] and xe (for [OO)- Then concate- nate two kinds of span representations---illee (Dozat and Manning, 2016) and attentive pooling- as the final one. For a span 8; consisting of tokens TST()): ..., TED(j) its bi-affine span representation
u FFN g(()))
u FFN g(()))
u FFN, g (g()))
u², u E RC1lH1 u" E RIGH+

---

Higher-order scores Each factor scores the joint distribution of the node types connected to it. For ter factor connects a subject vy. an object vg and a relation u the factor score fi E R1C++1 is:
For a relation of, the message could be passed from four factors, we list them by the source. From ter factor:
T (11) ΣΣ
De (e)) (e2) (e,, (2,)
h FFN (g(v)))
h = FFN (g(v)))
02
From sib factor:
h FFN (g(u,))
F = FFNJ" (h, o h3 hly)
S (r1)
For factor Z; E {sib, cop, gp) which connects relations, name them relation and b for simplicity. If relation a is u/ then relation is ak / and oith for sib, cop and 8P respectively.
(e (12()2)1))
From the cop factor:
We use a(a),g(b) to refer to the relation repre- sentations of relations a and b. The factor score fijk E +1|, is defined as:
Cipy (r1)
Skj (r2, r1))
ha FFN (g(a))
From the sp factor:
h FFN (g(b))
iik F (ha hb)
(r1)
higher-order inference In the model, com- puting the node distribution can be seen as doing posterior inference on a Conditional Random Field (CRF). MFVI iteratively updates factorized variational distribution Q to approx- imate the posterior label distribution. We use (e)(,, Qoy (e2) to refer to the probability of subject vf and object va has entity type ei and e2 respectively and Qry (r) represents the relation of has the relation type r. For simplicity, we use (e1), uf((2), m/, (ri), fly (e,, e2, r1), (ri to represent the first-order and higher-order scores when the subject og the object vg have entity type e1, e2, the relation (ut"), the relation b have relation types F1; respectively. Following the iterately updating of the distribution Qe. Qo. Qr- For subject vy- The message only passed from ter factor in the 1-th iteration is:
ΣΣ (o (12) (ff) 12 (12) ( (r2. r1))
The posterior distribution of entity e; with respect to the subject 8, and object o,
(e)() exp(u((e) Fx do (e) & explu)(e) + F1, (e))
Then the entity distribution is: Q' + Q¹ We initial the Q of subject vf. object ug. the relation oy by normalizing the unary potential exp(u, (uf), ), exp(uş), exp(uf,) respectively. The pos- terior distribution of the relation riy is:
Q' (r) x exp(uf,(r) + LterTH", (r) S Fiy (r) Le Cl. (r) + 1 Gl (r))
(e1)
The symbol 1,, 2 E [ter, cop, 8P} indicates whether the factor Z exists in the graph.
ΣΣ (62) (e)
(e.,e,)))
A.6 GCN
similarly, the message passed from ter factor to the object ud is:
Here is the introduction of the bascline GCN. As in HGERE, we also build the graph g (V,E) with subject, object and relation nodes, V Va Vv, For each relation node of V,, we build two edges connecting its subject node og E V. and object node vg E V, respectively. the
(02) =
Σ Σ O (e) (on.
(01,

---

SciERC
D23
Ent
Rel
Rel+
1/0
max
74.0
54.7
41.4
1/
73.6
54.5
41.5
120
attn
74.9
55.7
43.6
Table 8: F1 scores of HGERE (the tersibcop variant) with different aggregation functions on SciERC test set.
subject node
object node
relation
results shown in Table 7 we can see that the incre- ments of absolute Fl score on Rel+ metric from Backbone to HGERE are 2.1%, 2.2% 5% and 10% of training set respectively, which are much higher than 0.6% full training set.
Figure 3: Illustration of example graph of GCN
model convolution layers to update the node representations. We define the neighbor set A(v) node v is the nodes connected to The node representation update of I-th layer is as follow:
A.8
Effect of the number of HGNN layers
(1(11, v) w o
(v)
a'(11,0) exp (1),
65.7
66.0
67.2
65.3
EmeNoo exp (1)
43.6
- (v)+ Σ
42.0
39.4
vIE((()
SciERC
ACE2005
A.7
Performance with part of the training
Figure The change Fl scores with respect to the number of HGNN layers.
data
From Fig.4 we can see that using three HGNN layers performs the best while more layers lead worse results. We posit that this is because using more HGNN layers would suffer from the well- known over-smoothing problem (Cai and Wang,
ACI20005 (BERT,) B)
ratio
model
Ent
Rel
Rel+
Backbone
80.4
39.7
36.0
5%
HGERE
79.5
42.0
38.1
Backbone
83.9
51.3
47.2
2020).
10%
HGERE
84.2
53.3
49,4
A.9
Effect the aggregation function in message passing
Backbone
90.0
69.8
66.7
100%
HGERE
90.2
70.1
67.3
We study the influence of using different message aggregation functions. HGERE uses attention mechanism (attn) update node representations while also possible use max wax-pooling (max) or num-pooling (sum). Table shows that attn performs the best.
Table 7: F1 score of HGERE ACE2005 test set when
only provide 5% and 10% training samples.
From the main results, we can see that the HGERE shows a significantly greater improvement per- formance compared to the Backbone model on the SciERC dataset than on the ACE2005 dataset. We guess one of the reason is the size of the train- ing data. Because with more training data, models could learn enough knowledge from a large number of samples and reduce the demand of higher-order information. So we compare HGERE to Backbone with 5% and 10% of training data on the ACE2005 (BERT,) to see higher-order inference is more effectiveness with small training data. From the


Reviews:
[{"review_id": "2a3TTU2H6y", "reviewer": "Reviewer_g3Cy", "summary": "", "questions": "- Is this the first work that explicit graph neural network on the ERE task?\n\n- Beside HyperGNN and GCN, do you try other SOTA graph network modeling method?\n\n- Is there any other SOTA higher-order modeling approach baseline? MFVI seems too old…\n\n- Line 447. Report the Precision and Recall to make a clear comparison and analysis.\n\n- Could you report the computing costs.\n\n- Line 430. Is there any experiments to show the span pruning reduce the error propagation?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and questions regarding our work. To provide a better understanding of the significance of our work, let us first define higher-order models.\n\nIn our paper, **first-order** models refer to models that only utilize the features or distributions of individual variables. GNN models with a traditional graph backbone like GCN can be considered as **second-order** models since they utilize pairwise relationships. In contrast, our HGNN-based HGERE model captures features of hyperedges connecting three nodes (ternary edge in the paper), making it a **third-order** model. Similarly, CRFIE[1], which published in ACL2023,defines ternary factors connecting three nodes and is therefore also a third-order model. Note that some previous studies using GCN also claim to be higher-order models by arguing that one GCN layer encodes information about immediate neighbors and K layers encode Kth-order neighborhoods. However, according to our definition, they can only be considered as second-order models.\n>[1]: Modeling Instance Interactions for Joint Information Extraction with Neural High-Order Conditional Random Field\n  \n\n\n## Regarding the innovation\n1. As far as we know, CRFIE is the only previous approach to higher-order Entity and Relation Extraction, which is based on CRF+MFVI. We are the first to utilize HGNNs for higher-order modeling. The advantages of HGNNs over CRF+MFVI are discussed in the paragraph starting at line 098.\n2. In the field of IE, span-based models generally outperform token-based models but have higher computational complexity (especially for higher-order modeling). To address this problem, most previous span-based models employed entity identification as an initial step to filter entities. However, a major drawback of this approach is that errors from the first step propagate to downstream models, impacting the performance of the models. In contrast, our span pruner not only reduces the complexity of higher-order models but also significantly improves their performance (as demonstrated in Section 5.1). Although span pruning has been researched previously, it is novel to combine it with HGNNs in higher-order span-based ERE, improving both accuracy and efficiency.\n\n## Answering the questions\n\n1. We are the first to utilize **HGNN** for ERE. Note that HGNNs are different from and more general than GNNs.\n  \n2. We have tried GAT but it achieves similar performance compared to the GCN baseline in our paper. Meanwhile the state-of-the-art ERE model PL-marker, does not use GNN and achieves superior performance compared to previous GNN-based models. So in additional to PL-marker, we choose current baselines in the paper. \n  \n3. As mentioned above, there is limited research on higher-order models and there are no higher-order models available for comparison other than third-order MFVI.\n  \n4-5. We will report them in the final edition.\n   \n6. The experiments in Section 5.1 demonstrate the performance differences between models with an entity pruner vs. an entity identifier, which verify how the pruner reduces error propagation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The application of span pruning and Hyperbolic Graph Neural Networks (HGNN) for addressing the ERE task demonstrates good performance. It is interesting to see the improvement over PL-maker, and this works analyzes the improvement brought from GNN.", "reasons_to_reject": "Utilizing state-of-the-art Graph Neural Networks (GNNs) as a higher-order modeling technique is intriguing; however, the scope of innovation appears constrained. The work consider GCN and MFVI as baseline and shows the effect of higher-order model but the improvement seems limited. Also the paper should add more higher-order baselines.", "questions_to_authors": "- Is this the first work that explicit graph neural network on the ERE task?\n\n- Beside HyperGNN and GCN, do you try other SOTA graph network modeling method?\n\n- Is there any other SOTA higher-order modeling approach baseline? MFVI seems too old…\n\n- Line 447. Report the Precision and Recall to make a clear comparison and analysis.\n\n- Could you report the computing costs.\n\n- Line 430. Is there any experiments to show the span pruning reduce the error propagation?", "paper_topic_and_main_contributions": "This paper proposes a joint entity and relation extraction model, utilizing span pruning and GNN to make improvements over SOTA approach, PL-maker. The paper analyzes the effects of span pruning and HGNN separately. The results on three dataset shows the improvement.\n\nWeakness: Lack of innovation. The span pruning method is simple and there are some related works. There are also some related works based on higher-order modeling approaches. It is interesting to use SOTA GNN to replace MFVI but the innovation is limited. If you can make a deep analysis on more different SOTA higher-order models or graph neural networks, it will be better and more convincing.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "geKht8nOu5", "reviewer": "Reviewer_YBzL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our work.\n\n1. HIORE [1] is a preprint paper published in May 2023 and is therefore concurrent to our work.\n  \n2. While it is true that LLM-based IE is becoming more popular, it is important to embrace a diverse range of research directions and not focus on a single type of methodology. \nThere is also evidence showing that with sufficient training data, LLM-based IE methods still underperform traditional encoder-based IE methods.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Paper is well-written and the idea is intuitive.\n2. The results are good.", "reasons_to_reject": "1. There has been similar work [1] on GNN for capturing higher-order interactions. It weakens the contribution of this paper.\n2. This IE research paradigm is a bit outdated in the age of LLMs.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes HGERE, a joint entity and relation extraction model equipped with a span pruning mechanism and a higher-order interaction module.\nIt is built upon the state-of-the-art PL-marker, and achieves higher-order inference with hypergraph neural networks.\nExperiments on 3 datasets show the effectivesss.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] HIORE: Leveraging High-order Interactions for Unified Entity Relation Extraction"}, "extra_text": {}}, {"review_id": "R0luMg5QCg", "reviewer": "Reviewer_tBFS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your useful comments. To provide a better understanding of the significance of our work, let us first define higher-order models.\n\nIn our paper, **first-order** models refer to models that only utilize the features or distributions of individual variables. GNN models with a traditional graph backbone like GCN can be considered as **second-order** models since they utilize pairwise relationships. In contrast, our HGNN-based HGERE model captures features of hyperedges connecting three nodes (ternary edge in the paper), making it a **third-order** model. Similarly, CRFIE[1], which published in ACL2023,defines ternary factors connecting three nodes and is therefore also a third-order model. Note that some previous studies using GCN also claim to be higher-order models by arguing that one GCN layer encodes information about immediate neighbors and K layers encode Kth-order neighborhoods. However, according to our definition, they can only be considered as second-order models.\n>[1]: Modeling Instance Interactions for Joint Information Extraction with Neural High-Order Conditional Random Field\n\n  \n\n## Regarding using GNN in previous study\n\nNote that HGNNs are different from and more general than GNNs. GNNs are only capable of second-order modeling, while HGNNs are capable of higher-order modeling. We are the first to utilize HGNN for higher-order ERE.\n\n## Regarding levitated markers in previous study\n\nWe simply build our higher-order model upon a marker-based model and levitated marker itself is not our contribution.\n\n## Regarding the question about \"...hypergraphs are fully connected in the sense that no weights are assigned on the edges.\"\n\nThe hypergraph in the paper is not fully connected as that would result in very high complexity. We considered only two types of manually designed hyper-edges: *sub-obj-rel* and *rel-rel*. This is the most natural way for higher-order modeling of ERE as it connects potentially related nodes using hyperedges. Ordinary graphs are only capable of second-order modeling and cannot capture higher-order relationships such as *sub-obj-rel*.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the paper is well-written and easy to follow.\n- the results are strong.\n", "reasons_to_reject": "- though the model gets a strong performance, both using levitated markers and using graph neural networks for high-order interaction have been studied in previous tasks.\n- It seems that the hypergraphs are fully connected in the sense that no weights are assigned on the edges. it would be better to study other ways to build the graph (e.g., pruning some edges, even using ordinary graphs rather than hypergraphs).\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper develops an entity relation extraction model based on the LP-marker (which is derived from a pipeline extractor PURE). The key technical contribution is to make connections among entities and relations closer. Specifically, The method includes more candidates in the first entity extraction step and deploys a stronger relation extractor (though GNN with hypergraph) to determine which entities will finally be extracted (by consulting relation signals). ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "It would be better to discuss related work on learning with hypergraphs.\n[nips06]Learning with Hypergraphs: Clustering, Classification, and Embedding\n[neurips21]Edge Representation Learning with Hypergraphs"}}]

Author rebuttal:

Meta-review: This paper proposes HyperGraph neural network for joint entity and relation extraction to alleviate the error propagation in marker-based pipeline models, and capture higher-order interactions between multiple entities and relations. For all the reviewers, the most concern is its novelty and contribution. Comparing with prior methods, there are indeed something new in the proposed method. It would be nice if the authors could make precise revisions according to their responses.
Decision: Accept-Main