BiSPN: Generating Entity Set and Relation Set Coherently in One Pass
Authors: Yuxin He
Buzhou Tang
Venue: EMNLP 2023 Findings (2023)
Keywords: Information Extraction
Joint Entity-Relation Extraction
Non-autoregressive Generation
TL;DR: We propose a novel joint extraction model that can generate entity set and relation set in one pass, which is equipped with two special loss functions to maintain the coherence of generated sets.

Abstract: By modeling the interaction among instances and avoiding error propagation, Set Prediction Networks (SPNs) achieve state-of-the-art performance on the tasks of named entity recognition and relation triple extraction respectively. However, how to jointly extract entities and relation triples via SPNs remains an unexplored problem, where the main challenge is the maintenance of coherence between the predicted entity/relation sets during one-pass generation. In this work, we present Bipartite Set Prediction Network (BiSPN), a novel joint entity-relation extraction model that can efficiently generate entity set and relation set in parallel. To overcome the challenge of coherence, BiSPN is equipped with a novel bipartite consistency loss as well as an entity-relation linking loss during training. Experiments on three biomedical/clinical datasets and a general-domain dataset show that BiSPN achieves new state of the art in knowledge-intensive scene and performs competitively in general-domain, while being more efficient than two-stage joint extraction methods.

BiSPN: Generating Entity Set and Relation Set Coherently in One Pass
Yuxin He' and Buzhou Tang
'Department of Computer Science, Harbin Institute of Technology, Shenzhen, China Peng Cheng Laboratory, Shenzhen, China
55051047estu hit. edu. cn
tangbuzhou@gmail.com
Abstract
GSK-3
Bind
GSK-3
E-cadherin
By modeling the interaction among instances and avoiding emror propagation, Set Prediction Networks (SPNs) achieve state-of the-art per- formance on the tasks named entity recogni- tion and relation triple extraction respectively. However, how to jointly extract entities and relation triples SPNs remains an unex- plored problem, where the main challenge is the maintenance of coherence between the pre- dicted entity/relation sets during one-pass gen- eration. In this work, we present Bipartite Set Prediction Network (BiSPN), a novel joint entity-relation extraction model that effi- ciently generate entity set and relation set in parallel. To overcome the challenge of coher- ence. BiSPN is equipped with novel bipartite consistency loss as well as an entity-relation linking loss during training. Experiments on three biomedical//linical datasets and general- domain dataset show that BiSPN achieves new state of the art in knowledge- wwwlleeinnttnssve scene and performs competitively in general-domain, while being more efficient than two-stage joint extraction methods.
cacherinnn
E-cadherin
catenin
Entities
Relations
Interestingly, It has been described that GSX-3 can bind E-cadherin to phosphorylate which would result tighter interaction with catenin
Figure I: The target output of joint entity and relation extraction is essentially an entity set and a relation set that should be consistent with each other. Such co- herence is difficult to be guaranteed when generating entity/relation sets in parallel. And our work manages address this challenge.
the possibility of jointly solving the extraction of entities and relation triples with SPNs, which is promising but unexplored direction.
In this paper, we propose Bipartite Set Prediction Network (BiSPN), variant of SPNs, generate target entity set and relation set in one pass. It can not only avoid the negative effect of cascade error but also enjoy the henefit of high inference speed. However, it is challenged by the difficulty to maintain the coherence between the generated entity set and relation set, due its parallel design. As illustrated in Figure the head/tail entities of generated relation triples should be included in the generated entity set. The significance of this coherence is two-fold: 1) by requiring the gener- ated entity set to contain the head/tail entities, the recall of the generated entities is more guaranteed; 2) by restricting the head/tail entities within the generated entity set, the precision and recall of the generated triples more guaranteed.
1 Introduction
Extracting entities and relation triples from text is fundamental task of Information Extraction. There have been many efforts that decompose the prob- lem into separate tasks, i.e. named entity recogni- tion (NER) and relation triple extraction (RE), and solve them respectively. Among these efforts, Set Prediction Networks (SPNs) have demonstrated state-of-the-art performance NER (Tan al., 2021; Shen al., 2022) and RE (Sui et al., 2020; Tan et al., 2022).
Typically, SPNs leverage set of learnable queries to model the interaction among instances (entities or relation triples) via attention mechanism and generate the set of instances naturally. The suc- cess of SPNs on NER and RE inspires us to explore
Despite that, is difficult to maintain such con- sistency when generating the two sets in parallel, since instance queries are assumed to be of equal status and their hidden representations are updated
"Corresponding Author.

---

using bidirectional attention without any further restriction.
(MRC)-basee methods (Li al., 2019) that casts the task as multi-turn question answering prob- lem via manually designed templates; (4) autore- gressive generation- 2018; Lu et al., 2022) that reformulate the task as a requence-to-sevee sequence problem by lincarizing target entities/relations into a pointer sequence or augmented natural language.
To overcome this challenge, we come up with two novel solutions. The first one a Bipartite Consistency Loss function. It works by looking for a reference entity from the generated entity set for each relational subject/object and forcing the subject/object to simulate the reference tity. Symmetrically, it also find reference sub- ject/object for each entity classified as involved relation and force the entity to simulate the reference subject/object. Our second solution is an Entity-Reation Linking Loss function, which works in the hidden semantic space. By computing the linking scores between the projected represen- tations of entity queries and relation queries, it en- courages the model to learn the interaction between entity instances and relation triple instances. To sum up, our main contributions include:
methods (Zeng et al.,
Among them, only the methods based on table filling can extract entities and relation triples in one stage, all other methods perform multi-step prediction, suffering from cascade errors and low inference speed. In this paper, we provide a new choice for one-stage joint entity and relation extrac- tion, which is based on set prediction networks.
2.2 Set Prediction Networks
Set prediction networks are originally proposed for the task of object detection in computer vision (Carion et al., 2020). And they are successfully extended for information extraction by (Sui et al., 2020; Tan al., 2021; Shen al., 2022; Tan al., 2022).
We present BiSPN, the first SPN-based joint entity and relation extraction model, which is able to generate target entity set and relation set in one pass.
Generally, these methods employ a set of learn- able queries as additional input, model the interac- tion among instances (entities/relations) via self- attention among the queries and one-way attention between the queries and the textual context. How- ever, all these methods can only perform named entity recognition (Tan al., 2021; Shen et al., 2022) or relation triple extraction (Sui et al., 2020; Tan al., 2022), rather than jointly solve both of them.
To maintain the coherence between the gener- ated entity set and relation set, two novel loss functions, i.e., Bipartite Consistency Loss and Entity-Relation Linking Loss are introduced.
BiSPN outperforms SOTA methods on three biomedical/clinical datasets, which is intensive, and achieves competi- tive results on a general-domain benchmark. Besides, it infers much faster than two-stage joint extraction methods.
Methodology
3.1 Problem Formulation
2 Related Work
Given input sentence x = 21, 22, LL, of joint entity and relation extraction is predict the set of entities {e:) and the set of relation triples (r,) j=1 mentioned in the sentence. Here, L is the length of the input sentence, N₂ and N, are the numbers of target entities and target relation triples respectively. The i-th entity e, is denoted (slarl, end,, ()), where slarl,, end, are the start token index and end token index of the entity, t e To is the entity type label. The j-th relation triple Tj is denoted as (e),, t5, e²), where e and e¹ are the head entity (start); endyy ( and tail entity (start,, end', 19:1) of the relation triple, fz T, is the relation type label. We additionally define a null label for the set of entity types T, and the
LL. the aim
2.1 Joint Entity and Relation Extraction
The target of joint entity and relation extraction is to recognize all entities mentioned in given text and identify all entity pairs involved in relation. Exist- ing methods for joint entity and relation extraction fall into four categories: (1) span-based methods (Dixit and Al-Onaizan, 2019; Zhong and Chen, 2021; Ye et al., 2022) that enumerate spans and conduct sean-level classification to extract entities, enumerate and classify span pairs extract rela- tion triples: (2) table filling-based methods (Wang et al., 2020, 2021; Yan et al., 2021) that fill out a table for each entity/relation type via token pair classification; (3) machine reading comprehension

---

Predicted Entity Set
Predicted Relation Set
(ef.ef.ef)
Head-Tail Predictor
Relation Type Predictor
Entity Predictor
Entity Decoder
Relation Decoder
Rel View
Ent View
Shared Decoder
Bidirectional Self-Attention
x N
One- One-Way Self-Attention
Shared Encoder
Input Embeddings
Entity Queries
Relation Queries
Input Text
Figure 2: An overview of BiSPN, the proposed joint entity and relation extraction framework that is capable of generating target entity set and relation set coherently in one pass.
set of relation types Tr respectively, to indicate that entity relation is recognized.
we follow the work (Shen et al., 2022) to modify the bidirectional self-attention into way self- attention. Concretely, the upper right L (Mş M,) sub-matrix of the attention mask filled with negative infinity value so that the entity/relation queries become invisible to the token encodings, while the entity/relation queries can still attend to each other and the token encodings.
3.2 BiSPN
As illustrated in Figure 2, the proposed model mainly consists of shared encoder, a shared de- coder, an entity decoding module (in blue) and relation decoding module (in red).
After multiple one-way self-attention layers and fred-forward layers, the encoder outputs the textual token encodings as well the contextual entity/relation queries.
3.2.1 Shared Encoder
The encoder of BiSPN is essentially bidirectional pretrained language model (Devlin et al., 2019) with modified input style and attention design.
3.2.2 Shared Decoder
We first transform the input sentence into input embeddings X R and then concatenate X with a series of learnable entity queries Q. and relation queries Qr to form the model input X:
The shared decoder consists N decoding blocks. Each decoding block includes an one-way self- attention layer (as described above), a bidirec- tional self-attention layer and feed-forward layers. The one-way self-attention layer here functions as the cross-attention layer of Transformer decoder (Vaswani et al., 2017), which aggregates textual context for decoding. (The main difference be- tween one-way self-attention and cross-attention is that the contextual token encodings also get updated by one-way celf-attention.) The bidirec- tional self-attention layer updates the entity/relation
X [x:Qe: Qr] R Mll-MM+M.)xd (1)
where d is the model dimension, Me and M, are hyperparameters controlling the number of entity queries and the number of relation queries (Me Ne, M, > N,).
To prevent the randomly initialized queries from negatively affecting the contextual token encodings,

---

queries via modeling the interaction among them. After shared decoding, the decoder outputs the updated token representations H entity queries H and relation queries H
3.2.4 Relation Decoding Module
The relation decoding module consists of a relation- view projection layer, relation decoder, a head-tail predictor and a relation type predictor.
The relation-views projection layer and relation decoder work in the same manner as the entity- view projection layer and entity decoder, except that the relation decoder splits relation queries into head/tail queries before decoding:
3.2.3 Entity Decoding Module
The entity decoding module consists of an entity- view projection layer, an entity decoder and an entity predictor.
The entity-view projection layer first linearly transforms the token encodings H into entity- view:
[H,; HH' = Linear (
(10)
I = Linear(
(11)
(2) The entity decoder, which includes multiple layers of cross-attention and bidirectional self- attention, receives the transformed token encodings H. as decoding context and the entity queries He decoder input, and output the final representation of entity queries Hº:
H. = Linear(III)
Hh, H Hr RelationDeecoder( H H H HH)
(12)
The head-tail predictor then predicts the bound- ary and entity type of the head/tail entity associated with each relation queries. This process is similar to the entity prediction process (Equation 4-10). The only difference is that the entities queries be- comes the head/tail queries Ht/t and the token encodings is now in relation-vies view H.P.
He EntityDecoder((1111.
(3)
The entity predictor is responsible for predicting the boundary and entity type of each entity query. For each entity query. it first fuse the query rep- resentation with the transformed token encodings, and then calculate the probability of each token in the sentence being the start/end token of the corre- sponding entity:
The relation type predictor classifies the category of i-th relation query according to H7;
ps
Softmax (M
(13)
3.3 Prediction Loss
o = Linear (uneaa(ReluLLineati ([ç)) + Linear( H))))
To train the model, we should find the optimal assignment between the gold entity set and the gen- erated entity set, as well as the optimal assignment between the gold relation set and the generated re- lation set, which are calculated in the same way as in (Tan et al.. 2021: Shen et al., 2022) using the Hungarian algorithm (Kuhn, 1955).
(4)
pe Soft 8 (start, end}
(5)
where 5/ E R is vector of logits of each token being the start/end token of the entity associated with i-th entity query, P is the corresponding prob- ability distribution.
After the optimal assignments are obtained, we calculate the following prediction loss Lpred for each sample:
An MLP- classifier is leveraged to predict the type of the entity associated with the i-th entity query:
Me log pr (tartttttppt)))) + log pi pprl (en
Leu
Pdd = Coftmax(MLP(Iif)
(6)
log pe (t()
(14)
During inference, the predicted boundary and entity type corresponding to the k-tth entity query are calculated as:
Σ log pamn start + log pood (end
log prs p
(15)
score (i, = patart [i] + Pe []]
(7)
M,
(starts; endds) arg score O<jiic
(8)
Lit Clu
Σ
log Pf (6
(16)
# arg max P
Epra Len Liel
(17)
Note that, the entity predictor will filter out the entity whose predicted type label is Ø.
where (iii is the index of the gold entity assigned to the i-th generated entity, a(j) ) is the index of the

---

gold relation triple assigned to the j-th generated relation triple, Ltu represents the loss of head/tail entity prediction.
where 92 is the of indices of the entities classified as involved in relation.
We sum up Lent vrels Lel sent and Chas-rel to get the overall bipartite consistency loss Centerre--
3.4 Bipartite Consistency Loss
To calculate the bipartite consistency loss, we first find a reference entity from the generated entity set for each head/tail entity. A reference entity is defined as the entity most similar to the referring head/tail entity. Concretely, the similarity between Ca. the a-th generated entity, and .ift, the head/tail entity of the b-th generated relation triple is mea- sured in KI. divergence between the start/end/type probability distributions of ea and eb.
3.5 Entity- Linking Loss
While the bipartite consistency loss softly aligns the predicted distributions between the generated entity set and relation set during training, the entity- relation linking loss encourages BiSPN to model the interaction between entity queries and relation queries.
To this end, we first project the intermediate rep- resentations of entity queries and relation queries and then compute the linking scores between them via a Biaffine layer:
sim(ca; cb
Σ
(Ra (
8E(start,end.0"
where DKL (P) is the KL divergence between target distribution P and approximate distribution Q: Paan means p when eb is a head entity, otherwise Pr-M means p...A.
He Linear(H")
(23)
H" Linear(H")
(24)
Slink Biaffine HH) E E
(25)
We want every head/tail entity to simulate its reference entity, which equivalent to maximizing the similarity. Hence, the consistency loss in the relation + entity direction is computed as:
With the linking scores, we calculate the follow- ing binary cross-entropy loss:
plink iggoiid(Siing
(26)
Mr
Cell =
Σl max sim(e,, e MELI,M))
M Mr
1=
(18)
Llink =
Ml
link
log plink
max sim(e,, cf]
M,M,
i=1
j=1
(27)
JE(L.Me)
+ (1 - yog log(1 1 plink)
Symmetrically, also find a reference head/tail entity for each generated entity that is classified as having relation. The classification is conducted by a binary classifier, which is trained with a binary cross-entropy loss function:
where v 1 only if the gold entity assigned the i-th entity query appears in the gold relation triple assigned to the j-th relation query.
p has-rel = sigmoid(MLP(Ht))
(19)
Experiments
1
Me
4.1
Experiment Setups
Chas-rel
has-rel
log P
has-rel
Me
(20)
(1
y
has-rel
log(1
has-rel
Statistics
ACEO5
BioRelEx
ADE
Text2DT
Sentences
14525
2010
4272
500
where y has-rel 1 only the gold entity assigned
Avg sent. length
14.6
29.0
20.3
66.5
the i-th entity query is involved in relation.
Entity types
7
33
2
6
The consistency loss in the entity + relation
Relation types
1
direction is then calculated as follows:
Entities
38287
9871
17808
3280
Relations
7691
3235
6821
3196
sim( hit ea)
Σ
-DkI (PAM))))))))
Ent. per sent.
2.63
4.91
4.17
6.56
se(dtart,end,11]
Rel. sent.
0.53
1.61
1.60
6.39
Lent rrll
Σ
max
sim hit .;)
(21)
Table 1: Statistics of the ACE0S, BioRelEx, ADE and Tex12DT datasets.
JE(LMM,)
ic
Q
p
has-rel
2 0.5, E M2)]
(22)

---

ACE05
BioRelEx
ADE
Text2DT
Paradigm
Method
Ent-Fl
Rel-FI
Ent-F1
ReI-F1
Ent-FI
Rel-FI
Ent-F1
Rel-FI
KECI (2021)
87.4
66.1
90.7
81.7
Two-stage
MADR (2023)
91.8
80.1
PL-Marker (2022)
89.8
66.5
87.4
66.3
92.2
83.2
96.3
93.3
PFN (2021)
88.4
64.9
87.1
65.8
91.3
83.2
95.1
92.9
UniRE (2021)
88.8
64.3
87.3
65.6
91.6
82.9
95.4
92.5
TOPI (2021; 2022)
94.4
One-stage
BiSPN (Ours)
89.7
65.5
87.5
67.0
92.1
83.7
97.1
94.9
w/o Lenterel
89.2
64.0
86.1
66.2
91.4
83.5
96.8
94.4
w/o Link
89.5
65.1
87.4
66.7
91.8
83.5
97.1
94.6
w/o Centernel, Llink
88.9
63.4
85.7
66.0
90.9
83.0
96.2
94.1
Table 2: Main results on the ACEOS, BioRelEx,, ADE and Text2DT datasets. The bold font indicates the best score and the underline font indicates the second-best score.
Datasets. We experiment on one general-domain dataset (ACE05) and three mnowlegge-intensing datasets (BioRelEx, ADE, Tex12DT). ACEOS (Walker et al., 2006) includes a corpus of newswire, broadcast news, telephone conversa- tions. BioRelEx (Khachatrian et al., 2019) contains biomedical literature about binding interactions be- tween proteins and/or biomolecules. ADE (Gu- rulingappa et al., 2012) consists of medical reports describing drug-related adverse effects. Text2DT is originally an benchmark for medical Decision Trees extraction task in China Health Information Processing Conference 2022. And we only use its entity/relation annotation for experiments. See Table I for detailed statistics of the datasets.
ACE05
BioRelEx
Method
Rel-F1
sent/s
Rel-FI
sent/s
KEIC (2021)
66.1 I
15.7
UniRE (2021)
64.3
134.6
65.8
107.2
PL-Marker (2022)
66.5
32.0
66.4
21.9
BiSPN (Ours)
65.5
59.7
67.0
40.8
Table Inference speed comparison on the ACE05, BioRelEx datasets.
4.2 Implementation Details
We implement BiSPN with Pytorch (Paszke et al.. 2019) and experiments NVIDIA Tesla V100 GPUs. For ACE05, follow (Wang al., 2021; Ye et al., 2022) to initialize the shared encoder with BERT-base (Devlin et al., 2019). For BioRelEx and ADE, we follow (Haq et al., 2023; Lai et al., 2021) to initialize the shared encoder with BioBERT-base. For Text2DT, we initialize the shared encoder with Chinese-bert-wwm-- (Cui et al., 2021). The de- coding modules are randomly initialized. Follow- ing (Shen et al., 2022), freeze the encoder in the first epochs and unfreeze it in the remaining epochs. The learning rate decoding modules set to larger than the learning rate of the en- coder. We adopt an AdamW optimizer (Loshchilov and Hutter, 2017) equipped with a linear warm-up scheduler to tune the model. See Appendix A for details of hyperparameter tuning.
We additionally experiment the SciERC dataset, where follow the same setting in (Wang al.. 2021; Ye al., 2022). See Appendix B for the results.
Evaluation Metrics. Strict evaluation metrics are applied, where an entity is confirmed correct only if its boundary and entity type correctly predicted: relation triple is confirmed correct only if its relation type and head/tail entity are correctly predicted. For ACBOS and BioRelEx, we report the averaged Micro Fl scores over 3 random seeds. For ADE, we follows (Ji et al., 2020; Lai et al., 2021) to conduct 10-fold cross-validation and report the av- eraged Macro Fl scores. For Tex12DT, we follows the top-1 system on the evaluation task to ensemble 5 models trained with different random seeds and report the Micro F1 scores.
4.3 Compared Baselines
We compare BiSPN with several SOTA methods listed as follows.
KECI (Lai et al., 2021): knowledge-enhanned two-stage extraction model based on span graphs.

---

MADR (Haq et al., 2023): A pipeline of inde- pendent NER and RE models.
teek cres
PL-Marker (Ye et al., 2022): A span-based method that models the interrelation between spans by packing levitated markers the encoder.
we lask cee.
PFN (Yan et al., 2021): ): A partition filter network that models two-way interaction between NER and RE subtasks.
sap
UniRE (Wang al., 2021): A method based table filling. featured with a unified label space for one-stage joint entity and relation extraction.
cark
this you
TOP1: The top-1 system on the Tex12DT evalu- ation task, which combines PFN (Yan et al., 2021) with Efficient GlobalPointer (Su, 2022).
mark lesk ciea
york gal
Note that, we do not compare with SPN (Sui et al., 2020), UniRel (Tang et al., 2022) and QIDN (Tan al., 2022), since these methods can only extract relation triples and cannot recognize those entities uninvolved in relation.
Figure 3: Visualization of attention between entity queries and relation queries of a sample from BioRelEx. The input text this sample is "Moreover, in vitro binding of NF-B Spl its target DNA was not af- fected by the presence of K-12"
4.4 Main Results
Table summarizes the overall performance BiSPN and compared baselines on ACE05, BioRelEx, ADE and Tex12DT. In terms entity extraction, BiSPN performs competitively with or slightly better than SOTA methods on ACE05, BioRelEx and ADE, while outperforming the SOTA method PL-Marker by 0.8 Fl Tex12DT. In terms of relation extraction, BiSPN boosts SOTA performance by 0.7, 0.5 and 0.5 Fl on BioRelEx, ADE and Tex12DT respectively, verify- ing the effectiveness of our method on knowledge- intensive scene. However, although BiSPN out- performs SOTA one-stage methods by 0.6 Fl on ACE05, it is no match for the two-stage method PL-Marker on this general-domain dataset. We will look into the reason behind this in Section 4.6.2.
Rel [10051, 2050]
Rel 1
Rel 2 [1203, 302)
Method
12643, 5971
PL-Marker
66.5
63.8
57.9
BiSPN
65.5 (-1.0)
63.4 (-0.4)
58.2 (+0.3)
Table 4: Relation F1 scores BiSPN on ACB05 with different settings of knowledge density. The numbers in each bracket are the numbers of training. testing samples dataset under specific setting.
4.6 Analysis
4.6.1
Effects Cent+erel and Llink
We conduct ablation study and qualitative visualiza- tion to analyze how the bipartite consistency loss Lentevrel and entity-relation linking loss Llink work.
The results of ablation study are shown in Table 2. Without the bipartite consistency loss, the entity
4.5 Inference Efficiency
We compare the inference efficiency of KEIC, UniRE, PL-Marker, BiSPN on the BioRelEx and Tex12DT datasets. For fair comparison, the ex- periments are conducted on a server with In- tel(R) Xeon(R) E5-2698 CPUs and NVIDIA Tesla V100 GPUs. And we fix the batch size as 8 during evaluation. As shown in Table 3, BiSPN can pro- cess around 40~60 sentences per second, which is 2 times faster than the SOTA two-stage method PL-Marker. Although UniRE, SOTA one-stage method, is about 2.5 times faster than BiSPN, its performance of relation extraction is uncompetitive against BiSPN.
F1 scores drop by 0.5, 1.4, 0.7, 0.3 and the rela- tion Fl scores drop by 1.5, 0.8, 0.2, 0.5 on ACE0S, BioRelEx, ADE and Text2DT respectively. With- out the entity-relation linking loss, the entity Fl scores decrease by 0.2, 0.1, 0.3, and the relation Fl scores decrease by 0.4, 0.3, 0.2, 0.3 on the four datasets respectively. After removing the bipartite consistency loss and entity-relation linking loss to- gether, the entity Fl scores decrease by 0.8, 1.8, 1.2, 0.9 and the relation Fl scores decrease by 2. 1.0, 0.7, 0.8 on the four datasets respectively.
Three conclusions can be derived from these results: 1) both the bipartite consistency loss and

---

BESPN
w/n Auperend faot
Input Text
Eadiy
GPE: North
CPP:: North Koma
(laemaliers, ORG AFT, 15).
GPE: American
(tmmmators, CHREC-AR 11s9
North Noren has tebl Americcn Jasaselers alrrady has neclear moogres and Indveds hulld mere, senter L.N.
(Ir, ART, meggono
PER: maneeers
(R,ART, weepotn)
PER: Lawenaliees
ART,
UPI: It
WEA:: weapuee
(lommskkes,, ORG ARI, Amerlown)
weepots
(lwaellees, ORG APE, American
WEA: MARY
GPE U.S.
(t,, ART, Minee)
genenal transcriptien and TP115 bled dlivetlly TII:, stabilize
TELLA
Protei:: TIP
Prodein: TITAA
Protet:: TrI0
(TBP, bind. TFBA)
hind,
Proteis:
Proteis: TETMI
Protets: TELRA
(TTM Mixd, TFII))
(115) Mind, TFHAA)
(1DH1,, THU)
sesciation with the prinitatioo complec 1s, 16. dg.
and Burm
DNA: prometer
[TBP bind, prenuter)
Protcis congles: RNA palyeeroe 1I
(TRTIR bixd,
the cure of the RNA prbuuces
Proteia comples
Proteie-compppp complee: RNA pálmerrs
(TTIES sand. EUPY
DNA promoser
Proteie-comple:: IIA polymeras II
(TBP, Mard protootory (peoanster, TBP)
peciaiiiation complex
preitattetion comples
Figure 4: Cases from ACE05 and BioRelEx. False negative predictions are in blue.
entity-relation linking loss contribute the overall performance of BiSPN; 2) the bipartite consistency loss is much more effective than the entity-relation linking loss; 3) the effects of the two losses are not orthogonal but still complementary in some degree. To visualize the attention between entity queries and relation queries, we record the attention weight in the last bidirectional self-attention layer of shared decoder for a sample from BioRelExx. Since the original weight is bidirectional, we average the weight over two directions and conduct nor- malization to obtain the weight for visualization. As shown in Figure 3, without Cente vel and Ltink. the interaction between entity queries and relation queries is chaotic and pointless. After applying Lenterel or Liin separately, the relation queries as- sociated with gold relation triples tend to focus on the entity queries associated with gold entities. When applying the two loss functions together, the relation queries associated with gold relation triples further concentrate the entity queries whose tar- get entities are the same as their head/tail entities. This phenomenon coincides with the purpose of the two loss functions.
without relation are discarded, the performance gap between PL-Marker and BiSPN narrows from 1.0 to 0.4. When further discarding the samples with less than 2 relation triples, BiSPN even performs slightly better than PL-Marker. This reveals that the strength of BiSPN emerges in knowledge-intensive scene, which is reasonable, since BiSPN works by modeling interaction among knowledge instances.
4.7 Case Study
Figure 4 illustratee two test cases from ACE05 and BioRelEx respectively. In the first case, BiSPN without Lentevrel and Lrink fails to recognizes the WEA entity "more" and the PART-WHOLE rela- tion between "it" and "more", while BiSPN suc- cessfully extracts them by considering the context in entity-view and relation-view concurrently. Like- wise, in the second case, BiSPN successfully rec- ognizes the entity "promoter" after Lente vel and Llink is applied. However, it still fails to recognize "RNA polymerase Il preinitiation complex", which is complicated and may require domain knowledge for recognition.
Conclusion
4.6.2 Influence of Knowledge Density
In this work, we present BiSPN, novel joint entity relation extraction framework based on bipartite set prediction. generates entity set and relation set in a distributed manner, so as to avoid error prop- agation. To maintain the coherency between the generated entity set and relation set, we come up with two novel loss designs, namely bipartite con- sistency loss and entity-relation linking loss. The first one pulls closer the predicted boundary/type distributions of entities and head/tail entities, while the second one enforces the interaction between entity queries and relation queries. Extensive ex- periments demonstrate the advantage of BiSPN
The performance of BiSPN on ACE05 is not so good as its performance BioRelEx, ADE and Tex12DT. We hypothesize it is the sparsity of rela- tion triples that hinders the learning of BiSPN. As listed Table the average number of relations per sentence is 0.53 on ACE05. In contrast, the average numbers of relations per sentence are 1.61, 1.60 and 6.39 on the other three datasets.
To verify our hypothesis, we filter samples ac- cording to the number of relations they contain and experiment on different versions of the filtered dataset. As shown in Table 4, when the samples

---

knowledge-intensive scene, as well as the effective- ness of the proposed bipartite consistency loss and entity-relation linking loss.
Kalpit Dixit and Yaser Al-Onaizan. 2019. Span-level
model for relation extraction. In Proceedings of the 57th Annual Meeting of the Association for Computa- rional Linguistics, pages 5308-5314 14, Florence, Italy, Association for Computational Linguistics.
Limitations
As mentioned in Section 4.6.2, the performance of our BiSPN framework can be hindered when the distribution of relation triples are overly sparse in dataset. This suggests that BiSPN is a better choice for biomedical and clinical domains but not the general-domain, where knowledge sparsity is common.
Harsha Guralingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmman-AAiitus, and Luca Toldo. 2012. Development of benchmark corpus to support the automatic extraction of drug- related adverse effects from medical case reports. Journal of Biomedical Informatics, 45(5):885-892. Text Mining and Natural Language Processing in Pharmacogenomics.
Another limitation of BiSPN its reliance on a fixed number of entity/relation queries. Although is possible to set the number of queries larger in or- der to let BiSPN generalize to longer text input, the cost is additional memory and time consumption that grows quadratically. To effectively address this, future work can draw lessons from the field of dynamic neural network and explore dynamic selection of instance queries.
Hasham UI Haq, Veysel Kocaman, and David Talby. 2023. Mining Adverse Drug Reactions from Unstruc- tured Mediums at Scale, pages 361-375. Springer International Publishing, Cham.
Bin Ji, Jie Yu, Shasha Li, Jun Ma, Qingbo Wu, Yusong Tan, and Huijun Liu. 2020. Span- Span-based joint en- tity and relation extraction with attention- hased span- specific and contextual semantic representations. In Proceedings of the 28th International Conference on Computational Linguistics, pages 88-99, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
Acknowledgments
We thank the reviewers for their valuable sugges- tions. This study is partially supported by National Key R&D Program of China (2021ZD01 13402
Hrant Khachatrian, Lilit Nersisyan, Karen Ham- bardzumyan, Tigran Galstyan, Anna Hakobyan, Ar- sen Arakelyan, Andrey Rzhetsky, and Aram Galstyan. 2019. BioRelEx 1.0: Biological relation extraction benchmark. In Proceedings of the 18th BioNLP Workshop and Shared Task, pages 176-190, Florence, Italy. Association for Computational Linguisties.
). National Natural Science Foundation of China (62276082), Major Key Project of PCL (PCL2021A06) Shenzhen Soft Science Research
and the Fundamental Research Fund for the Central
Program Project (No. 200.XX22220705152.8..
H. W. Kuhn. 1955, The hungarian method for the assign- ment problem. Naval Research Logistics Quarterly. 2112))833997
Universities (HIT.D2JJ.2033117)
References
Tuan Lai, Heng Ji, ChengXiang Zhai, and Quan Hung Tran. 2021. Joint biomedical entity and relation extraction with knowledge-enhanced collective in- ference. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the I 11th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 6248-6260, Online, Association for Computa- tional Linguistics.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Computer Vision - ECCY 2020. pages 213-229, Cham. Springer International Pub- lishing.
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. 2021. Pre- Pre-training with whole word masking for chinese bert. IEEE Transactions on Audio, Speech and Language Processing.
Xiaoya Li, Fan Yin, Zijun Sun, Xiayu L., Arianna Yuan, Duo Chai, Mingxin Zhou, and Jíwei Li. 2019. Entity- relation extraction as multi-tur turn question answering. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 1340- 1350, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-48 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ilya Loshchilov and Frank Hutter. 2017.
De-
coupled weight decay regularization. Cite arxiv:171 ence paper at ICLR 2019.
Published as a confer-

---

Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu, 2022. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Anmual Meet- ing of the Association for Computational Linguistics (Volume I: Long Pupers), pages 5755-5772, Dublin, Ireland. Association for Computational I Linguisties.
Conference on Neural Information Processing Sys- tems, NIPS'17, page 6000-6010, Red Hook, NY. USA. Curran Associates Inc.
Christopher Walker, Stephanie Strassel, and Kazuaki Maeda. 2006. The automatic content extraction (ace) program tasks, data, and evaluation. In Linguistic Data Consortium, page 57, Philadelphia.
Adam Paszke, Sam Gross. Francisco Massa, Adam Lever, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf. Edward Yang. Zachary Vito, Martin Raison, Alykhan Te- jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelz- imer, F. d'Alché Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc.
Yijun Wang, Changzhi Sun, Yuanbin Wu, Hao Zhou, Lei Li, and Junchi Yan. 2021. UniRE: A unified label space for entity relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 220-231, Online. Association for Computational Linguistics.
Yucheng Wang, Bowen Yu. Yueyang Zhang. Tingwen Liu, Hongsong Zhu, and Limin Sun. 2020. TPL.inke: Single-stage joint extraction of entities and relations through token pair linking. In Proceedings of the 28th International Conference on Computational Lin- guisties. pages 1572-1582, Barcelona, Spain (On- line). International Committee on Computational Lin- guistics.
Yongliang Shen, Xiaobin Wang, Zeqi Tan, Guangwei Xu, Pengjun Xie, Fei Huang, Weiming Lu, and Yuet- ing Zhuang. 2022. Parallel instance query network for named entity recognition. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics /Volume 1: Long Papers), pages 947-961, Dublin, Ireland. Association for Computa- tional Linguistics.
Zhiheng Yan, Chong Zhang, Jinlan Fu, Qi Zhang, and Zhongyu Wei. 2021. A partition filter network for joint entity and relation extraction. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing, pages 185-197, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Jianlin Su. 2022. Efficient globalpointer.
Dianbo Sui, Yubo Chen, Kang Liu, Jun Zhao. Xian- grong Zeng, and Shengping Liu. 2020. Joint En- tity and Relation Extraction with Set Prediction Net- works. arXiv e-prints, page arXiv:2011.016
Deming Ye, Yankai Lin, Peng Li, and Maosong Sun. 2022. Packed levitated marker for entity and relation extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Compulational Linguistics (Volume Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4904 4917. Association for Computational Linguistics.
Zeqi Tan, Yongliang Shen, Xuming Hu, Wenqi Zhang, Xiaoxia Cheng, Weiming Lu, and Yucting Zhuang. 2022. Query-based instance discrimination network for relational triple extraction. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pages 7677-7690, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Xiangrong Zeng, Daojian Zeng, Shizhu He, Kang Liu, and Jun Zhao. 2018. Extracting relational facts by an end-to-end neural model with copy mechanism. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 506-514 514, Melbourne, Australia. Association for Computational Linguistics.
Zeqi Tan, Yongliang Shen, and Shuai Zhang. 2021. A sequence-to-set network for nested named entity recognition. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI-21.
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 50-61. Online. Association for Computational Linguistics.
Wei Tang, Benfeng Xu, Yuyue Zhao, Zhendong Mao, Yifeng Liu, Yong Liao, and Haiyong Xie. 2022. UniRel: Unified representation and interaction for
joint relational triple extraction. In Proceedings of the 2022 Conference Empirical Methods in Nat- ural Language Processing, pages 7087-7099, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguisties.
A Hyperparameter Configuration
Ashish Vaswami, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Eukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Proceedings of the 31st International
We tune the hyperparameters for each dataset by manually trying different values of cach hyperpa- rameter within a specific interval and choosing the

---

value that results in the highest relation Fl the development (For ADE, the validation of its first fold of data employed as the develop- ment set). After trial, we find optimal to set the numbers of shared decoder layers, entity decoder layers and relation decoder layers as 4, 1, for all datasets. The trial intervals and final configuration of other hyperparameters are shown in Table 5.
B Experiment Results the SciERC dataset
Here, we append the results of additional exper- iment on the SciERC dataset. As shown in Ta- ble 6, in terms of entity recognition, BiSPN derperforms SOTA two-stage method PL-Marker by Ent-F1, but still outperforms SOTA one-stage methods (PFN, UniRE) substantially. In terms of relation triple extraction, BiSPN establishes new SOTA (Rel-Fi1) on the SciERC dataset. In terms of inference speed, BiSPN is about faster than PL- Marker, but slower than other one-stage methods. The results after ablating the consistency loss and the linking loss verify the effectiveness of them the dataset.

---

Parameter
Trial Interval
ACE05
BioRelEx
ADE
Text2DT
SciERC
Epochs
[50, 100]
70
75
70
80
70
Warmup Rate
[0.01, 0.2]
0.1
0.1
0.1
0.1
0.1
Encoder Ir
[le-5, 5c-5]
2e-5
2c-5
2c-5
2c-5
2e-5
Decoder Ir
[le-5, 5c-5]
4c-5
4c-5
4e-5
4e-5
4e-5
Batch Size
[2, 64]
8
8
Me
[20, 40]
40
30
30
30
35
M,
[20, 40]
35
25
25
30
30
a [le-3, le-2]
le-4
le-4
1e-3
1e-4
B
[1e-5, le-2]
le-3
1e-4
le44
1e-3
Table Configuration of Hyperparameters. Ir represents the initial learning rate. Me. M, are the number of entity queries and the number of relation queries respectively. o, 3 are the weights of the hipartite consistency loss and entity-relation linking loss respectively.
Method
Ent-F1
Rel-F1
sent/s
PFN (2021)
66.8
38.4
55.1
UniRE (2021)
68.4
36.9
79.6
PL-Marker (2022)
69.9
41.6
18.5
BiSPN (Ours)
68.9
42.0
37.2
w/o Lent
67.7
41.4
w/o Llink
68.0
41.3
w/o Cente-rel, Liink
67.1
40.8
Table Results on the SciERC dataset.


Reviews:
[{"review_id": "GqgRo7SHSO", "reviewer": "Reviewer_DE1H", "summary": "", "questions": "1. See reasons to reject 2.\n\n2. Did you also consider other knowledge intensive datasets, e.g., do you know how the method works on SciERC?\n\n3. Why should M_e be much larger than N_e (similarly for relations)?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. Here are our responses to your questions:\n\n**Answer to Question 1:**\n\nAs stated in lines 160-163, a null label ∅ is employed to indicate that no entity or no relation is found by an entity/relation query. When calculating the Prediction Loss, we assign the null label to entity/relation queries that are unassociated with gold entities/relations. During inference, the entity predictor filters out any entity whose predicted label is ∅ (lines 250-251) and the relation type predictor filters out relation triples similarly. In practice, the model additionally filters out any predicted entity or relation triple with boundary score below a predefined threshold.\n\n**Answer to Question 2:**\n\nBy the time we submitted the paper, we had not yet experimented on the SciERC dataset due to time limit. We carried out experiments on SciERC afterward. The results are as follows:\n\n_Main results on the SciERC dataset_\n\n| Method                                                                    | Ent-F1 | Rel-F1 | sent/s |\n| :------------------------------------------------------------------------ | :----: | :----: | :----: |\n| PFN                                                                       |  66.8  |  38.4  |  55.1  |\n| UniRE                                                                     |  68.4  |  36.9  |  79.6  |\n| PL-Marker                                                                 |  69.9  |  41.6  |  18.5  |\n| BiSPN (Ours)                                                              |  68.9  |  42.0  |  37.2  |\n| w/o consistency loss                                             |  67.7  |  41.4  |   -    |\n| w/o linking loss                                                      |  68.0  |  41.3  |   -    |\n| w/o both                                                                  |  67.1  |  40.8  |   -    |\n\nIn terms of entity recognition, BiSPN underperforms SOTA two-stage method PL-Marker by $1.0$ Ent-F1, but still outperforms SOTA one-stage methods (PFN, UniRE) substantially. In terms of relation triple extraction, BiSPN establishes new SOTA ($42.0$ Rel-F1) on the SciERC dataset. In terms of inference speed, BiSPN is about $2\\times$ faster than PL-Marker, but slower than other one-stage methods. The results after ablating the consistency loss and the linking loss verify the effectiveness of them on the dataset.\n\n**Answer to Question 3:**\n\nIn fact, for a specific corpus, $M_e$ ($M_r$) should be larger than the maximum number of entities (relation triples) occurring in a single sentence of the corpus, so that our method can be valid for all sentences of the corpus.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This is a solid paper on joint entity/relation extraction with impressive results.\n\n2. They have done a great job in explaining the details of the methods as well as the experiments.\n\n3. The ablation studies, visualization, and case studies help confirm their claims.\n\n4. They also show that their method is efficient compared to two-stage methods.", "reasons_to_reject": "1. I'm worried about the generalizability of this method beyond relatively old/small LMs such as BERT. However, the smaller LMs lead to better efficiency which is one of the goals of the paper.\n\n2. I couldn't find how exactly their method works during inference. Adding an explanation will help (how the many relation queries and entity queries are mapped to actual predictions).\n\n3. The paper needs more proofreading throughout, but I'm sure this can be done in the updated version (see example typos below).", "questions_to_authors": "1. See reasons to reject 2.\n\n2. Did you also consider other knowledge intensive datasets, e.g., do you know how the method works on SciERC?\n\n3. Why should M_e be much larger than N_e (similarly for relations)?", "paper_topic_and_main_contributions": "This paper proposes a new method for joint entity and relation extraction. Their work is based on Set Prediction Networks (SPN) that uses a set of learnable queries that model the interaction between different relations/entities using attention mechanism. SPN has been applied to the two tasks of entity extraction and relation extraction separately, but this paper does that jointly. Their approach generates both entities and relations in one pass, but the challenge is to maintain the consistency between the two sets of outputs (e.g., entities in a triple should be in the list of extracted entities). They propose two auxiliary loss functions:\n\n1. Bipartite consistency loss: For each sub/obj in a triple, it looks for a reference entity and encourages their representations to be similar to each other. Similarly, for each entity, it looks for an entity in the extracted triples and encourages their representation to be similar to each other.\n\n2. Entity-Relation linking loss: This loss function encourages the representation of the entity queries and relation queries to be consistent. In particular, when a relation query and entity query are relevant (the gold entity associated with the entity query is present in the gold triple associated with the relation query), their representations are encouraged to be similar.\n\nThey perform experiments on ACE05 and three biomedical datasets. Their model works comparably or better than SOTA methods. Their methods achieves new SOTA on the three biomedical datasets, which are knowledge-intensive (multiple relations per sentence). However, their method gets worse results on ACE05. Through their experiments, they show that this is actually because ACE05 doesn't have many relations per sentence while their method is effective when more relations are involved in each sentence.\n\nThey perform important ablation studies showing that both proposed losses are useful. They also perform visualization and look into case studies to confirm their claim.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Figure 1: difficult to guaranteed => difficult to be guaranteed\n\nLine 53: included the => included in the\n\n101: all entity => all entities"}, "extra_text": {}}, {"review_id": "4Dhi2w13yk", "reviewer": "Reviewer_w2i7", "summary": "", "questions": "Please refer to \"Reasons To Reject\".", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We hope to address your questions with the following responses:\n\n**Answer to Question 1:**\n\nAlthough our method does not surpass the SOTA two-stage method (PL-Marker) on the general domain dataset ACE05, it performs better than SOTA one-stage methods (PFN, UniRE) on the dataset. In this sence, the generalization ability of our method in the general domain should not be overlooked.\n\n**Answer to Question 2:**\n\nBecause of the page limit of EMNLP, we mainly presented the main contributions of our paper, and did not conduct an error analysis. However, if the paper gets accepted for publication, we will include an error analysis to facilitate future research.\n\n**Answer to Question 3:**\n\nIn spite of some typos, the overall writing of our paper is good, as suggested by other reviewers. And we do not think the writing of our paper prevents it from publication as we will fix these typos carefully.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of this paper is intuitive.\n\n2. This paper conducts extensive experiments and provides detailed analysis.", "reasons_to_reject": "1. The generalization ability of the proposed model in the general domain appears to be limited.\n\n2. It would be better to include an error analysis to further explore the limitations of the proposed model.\n\n3. The writing of this paper needs to be further polished.", "questions_to_authors": "Please refer to \"Reasons To Reject\".", "paper_topic_and_main_contributions": "This paper proposes a joint entity-relation extraction model, Bipartite Set Prediction Network (BiSPN). Specifically, BiSPN exploits a novel bipartite consistency loss and an entity-relation linking loss to maintain the coherence between the generated entity set and relation set. Experiments on biomedical/clinical datasets and a general-domain dataset show the effectiveness of BiSPN.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Line 53: “should be included the” --> “should be included in the”\n\n2. Line 218: “transform”-->“transforms”"}, "extra_text": {}}, {"review_id": "EFb7vD7nOB", "reviewer": "Reviewer_6vNJ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We hope to address your concerns with the following responses:\n\n- Our work is an essential step to successfully apply set prediction networks on the joint extraction of entities and relations. Although the idea of adding a consistency loss is natural, it requires ingenious design and is not within easy reach.\n\n- We do not experiment on ACE2004 because it shares a similar corpus with ACE2005 and we believe ACE2005 is more representative than ACE2004. For the results on the SciERC dataset, please refer to our responses to Reviewer DE1H. In addition, the Rel-F1 metric used in our work (lines 372-377) is in fact the Rel+ metric that you mention, which is the most comprehensive metric for relation triple extraction.\n\n- As demonstrated by Table 3 in our paper, BiSPN achieves a better trade-off between accuracy and efficiency compared to the SOTA two-stage PL-marker and the SOTA one-stage method UniRE, which is a significant advantage of our method.\n\n- The results of PL-marker on the ADE dataset (shown below) will be included in a revised version of our paper. We can see that BiSPN outperforms PL-Marker by $0.5$ Rel-F1, while being competitive with PL-Marker in terms of Ent-F1.\n\n_A Comparison between PL-Marker and BiSPN on ADE_\n\n| Method       | Ent-F1 | Rel-F1 |\n| :----------- | :----: | :----: |\n| PL-Marker    |  92.2  |  83.2  |\n| BiSPN (Ours) |  92.1  |  83.7  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This article is well-written and very easy to follow, making it accessible for readers to understand the research and its contributions.\n\n-The experimental results are promising, and the paper provides sufficient experiments and a detailed ablation study, which enhances the credibility of the proposed method and its performance evaluation.", "reasons_to_reject": "- This article is relatively incremental. The author claims that using a set prediction network for the joint entity and relation extraction task is less explored. However, this simply involves combining the set prediction network for entity prediction and the set prediction network for relation prediction. The idea of adding a consistency loss is also natural and relatively trivial.\n\n- Some commonly used datasets, such as ACE2004 and SciERC, were not included in the results presented by the authors. I want to know why the authors did not perform validation on these datasets as well. Additionally, many papers now provide results using the Rel+ metric. I would like to know why the authors did not include this metric in their evaluation\n\n- Despite the promising experimental results, the proposed method does not demonstrate a significant advantage over the PL-marker; The authors should include the results of the PL-marker on the ADE dataset for comparison; Although it performs better in terms of speed compared to the two-pass model, it is significantly slower when compared to one-pass models such as UniRE.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors combined a set prediction network (SPN) for predicting entities and another SPN for predicting relation triplets. They integrated these two SPNs together and used consistency loss to enhance the predictive results.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": " "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a joint extraction method of entities and relation triples by combining set prediction networks (SPNs) and a new loss function that promotes global consistency. The effectiveness of the work is demonstrated on three biomedical datasets, where SOTA performance was achieved; however, their method didn't outperform baselines on ACE05. Through their experiments, the paper argues that this is because ACE05 doesn't have many relations per sentence while their method is effective when more relations are involved in each sentence. As pointed out by one of our reviewers, some commonly used datasets, such as ACE2004 and SciERC, were not included in the paper; many recent papers now use the Rel+ metric, which is also not presented in this paper.

The soundness of the work is well-perceived by all of the reviewers but reviewers split in terms of excitement. The general idea of joint extraction of entities and relations has been explored in the field for a long time, in training, in inference, or in both training and inference, so from that angle, the idea is not exciting enough. The novelty mainly resides only in the choice of SPN and the newly defined consistency loss function. The lack of analysis on other commonly used datasets and metrics is another limitation of the work.
Decision: Accept-Findings