Document-level Relationship Extraction by Bidirectional Constraints of Beta Rules
Authors: Yichun Liu
Zizhong Zhu
Xiaowang Zhang
Zhiyong Feng
Daoqi Chen
Yaxin Li
Venue: EMNLP 2023 Main (2023)
Keywords: Document-level Relation Extraction
Logical Consistency
Beta Distribution
Bidirectional Constraints
TL;DR: We propose a rule-constrained framework that utilizes the learning of beta rules and logical bidirectional constraints to assist in document-level relationship extraction.

Abstract: Document-level Relation Extraction (DocRE) aims to extract relations among entity pairs in documents. Some works introduce logic constraints into DocRE, addressing the issues of opacity and weak logic in original DocRE models. However, they only focus on forward logic constraints and the rules mined in these works often suffer from pseudo rules with high standard-confidence but low support. In this paper, we proposes Bidirectional Constraints of Beta Rules(BCBR), a novel logic constraint framework. BCBR first introduces a new rule miner which model rules by beta contribtion. Then forward and reverse logic constraints are constructed based on beta rules. Finally, BCBR reconstruct rule consistency loss by bidirectional constraints to regulate the output of the DocRE model.  Experiments show that BCBR outperforms original DocRE models in terms of relation extraction performance ($\sim$2.7 F1 score) and logical consistency($\sim$3.1 logic score). Furthermore, BCBR consistently outperforms two other logic constraint frameworks.

Document-level Relation Extraction (DocRE) intends extract relationships from documents. Some works introduce logic constraints into DocRE, addressing the issues of opacity and weak logic in original DocRE models. How- ever, they only focus on forward logic con- straints and the rules mined in these works of- ten suffer from pseudo rules with high standard- confidence but low support. this paper, we proposes Bidirectional Constraints of Beta Rules(BCBR), a novel logic constraint frame- work. BCBR first introduces new rule miner which model rules by beta contribtion. Then forward and reverse logic constraints are con- structed based on beta rules. Finally, BCBR reconstruct rule consistency loss by bidirec- tional constraints to regulate the output of the DocRE model. Experiments show that BCBR outperforms original DocRE models on relation extraction performance (~2.77 FI) and logic consistency(-33. Logic). Furthermore, BCBR consistently outperforms two other logic con- straint frameworks. Our code is available at Document-level Relationship Extraction by Bidirectional Constraints of Beta Rules
Yichun Liu Zizhong Zhu?; Xiaowang Zhang*1
Zhiyong Feng', Daoqi Chen', Yaxin Li
CColleee of Intelligence and Computing, Tianjin University 2 School of New Media and Communication, Tianjin University
Abstract
Document: [1] Porsche is currently enjoying record profits after two-year downturn, but the export-reliant company has hit pothole in Argentina. [2] ... the Argentine govermment announced Wednesday.
Entities: Porsche, Argentina, Argentine
Relations: based inO(Porsche, Argentina), gpee (Argentina, Argentine)
Rule: based in0-x(eo: ez) based in0(eg. e1) A gpe0 (C1, C2)
Prediction:
Iussd bodl
Argentina
Spep
Porsche
A Argentine
Figure I: A case of logic constraint DocRE. Different colors represent different entities. Solid lines represent the relationships predicted by a general DocRE model, while dotted lines represent the relationships predicted by rule-based constraints.
Based on the use of rules, we can classify pre- vious DocRE works into two categories: plain DocRE models and logic constraint DocRE models. In plain DocRE models, attention mainly given to learning more powerful implicit representations.
1 Introduction
In recent years, DocRE attracts significant attention from researchers, with its intention to distinguish the relations between entity pairs in the documents. It's not limited to sentence-level relation extrac- tion (Zeng et al., 2014; Zhang al., 2017; Han et al., 2018; Wang et al., 2021). It aims to uncover the dependencies between entities in different sen- tences of one document (Zhou et al., 2021; Ma et al., 2023). The challenges in DocRE mainly include two aspects: first, it's difficult to capture complex long-range dependencies between entity pairs in documents; second, it's prone to errors in logical reasoning due lack of logic. To ad- dress the aforementioned challenges, the academic community has made a lot of efforts.
These methods include models based on sequence encoders (Wang et al., 2019; Xu et al., 202 la; Zhou et al., 2021), and models based on graph encoders (Zeng et al., 2020; Christopoulou et al., 2019). Although these methods have achieved de- cent results, their inferences are non-trame con-transparent and lack logic. making them prone to errors logical inference. Meanwhile, the combination relation extraction frameworks and logical rules has alleviated the issues of low transparency and weak logic. As the Figure/ shows, we can derive two relations based in0 (Porsche, Argentina and gpe0 (Argenlina, Argentine) from the doc- ument. When the rule is applied,we can pre- diet the relation based_in0- between Porsche and Argentina. LogiRE(Ru et al., 2021) is the first work that introduces logical rules into
These authors contributed equally.
Corresponding authors

---

document-level relation extraction. It employed the pnpectaatoonMaaimizzttiii (EM) algorithm to iteratively update the rule generator and relation extractor, optimizing the results relation extrac- tion. However, rule generator and relation extractor are in isolation. The EM algorithm enables joint optimization of the two models, but it can still lead to suboptimal results. MILR(Fan et al., 2022) ad- dresses the issue of suboptimality by jointly train- ing the relation classification and rule consistency losses. But MILR utilizes confidence- based meth- ods to mine rules, which can lead to pscudo rules with high standard- standard-confidence confidence but low support, af- fecting the effectiveness of relation extraction.
passes LogiRE and MILR in terms of both relation extraction performance and logic con- sistency.
2 Related Work
2.1 Rule Learning
Rule learning is the foundation of logic constraint DocRE. Rule learning is primarily applied in the field of knowledge graphs, but DocRE can draw on its ideas. Currently, rule learning methods can be divided into three types: symbol-based rule learn- ing, embedding-based rule learning, and differen- tiable rule learning based on TensorLog (Cohen, 2016). Symbol-based rule learning aims to mine rule paths of high frequency knowledge graphs more efficiently. Galárraga al. proposes the open- world assumption and utilizes pruning strategy mine rules. Meilicke et al. adopts a bottom-up rule generation approach. Embedding- based rule learning focuses on learning more powerful em- beddings for entities and relations. Omran et al. calculates the similarity between rule head and rule body to select better rules. Zhang et al. iteratively learns embeddings and rules to generate the rule set. TensorLog-beee based methods transform the rule learn- ing process into differentiable process, allowing neural network models to generate rules. For ex- ample, Sadeghian et al.; Sadeghian et al. trains a rule miner by using bidirectional RNN model, and Xu et al.utilizes transformer model.
In the paper, we propose BCBR, novel frame- work which assists relation extraction with the help of logical rules. BCBR models the bidirectional constraints of beta rules and optimizes relation ex- traction through rule consistency loss. (1) The prior rule set derived from the documents different from the one extracted from the knowledge graph. Textual data has a relatively small volume, and the inceredo correlations are low, leading to sparsity in the prior rule set. Thus, general rule miners in previous works cause the prevalence of pseudo rules with high standard-conffdencee but low support. To tackle this problem, we utilize the beta distribution to model the rules and consider both their successful predictions and failures to filter the rules. (2) In addition, we discovered that the constraints between rule head and rule body are bidirectional, while previous methods often only considered the forward constraints from rule body to rule head. Therefore, introduce reverse con- straints from rule head rule body. (3) Finally, based on above constraints, we reconstruct the rule consistency loss to enhance the performance of the original DocRE models. We summarize our contributions as follows:
2.2 Document-level Relation Extraction
Previous works on DocRE can be divided into two categories: plain DocRE and logic constraint DocRE. Plain document-level relation extraction focuses on learning more powerful representations (Zheng et al., 2018; Nguyen and Verspoor, 201 8), There are methods based on sequence models that introduce pre-trained models to generate better rep- resentations (Wang al., 2019; Ye et al., 2020: Xu et al., 2021b), Zhou et al. sets an adaptive threshold and uses attention to guide context pooling. Ma et al. uses evidence information as supervisory signal to guide attention modules. Graph-based methods model entities and relations as nodes and edges in graph and use graph algorithms to gener- ate better representations (Zeng et al., 2020; Wang al., 2020).
To our knowledge, we first propose rule miner that utilizes the Beta distribution to model rules.
We introduce reverse logic constraint en- sure that the output of DocRE models satisfies the necessity of rules.
We model bidirectional logic constraints as reasonable probability patterns and turn them into rule consistency loss.
However, previous works lack transparency and logic, making them prone to errors in logical in- ference. Currently, research on logic constraint
Our experiments demonstrate that BCBR sur-

---

Document
Rules
Gold Annotations
[1] Heids, the world funous from the Swies Alps who appeaeed
Sy.created (e() drected es) dedemrrreeeddbye e.) + moreated_by
create by(Hleidi. Markus).
years ago... [2].. fanded by the Zunch based film compony Vega and directed by German film maker Markus Imboden, to launched Germany Tbursday...
directed, by(Heidi, Markus),
Forward Constraints
Reverse Constraints
Beta Rule Miner
Backbone
1
X-Bela.81
c
Towky
Probebiliy denslty
1900 LOG)
1500 2003
5000
9,,
(200, GOGO
2,
5
9 C3
Leta
Lens
global
Joint Traning
Figure 2: The framework of BCBR. The left a plain DocRE model, the right is the Beta rule miner and the middle is the bidirectional logic constraint module and joint training module.
document-level relation extraction is limited. There are two noteworthy works in this area: Log et al., 2021 annd MILR (Fan et al., 2022). LogiRE involves two modules, the rule generator and the relation extractor. It uses the EM algorithm to ef- ficiently maximize the overall likelihood. But this method often leads to suboptimal results due the isolation between rule generator and relation tractor. To address this limitation, MILR constructs a joint training framework that combines rule con- sistency loss and relation classification loss of the backbone model. Previous works only applied for- ward logic constraints, while our works introduce reverse logic constraints and enhance the result of backbone model.
Logic Rules We define binary variable r (ch, ct) to indicate the existence of r E IR between en and 6₁- When r is true, r (ex, 6q) = 1; otherwise, (eh, ex) = 0. A rule consists of rule head and rule body. The rule head is denoted as Thead (co, ei), and the rule body is defined as the conjunction of I binary variables, denoted Tbody (Co, e1)- We define the rule set as S and the pattern of rules is as follows:
Thead (co, e1) f- r1 (eo, An. (co..)))... (I)
where 6 E E, represents the rule length, and Thead (eh, et) and ri (e: 1, 1,e;) are referred as the head atom and the body atom, respectively.
On the basis of Closed World Assump- tion(CWA)
3 Method
et al., 2015), introduce two concepts: standard confidence and head cover- age. Standard confidence refers to the conditional probability of rule head being satisfied given that the rule body is satisfied. The standard confidence of rule can be modeled as the following condi- tional probability distribution:
In this chapter, introduce our framework Bidirectional Constraints of Beta Rules(BCBR) (Fig.2). We define concepts related to DocRE and rulcs(Scc.3.l.. Then We propose novel rule ex- traction method(Sec.3.2) and model bidirectional logic constraints based on rulcs(Sec.3.3)) Finally, we construct rule consistency loss and jointly train with relation classification loss to enhance relation extraction performance(Sec.3.4)
C (Thead Tbody)
Psc (s)
C (Pbody)
(-) represents counter.
3.1 Prelininaries
Head coverage refers to the conditional probabil- ity of rule body being satisfied given that the rule head is satisfied. The head coverage of rule can be modeled the following conditional probability distribution:
Document-level Relation Extraction Given a document D and entities & ( Entities consti- tute entity pairs (Cn, f)i which en and e; indicate the head entity and tail entity, respec- tively. The task of DocRE is distinguish the relation between each entity pair (eh, et), where r R and IR RU (WA). . R a set of relations and ACA indicates there is no relation on the entity pair.
C. (Fhead A Thody (Thead)
Phe (s)
(3)
Backbone Model Paradigm Our approach in- volves using logic rules to assist the original

---

DocRE model, which can be generalized any backbone DocRE model. Therefore, we define the paradigm of the backbone model here. For all en- tity pairs (en, ft) in the document, the backbone model generates a score 9 (Ch, Qt) for their relation r. The probability of this triple being true is defined follows:
Algorithm 1 Beta Rule Miner
Input: training set's labels: T, rule template set generated by labels Stemplates lower bound of integration for Beta distribution k, rule fitness threshold; 1)
Output: high quality rules : S
5 {}
2: for s in Stemplate do
P (r en, = (9 (eh, et))
(4)
3:
0g =1
where a (+) is sigmoid function.
4:
B. 6,=1
for TD in T do
During training, the backbone model uses bi- nary cross-entropy loss or adaptive threshold loss to compute the relation classification loss Cats- Dur- ing inference, the model sets global threshold uses learned adaptive threshold to determine whether the triple (ch, et) holds:
if rbody (co, et) in To and l'haad (co, qi) in
To then
Os +=
8:
else
if Tbody (eo, et) in To and
Thead (co, e1) not in To then
9:
3, +=1 end if
Pr (eh: ex) I (P (r en, ex) >0
(5)
10:
11:
end for
where I (-) represents the indicator function, and 9 represents a threshold. If the probability of the triple being true satisfies the threshold, then Pr (Ch, Gx) 1, indicating that the triple (cA, ci) holds. Conversely, if the probability does not sat- isfy the threshold, then Pr (ch, et) 0, indicating that the triple (eh, ex) does hold.
12:
Ps II (P, (a> 1)
13:
if P. == then
14:
S.add(s)
15: end
16: end for
17: return S
3.2 Beta Rule Miner
indicating that the rule does not hold. The proba- bility density function of the Beta distribution for rule is given by:
Rule mining methods on knowledge graphs are mainly based on the large-scale and data-intensive essence of knowledge graphs. However, when these methods are transferred document data (Ru et al., 2021; Fan et al., 2022), they still rely on confidence to filter rulcs. leads to a inadapt- able phenomenon that there are massive pseudo rules with high port. Therefore, we abandon the approach of using confidence support alone and instead use the Beta distribution to model rules. In this section, propose a new rule mining method called beta rule miner.
fs (r; as B₂) -0 (1 x)33
(7)
B (as, Bo)
where E [0, 1] and B(-) represents the beta func- tion. Next, calculate the integration of the Beta distribution for rule (rule fitness). It determines whether rule s a high-quality rule or not.
but low sup-
Ps > k) = S So (æ; ce. 2a) dr Pr I (P. (x k)
(8a)
(8b)
The Beta distribution Beta(a,,,3) B.) for rule s has two parameters, which we set as follows:
where k is the lower bound of integration for Beta distribution and 17 is the threshold for rule fitness. high-quality rule when the integra-
a, = C = 1)+ +1
(6a)
We select tion of its Beta distribution satisfies the threshold.
B, - C +1
(6b)
As shown in Algorithm 1, summarize how to extract high quality rules. For each rule s. cal- culates as and B, (lines 5-11). Then computes Ps (a > kt) using equations (7) and (8a) (line 12) and add high connddarr--ofidennee rules to the rule set using equation (8b) (lines 13-17).
where ((s) represents whether rule holds. Tak- ing the example of mining high-confidence rules, if both rbody and Phoad exist for rule s, then ip(s) = 1,
indicating that the rule holds. Conversely, if Thody exists for rule s but Thead does not, then (ps)) (s) = O,

---

3.3 Bidirectional logic constraints
Liquation (10b), which states that if any body atom does not exist, then Thead does not exist.
We utilize the above rules to impose constraints on the DocRE task. However, previous methods only employed forward logic constraints from Tbody to Fan al., 2022). They could not leverage the reverse logic constraints from Thead to Tbody due to the uncertainty of rule body atoms. BCBR models the reverse logic constraints based on head- coverage rules, thereby compensating for the loss of constraint conditions. Below, we provide a de- tailed explanation of the modeling process for bidi- rectional logic constraints:
Thead (eo, e1) - 1ı (eo, e1) A A (E(1,,
(10a)
Thead (co: Qi) <- mm (1), V V m (ct 1;4)
(10b)
We model the ideal probability form of the re- verse constraint Equation (11):
P (Thead (eo, ei)) S bhead min (P (ri e++1) )) if min (P (ri (es, e++1)))
(11)
Forward logic constraints Forward logic con- straints exist in high standard-confidence rules. As shown in Equation (2), when Tbody occurs and rhead simultaneously occurs, it considered to satisfy the forward logic constraint. Conversely, Thead does not occur, it is considered not to satisfy the forward logic constraint. It represents the suffi- ciency of Mbody for Thead. We model the ideal form of forward constraints follows:
where bhead represents the rule fitness of high head- coverage rules. The inverse constraint is only gen- erated high head-coverage rules. When the score of weakest body atom r; in Tbody is less than 0, the reverse constraint of the rule comes into effect. It constrains that the existence probability of Thead must be less than the existence probability of rj.
3.4 Rule consistency loss
In addition to the original relation classification loss Lds of backbone models, construct rule consistency loss based on the bidirectional con- straints of beta rules. This loss jointly trained with Lets to improve the logical consistency and performance of relation extraction.
P (Thead (eo, et)) 2 beons min (P(ri (ri (e;; e4+1) if min (P (rs (ei, ex+1)))
(9) where beonf represents the rule fitness of high standard-confideen confidence rules, I denotes the length of the rule, and is a threshold. Forward constraints are only generated in high standard-confidencc rules. When the score of the weakest body atom Ti in Foody is greater than 8, the forward constraint of the rule comes into play. It constrains that the probability of Thead being present is greater than the probability of r; being present.
The rule consistency loss is derived from the bidi- rectional constraints of beta rules and consists of two parts: forward loss generated by high standard- confidence rules and reverse loss generated by high head-coverage rules. The loss function is formu- lated as shown in equations (12a) and (12b).
Reverse logic constraints Reverse constraints ex- ist in high head-coverage rules. As shown in the probability model in Equation (3), when l'head is present, Roody is also expected to be present. It is referred as satisfying the reverse constraint. Con-
Lec Σ Σ
max (0, (log (Deonf) + log (min 8ES deD
(P (ri e (e -
log (P (Thead eo:
et)))) Prmin (eh, et)
(12a)
Lho = Σ Σ max (0, - (log (bhoad) log (min
versely, if Thody is not present, it is considered as
not satisfying the reverse constraint. It represents the necessity of rbody for Thead- The reverse con- straint is formulated as shown in Equation (10a). Reverse constraint differs from the rule form of for- ward constraint shown in Equation (1), as it derives Thody from Thead- Thody contains multiple uncer- tain body atoms because the entities connecting the triple may not exist. But conjunction rules require to consider each triple in constructing the constraint probability model. So We use De Morgan's 's laws for (10a) and obtain a disjunctive rule as shown in
SCS.c dCD
(P (rv c;)) + log (P (Thoad co:
e1))) Prmin (eh, et)
(12b)
where Sse and She represent the sets of high standard- conndarr-coofieence rules and high head-coverage rules, respectively. beonf and bhead are their rule fitness. Prom (Ch, C1) is an indicator function men- tioned in equation (5), which takes if the weakest triple in the rule body holds true, and otherwise.
We combine the bidirectional constraint losses

---

into a unified loss, which is jointly computed with
is used to assess the adherence of our predictions to the golden rule.
dds. The formulation is follows:
Baselines To verify the generalizability of our method as plugin model for DocRE, we select the following four models as backbone models: BiL- STM(Yao et al., 2019), GAIN(Zeng al., 2020), ATLOPPZZou et al., 2021), and DREEAM(Ma et al., 2023). For faireess, choose bert-base- cased as the pretraining model for GAIN, ATLOP, and DREEAM. Mcanwhile, we also compare our model BCBR with other logic constraint DocRE models - LogiRE(Ru et al., 2021) and MILR(Fan et al., 2022)'.
gllobal = Lets + X (Eso + Lhc)
(13)
where À is a relaxation factor that reflects the weight of the rule consistency loss.
4
Experiments
4.1 Datasets
To demonstrate the ability of our method to gener- alize, we conducted evaluations three datasets for document-level relation extraction. including DWIEoZaporojets al., 2021). DocRED(Yao et al., 2019), and Re-DocRED(Tan et al., 2022). The de- tailed information of datasets are shown in Table 1.
Implementation Details For fairness, we conduct experiments based on the recommended parame- ters in the baselines. We average the results over five different random seeds. The specific hyper- parameter settings for the new parameters intro- duced by BCBR are provided in Appendix A. All models were implemented using Py PyTorch 1.8.1 and trained Quadro RTX 6000 GPU.
DWIE This dataset is a human-annotated collec- tion used for document- tion, which includes DocRE. It contains gold rule labels, which can be used to evaluate the logical consistency of the output of DocRE models.
information extrac-
DocRED It's popular large-scale DocRE dataset, which is sourced from Wikipedia articles. is the most widely used dataset for DocRE, and the ma- jority of methods are experimented on this dataset. Re-DocRED It analyzes the causes and impacts of false negatives in the DocRED dataset and re- annotates 4,053 documents. Compared to the Do- cRED dataset, most document-level relation ex- traction methods show significant improvement in performance on this dataset.
4.3 Results & Discussions
Results on DWIE We can observe results on DWIE in Table 2. Among all baseline models, our BCBR model consistently outperforms Lo- giRE and MILR, indicating its strong generality, making it compatible with the majority of DocRE models. Building upon the state-of-the-an base- line model, DREEAM, BCBR achieves 3.33% Ign F1, 3.34% Fl and 4.02% Logic improvements on test set, reaching state-of-the-art performance. In comparison to LogiRE and MILR, BCBR achieves 1.94% Ign F1, 1.40% F1 and 2.83% Logic improve- ments. It demonstrates that BCBR has achieved significant improvements in both relation extrac- tion performance and rule consistency. Meanwhile, We conducted a comparative experiment between our beta rule miner and a general rule miner, as detailed in the Appendix B.
Dataset
#Doc.
#Rel.
Ant.
Train
602
27.4
DWIE
Dev
98
65
28.4
Test
99
26.5
Train
3053
19.5
DocRED
Dev
1000
96
19.6
Test
1000
19.5
Train
3053
19.4
Re-DocRED
Dev
500
19.4
Test 500
19.6
Results DocRED The experimental results on DocRED presented in Table 2. Apart from DWIE, we only include the performance of strong baselines on other datasets. LogiRE does not ex- hibit significant improvements on the DocRED dataset, primarily due to the presence of a large number of false negative labels. The EM algo- rithm used in LogiRE leads to overfitting issucs. On the other hand, MILR and BCBR perform rela-
Table 1: Statistics of datasets.
4.2 Experimental Setups
Metrics Following the experimental settings of (Ru et al., 2021) and (Fan al., 2022), we evaluate our method using three metrics: F1, Ign F1, and Logic. The Ign Fl score excludes relation triplets that involved by cither train set or dev set, prevent- ing leakage of information from the test set. Logic
Our implementation.

---

Dev
Test
model
Ign Fl
F1
Logic
Ign F1
F1
Logic
BiLSTM
40.46
51.92
64.87
42.03
54.47
64.41
LTMiLoggiRE
12.59(+2.13)
53.83(+1.91)
73.37(+8.50)
43.65(+1.62)
55.14(+0.67)
77.11(+12.70)
BILSTM+MILK
43,03(+2.57)
53.90(+1.98)
74.66(19.79)
43.80(+1..7))
55,48(+1001)
77.69(+13.28)
Bil BLSM++BCER
43.71(+3.25)
54.61(+2.69)
76.01(+11.14)
45.46((57.43)
57.13(+2.66)
79.85(+15444)
GAIN
58.63
62.55
78.30
62.37
67.57
86.19
GAIN+LogiRE
60.12(+1.49)
63.91
87.86(49.56)
64.43(+2.06)
69.40(+188))
91.22(+5.02)
GAIN+MILR
60,44(+1.81)
64.03(+1.44)
83. 83.59(+5.22) 222)
19(2.82))
70.17( (7+2.60))
87.67(+1
GAIN+BCBR
61.37(+2.74)
64.83(22.28)
88.29
66.72(44.35)
71.25(43.68)
91.69(+5.40)
ATLOP
59.03
64.82
81,98
62.09
69.94
82.76
ATLOP+LogiRE
60.24(+1.21)
66. 66.76(+1.94)
86.98(+5.00) 00)
64.11(+2.02)
71.78(+1.84)
86.07(+3.31)
ATLOP+MILR
59.58(+0.55)
65.51(+0.69)
86..2(++4.34)
65.08(+2.99)
86. 86.94(+4.11) 7185(+1991)
ATLOP+BCBR
60,91(+1.88)
66.44(+1.62)
87.13(+5.45)
66,25(+4.16)
73. 73.19()3.25)
90.27(+7.51)
DREEAM
60.84
66.07
82.43
64.82
71.44
84.78
DRERAM+LogiRE
61.53(+0.69)
(60.77)))77)
84.06(+1.63)
65.79(+0.977)
73.02(+1158))
85.277+0049)
DREEAM+MILR
61.39(+0.55)
66.51 (+0.44)
83.49(+1.06)
66.21(+1.39)
73.38(+1.94)
85.97(+1.19)
DREEAM+BCBR
62.23(+1.39)
(8.07()2.00)
84.69(+2.26)
68.15())333)
74.78(+3.34)
86.07(+4.02)
Table 2: Main results DWIE(%).
Test
Test
model
model
Ign Fl
Fi
Ign
F1
GAIN
57.93
60.07
GAIN
69.77
70.59
GogiEE
58.62(+0.69)
60.61(+0.54)
GAIN LogiRE
70.53(+0.76)
71.48(+0.89)
GAIN+MILR
(8.85(+0.92)
61.01(+4)96)
GAIN+MILR
70.82(+1.00
71.78(+1.19)
GAIN+BCBR
59.36(+1.43)
61.37(+1.30)
GAIN+BCBR
71.57(+1.80)
72.34(+1.75)
ATLOP
58.28
60.29
ATLOP
70.86
71.68
ATLOP+LogiRE
58.52(+0.24)
60.41 12)
ATLOP+LogiRE
71.83(+0.97)
72.77(+1.08
ATLOP+MILR
59.07(+0.79)
60.98(+0.69)
ATLOP+MILR
71.86(+1.00)
72.58(+0.90)
ATL.PPBCCR
19.89(+1.61)
61.63(+1.44)
ATI.PPBCER
72.43(+1.57)
73.22(+1.54)
DREEAM
59.08
60.86
DREEAM
71.45
72.16
DREEAM+LogiRE
+9.29)
61.03(+0.17)
DREEAM+LogiRE
72.23(+0.78)
72.92(+0.76)
DREEAM+MILR
60. 1.005)
61.78(+0.92)
DREEAM-MILR
72.28(+0.83)
73. (3.03(+0.87)
DREEAM-BCBR
60.77(+1.59)
(1.59))))53)
DREEAM-BCBR
72.74(+1.29)
73.50(+1.34)
Table Main results on DocRED(%).
Table Main results on Re-DocRED(%))
4.4 Ablation study
tively better as they jointly train with DocRE mod- els. BCBR achieves the best results on this dataset, with 1.53% Ign Fl and 1.59% F1 improvements. It demonstrates that BCBR performs better than LogiRE and MILR DocRED.
To demonstrate the effectiveness of each compo- nent of the BCBR framework, conduct abla- tion experiments, and the experimental results are shown Table We use the DWIE dataset and perform the experiments on the strongest baseline model DREEAM. the table, BR and BC refer to the Beta Rule and Bidirectional Constraint, re- spectively. We exclude the beta rules using the original rule miner and exclude the bidirectional constraint using the rule forward constraint. From the table, can observe that when exclude one of the components, our method still outperforms the baseline approach. This indicates that both com- ponents are effective. The quality of rules and the comprehensiveness of logic constraints are both
Results on Re-DocRED The results on Re- DocRED can be seen from Table 4. Due to the resolution of false-negative labels in DocRED, most relation extraction models exhibit significant improvements this dataset. BCBR achieves 1.34% Ign F1 and 1.29% F1 improvement, which is slightly higher than the improvement DocRED, By this, we can conclude that the BCBR can assist the backbone model more effectively when a major- ity of the Salse-negative label issues are resolved.

---

All long, Buyern Munich have been consumed with achirving reaching the final Champions Leugue..[[2] [2] After finishing a distant second place Dorimund the Bundesliga and dismally losing them 5-2 in the German Cup final...
Predictions of DREEAM:
Document
95
Bayern
Dortmund
Rule
Sx: vs(O f2) - vx(e,,
Predictions of BCBR:
Prompt: ...If there is relationship [se) from Bayern to Dortmunnd piease choose relationship Dortound (won vsl... None. Output: (vs)
following relationships: (us),
ChatGP
Rayorn
Dortmund
Modi's fourth Indlan lender's schedtle in the US capital includes holding talks with President Obama...[2] ..
taking office as prime minister,
Prodictions of
Documents
investment India, particularly the onergy
Modi
Indion
India
sector.
pept
Rule
Saci head gov head, (ev. et) gepū(er.es)
Predictions of BCBR:
Prompt: if there is relationship (head of gov-x) from Mlodi to Indians, and relationhip (gep0] from Indiam India, please chouse the rehitionshp from Modi to India from the following relationships: (bead of guv-x), (head of gov). [head of state), (head of statexx)..... None. Outpat: (head of gov-xx
gow
ChatGPT
India
bead goes
pep0
[1] Berlin court rules Google Street View legal in Germany. h L2]s Tuesday, the Berlin State Supreme Court (Kammergericca) announced iis... [3] allow Germans to opt-out of the service to have their house obfiascared as well.
Predictions of DRREAM:
Decuments
Baned beb
Berlin court apencc of
sppo
Rule
Sac magency of (x-xxee, ex) -based inD(e,, -gepl((e,..))
Germany
Predictions of BCBR:
Prompt: ...if there relationship (gep0) from Germany Germans, and not a Germany please choose the relationship from Berlin court to Germans from the following relationships: (gep0), [agency_offae _ffx)..... None.
[based] in0) from Berlin
boand Berlin court
CharGPT
Outpat: (fgxncy_ofxx)
Germany
2000
Germans
Figure 3: Several BCBR inference cases on DWIE and the predictions of the large language model-ChatGPT them. Different colors represent different entities. Green solid lines represent correct predictions, red solid lines represent incorrect predictions, and gray dotted lines represent non-existent relations that correspond to rules.
crucial factors.
also choose the correct answer, which indicates that general DocRE models cannot even perform simple logical inference, but there is no signifi- cant gap between our framework and ChatGPT in simple logical inference. In the second example, BREEAM also cannot predict the head_ot fidddof__pe of_e re- lation from Modi to India, while ChatGPT makes an incorrect prediction. This fully demonstrates the superiority of our method in complex logical infer- ence. In the third example, the rule used is different from the front two. It is rule that satisfies the verse constraint. We can infer the non-existence of the rule head agency_of by the non-existence of the rule body atom fresd_inn0 Both BREEAM and ChatGPT cannot satisfy reverse constraint and make an incorrect prediction, which reflects the unique advantage of our method in reverse con- straints. However, BCBR is not in conflict with ChatGPT, as logic constraints can enhance the rea- soning ability of large models. Thus, ChatGPT can work together with logic constraints improve the performance of DocRE. The integration will be a interesting direction in future research.
Dev
Test
model
Ign F1
F1
Ign
F1
DREEAM-BCBR
62,23
68.07
68.15
74.78
DREEAM+BC
61.94
67.86
67.74
74.42
DREFAM-BR
60.94
67.19
66.57
73.55
DREEAM
60.83
66.07
64.82
71.44
Table 5: Ablation study on the DWIE dataset (6).
4.5
Case study & LLM
We list some rules mined by our Beta Rule Miner and their beta scores in Table 6. From the table, can learn about various rule patterns that we can mine. Then we present several inference cases of BCBR framework on the DWIE dataset, as shown in Figure 3. We compare the results of BCBR with the strongest baseline- - BREEAM and highlight the advantages of using logic constraints. We also com- pare ours to the outputs of large language models to demonstrate the significance of our task during the era of large language models. In the first case, BREEAM can only predict the relation us from Bayern Dortmund, while our BCBR can pre- dict the relation us from Dortmund to Bayern due to the assistance of rules. However, ChatGPT
Conclusion
In this paper, we propose a novel logic constraint framework BCBR, which utilises bidirectional

---

Rule Patterns
The Mined Beta Rules With Their Bcta Scores
Thona(Co, 61) 6- ri (1p,e1)
agent of (ea,ei) ei) <- minister of (cp,ei)
0.99
Peeaa 02) t ri (Cu, 61) A 12 (2)
in0-x (eu, ex) + in0 (eu, A (es, 62)
0.96
Thead Tho (1)+ &- mm (ep,ci)
-head_of of e1) [ <- -emberr_ee off (ea, e1)
0.99
Theod Co, 02) f- mi (eo, ei) m2 (2),
ugend_of (eo, e2) +- aagency_of (eo, e1) V "gee0 (es,
0.99
Table 6: Case study of rules mined by beta rule miner.
logic constraints of beta nules to regulate the output of DocRE. We are the first to propose the use of beta distribution for modeling rules, which effec- tively solves the problem of pseudo-rules. Then we model the reverse logic constraints and utilize bidi- rectional constraints of beta rules to construct rule consistency loss. By jointly training with relation classification loss, we improve the performance of DocRE. Experimental results on multiple datasets demonstrate that BCBR outperforms baseline mod- els and other logic constraint frameworks.
the 2022 Conference on Empirical Methods Natu- ral Language Processing. EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 10311-10325 Association for Computational Lin- guistics.
Luis Galárraga, Christina Teffioudi, Katja Hose, and Fabian M. Suchanek. 2015. Fast rule mining in tological knowledge bases with AMIE+. VLDB J., 24(6):707-730.
Xu Han, Pengfei Yu, Zhiyuan Liu, Maosong Sun, and Peng Li. 2018. Hierarchical relation extraction with coarse-to- grained attention. In Proceedings of the 2018 Conference on Empirical Methods in Natu- rul Language Processing, Brussels, Belgium, Octo- ber 31 November 2018, pages 2236-2245. Asso- ciation for Computational Linguistics.
Limitations
Our BCBR brings additional rule consistency loss, resulting in significant increase in training time. We need traverse all rules when processing each document to generate rule consistency loss. It leads to a significant increase in time cost. We will opti- mize the code structure in future work to achieve convergence of the model in a relatively short pe- riod of time.
Youmi Ma, An Wang, and Naoaki Okazaki. 2023. DREEAM: guiding attention with evidence for im- proving document-level relation extraction. In Pro- ceedings of the 17th Conference of the European Chapter of the Association for Computational Lin- guistics, FACI. 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 1963-1975, Association for Computa- tional Linguistics.
Acknowledgements
Christian Meilicke, Melisachew Wudage Chekol, Daniel Ruffinelli, and Heiner Stuckenschmict. 2019. Any- time bottom-up rule learning for knowledge graph completion. In Proceedings of the Twenty-Fighth International Joint Conference Artificial Intelli- gence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 3137-3143. ijcai.org.
This work was supported by National Natural Sci- ence Foundation of China (NSFC) (61972455) and the Project of Science and Technology Research and Development Plan of China Railway Corpora- tion (N2023J044).
Dat Quoe Nguyen and Karin Verspoor. 2018. Convo- lutional neural networks for chemical-disease rela- tion extraction are improved with character-based word embeddings. In Proceedings of the BioNLP 2018 workshop, Melbourne, Australia, July 19, 2018, pages 129-136. Association for Computational Lin- guistics.
References
Fenia Christopoulou, Makoto Miwa, and Sophia Ana- niadou. 2019. Connecting the dots: Document- level neural relation extraction with edge-oriented graphs. In Proceedings the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural I Language Processing, EMNI.P-ICNNIP 2019, Hong Kong, China, November 3-7, 2019, pages 4924-4935. Association for Computational Linguistics.
Pouya Ghiasnezhad Omran, Kewen Wang, and Zhe Wang. 2021. An embedding-basee approach to rule learning in knowledge graphs. IEEE Trans. Knowl. Data Eng., 33(4): 1388):134881335)
William W. Cohen. 2016. Tensorlog: A differentiable deductive database. CoRR, abs/1605.06523.
Dongyu Ru, Changzhi Sun, Jiangtao Feng, Lin Qiu, Hao Zhou, Weinan Zhang, Yong Yu, and Lei Li, 2021. Learning logic rules for document-level relation ex- traction. I Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing,
Shengda Fan, Shasha Mo, and Jianwei Niu. 2022. Boosting document-level relation extraction by min- ing and injecting logical rules. In Proceedings of

---

EMNLP 2021, Virtual Event Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 1239- 1250. Association for Computational Linguisties.
document-leve relation extraction dataset. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, ACI. 2019, Florence, htly, July 28- August 2019, Volume I: Long Pa- pers, pages 764-777. Association for Computational Linguisties.
Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. 2019. DRUM: end-to- end differentiable rule mining on knowledge graphs. In Advances Neural Information Processing Sys- tems 32: Anmual Conference on Neural Information Processing Systems 2019, NewrlPS 2019, December 3-14. 2019, Vancouver. BC, Canada, pages 15321 15331.
Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020. Corefer- ential reasoning learning for language representation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020, pages 7170- 7186. Association for Computational Linguistics.
Qingyu Tan, Lu Xu, Lidong Bing, and Hwee Tou Ng. 2022. Revisiting docred- addressing the overlooked false negative problem in relation extraction. CoRR, abs/2205. 12696.
Klim Zaporojets. Johannes Deleu, Chris Develder, and Thomas Demeester. 2021. DWIE: an entity- centric dataset for multi-task document-level information extraction. Inf. Process. Manag. 58(4):002563
Difeng Wang, Wei Hu, Ermei Cao, and Weijian Sun. 2020. Global-to-loce neural networks for document- level relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2020, Online, November 16-20, 2020, pages 371 3712-. Association for Com- putational Linguistics.
Daojjan Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via con- volutional deep neural network. In COLING 2014, 25th International Conference on Computational Lin- guisties, Proceedings of the Conference: Technical Papers, August 23-29, 2014, Dublin, Ireland, pages 2335-2344, ACL.
Hong Wang, Christfried Focke, Rob Sylvester, Nilesh Mishra, and William Yang Wang. 2019. Fine- tune bert for docred with two-step process. CoRR, abs/1909 1999. 1189..
Shuang Zeng. Runxin Xu, Baobao Chang, and Lei Li. 2020. Double graph based reasoning for document- level relation extraction.
Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2020, Online, November 16-20, 2020, pages 1630-1640. Association for Com- putational Linguistics.
Yijun Wang, Changzhi Sun, Yuanbin Wu, Hao Zhou, Lei Li, and Junchi Yan. 2021. ENPAR: enhancing entity and entity pair representations for joint entity relation extraction. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online, April 19- 23, 2021, pages 2877-2887. Association for Computational Linguistics.
Wen Zhang, Bibek Paudel, Liang Wang, Jiaoyan Chen, Hai Zhu, Wei Zhang, Abraham Bersstein, and Hua- jun Chen, 2019. Iteratively learning embeddings and rules for knowledge graph reasoning. In The World Wide Web Conference, WWW 2019. San Francisco, CA. USA, May 13-17, 2019, pages 2366-2377. ACM.
Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021a. Entity structure within and throughout: Modeling mention dependencies for document-level relation extraction. CoRR, abs/2102.10249.
Yuhao Zhang, Victor Zhong. Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware awwae attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen. Denmark, September 9-11, 2017, pages 45-45. Association for Computational Linguis- tics.
Benfeng Xu, Quan Wang, Yajuan Lyu, Yong Zhu, and Zhendong Mao. 2021b. Entity structure within and throughout: Modeling mention dependencies for document-less level relation extraction. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Ap- plications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Ar- tificial Intelligenc, EAAT 2021, Virtual Event, Febru- ary 2-9, 2021, pages 14149-1441 141199.11...1111 AAAI Press.
Wei Zheng, Hongfei Lin, Zhiheng Li, Xiaoxia Liu, Zhengguang Li, Bo Xu, Yijia Zhang, Zhihao Yang. and Jian Wang. 2018. An effective neural model extracting document level chemical-indhen disease relations from biomedical literature. J. Biomed. In- formatics, 83:1-9,
Zezhong Xu, Peng Ye, Hui Chen, Meng Zhao, Huajun Chen, and Wen Zhang. 2022. Ruleformer: Context- aware differentiable rule mining over knowledge graph. CoRR, 05.82529905815
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. 2021. Document-lewe relation extraction with adaptive thresholding and localized context pool- ing. In Thirty-Fiftt AAAI Conference Intelligence, AAAI 2021, Thirty-Thing Thind Conference on Innovative Applicutions of Artificial Intelligence,
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. Docred: A large-scale
Artificial

---

IAAI 2021, The Eleventh Symposium Educational Advances in Artificial Intelligence, EAAI 2021, Vir- tual Event, February 2-9, 2021, pages 14612-14620. AAAI Press.
mostly scattered in the 101 to 500 support interval, and there are no low-support rules scattered in the to support interval. contrast, the general rule miner has a large number of rules scattered in the I to 10 support interval. It reflects that the high quality of rules mined by the Beta rulc miner is much higher than that of the general rule miner. In addition, we can observe the proportion between (eta(SC) and Beta(HC), which indicates that the rules satisfying the reverse constraint cannot be ig- nored. If only high-standard- sttandard--cofidence rules are used to constrain the relation extraction process. a large amount of consistency information among rules will be lost, leading to decline in the perfor- mance of relation extraction.
A Hyper-Paramettr Settings
We detail the hyperparameter settings BCBR on different datasets in Table 7.
Hyper-param
DWIE
DocRED
Re-DocRED
maxL
2
epoch
70
100
100
aac
0.9
0.8
0.9
khc
0.9
0.8
0.9
n
0.9
0.9
0.95
X
le-3
1e-4
lc-4
Table Hyper-parameter settings on different datasets.
160
140
General
Beta(HC)
120
Beta(SC)
60
40
20
0
0<x010
10<x5100
100->5500
x>500
Figure 4: Comparison of the amount of rules mined by different rule miners across different intervals. The gray cluster represents the rules generated by a general rule miner, while the red and blue clusters represent the www.ssttaadddnntteetee
rules and high-head-coverape rules generated by the Beta rule miner, respectively.
B Rule miner comparison
We analyzed the distribution of rules mined by the Beta rule miner and the general rule miner at different support intervals DWIE dataset. Beta(SC) and Beta(HC) represent the high- standard-confidence rules and high-head-coveragge rules extracted by the Beta rule extractor, respec- tively. The results are shown in Figure 4. We can observe that the rules mined by Beta rule miner are


Reviews:
[{"review_id": "wwCrJaEnCd", "reviewer": "Reviewer_akdh", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1:\nA thorough examination of the extracted rule could be incorporated. While a comparison of the rule miners is presented in the appendix, the addition of a comprehensive case study or more detailed statistical analysis could further enrich the content.\n\nA1:\nThank you for reviewing and providing your suggestions. The focus of our research lies in rule-assisted document-level relation extraction and we conducted a detailed case study on this task in Chapter 4.5 of our paper. For the case study of rules alone, we will provide a detailed analysis of rule form cases and examples as follow:\n\nrhead(e0, e1) ← r1(e0, e1)                      |                agent_of (e0, e1) ← minister_of (e0, e1)\n\nrhead(e0, e2) ← r1(e0, e1) ∧ r2(e1, e2)           |       in0-x (e0, e2) ← in0 (e0, e1) ∧ gpe0 (e1, e2)\n\n¬rhead(e0, e1) ← ¬r1(e0, e1)                    |             ¬head_of (e0, e1) ← ¬member_of (e0, e1)\n\n¬rhead(e0, e2) ← ¬r1(e0, e1) ∨ ¬r2(e1, e2)      |     ¬agent_of -x (e0, e2) ← ¬agency_of (e0, e1) ∨ ¬gpe0 (e1, e2) \n\nThe first two rule forms are high-confidence rules that can be used for positive constraints, while the last two rule forms are high-head coverage rules that can be used for reverse constraints. In the final submission, we will supplement it in the appendix in the form of a table.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a novel logic constraint framework, BCBR, which addresses the limitations of existing approaches in DocRE. Incorporating forward and reverse logic constraints based on beta rules is a unique and promising contribution.\n2. The paper supports its claims with experimental results, demonstrating that BCBR outperforms existing DocRE models in performance and logical consistency. ", "reasons_to_reject": "1. A thorough examination of the extracted rule could be incorporated. While a comparison of the relu miners is presented in the appendix, the addition of a comprehensive case study or more detailed statistical analysis could further enrich the content.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a novel framework called \"Bidirectional Constraints of Beta Rules\" (BCBR) to address the challenges in Document-level Relationship Extraction (DocRE). Existing approaches using logic constraints suffer from opacity and weak logic. The BCBR framework introduces a new rule miner based on beta contribution to model rules. It then constructs forward and reverse logic constraints using these beta rules. By using bidirectional constraints, BCBR regulates the output of the DocRE model and improves relation extraction performance and logical consistency. Experimental results show that BCBR outperforms existing DocRE models by achieving higher F1 scores and logic scores. ", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "KxxqFtQURm", "reviewer": "Reviewer_rFoA", "summary": "", "questions": "- Just a minor suggestion, if we provide the rules mined by BR to ChatGPT, will it make better relation extraction predictions?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1:\nAlthough the improvements over ATLOP/GAIN/DREEAM are significant. The improvement over LogiRE and MILR seems marginal on DocRED and Re-DocRED, with <0.5 Test F1. Considering this paper also adopts techniques from these two papers such as forward constraint loss, the marginal improvement weakens the contribution of this paper.\n\nA1:\nThank you for your comments on our paper.\n \n(1)\tThe improvement of BCBR on the DocRED dataset and Re-DocRED dataset is indeed relatively small. Our method mitigates the issue of pseudo rules in both datasets and improves performance by complementing reverse constraints. The reasons for our minor improvements on these two datasets are as follows:\n(a) \tOur method improves by 1.54/1.57(Test Ign F1/F1) based on ATLOP and by 1.34/1.29(Test Ign F1/F1) based on DREEAM. Compared to the existing two logical methods, there is also a certain performance improvement. It proves the effectiveness of our approach.\n(b)\tDWIE is a dataset containing golden rules, while DocRED and Re-DocRED comes from remote supervision, the quality of the two datasets in terms of logic is not as good as the DWIE dataset. \n\n(2)\tCompared to LogiRE and MILR, our contributions mainly lie in the following two aspects:\n(a)\tAs far as we know, we first propose a rule miner that utilizes the Beta distribution to model rules. The regular rule miners in LogiRE and MILR may make mistakes when facing pseudo rules, but ours will not.\n(b)\tWe introduce reverse logic constraint to ensure that the output of DocRE models satisfies the necessity of rules. The forward logic constraint in LogiRE and MILR only solves half of the rule constraint problem.\n\nQ2:\nAccording to the ablation studies, removing BR doesn't seem to affect the performance by a lot. DREEAM (-BR) only reduces performance by 0.29/0.21 Dev Ign F1/F1. This is quite unexpected, because when the quality of rules decreases, the reverse constraint loss may also be less effective.\n\nA2:\nThank you for your comments. We think the occurrence of the aforementioned situation can be attributed to the following reasons.\n\n(1)\tThe BR module alleviates the issue of pseudo rules and is an essential part of our framework. The DWIE dataset contains golden rules with good logical consistency, resulting in relatively fewer occurrences of pseudo rules(It still exists). As a result, the BR module has fewer problems to address in this scenario, and the contribution of the BC module becomes more significant. In DocRED dataset and Re-DocRED dataset, the contribution of the BR component would be equal to or even surpass that of BC.\n\n(2)\tWhen using the BR module alone, as shown in Table 5 \"-BC\", there is an obvious significant improvement on the test dataset (Ign F1 increased from 64.82 to 66.57, F1 increased from 71.44 to 73.55). Similarly, when using the BC module alone, as shown in Table 5 \"-BR\", there is an improvement compared to the baseline on the test dataset (Ign F1 increased from 64.82 to 67.74, F1 increased from 71.44 to 74.42). It confirms the effectiveness of both modules from another perspective. The combined use of the BR and BC modules does not achieve the additive effect, which may be due to the consistent role of them in the rule learning process.  \nThank you for raising this question again, as it may contribute to further in-depth research on the BR module and BC module.\n\nQ3: Just a minor suggestion, if we provide the rules mined by BR to ChatGPT, will it make better relation extraction predictions?\n\nA3: We have previously conducted similar experiments, and ChatGPT cannot directly comprehend our mined rules. We need to describe them in natural language for ChatGPT to understand. However, this approach essentially treats the rules as background knowledge, lacking strong constraint effects. While it impacts the results, its influence is not as significant as using rules directly. Exploring methods to effectively constrain the output of large models using rules represents an intriguing direction and is a topic of interest for our future research work.\n\nReply for Typos Grammar Style And Presentation Improvements:Thank you for the feedback and suggestions. We have made the corrections to the errors you mentioned and conducted another round of proofreading for the paper's format and grammar.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The two techniques of introducing beta distribution in rule mining (BR) and training DocRE with reverse logic constraints (BC) are novel. The BC technique is quite effective.\n- The paper conducts extensive experiments on both the DocRE performance and the logic consistency.", "reasons_to_reject": "- Although the improvements over ATLOP/GAIN/DREEAM are significant. The improvement over LogiRE and MILR seems marginal on DocRED and re-DocRED, with <0.5 Test F1. Considering this paper also adopts techniques from these two papers such as forward constraint loss, the marginal improvement weakens the contribution of this paper. \n\n- According to the ablation studies, removing BR doesn't seem to affect the performance by a lot. DREEAM (-BR) only reduces performance by 0.29/0.21 Dev Ign F1/F1. This is quite unexpected, because when the quality of rules decreases, the reverse constraint loss may also be less effective.", "questions_to_authors": "- Just a minor suggestion, if we provide the rules mined by BR to ChatGPT, will it make better relation extraction predictions?", "paper_topic_and_main_contributions": "This paper presents a new model called BCBR for DocRE. Specifically, it:\n(1) mines rules utilizing beta distribution\n- previous models only use confidence to mine rules\n(2) combine forward and reverse logic constraints to add two additional losses to the training of the DocRE model\n- previous models only consider forward rules (due to the low quality of rules)\n\nThis paper conducts experiments on DWIE, DocRED, and re-DocRED. In all three datasets, the BCBR techniques significantly improve two existing models: ATLOP and DREEAM. It also outperforms two existing techniques: LogiRE and MILR that leverage logic rules to improve DocRE.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The caption of Table 3 and Table 4 should be exchanged.\n- In Table 3 and Table 4, the position of \"Ign F1\" and \"F1\" should also be exchanged (F1 should be higher than Ign F1)"}, "extra_text": {}}, {"review_id": "1bxns9Sg6G", "reviewer": "Reviewer_4D7h", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1:\nA more detailed explanation of the bidirectional logic constraints is needed. This can greatly enhance the clarity and understanding of your approach.\n\nA1: \nThank you for your suggestion. We will add more detailed explanations for bidirectional logical rules in Section 3.3. The explanatory content is as follows:\nBidirectional constraints encompass two aspects. Positive constraints entail the rule's body imposing constraints on the rule's head, indicating that if the rule's body exists, the rule's head must also exist. However, if the rule's body does not exist, there is no constraint on the rule's head. Conversely, reverse constraints involve the rule's head imposing constraints on the rule's body, signifying that if the rule's body does not exist, the rule's head must not exist.\n \nQ2:\nThe inclusion of additional baselines, particularly on datasets like Re-DocRED and DocRED, is advisable. A more extensive baseline comparison would contribute to a more comprehensive evaluation of your proposed approach.\n\nA2:\nThank you for your review and suggestions. We will add a new baseline, GAIN, for the DocRED dataset and Re-DocRED dataset to make the experiments more comprehensive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper is well-written and easy to follow.\n2. The proposed BCBR outperforms both LogiRE and MILR, excelling in both relation extraction performance and logical consistency.", "reasons_to_reject": "1. A more detailed explanation of the bidirectional logic constraints is needed. This can greatly enhance the clarity and understanding of your approach.\n2.  The inclusion of additional baselines, particularly on datasets like Re-DocRED and DocRED, is advisable. A more extensive baseline comparison would contribute to a more comprehensive evaluation of your proposed approach.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Certain previous studies only focused on the forward logic constraints in document-level relation extraction. In this paper, the authors present an innovative approach termed \"Bidirectional Constraints of Beta Rules\" (BCBR), which incorporates both forward and reverse logic constraints to assist relation extraction. BCBR reconstructs rule consistency loss by bidirectional constraints to regulate the output of the DocRE model. Extensive experiment demonstrate the superior performance of BCBR compared to the original DocRED models.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree that this is a sound and exciting paper with good results and novelty. Ablations studies should be added to improve the paper.
Decision: Accept-Main