Distantly-Supervised Joint Extraction with Noise-Robust Learning (2024)

Abstract: Joint entity and relation extraction is a process that identifies entity pairs and their relations using a single model. We focus on the problem of joint extraction in distantly-labeled data, whose labels are generated by aligning entity mentions with the corresponding entity and relation tags using a knowledge base (KB). One key challenge is the presence of noisy labels arising from both incorrect entity and relation annotations, which significantly impairs the quality of supervised learning. Existing approaches, either considering only one source of noise or making decisions using external knowledge, cannot well-utilize significant information in the training data. We propose DENRL, a generalizable framework that 1) incorporates a lightweight transformer backbone into a sequence labeling scheme for joint tagging, and 2) employs a noise-robust framework that regularizes the tagging model with significant relation patterns and entity-relation dependencies, then iteratively self-adapts to instances with less noise from both sources. Surprisingly, experiments1 on two benchmark datasets show that DENRL, using merely its own parametric distribution and simple data-driven heuristics, outperforms large language model-based baselines by a large margin with better interpretability.

heading: 1 Introduction
text: Joint extraction aims to detect entities along with their relations using a single model (see Figure 1), which is a critical step in automatic knowledge base construction (Yu et al., 2020). In order to cheaply acquire a large amount of labeled joint training data, distant supervision (DS) (Mintz et al., 2009) was proposed to automatically generate training data by aligning knowledge base (KB) with an unlabeled corpus. It assumes that if an entity pair has
1Our code is available at https://github.com/yul091/ DENRL.
NEC Group Internal Use Only© NEC Laboratories America 20217
[Bill Gates] and [Allen] founded the [Microsoft] in [Albuquerque], [New Mexico].
1Founder_of
4Located_in
a relationship in a KB, all sentences that contain this pair express the corresponding relation.
Nevertheless, DS brings plenty of noisy labels which significantly degrade the performance of the joint extraction models. For example, given a sentence “Bill Gates lived in Albuquerque” and the sentence in Figure 1, DS may assign the relation type between “Bill Gates” and “Albuquerque” as Place_lived for both sentences. The words “lived in” in the first sentence is the pattern that explains the relation type, thus it is correctly labeled. While the second sentence is noisy due to the lack of corresponding relation pattern. Moreover, due to the ambiguity and limited coverage over entities in open-domain KBs, DS also generates noisy and incomplete entity labels. In some cases, DS may lead to over 30% noisy instances (Mintz et al., 2009), making it impossible to learn useful features.
Previous studies for handling such noisy labels consider either weakly-labeled entities, i.e., distantly-supervised named entity recognition (NER) (Shaalan, 2014), or noisy relation labels, i.e., distantly-supervised relation extraction (RE) (Rink and Harabagiu, 2010), where they focus on designing novel hand-crafted relation features (Yu et al., 2020), neural architectures (Chen et al., 2020), and tagging scheme (Dai et al., 2019) to improve relation extraction performance. Additionally, In-Context Learning (ICL)
10202
using external knowledge of Large Language Models (LLMs) (Pang et al., 2023) is popular. However, they are resource-demanding, sensitive to prompt design, and may struggle with complex tasks.
To cheaply mitigate both noise sources, we propose DENRL—Distantly-supervised joint Extraction with Noise-Robust Learning. DENRL assumes that 1) reliable relation labels, whose relation patterns significantly indicate the relationship between entity pairs, should be explained by a model, and 2) reliable relation labels also implicitly indicate reliable entity tags of the corresponding entity pairs. Specifically, DENRL applies Bag-of-word Regularization (BR) to guide a model to attend to significant relation patterns that explain correct relation labels, and Ontologybased Logic Fusion (OLF) that teaches underlying entity-relation dependencies with Probabilistic Soft Logic (PSL) (Bach et al., 2017). These two information sources are integrated to form a noiserobust loss, which regularizes a tagging model to learn from instances with correct entity and relation labels. Next, if a learned model clearly locates the relation patterns and understands entityrelation logic of candidate instances, they are selected for subsequent adaptive learning. We further sample negative instances that contain corresponding head or tail entities of recognized patterns in those candidates to reduce entity noise. We iteratively learn an interpretable model and select highquality instances. These two-fold steps are mutually reinforced—a more interpretable model helps select a higher quality subset, and vice versa.
Given the superiority of unified joint extraction methods, we introduce a sequence labeling (Zheng et al., 2017) method to tag entities and their relations simultaneously as token classification. We incorporate a BERT (Devlin et al., 2019) backbone that learns rich feature representations into the tagging scheme to benefit the information propagation between relations and entities. The transformer attention mechanism builds a direct connection between words and contributes to extracting long-range relations (Li et al., 2022, 2023a). Its multi-head attention weights indicate interactions between each pair of words, which is further leveraged by self-matching to produce position-aware representations. These representations are finally used to decode different tagging results and extract all entities together with their relations.
Our contributions are summarized as follows:
NEC Group Internal Use Only© NEC Laboratories America 20219
Tagging Examples
Bill Gates Allen founded the Microsoft in Albuquerque , New Mexico
and
• Our work introduces a novel framework for distantly-supervised joint extraction. This innovation lies in identifying and addressing multi-source noise arising from both entity and relation annotations, and a unified joint tagging scheme adaptable to various backbones.
• Our method DENRL is generalizable and effective and offers a cost-effective alternative to predominant LLMs that use a much larger backbone.
• Our comprehensive experiments show that DENRL is interpretible and well-motivated.

heading: 2 Joint Extraction Architecture
text: We incorporate a pre-trained BERT backbone into our sequence tagging scheme to jointly extract entities and their relations (see Figure 3).

heading: 2.1 Tagging Scheme
text: To extract both entities (mention and type) and relations, we tag quadruplets {e1, tag1, e2, re} for each start position p and define “BIO” signs to encode positions (see Figure 2). Here, e1 is the detected entity at p (head entity), tag1 is the entity type of e1, e2 is another detected entity that has a relationship with e1 (tail entity), and re is the predicted relation type between e1 and e2. For a T -token sentence, we annotate T different tag sequences according to different start positions.
For each tag sequence, if p is the start of an entity (this sequence is an instance), the entity type is labeled at p, and other entities that have relationship to the entity at p are labeled with relation types. The rest of tokens are labeled “O” (Outside), meaning they do not correspond to the head entity. In this way, each tag sequence will produce a relation quadruplet. For example, if p is 7, the head entity is
NEC Group Internal Use Only© NEC Laboratories America 2021 12
Joint Extraction with Distant Supervision [Final Version]
“Microsoft” and its tag is ORG. Other entities, such as “Albuquerque” and “New Mexico”, are labeled as L_I and L_I indicating their (unidirectional) relations with “Microsoft”. If p is 9, the head entity “Albuquerque” has no relationship with other entities, thus only the entity type LOC is labeled. If p is 13, all tokens are labeled as “O” because there is no entity at the head position to attend to.
We define instances that contain at least one relation as positive instances (e.g., p is 7), and those without relations as negative instances (e.g., p is 9). “BIO” (Begin, Inside, Outside) signs are used to indicate the position information of tokens in each entity for both entity and relation type annotation to extract multi-word entities. Note that we do not need the tail entity type, because every entity will be queried and we can obtain all entity types as well as their relations from the T tag sequences.

heading: 2.2 Tagging Model
text: Encoder with Self-Matching We follow BERT (Devlin et al., 2019) to use a multilayer transformer (Vaswani et al., 2017) that takes an input sequence S = {w1, ...wT } and converts it into token-level representations h0 = {ht}Tt=1, where ht ∈ Rd is a d-dimensional vector corresponding to the t-th token in S. The model applies L transformer layers over the hidden vectors to produce contextual representations: hl = TRANSFORMER(l)(hl−1), l ∈ [1, L]. Each layer contains a Multi-Head SelfAttention (MHSA) layer followed by a Feed-
Forward Network (FFN) over the previous hidden state hl−1. The final representations hL ∈ RT×d integrate the contextual information of all previous tokens but are inadequate for decoding a T -tag sequence, since for each position p we still need to encode e1 and its overlapping relations re with other entities e2.
We define self-matching (Tan et al., 2018) that calculates position-attention at between tokens at start position p as well as each target position t:
at = softmax( { atj }T j=1 )
s.t. atj = w ⊤(hLp + h L t + h L j ),
(1)
where w ∈ Rd is a parameter to be learned, hp, ht, hj ∈ Rd are hidden states at position p, t, j, respectively. atj is the score computed by comparing hp and ht with each hidden state hj . at ∈ RT is the softmax attention produced by normalizing atj . The start hidden state hp serves as comparing with the sentence representations to encode position information, and ht matches the sentence representations against itself to collect context information. The position-aware representation mt ∈ RT×d is an attention-weighted sentence vector:
mt = a ⊤ t h L. (2)
We concatenate ht and mt to generate positionaware and context-aware representations {xt}Tt=1:
xt = [ht;mt]. (3)
For each start position, self-matching produces different sentence representations and thus can model different tag sequences of a sentence.
CRF Decoder CRF (Lafferty et al., 2001) considers the correlations between labels in neighborhoods and jointly decodes the best chain of labels, which benefits sequence labeling models. For each position-aware representation xt, the input sequence scores Z = {zt}Tt=1 is generated by:
zt = W xxt, (4)
where zt ∈ RV is tag score of the t-th token, V is the number of distinct tags, and zjt is the score of the j-th tag at position t.
For a sequence of labels y = {y1, ..., yT }, the decoding score score(Z,y) is the sum of transition score from tag yt to tag yt+1, plus the input score zytt for each token position t. The conditional probability p(y|Z) is the softmax of score(Z,y) over all possible label sequences y′ for Z. We maximize the log-likelihood of correct tag sequences during training:
Lc = ∑
i log p(y|Z). (5)
Decoding searches for the tag sequence y∗ that maximizes the decoding score. The best tag sequence y∗ is computed using the Viterbi algorithm.

heading: 3 Noise-Robust Learning
text: To reduce the impact of noisy labels on tagging performance, we introduce Bow Regularization (BR) to attend to confident relation patterns for reducing relation noise and Ontology-based Logic Fusion (OLF) to increase entity-relation coherence for reducing entity noise. Finally, we employ SelfAdaptive Learning (SAL) to iteratively train on instances that can be explained by the model.

heading: 3.1 Bag-of-word Regularization (BR)
text: Originally proposed as a pattern-attentive loss, attention regularization (Jia et al., 2019) has been shown effective for reducing relation noise, yet it only considers attention over a single relation pattern and neglects models’ position-awareness, thus cannot identify overlapping relations. We formulate target attention distribution by introducing BoW frequency as an oracle to learn informative relations. For an input sentence S, an entity pair (e1, e2) in S, a relation label re, and a relation pattern p that explains the relation re of
e1 and e2, we define BoW frequency (i.e., pattern significance) as the corresponding guidance score ap conditional on pattern p. Take the relation Contains as an example, its BoW is a set of tokens that appear in a corresponding pattern set {“capital of”, “section in”, “areas of”, ...}. The motivation is to guide the model to explore new high-quality patterns p, e.g., “section of ”, “areas in”, etc. The guidance aI for an instance I is the average of ap regarding all patterns m corresponding to each relation type re:
ap = softmax({BoWt}Tt=1), aI = AvgPooling (ap1 , · · · ,ap|RI|) ,
(6)
where BoWt represents the BoW frequency of wt under relation re if wt belongs to corresponding relation pattern words or 1 if it belongs to entity words, e.g., f(“of”|Contains) = 2. |RI | is the number of distinct relation types in instance I.
We expect a joint tagger to approximate its position-attention aS to aI , where aS = AvgPooling (a1, . . . ,aT ) is the average pooling of model’s position-attention at defined in Equation (1) for each position j in S. We compute the mean squared error (MSE) as the objective:
LBR = MSE(aI ,aS) = ∑ (aI − aS)2. (7)

heading: 3.2 Ontology-Based Logic Fusion (OLF)
text: Probabilistic Soft Logic (PSL) (Bach et al., 2017) uses soft truth values for predicates in an interval between [0, 1], which represents our token classification probability p(yt|wt) as a convex optimization problem. Inspired by Wang and Pan (2020); Kirsch et al. (2020) that considers relation logic as rules for inference, we adapt PSL to entity-relation dependency rules according to data ontology using human annotation (see details in Table 8 and Table 9), e.g., relation type Founder_of should entail (head) entity type PERSON. Training instances that violate any of these rules are penalized to enhance comprehension of entity-relation coherence. Suppose BR guides a model to recognize confident relations, OLF further helps explore instances with reliable entity labels, especially when no relations exist in them.
Particularly, we define Logic Distance based on a model’s softmax scores over the head entity given its predicted relation type to measure how severely it violates logic rules. For a training instance, we define an atom l as each tag and the interpretation
Algorithm 1 Logic Distance Calculation D Input: Softmax p(y|ei), Prediction ŷi, i ∈ {1, 2},
PSL rulesR w.r.t. ontology; Output: Distance d;
1: Initialize d← 1; Satisfied← False; 2: for each r : lre → lent ∈ R ∧ ŷ2 == lre do 3: y1 ← lent; 4: d′ ← max {p(ŷ2|e2)− p(y1|e1), 0}; 5: d← min {d′, d}; 6: Satisfied← True; 7: if Satisfied == False then 8: d← 0.
I(l) as the soft truth value for the atom. For each rule r : RELATION → ENTITY, the distance to satisfaction dr(I) under the interpretation I is:
dr(I) = max {0, I(lre)− I(lent)}. (8)
PSL determines a rule r as satisfied when the truth value of I(lre) − I(lent) ≥ 0. For each instance I, we set lent as (head) entity type and lre as relation type. This equation indicates that the smaller I(lent) is, the larger the penalty it has. We compute the distance to satisfaction for each rule r and use the smallest one as a penalty because at least one rule needs to be satisfied.
We learn a distance function D(·, ·) that minimizes all possible PSL rule grounding results, as described in Algorithm 1. D(·, ·) should return 0 if at least one PSL rule is satisfied. The prediction probability p(y|e1) over head entity e1 is regarded as the interpretation I(lent) of ground atom lent, so as p(y|e2) over tail entity e2 for I(lre) of lre. If no rules are satisfied, the distance is set as 0. We formulate the distance to satisfaction as a regularization term to penalize inconsistent predictions:
LOLF = ∑ D(R; {(p(y|ei), ŷi)}), (9)
where p(y|ei) is the softmax probability of zti in Equation (4) for position ti of ei in S, and LOLF is the sum of D(·, ·) over all entity-relation pairs (e1, e2) in instance I. We finalize a noise-robust loss function by summing up (5), (7) and (9):
L = Lc + αLBR + βLOLF , (10)
where α, β are two balancing hyper-parameters.

heading: 3.3 Self-Adaptive Learning (SAL)
text: Self-adaptive learning (Jia et al., 2019) aims to iteratively select high-quality instances with informative relation patterns p and entity tags. In each
training epoch, more precisely labeled instances are chosen to guide a model to attend to informative evidence for joint extraction. For instance selection, more versatile patterns are required to select clean labels and to discover more confident relation patterns. According to the attention mechanism and entity-relation logic, a trained tagger can tell the importance of each word for identifying the entity pair along with their relationship, and predict reasonable entity-relation label pairs. For instance I, if 1) the model’s attention weights do not match the target attention that explains the relation types in I, or 2) its confidence distribution over entity and relation tags violates the logic dependencies, this instance is likely a false alarm. We add up both BR and OLF loss for an instance I to measure its fitness u(I), i.e., how likely it is correctly labeled:
u = σ[MSE(aI ,aS) +D(R; I)], (11)
where σ is the sigmoid function that bounds u in the range [0, 1]. The lower u is, the more confident an instance I is. We compute fitness scores for all training instances and select those whose score is smaller than a predefined threshold τ .
Because trustable relation labels also indicate trustable entity tags, we further consider Entity Selection (ES), i.e., selecting negative instances containing either the head or tail entity corresponding to each relation pattern in the selected positive candidates. Specifically, we consider relation pattern p as the text between two entities in an instance. We build an initial trustable pattern set P by counting all patterns up and selecting the top 10% frequent patterns for each relation type. Next, we redistribute the training dataset D based on P , where all positive instances that match patterns in P as well as negative instances that contain the head entity or tail entity of these patterns are retained to train the model for a few epochs. Finally, we select more reliable instances according to fitness scores over D, from which we extract new trustable patterns to enrich P . These new confident instances are learned in the subsequent iteration. We repeat the above procedure until the validation F1 converges.

heading: 4 Experiments

heading: 4.1 Datasets and Evaluation
text: We evaluate the performance of DENRL on two public datasets: (1) NYT (Riedel et al., 2010). We use the human-annotated test dataset (Jia et al.,
2019) including 1,024 sentences with 3,280 instances and 3,880 quadruplets. The training data is automatically generated by DS (aligning entity pairs from Freebase with handcrafted rules), including 235k sentences with 692k instances and 353k quadruplets. (2) Wiki-KBP (Ling and Weld, 2012). Its test set is manually annotated in 2013 KBP slot filling assessment results (Ellis et al., 2013) containing 289 sentences with 919 instances and 1092 quadruplets. The training data is generated by DS (Liu et al., 2017) including 75k sentences with 145k instances and 115k quadruplets.
We evaluate the extracted quadruplets for each sentence in terms of Precision (Prec.), Recall (Rec.), and F1. A quadruplet {e1, tag1, e2, re} is marked correct if the relation type re, two entities e1, e2, and head entity type tag1 are all matched. Note that negative quadruplets with “None” relation are also considered for evaluating prediction accuracy. We build a validation set by randomly sampling 10% sentences from the test set.

heading: 4.2 Baselines
text: We compare DENRL with the following baselines: LSTM-CRF (Zheng et al., 2017) that converts joint extraction to a sequence labeling problem based on a novel tagging scheme. PA-LSTM-CRF (Dai et al., 2019), which uses sequence tagging to jointly extract entities and overlapping relations. OneIE (Lin et al., 2020), a table-filling approach that uses an RNN table encoder to learn sequence features for NER and a pre-trained BERT sequence encoder to learn table features for RE.
PURE (Zhong and Chen, 2021), a pipeline approach that uses a pre-trained BERT entity model to first recognize entities and then employs a relation model to detect underlying relations. CoType (Ren et al., 2017), a feature-based method that handles noisy labels based on multi-instance learning, assuming at least one mention is correct. CNN+RL (Feng et al., 2018) that trains an instance selector and a CNN classifier using reinforcement learning. PCNN+RL (Qin et al., 2018), a baseline whose RL method used to detect and remove noise instances is independent of the training of RE systems. ARNOR (Jia et al., 2019) which uses attention regularization and bootstrap learning to reduce noise for DS RE. FAN (Hao et al., 2021), an adversarial method including a BERT encoder to reduce noise for DS RE. SENT (Ma et al., 2021), a negative training method that selects complementary labels and re-labels noisy instances with BERT for DS RE. Llama-ICL (Pang et al., 2023), we follow the basic prompt with two demonstration examples using Llama, each as a pair of input text and extracted triplets. GPT-4-ICL (Pang et al., 2023), the same setting as Llama-ICL but with GPT-4 as the backbone.

heading: 4.3 Implementation Details
text: For DENRL and baselines using pre-trained BERT, we use the pre-trained bert-large-cased from Hugging Face. For baselines using LSTM, we apply a single layer with a hidden size of 256. We also ex-
tend DENRL to two other backbones: T5 and GPT2, and demonstrate the generalizability of DENRL in Appendix C. For Llama-ICL, we use Llama-27B (Touvron et al., 2023). The prompt configuration and training setup for both Llama-2 and GPT-4 are detailed in Appendix A.

heading: 4.4 Overall Results
text: As shown in Table 1, DENRL (triplet) denotes ignoring head entity type tag1 when computing correctness, because all baselines only extract triplets {e1, e2, re}. The results of triplet and quadruplet have little difference, indicating that DENRL predicts precise entity types. DENRL significantly outperforms all baselines in precision and F1 metrics. Specifically, it achieves roughly 5∼20% F1 improvement on NYT (3∼6% on Wiki-KBP) over the other denoising methods—CoType, CNN+RL, ARNOR, FAN, SENT. Compared to LSTM-CRF which also trains on selected subsets, DENRL achieves 31% recall improvements on NYT (15% on Wiki-KBP) with still better precision, suggesting that we explore more diverse entity and relation patterns. Compared to the sequence tagging approach PA-LSTM-CRF, DENRL achieves improvements of 32% in precision and over 18% F1 improvement. DENRL also outperforms baselines using pre-trained transformers (OneIE, PURE, FAN, SENT) or LLMs (Llama-2-ICL, GPT-4-ICL), showing our noise-robust learning effectively reduces the impact of mislabeled instances on joint extraction performance.

heading: 4.5 Ablation Study
text: We investigate the effectiveness of several components of DENRL on NYT dataset, as shown in Table 2. Before noise reduction, we first evaluate the impact of CRF layer by substituting it
with an FC layer. We found it improves the final performance by over 1% F1. We then build an initial redistributed dataset (via IDR), which helps the joint model earn over 2% improvement in F1 and a sharp 28% precision increase compared to BERT+CRF. This suggests the original DS dataset contains plenty of noise, thus a simple filtering method would effectively improve the performance.
However, this initial data induces poor recall performance, which means a large proportion of true positives with long-tail patterns are mistakenly regarded as false negatives. Assuming that some relation patterns in the training data are too rare to guide the model to learn to attend them, we employ BR to training and achieve 5% recall increases with a slight decline in precision, inducing another 2% F1 improvement. This shows the effect of guiding the model to understand important feature words for identifying relations.
After we introduce OLF to training, both precision and recall improve by about 2%, leading to another 2% F1 improvement, proving that logic rules guide a model to learn the entity-relation dependencies and further reduce entity labeling noise.
After we obtain an initial model trained by BR and OLF, we continue SAL where DENRL collects more confident long-tail patterns to mitigate false negatives and finally achieves 5% F1 improvement. We observe that BR better facilitates SAL than OLF regarding the recall increase, as BoW helps explore more high-quality relation instances and reduces false negatives. Additionally, BR+OLF outperforms both BR and OLF. This proves our assumption that significant relation patterns and entity-relation dependencies can reinforce each other during training, i.e., understanding significant relation patterns facilitates the predictions of correct entity-relation dependencies, and vice versa.

heading: 4.6 Interpretability Study
text: To understand the effect of attention and logic guidance, we select some instances from the test set and visualize their attention weights, as well as the model’s softmax probability distribution over all la-
Attention & SoftMax Distribution (GPT-2+PA+CRF)
Attention & SoftMax Distribution (TAILOR)
bels. As shown in Figure 4, BERT+CRF, which is trained on original noisy data without BR or OLF, only focuses on entity pairs and makes wrong predictions. Its logic distance for r : Founder_of → PERSON is dr(I) = max {0, 0.7− 0.4} = 0.3. While DENRL precisely captures important words and correctly predicts the relation. The logic distance for r : Company_worked → PERSON is dr(I) = max {0, 0.8− 0.8} = 0 < 0.3, suggesting the effect of OLF.
Suppose the start position is at “Lyor”, for a target position at “Warner”, the model should pay more attention to (1) the pattern words “chief executive of” to determine their correct relation type Company_worked, and (2) the entity words “Lyor Cohen” to determine the correct head entity type. This is exactly how attention guidance is built in BR. During BR, for the relation Company_worked, its pattern set contains corresponding patterns, e.g., {‘CEO at’, ‘manager in’, ‘chief executive of’, ...} and the BoW will count tokens and their frequencies, e.g., {‘chief’: 8, ‘manager’: 14, ‘at’: 7, ‘of’: 16, }. Consequently, the guidance scores for the words “chief”, “executive”, and “of” become large (after softmax computation) as they appear in the BoW with high frequencies. Also, the head entity tokens “Lyor”, and “Cohen” will be given a score of 1. In this way, both “Lyor Cohen” and “chief executive of” will be assigned a high guidance score, and we want the model’s attention to approximate such guidance distribution (by minimizing their discrepancy) so that it can better identify correct relations.
We further check the performance of DENRL on negative test cases that do not contain relations from NYT dataset. After selecting confident candi-
dates in each epoch, we further choose additional trustable negative instances that contain either the head or tail entity corresponding to each relation pattern in the selected positive candidates during bootstrap. We compare the results between methods with and without entity selection, as shown in Table 3. The improved performance with ES demonstrates that a trustable relation pattern also indicates reliable entity labels, and partially explains the overall superiority of DENRL.

heading: 4.7 Efficiency Analysis
text: While DENRL considers the position-attentive loss calculated through traversing transformer logits on different start positions, it does not significantly inflate the training overhead. For a sentence of n tokens, the time-intensive self-attention operations (squared complexity) are executed just once per sentence. The resultant hidden outputs are used to perform self-matching and CRF decoding regarding each start token, which also has an O(n2)
complexity but with few extra trainable parameters introduced. This layered approach ensures a manageable overall computational overhead. Table 4 shows the average GPU hours per training epoch for each method. We observe that DS methods consume more time compared to their normal counterpart, e.g., ARNOR takes up to ×1.6 the overhead of LSTM-CRF. DENRL, though consumes more time compared to BERT+CRF, is more efficient than other DS methods using PLMs.

heading: 5 Related Work
text: Joint extraction. Entities and relations extraction is important to construct a KB. Traditional methods treat this problem as two separate tasks, i.e., NER and RE. Joint extraction detects entities and their relations using a single model which effectively integrates the information of entities and relations, and therefore achieves better results in both subtasks (Zheng et al., 2017). Among them, unified methods tag entities and relations simultaneously, e.g., Zheng et al. (2017) propose a novel tagging scheme that converts joint extraction to a sequence labeling problem; Dai et al. (2019) introduce query position and sequential tagging to extract overlapping relations. These methods avoid producing redundant information compared to the parametersharing neural models (Gupta et al., 2016) and require no hand-crafted features that are used in the structured systems (Yu et al., 2020; Ren et al., 2017).
LLMs in open-domain IE. LLMs have shown significant promise in the field of open-domain Information Extraction (IE). Recent surveys (Xu et al., 2023) have highlighted the diverse prompts, paradigms, backbones, and datasets employed in this domain. These studies reveal that LLMs, through their ability to understand and generate human-like text, can effectively extract structured information from unstructured data sources across various domains (Pang et al., 2023). The adaptation of LLMs to different IE tasks, including NER, RE, and event extraction, demonstrates their versatility and effectiveness. Moreover, the integration of LLMs with prompt-based learning paradigms has further enhanced their performance by leveraging the contextual knowledge embedded within these models.
Distantly supervision. Previous studies on distantly-supervised NER rely on simple tricks such as early stopping (Liang et al., 2020) and
multi-type entity labeling (Shang et al., 2018; Meng et al., 2021). For distantly-supervised RE, existing methods include multi-instance learning (Lin et al., 2016) that models noise problem on a bag of instances, reinforcement learning (RL) (Nooralahzadeh et al., 2019; Hu et al., 2021), adversarial (Chen et al., 2021; Hao et al., 2021) or probabilistic learning (Liu et al., 2022; Li et al., 2023b) that selects trustable instances, and pattern-based methods (Ratner et al., 2016; Shang et al., 2022) that directly model the DS labeling process to find noise patterns, e.g., Feng et al. (2018) propose a pattern extractor based on RL and use extracted patterns as features for RE.
Probabilistic soft logic. In recent years, PSL rules have been applied to machine learning topics, including model interpretability (Hu et al., 2016), probability reasoning (Dellert, 2020), sentiment analysis (Gridach, 2020), and temporal relation extraction (Zhou et al., 2021). We are the first to model entity-relation dependencies by designing ontology-based PSL.

heading: 6 Conclusions
text: We propose DENRL, a noise-robust learning framework for distantly-supervised joint extraction, which consists of a transformer backbone, a new loss function and a self-adaptive learning step. Specifically, we use Bag-of-word regularization and logic fusion to learn important relation patterns and entity-relation dependencies. The regularized model is able to select trustable instances and build a versatile relation pattern set. A self-adaptive learning procedure then iteratively improves the model and dynamically maintains trustable pattern set to reduce both entity and relation noise. In the future, we aim to explore more complex patterns when configuring pattern sets. We will also evaluate our framework on other tasks such as event extraction and open information extraction.
Limitations
We incorporate a BERT backbone into a sequence tagging scheme for distantly-supervised joint extraction. While our current framework is built upon BERT due to computation resource constraints, it’s designed with flexibility in mind. It can be easily adapted to other transformers such as GPT-2, T5, and even LLMs like Llama2, as the only difference is the computation of the transformer’s final representations, which is the very first step before our
architecture designs. Though achieving state-ofthe-art performance compared to other DS methods, DENRL can be computation costly due to the position-attentive loss computed on multiple start positions. We further conduct an efficiency analysis in Section 4.7, demonstrating a relatively small training overhead of DENRL compared to other DS methods using PLMs.
Moreover, we focus on relations within a sentence and regard words between an entity pair as relation patterns. In our future work, we aim to consider relations beyond the sentence boundary for DS joint extraction to better adapt to real-world information extraction scenarios.
Furthermore, although our OLF is a one-time effort and can benefit future training, it is still handcrafted based on ontology, and we aim to design a probabilistic method such as model uncertainty to quantify more comprehensive underlying relationentity dependencies in the future.

heading: Acknowledgment
text: This research was supported by the National Science Foundation under Grants CNS Career 2230968, CPS 2230969, CNS 2300525, CNS 2343653, CNS 2312397.

heading: B Study on normal IE datasets
text: We also evaluate DENRL and Llama-ICL in two popular IE datasets—ACE (Walker and et al., 2006) and SciERC (Luan et al., 2018). As shown in Table 5, we observe that DENRL still achieves a significant (over 5%) improvement in F1 over LlamaICL, suggesting the generalizability and robustness
of our methodology across diverse datasets. We will enrich the evaluation in our final version with the new results.

heading: C Generalizability across Backbones
text: To demonstrate the generalizability of our noise reduction approach, we extend DENRL to additional two transformer backbones: T5 (Raffel et al., 2020) and GPT-2 (Radford et al., 2019). Table 6 illustrates the concrete PLMs and their number of parameters for a rather fair comparison. As shown in Table 7, we can see DENRL with different backbones all achieve around 68% F1 score on the NYT dataset, significantly outperforming existing methods. This consistent superiority highlights that the efficacy of our method is agnostic to the backbone selection.

heading: D OLF Rules
text: We create the OLF rules based on human annotation, as in this work we focus on open-domain IE scenarios and it is straightforward to identify such logic dependencies (apparently, this method can be substituted by querying LLM for domain-specific scenarios if human expertise is not available). Table 8 and Table 9 summarize our OLF rules for the NYT and Wiki-KBP datasets, respectively. Our OLF can be easily built given the target dataset. It is a one-time effort and can be reused for subsequent fine-tuning.

heading: E Case Study
text: To show that BR explores versatile patterns to enrich pattern set P , we summarize both highfrequency patterns obtained by IDR and meaning-
ful long-tail patterns discovered during SAL, and statistic their average fitness (see Table 10). Some long-tail patterns are not similar syntactically but still have over 0.5 average fitness scores, meaning the model learns useful semantic correlations between related feature words.

heading: Future Work
text: In the future, we aim to explore more complex patterns when configuring pattern sets. We will also evaluate our framework on other tasks such as event extraction and open information extraction. Limitations
We incorporate a BERT backbone into a sequence tagging scheme for distantly-supervised joint extraction. While our current framework is built upon BERT due to computation resource constraints, it’s designed with flexibility in mind. It can be easily adapted to other transformers such as GPT-2, T5, and even LLMs like Llama2, as the only difference is the computation of the transformer’s final representations, which is the very first step before our
architecture designs. Though achieving state-ofthe-art performance compared to other DS methods, DENRL can be computation costly due to the position-attentive loss computed on multiple start positions. We further conduct an efficiency analysis in Section 4. 7, demonstrating a relatively small training overhead of DENRL compared to other DS methods using PLMs. Moreover, we focus on relations within a sentence and regard words between an entity pair as relation patterns. In our future work, we aim to consider relations beyond the sentence boundary for DS joint extraction to better adapt to real-world information extraction scenarios. Furthermore, although our OLF is a one-time effort and can benefit future training, it is still handcrafted based on ontology, and we aim to design a probabilistic method such as model uncertainty to quantify more comprehensive underlying relationentity dependencies in the future