Uncertainty-Aware Bootstrap Learning for Joint Extraction on Distantly-Supervised Data
Year: 2023

Abstract:
Jointly extracting entity pairs and their relations is challenging when working on distantlysupervised data with ambiguous or noisy labels. To mitigate such impact, we propose uncertainty-aware bootstrap learning, which is motivated by the intuition that the higher uncertainty of an instance, the more likely the model confidence is inconsistent with the ground truths. Specifically, we first explore instance-level data uncertainty to create an initial high-confident examples. Such subset serves as filtering noisy instances and facilitating the model to converge fast at the early stage. During bootstrap learning, we propose self-ensembling as a regularizer to alleviate inter-model uncertainty produced by noisy labels. We further define probability variance of joint tagging probabilities to estimate innermodel parametric uncertainty, which is used to select and build up new reliable training instances for the next iteration. Experimental results on two large datasets reveal that our approach outperforms existing strong baselines and related methods.

heading: 1 Introduction
text: Joint extraction involves extracting multiple types of entities and relations between them using a single model, which is necessary in automatic knowledge base construction (Yu et al., 2020). One way to cheaply acquire a large amount of labeled data for training joint extraction models is through distant supervision (DS) (Mintz et al., 2009). DS involves aligning a knowledge base (KB) with an unlabeled corpus using hand-crafted rules or logic constraints. Due to the lack of human annotators, DS brings a large proportion of noisy labels, e.g., over 30% noisy instances in some cases (Mintz et al., 2009), making it impossible to learn useful features. The noise can be either false relations due to the aforementioned rule-based matching assumption or wrong entity tags due to limited coverage over entities in open-domain KBs.
Existing distantly-supervised approaches model noise relying either on heuristics such as reinforcement learning (RL) (Nooralahzadeh et al., 2019; Hu et al., 2021) and adversarial learning (Chen et al., 2021), or pattern-based methods (Jia et al., 2019; Shang et al., 2022) to select trustable instances. Nevertheless, these methods require designing heuristics or hand-crafted patterns which may encourage a model to leverage spurious features without considering the confidence or uncertainty of its predictions.
In response to these problems, we propose UnBED—Uncertainty-aware Bootstrap learning for joint Extraction on Distantly-supervised data. UnBED assumes that 1) low data uncertainty indicates reliable instances using a pre-trained language model (PLM) in the initial stage, 2) model should be aware of trustable entity and relation labels regarding its uncertainty after training. Our bootstrap serves uncertainty as a principle to mitigate the impact of noise labels on model learning and validate input sequences to control the number of training examples in each step. Particularly, we quantify data uncertainty of an instance according to its winning score (Hendrycks and Gimpel, 2017) and entropy (Shannon, 1948). We define averaged maximum probability that is estimated by a joint PLM over each token in a sequence to adapt previous techniques in joint extraction scheme. Instances with low data uncertainty are collected to form an initial subset, which is used to tune the joint PLM tagger and facilitate fast convergence. Then, we define parametric uncertainty in two perspectives—inter-model and innermodel uncertainty. The former is quantified by selfensembling (Wang and Wang, 2022) and serves as a regularizer to improve model robustness against noisy labels during training. The latter is captured by probability variance in MC Dropout (Gal and Ghahramani, 2016) for selecting new confident instances for the next training iteration. Such two-
1349
fold model uncertainties reinforce with each other to guide the model to iteratively improve its robustness and learn from reliable knowledge.

heading: 2 Related Work
text: Joint Extraction Methods Joint extraction detects entities and their relations using a single model, which effectively integrates the information from both sources and therefore achieves better results in both subtasks compared to pipelined methods (Zheng et al., 2017). For example, unified methods tag entities and relation simultaneously, e.g., (Zheng et al., 2017) proposes a novel tagging scheme which converts joint extraction to a sequence labeling problem; (Dai et al., 2019) introduces query position and sequential tagging to extract overlapping relations. Such methods avoid producing redundant information compared to parameter-sharing neural models (Miwa and Bansal, 2016; Gupta et al., 2016), and require no hand-crafted features that are used in structured systems (Yu et al., 2020).
To address the challenge of learning from DS, pre-trained transformers (e.g., BERT, GPT-2) have gain much attention. They model strong expressive context-aware representations for text sequence through multiple attention layers, and achieve stateof-the-art performance on various NLP tasks (Radford et al., 2019; Devlin et al., 2019; Li et al., 2022). They can be cheaply fine-tuned to solve different downstream tasks including NER and RC. Specifically, BERT is trained on large English corpus using masked language modeling. The multi-head attention weights indicate interactions between each pair of words and its hidden states integrate semantic information of the whole sentence, which are used to decode different tagging results.
Uncertainty Methods Uncertainty generally comes from two sources—aleatoric uncertainty and epistemic uncertainty. The former is also referred to as data uncertainty, describing noise inherent in the data generation. Methods mitigating such uncertainty include data interpolation (Dong et al., 2018), winning score, and temperature scale (Guo et al., 2017). The latter is also called model uncertainty, describing whether the structure choice and model parameters best describe the data distribution. One main solution to mitigate model uncertainty is Bayesian Neural Network (BNN) (Klein et al., 2017) that puts a prior distribution on its weights. To save computational cost, Monte Carlo
UnBED: Joint Tagger Joint extraction as a token classification problem
Joint Extraction Model
CRF
Selfmatching
ct at
B-ORG O
HtHp
Input Sentence Microsoft in
query position p
…
…
zt
BERT
• Tagging scheme: tag n sequences according to different query position p following (Dai et al., 2019)
• Backbone: BERT + self-matching + CRF decoder • BERT encode a sentence X into token-level
representations h • For ach query p, self-matching is applied to
calculate the position-attention between token at p and each token at target position t
• CRF considers the correlations between labels in neighborhoods and jointly decodes the best chain of labels
Figure 1: Joint extraction as a token classification task.
dropout is proposed as an approximation of variational Bayesian inference (Gal and Ghahramani, 2016), realized by training models with dropout layers and testing with stochastic inference to quantify probability variance. Besides BNN, selfensembling (Wang and Wang, 2022) which measures the outputs variance between models with the same architecture has been shown effective to reduce parametric uncertainty across models.

heading: 3 Joint Extraction Model
text: Tagging Scheme For an input sequence X = {x1, ..., xn}, we tag n sequences according to different query position p following (Dai et al., 2019). If p is the start of an entity (query entity e1), the sequence is an instance. The entity type is labeled at p and other entities e2 which have relationship with the query entity are labeled with relation types re. The rest of tokens are labeled “O” (Outside), meaning they do not correspond to the query entity. Accordingly, we convert joint extraction into a token classification task and extract relation triplets {e1, re, e2} in each instance, as shown in Figure 1.
Position-Attentive Encoder we use BERT (Devlin et al., 2019) to encode a sentence X into tokenlevel representations h = {h1, ..,hn}, where hi ∈ Rd is a d-dimensional vector corresponding to the i-th token in X . For each query p, self-matching is applied to calculate the position-attention at ∈ RT between token at p and each token at target position t, which compares the sentence representations against itself to collect context information (Tan et al., 2018). The produced position-aware representation ct ∈ RT×d is an attention-weighted sentence vector ct = a⊤t h. Finally, we concatenate ht and ct to generate position-aware and context-
aware representations ut = [ht|ct]. CRF Decoder (Lafferty et al., 2001) For each position-aware representation ut, we first learn a linear transformation zt = Wut ∈ RC to represent tag scores for the t-th token. Here C is the number of distinct tags. For an instance with labels y = {y1, ..., yn}, the decoding score s(z,y) is the sum of transition score from tag yt to tag yt+1 plus the input score z yt t . The conditional probability p(y|z) is the softmax over s(z,y) for all possible label sequences y′. We maximize the loglikelihood of correct tag sequences during training Lc = ∑ log p(y|z).

heading: 4 Uncertainty-Aware Bootstrap Learning
text: Motivation One of the main challenges in bootstrap learning is to evaluate the “correctness” of a labeled instance. We consider this problem from an uncertainty perspective and assume instances with lower uncertainty are more likely to be correctly labeled. In this section, we first propose instancelevel data uncertainty which is used to filter noisy examples and build an initial subset. Then, we introduce our two-fold model uncertainties which helps iteratively mitigate DS effect and build up trustable examples during bootstrap learning.

heading: 4.1 Data Uncertainty
text: Presenting examples in an easy-to-hard order at different training stages can benefit models (Platanios et al., 2019; Zhou et al., 2020), we propose data uncertainty as a way to quantify the “hardness” of an instance. To better estimate the data uncertainty, we use pre-trained language models (PLMs) to generate tag probability for each token in a sequence. Our intuition is that higher uncertain inputs are “harder” to be generated by a PLM, as it already has rationales of language. Accordingly, we propose two data uncertainties, which can be used individually or combined together:
Winning Score (WS) The maximum softmax probability reflects data uncertainty of an input (Hendrycks and Gimpel, 2017). Given an input instance I = {x1, ..., xn}, we define data uncertainty ud(I) as the minus averaged token classification winning score:
ud(I) = − 1n ∑n t=1maxc∈[1,C] P (yt = c|xt) (1)
Entropy Shannon entropy (Shannon, 1948) is widely used to reflect information uncertainty. We
propose data uncertainty ud(I) as the averaged token classification entropy:
ud(I) = 1n ∑n t=1 ∑C c=1 P (yt = c|xt) logP (yt = c|xt)
(2) We filter out examples with high uncertainty scores and build an initial subset with “simple” examples. At the early training stage, a model is not aware of what a decent distribution P (y|x) should be, thus data uncertainty facilitates it to converge fast by tuning on a fairly “simple” subset.

heading: 4.2 Model Uncertainty
text: In our bootstrap learning, we define model uncertainty, i.e., epistemic uncertainty (Kendall and Gal, 2017), to measure whether model parameters can best describe the data distribution following (Zhou et al., 2020). A small model uncertainty indicates the model is confident that the current training data has been well learned (Wang et al., 2019). We adopt Monte Carlo Dropout (Gal and Ghahramani, 2016) to approximate Bayesian inference which captures inner-model parametric uncertainty. Specifically, we perform K forward passes through our joint model. In each pass, part of network neurons θ are randomly deactivated. Finally, we yield K samples on model parameters {θ̂1, ..., θ̂K}. We use the averaged token classification Probability Variance (PV) (Shelmanov et al., 2021) over all tags for instance I:
um(θ) = 1n ∑n t=1 ∑C c=1 Var [ P (yt = c|xt, θ̂k) ]K k=1
(3) where Var[.] is the variance of distribution over the K passes following the common settings in (Dong et al., 2018; Xiao and Wang, 2019). Accordingly, model is aware of its confidence over each instance and how likely the label is noisy.

heading: 4.3 Training Strategy
text: Uncertainty-Aware Loss Besides MC Dropout which measures parametric uncertainty within a model, we also consider mitigating parametric uncertainty between models to stabilize the weights during training. Specifically, we use selfensembling (He et al., 2020; Wang and Wang, 2022) to calculate the loss between the same models to improve model robustness and reduce the label noise effect on model performance.
Algorithm 1 Bootstrap Learning
Input: Original dataset D = {(In, yn)}Nn=1, two joint models f1, f2 with parameters θ1, θ2;
1: Compute data uncertainty ud(I) for each instance I in D; 2: Initial dataset C ← Select data pairs (In, yn) such that ud(I) < τd from D; 3: for epoch e = 1, ... do 4: Train f1, f2 on C using Eq. (5); 5: Calculate model uncertainty um(θ1) on D; 6: C ← Select data pairs (In, yn) such that
um(I; θ1) < τm from D;
We create another joint model with identical framework, e.g., architecture, loss functions, hyperparameters, and compute a self-ensemble loss Le to minimize the difference between two outputs from the two models regarding the same inputs:
Le = ∑ KL(f(I; θ1), f(I; θ2)) (4)
where KL(.) is the Kullback-Leibler divergence between two probabilistic distributions, θ1, θ2 denote the parameters of first and second models. We formulate our final uncertainty-aware objective L as the sum of CRF and self-ensemble loss:
L = Lc + αLe (5)
where α denotes the weight of self-ensembling, and Lc means the token classification loss.
Bootstrap Learning Procedure To mitigate the DS effect on model performance, we propose a twofold bootstrap learning strategy (see Algorithm 1). Specifically, we first apply data uncertainty to filter “harder” examples and redistribute a reliable initial training dataM. Then, we iteratively feed examples following an easy-to-hard order to the model. In each training iteration, we regularize the joint model with self-ensembling loss to reduce the impact of noisy labels on model parameters. Then we use probability variance to select new confident training instances D′ that can be explained by the model as the next training inputs. The more certain examples are selected, the more likely the model will learn beneficial information and will converge faster. We repeat the above procedure until the F1 score on the validation set converges.

heading: 5 Experiments

heading: 5.1 Setup
text: We evaluate the performance of UnBED on two datasets, NYT and Wiki-KBP. The NYT (Riedel et al., 2010) dataset collects news from New York Times and its training data is automatically labeled by DS. We use the revised test dataset (Jia et al., 2019) that is manually annotated to ensure quality. The Wiki-KBP (Ling and Weld, 2012) dataset collects articles from Wikipedia. Its training data is labeled by DS (Liu et al., 2017), and the test set is manually annotated (Ellis et al., 2013).
We compare UnBED with the following baselines: ARNOR (Jia et al., 2019), a pattern-based method to reduce noise for distantly-supervised triplet extraction. PURE (Zhong and Chen, 2021), a pipeline approach that uses pre-trained BERT entity model to first recognize entities and then employs a relation model to detect underlying relations. FAN (Hao et al., 2021), an adversarial method including a transformers encoder to reduce noise for distantly-supervised triplet extraction.
Evaluation We evaluate the extracted triplets for each sentence based on Precision (Prec.), Recall (Rec.), and F1. A triplet {e1, re, e2} is marked correct if the relation type re, two entities e1, e2 are all correct. We build a validation set by randomly sampling 10% sentences from the test set.
Implementation Details We use Hugging Face bert-large-uncased (Devlin et al., 2019) pre-trained model as backbone. For ARNOR, the hidden vector size is set to 300. In regularization training, we find optimal parameters α as 1 for both datasets. We implement UnBED and all baselines in PyTorch, with Adam optimizer, initial learning rate 10−5, dropout rate 0.1, and batch size 8. For initial subset configuration, we choose data uncertainty threshold 0.5. For bootstrap learning, an empirical model uncertainty threshold is set to 0.6 with the best validation F1.

heading: 5.2 Overall Results
text: As shown in Table 1, UnBED significantly outperforms all baselines in precision and F1 metric. Specifically, UnBED achieves 8% F1 improvement on NYT (3% on Wiki-KBP) over denoising approaches—ARNOR and FAN. Our approach also outperforms baselines using pretrained transformers (PURE and FAN), showing that uncertainty-aware bootstrap learning effectively reduces the impact of noisy labels.

heading: 5.3 Further Analysis
text: We analyze the functionality of different components in Figure 2. We observe that both the entropyPV and vanilla-PV outperform the baseline (joint model directly trained on the original DS dataset) in terms of F1 (5∼7% increase), demonstrating the effect of filtering noisy labels and selecting trustable instance using probability variance. Besides, self-ensembling further enhances the performance in later training stage (2∼4 F1 increase), proving that mitigating the inter-model uncertainty benefits model robustness against noisy labels.

heading: 6 Conclusions
text: In this paper, we propose a novel uncertaintyaware bootstrap learning framework for distantlysupervised joint extraction. Specifically, we define data uncertainty in generally token classification to filter out highly-error-prone instances and build an initial high-confident subset, which is used to tune the joint extraction model for fast convergence. We then propose a two-fold bootstrap learning procedure which iteratively mitigates the DS impact on model robustness and selects new trustable training instances. Experimental results on two benchmark datasets show that UnBED significantly out-
performs other denoising techniques.
Limitations
In this work we propose an uncertainty-aware bootstrap learning framework for joint extraction. Though it achieves state-of-the-art performance compared to other denoising techniques, UnBED requires large training resources considering the ensemble loss calculated between two large PLMs and the probability variance calculated on the PLM joint extraction model. In our future work, we hope to incorporate pruning techniques during training to improve the efficiency. We will also consider more complex relations between entities, e.g., relations beyond the sentence boundary, to fit in real-world information extraction scenarios.

heading: Acknowledgements
text: This work was supported by NSF CNS 2135625, CPS 2038727, CNS Career 1750263, and a Darpa Shell grant.

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 7
7 A2. Did you discuss any potential risks of your work? We study open-domain information extraction for researches in this area

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 5
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? No response.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 5
C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? No response.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 5
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In our future work, we hope to incorporate pruning techniques during training to improve the efficiency. We will also consider more complex relations between entities, e. g. , relations beyond the sentence boundary, to fit in real-world information extraction scenarios