CoAug: Combining Augmentation of Labels and Labelling Rules
Year: 2023

Abstract:
Collecting labeled data for Named Entity Recognition (NER) tasks is challenging due to the high cost of manual annotations. Instead, researchers have proposed few-shot self-training and rule-augmentation techniques to minimize the reliance on large datasets. However, inductive biases and restricted logical language lexicon, respectively, can limit the ability of these models to perform well. In this work, we propose CoAug, a co-augmentation framework that allows us to improve few-shot models and ruleaugmentation models by bootstrapping predictions from each model. By leveraging rules and neural model predictions to train our models, we complement the benefits of each and achieve the best of both worlds. In our experiments, we show that our best CoAug model can outperform strong weak-supervision-based NER models at least by 6.5 F1 points on the BC5CDR, NCBI-Disease, WikiGold, and CoNLL-2003 datasets.1

heading: 1 Introduction
text: Named Entity Recognition (NER) is the task of identifying entity spans of specific types in a given document. While deep learning has led to the development of highly performant supervised NER models (Ma and Hovy, 2016; Lample et al., 2016; Devlin et al., 2019), their performance is contingent on the availability of high-quality large labeled datasets, which is often expensive to collect. Moreover, it is impractical to assume the availability of large datasets for all domains. Hence, learning from limited labeled data is a pressing challenge in named entity recognition research. The majority of research in this area can be broadly classified into two distinct paradigms: few-shot learning with pre-trained language models (LMs) and weak supervision methods that utilize heuristic rules for entity extraction.
∗ Work done during an internship at Bosch Research. 1 Code: https://github.com/boschresearch/CoAug
In few-shot learning, models are trained to identify novel entities given just a few labeled examples for each entity type. While pretrained LMs have been explored for this setting, their susceptibility to overfitting on small datasets results in poor performance. Consequently, recent works improve recognition using prototypical networks (ProtoBERT, Tänzer et al., 2022), improved representations from self-supervised pre-training of LMs (QuIP, Jia et al., 2022), and self-training (Huang et al., 2021). In the iterative learning process of self-training, many candidate entities are extracted and added into the training set for future iterations. However, premature models from initial iterations also add erroneous entities to the training set, resulting in models whose performance lags behind fully-supervised models that utilize large labeled datasets.
On the other hand, rule-based weak supervision methods utilize heuristic rules and manual lexicons (Shang et al., 2018; Peng et al., 2019) developed by domain experts to supervise entity recognition models. However, experts may find it challenging to enumerate all possible heuristics, which can limit the diversity of identified entities in docu-
9062
ments. In recent work, TaLLOR (Li et al., 2021) overcomes this limitation by automatically learning rules given unlabeled data and an initial set of seed rules (tens of rules). Nonetheless, while rule-based methods offer high precision, their performance is constrained by the logical language specified by the developer, which limits the set of identifiable entities. Moreover, learning rules can fail to identify entities in new linguistic contexts that would otherwise be known.
We hypothesize that the two paradigms of fewshot learning and rule-based weak supervision can effectively complement each other, as neural models are skilled at identifying candidates from different linguistic contexts but lack precision, while rulebased methods can identify accurate candidates with precision but lack the flexibility to identify entities in different contexts. Therefore, in this work, we propose Co-Augmentation (CoAug), as shown in Figure 1, an iterative bootstrapping framework that effectively combines neural models, rule-based weak supervision methods, and unlabeled data.
Our proposed framework draws inspiration from co-training (Blum and Mitchell, 1998), but it has its own unique approach. Like co-training, CoAug aims to combine two distinct inductive biases in limited labeled data settings. Unlike co-training, instead of improving two models that use different feature sets individually by bootstrapping labels from each other, CoAug accomplishes the same goal by using two models that use different forms of supervision to expand the same label set. Additionally, in each iteration of CoAug, both classifiers are trained with the predictions made by both models, rather than just one. Our choice allows the framework to function from really small initial training sets for the individual models.
We evaluate our approach on four named entity recognition datasets that span general and science domains. Our results indicate that (a) CoAug consistently improves performance over self-training ruleaugmentation and few-shot models while being highly precise, (b) utilizing stronger pre-training for the neural models leads to improved performance of models in our framework. In summary, our contributions are as follows:
• We present CoAug, a co-augmentation framework that leverages both rule-augmentation and label-augmentation approaches for NER.
• Experimental results show that CoAug can perform better than prior rule-based methods on
four datasets in two domains. • We provide a brief analysis of factors that con-
tribute towards the success of CoAug.

heading: 2 CoAug
text: In this work, we consider a setting where we have access to an initial set of seed rules, S , and a large unlabeled corpus, U , to perform the named entity recognition task. Applying the rules, S, on U provides the initial set of labeled examples, L, to train models in our framework.
Our framework, CoAug (short for CoAugmentation), iteratively improves the performance of two models by leveraging the bootstrapped predictions on unlabeled data by each model. Given that prior work in low-resource NER focuses on two parallel tracks of rule-augmentation and few-shot learning methods that do not interact with each other, we instantiate CoAug with a rule-augmentation model and a few-shot model to leverage the best of both paradigms. We refer to these components of our framework as Rule Augmenter and Label Augmenter (Figure 1). In the subsections below, we describe the Rule Augmenter and Label Augmenter modules.

heading: 2.1 Rule Augmenter
text: Algorithm 1 TaLLOR Require: U = {x1:N} unlabeled examples Require: R = {S} rules initialized with seed rules Require: C = {c1:M} candidate rules
Initialize: L = {} for t in (1, . . . , T ) do
// Apply rules to get weak-label set W = RULEAPPLIER(R,U) // Filter accurate examples W = LABELSELECTOR(W) L = L ∪W U = U \ L // Train NEURAL NER MODEL M ← TRAIN(M , L) // Label using NEURAL NER MODEL LM ← PREDICT(M,U) // Select High-precision Rules RS ← RULESELECTOR(LM , C) C = C \ RS R← R∪RS
end for
The primary function of the Rule Augmenter is to automatically learn labeling rules from unlabeled data and use them to generate weak labels for training a neural model. In this work, we instantiate the rule augmenter module using the TaLLOR framework. Accordingly, our rule augmenter has
the following subcomponents: (a) RULE APPLIER that applies rules over unlabeled data to generate weak labels, (b) LABEL SELECTOR that filters the most accurate examples based on the similarity of averaged token-level BERT (Devlin et al., 2019) representations of proposed entities to the representations of previously identified entities of the same label in the training set, (c) NEURAL NER MODEL that is trained on the accurate instances and proposes new entities in the unlabeled data that can be used to develop new rules, and (d) RULE SELECTOR that scores candidate labeling rules and selects high-precision rules that satisfy the predictions from the NEURAL NER MODEL. We summarize the iterative process of automatic rule identification by TaLLOR in Algorithm 1.

heading: 2.2 Label Augmenter
text: The Label Augmenter module consists of a NEURAL MODEL that learns to perform entity recognition with minimal supervision and LABEL SELECTOR that selectively adds the weak labels proposed by the NEURAL MODEL into the training set for the next iteration.
Algorithm 2 Label Augmenter Require: U = {x1:N} unlabeled examples Require: L = {S} rules initialized with seed rules Require: β0, β1 ▷ initial threshold and increment
Initialize: L = R(U) for t in (1, . . . , T ) do
// Train NEURAL MODEL M ← TRAIN(M ,L) // Label using NEURAL MODEL LM ← PREDICT(M,U) // Select Examples Using Adaptive Threshold LM ← LABELSELECTOR(LM , β0 + t× β1) L = L ∪ LM
end for
In this work, we experiment with two instantiations of the NEURAL MODEL using recent few-shot NER models, namely, ProtoBERT and QuIP. We use an adaptive threshold for the Label Selector to filter out low-quality, weakly labeled instances. Initially, we add 20% of the proposed instances from the Neural Model to the training set. Then, as the model becomes more confident in its predictions over iterations, we gradually increase the proportion of instances incorporated, with a 5% increase per iteration. We summarize the label augmenter algorithm in Algorithm 2.
We provide an outline for the CoAug algo-
Algorithm 3 CoAug algorithm Require: U = {x1:N} unlabeled examples Require: R = {S} rules initialized with seed rules Require: RuleAugmenterM1, LabelAugmenterM2 L = R(U) for t in (1, . . . , T ) do U = U \ L // Training the Rule Augmenter section M1 ← TRAIN(M1 , L) R← R∪ UPDATERULES(M1)
▷ Select high-precision rules L = L ∪R(U) ▷ Add examples after applying rules // Training the Label Augmenter section M2 ← TRAIN(M2 , L) W ← HIGHCONFWEAKLABEL(M2 , U ) ▷ Select high-confident weak-labels L = L ∪W
end for
rithm in Algorithm 3. In each training iteration, we alternatively train the Rule Augmenter and Label Augmenter models. Different from cotraining (Blum and Mitchell, 1998), in CoAug, the Rule-Augmenter (Label-Augmenter) utilizes the examples that have been labeled by the Rule-Augmenter (Label-Augmenter) and the Label-Augmenter (Rule-Augmenter) to improve its entity recognition performance over iterations.

heading: 3 Experiments

heading: 3.1 Experimental Settings
text: We evaluate our framework on four popular datasets that are composed of two science-domain and two general-domain datasets. Following Li et al. (2021), we utilize the training data without labels as our unlabeled data. Further, for all experiments, we use a set of 20 initial seed rules. These rules specify highly frequent entities for each category within a dataset.
BC5CDR (Li et al., 2016) contains 1,500 PubMed abstracts with manual annotations for disease and chemical entity mentions. The abstracts are split equally among train, dev, and test sets (500/500/500).
NCBI-Disease (Doğan et al., 2014) contains 793 PubMed abstracts with manual annotations for disease entity mentions. The abstracts are split as 593/100/100 for train, dev, and test sets.
CoNLL2003 (Tjong Kim Sang and De Meulder, 2003) contains about 20,744 sentences from Reuters news articles. We split the data into 14,987/3,469/3,685 sentences for the train, dev, and test set. Additionally, for our experiments, we only
consider the Person, Location, and Organization entities2 following Li et al. (2021).
WikiGold (Balasuriya et al., 2009) contains 1,696 sentences from Wikipedia articles with annotations for Person, Location, and Organization entity categories similar to CoNLL2003. We split the dataset into 1,142/280/274 sentences for the train, dev, and test sets.
We evaluate two instantiations of the CoAug framework where the Rule Augmenter uses TaLLOR, and the Label Augmenter uses either ProtoBERT/QuIP. For baselines, our main experiments compare CoAug against TaLLOR, self-trained ProtoBERT, and self-trained QuIP. Our code is implemented in Pytorch (Paszke et al., 2019) using the Huggingface library (Wolf et al., 2020). For the Rule Augmenter section, all experimental hyperparameters follow that from Li et al. (2021). Notably, we use the same hyperparameters for the NCBI-Disease, and WikiGold datasets as Li et al. (2021) did for BC5CDR and CoNLL2003. For science-domain datasets, we utilize SciBERT-base (Beltagy et al., 2019) as the base for the ProtoBERT model and BERT-base (Devlin et al., 2019) otherwise. We do not make any such distinctions for QuIP as it is a specially fine-tuned RoBERTa-large (Liu et al., 2019) model designed to perform well on extraction-based tasks (Jia et al., 2022). We report the hyperparameters used for all experiments in more detail in Appendix C.

heading: 3.2 Results and Analysis

heading: 3.2.1 Main Results
text: Table 1 reports the test set F1 scores for all models on each of the four datasets. We observe that CoAug with QuIP/ProtoBERT outperforms TaLLOR on all 4 datasets substantially (average F1 on WikiGold for
2skipping entities from the Miscellaneous category.
CoAug is more than 2× TaLLOR). Further, we also observe that utilizing the co-augmentation framework as opposed to self-training also aids models to produce similar results more reliably, as indicated by the variance of the results (in 3 out of 4 datasets). Further, we also observe that utilizing larger few-shot models, such as QuIP (which has a RoBERTa-large base), is complementary to our framework and continues to push the NER performance further. On comparing with QuIP, we observe that CoAug with QuIP performs better on 3 out of 4 datasets.
However, on the NCBI-Disease dataset, we observe that QuIP outperforms CoAug by a considerable margin. On analysis, we identify that QuIP adds too many incorrect instances during the initial few iterations for this dataset. Consequently, the rule augmenter selects rules that lose precision, and the overall quality of examples in CoAug deteriorates. Nonetheless, since entity recognition for this dataset is hard for TaLLOR as well, we observe some improvement from using CoAug. Future work should look to address the issue of controlling candidates from neural models in order to maintain the reliability of the high-precision set.
In Figure 2, we identify that the success of CoAug over high-precision rule-augmentation approaches, such as TaLLOR, lies in its ability to identify more instances in the unlabeled that improve precision as well as recall over TaLLOR.

heading: 3.2.2 Effect of Task-aligned Pre-training
text: In this subsection, we analyze the contribution of pre-training strategies towards the performance of CoAug. Specifically, we ablate the effect of changing the pre-training initialization from QuIP to that of RoBERTa-large, the base model for QuIP. As shown in Table 2, the performance of CoAug with RoBERTa-large lags far behind the performance
of CoAug with QuIP. On BC5CDR, we observe that the CoAug with RoBERTa-large performs poorly in comparison to TaLLOR as well. This indicates that any form of task-aligned pre-training, such as QuIP, can help design NER models for a diverse domain of tasks which corroborates some of the earlier work in task-adaptive pre-training (Gururangan et al., 2020).

heading: 4 Conclusion
text: In this work, we introduce CoAug, a coaugmentation framework that utilizes unlabeled data to train rule-augmentation and neuralaugmentation models to become better NER taggers. Our results on datasets from two domains demonstrate the effectiveness of CoAug for lowresource domains. Our analysis reveals that CoAug is able to perform better than weak-supervision methods like TaLLOR because of an ability to find more positive instances while maintaining high precision. Further analysis shows the importance of factors such as the strength of pre-training that can contribute towards the success of models in domain-specific datasets.
Limitations
We observe that although CoAug outperforms baselines on multiple datasets, it is still prone to errors
that emerge from the bootstrapping process. Specifically, our framework utilizes models to augment weak labels to the training set, and if the proposals are extremely noisy, training on noisy examples in future iterations will further exacerbate the ability of the framework to identify entities with high precision. Incorporating constraints to preserve the quality of the pseudo-labeled data (Shrivastava et al., 2012) is an exciting direction for future work in low-resource named-entity recognition.
Ethics Statement
All our experiments are performed over publicly available datasets. We do not use any identifiable information about crowd workers who provide annotations for these datasets. Neither do we perform any additional annotations or human evaluations in this work. We do not foresee any risks using CoAug if the inputs to our model are designed as per our procedure. However, our models may exhibit unwanted biases that are inherent in pre-trained language models. This aspect is beyond the scope of the current work.

heading: Acknowledgement
text: We would like to express our appreciation to Haibo Ding for the insightful discussions and helpful suggestions during the initial phases of this project.

heading: A Related Work
text: Weakly-supervised NER. Utilizing distant supervision in the form of knowledge bases or typed lexicons dates back to the work of Mintz et al. (2009). However, obtaining pre-defined lexicons for all domains is challenging. Therefore, more recent work proposes to use manually-defined labeling functions to provide weak labels for documents at scale (Bach et al., 2017). Safranchik et al. (2020); Lison et al. (2020) have proposed improved techniques for leveraging such labeling functions to derive weak labels for entities. However, exhaustively defining labeling functions to identify entities can be a cumbersome task, even for domain experts. Hence, TaLLOR (Li et al., 2021) introduces an automatic technique for learning rules through an iterative approach of proposing weak labels and new rules for entity extraction. More recently, GeNER (Kim et al., 2021) utilizes DensePhrases (Lee et al., 2021) to query Wikipedia for documents that contain entities from desired categories. However, Wikipedia may not contain enough information for new emerging domains or even new
languages. In contrast, CoAug can be applied in such situations as well using few rules and some unlabeled datasets.
Co-training. In co-training (Blum and Mitchell, 1998), given two views of an input that are conditionally independent of each other given the true label, classifiers learned over both views can be improved by bootstrapping the performance of each view iteratively with unlabeled data. Some recent studies suggest, however, that the conditional independence assumption of the views can be relaxed when the models are “different enough" (Balcan et al., 2004; Goldman and Zhou, 2000). Within language processing methods, co-training has been used for cross-lingual adaptation (Yu and Joty, 2021) and improving prompt-based techniques (Lang et al., 2022). In our work, we improve named entity recognition with a combination of rule-augmentation and neural-augmentation techniques.

heading: B Background
text: Re-iterating, we consider a setting where we have access to an initial set of seed rules, S , and a large unlabeled corpus, U , to perform the named entity recognition tasks. Applying the initial set of rules on U provides an initial set of labeled examples, L, to train models in our framework.
B.1 TaLLOR Our work primarily builds on top of the TaLLOR framework introduced in Li et al. (2021). In TaLLOR, a neural model is trained on L to provide weak labels for the potential entities present in U . Based on the computed weak labels, a Rule Selector module proposes new labeling rules that align well with the weak labels while maintaining high precision for entity recognition. Finally, the newly proposed rules are used to label more examples in U , and the process is repeated over many iterations. At the end of training, TaLLOR is evaluated by the neural model’s performance on the test set of the corresponding task. For more details on TaLLOR, we refer the reader to (Li et al., 2021).

heading: C Experiment Hyperparameters
text: Across all datasets, we limit the span of the entities to 5 tokens.
Following Li et al. (2021), the neural model in the Rule-Augmentation model is initialized with a BERT-base/ SciBERT-base model depending on
the domain of the dataset. During training, we use a minibatch size of 32 with the Adam optimizer (Kingma and Ba, 2015), a learning rate of 2e−5, and perform gradient clipping (clipped at norm of 5.0) to stabilize training.
For the Label-Augmenter model, we utilize two models: ProtoBERT and QuIP. Since these models have different characteristics, we utilize a different set of hyperparameters to fine-tune each model for our task. Specifically, for the ProtoBERT model, we use the AdamW (Loshchilov and Hutter, 2019) optimizer, a learning rate of 1e−4, and apply weight decay of 1e−2 for all parameters except the layer-norm weights. For QuIP, we follow the recommendations from Jia et al. (2022) and adopt a learning rate of 2e−5 with the AdamW optimizer for fine-tuning. Further, we initialize the token prediction head for the NER task using question prompt embeddings from this model. The set of questions we use for the different datasets has been summarized in Table 3.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: A1. Did you describe the limitations of your work?
text: Left blank.

heading: A2. Did you discuss any potential risks of your work?
text: Left blank.

heading: A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.

heading: A4. Have you used AI writing assistants when working on this paper?
text: Left blank.

heading: B Did you use or create scientific artifacts?
text: Left blank.

heading: B1. Did you cite the creators of artifacts you used?
text: Left blank.

heading: B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Left blank.

heading: B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Left blank.

heading: B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Left blank.

heading: B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.

heading: C Did you run computational experiments?
text: Left blank.

heading: C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Left blank.

heading: C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.

heading: D Did you use human annotators (e.g., crowdworkers) or research with human participants?
text: Left blank.

heading: D1. Did you report the full text of instructions given to participants, including e.g., screenshots,
text: disclaimers of any risks to participants or annotators, etc.? Left blank.

heading: D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students)
text: and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Left blank.

heading: D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to
text: crowdworkers explain how the data would be used? Left blank.

heading: D4. Was the data collection protocol approved (or determined exempt) by an ethics review board?
text: Left blank.

heading: D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data?
text: Left blank.

heading: Future Work
text: Specifically, our framework utilizes models to augment weak labels to the training set, and if the proposals are extremely noisy, training on noisy examples in future iterations will further exacerbate the ability of the framework to identify entities with high precision. Incorporating constraints to preserve the quality of the pseudo-labeled data (Shrivastava et al. , 2012) is an exciting direction for future work in low-resource named-entity recognition