Improving Low-resource Named Entity Recognition with Graph Propagated Data Augmentation
Year: 2023

Abstract:
Data augmentation is an effective solution to improve model performance and robustness for low-resource named entity recognition (NER). However, synthetic data often suffer from poor diversity, which leads to performance limitations. In this paper, we propose a novel Graph Propagated Data Augmentation (GPDA) framework for Named Entity Recognition (NER), leveraging graph propagation to build relationships between labeled data and unlabeled natural texts. By projecting the annotations from the labeled text to the unlabeled text, the unlabeled texts are partially labeled, which has more diversity rather than synthetic annotated data. To strengthen the propagation precision, a simple search engine built on Wikipedia is utilized to fetch related texts of labeled data and to propagate the entity labels to them in the light of the anchor links. Besides, we construct and perform experiments on a real-world lowresource dataset of the E-commerce domain, which will be publicly available to facilitate the low-resource NER research. Experimental results show that GPDA presents substantial improvements over previous data augmentation methods on multiple low-resource NER datasets. 1

heading: 1 Introduction
text: Data augmentation is an effective solution to improve model performance and robustness, and is especially useful when the labeled data is scarce. In computer vision and speech, simple hand-crafted manipulations (Zhong et al., 2020; Zhang et al., 2018) are widely used to generate synthetic data that preserve the original information. However, ∗ The email of the authors are: Jiong Cai (caijiong@shanghaitech.edu.cn), Shen Huang (pangda@alibabainc.com), Yong Jiang (yongjiang.jy@alibaba-inc.com), Zeqi Tan (zqtan@zju.edu.cn), Penjun Xie (chengchen.xpj@alibabainc.com) and Kewei Tu (tukw@shanghaitech.edu.cn). Yong Jiang and Kewei Tu are the corresponding authors. 1Our code is publicly available at https://github.com/ modelscope/AdaSeq/tree/master/examples/GPDA.
when applied to natural language processing (NLP), it is challenging to edit a sentence without changing its syntax or semantics.
There are two successful attempts of applying data augmentation on sentence-level NLP tasks. One is manipulating a few words in the original sentence, which can be based on synonym replacement (Zhang et al., 2015; Kobayashi, 2018; Wu et al., 2019; Wei and Zou, 2019), random insertion or deletion (Wei and Zou, 2019), random swap (Şahin and Steedman, 2018; Wei and Zou, 2019; Min et al., 2020). The other is generating the whole sentence with the help of backtranslation (Yu et al., 2018; Dong et al., 2017; Iyyer et al., 2018), sequence to sequence models (Kurata et al., 2016; Hou et al., 2018) or pre-trained language models (Kumar et al., 2020). However, when applied to token-level tasks such as NER, these methods suffer heavily from token-label misalignment or erroneous label propagation.
To overcome the issue of token-label misalignment, Dai and Adel (2020) extend the replacement from token-level to entity-level with entities of the same class, which proves to be a simple but strong augmentation method for NER. Li et al. (2020) adopt a seq2seq model to conditionally generate contexts while leaving entities / aspect terms unchanged. Ding et al. (2020) exploit an auto-regressive language model to annotate entities while treating NER as a text tagging task. Zhou et al. (2022) utilize labeled sequence linearization to enable masked entity language model to explicitly condition on label information when predicting masked entity tokens. Still, these methods generate synthetic data, which inevitably introduces incoherence, semantic errors and lacking in diversity.
In this work, we investigate data augmentation with natural texts instead of synthetic ones. We are inspired by the fact that professional annotators usually understand the semantics of an entity through its rich context. However, in low-resource
110
NER, the semantic information of a specific entity is relatively limited due to fewer annotations. To this end, we propose to improve the NER models by mining richer contexts for the existing labeled entities. More particularly, we propose a Graph Propagation based Data Augmentation (GPDA) framework for NER, leveraging graph propagation to build relationships between labeled data and unlabeled natural texts. The unlabeled texts are accurately and partially labeled according to their connected labeled data, which has more diversity rather than synthetic hand-crafted annotations. Furthermore, not restricted to the existing annotated entities in the training data, we explore external entities from the unlabeled text by leveraging consistency-restricted self-training.
The contributions of GPDA can be concluded:
• We propose a data augmentation framework that utilizes graph propagation with natural texts for augmentation, which is rarely investigated in previous work (Section 2);
• We utilize a simple Wikipedia-based search engine to build the graph with two retrieval methods (Section 2.2);
• With consistency-restricted self-training, we further make the most efficient utilization of externally explored unlabeled text (Section 2.3);
• By conducting experiments on both public datasets and a real-world multilingual lowresource dataset, GPDA achieves substantial improvements over previous data augmentation methods (Section 3).

heading: 2 Method
text: Fig. 1 presents the workflow of our proposed data augmentation framework. First, we build a graph between labeled data nodes and unlabeled text nodes according to their textual similarity. Then, the entity annotations are propagated to obtain augmented data. Finally, the marginalized likelihood for conditional random field (CRF) (Tsuboi et al., 2008) is applied during the training phase as the augmented data are partially labeled. Moreover, we adopt the consistency-restricted self-training strategy to further improve the model performance.

heading: 2.1 NER with Pure Labeled Data
text: We take NER as a sequence labeling problem, which predicts a label sequence y =
{y1, · · · , yn|yi ∈ Y} at each position for the input tokens x = {x1, · · · , xn}, where Y denotes the label set. The sequence labeling model feeds the input x into a transformer-based encoder (such as BERT (Devlin et al., 2019)) which creates contextualized embeddings ri for each token. Then a linear-chain CRF layer that captures dependencies between neighboring labels is applied to predict the probability distribution:
ψ(yi−1, yi, ri) = exp(W Ty ri + byi−1yi)
Pθ(y|x) =
n∏ i=1 ψ(yi−1, yi, ri)
∑ y′∈Y(x) n∏ i=1 ψ(y′i−1, y ′ i, ri)
Unified Training Objective Instead of directly minimizing the negative log-likelihood, we unify the training objectives in Section 2.1, 2.2 and 2.3. Specifically, we compute the marginal probability of each token Pθ(yi|x) with the forward-backward algorithm.
α(yi) = ∑
{y0,...,yi−1}
i∏
k=1
ψ(yk−1, yk, rk)
β(yi) = ∑
{yi+1,...,yn}
n∏
k=i+1
ψ(yk−1, yk, rk)
Pθ(yi|x) ∝ α(yi)× β(yi)
The marginal distributions can be computed efficiently. Given a partially annotated label sequence y∗ = {∗, . . . , yi, . . . , ∗} that ∗ denotes the label that is not observed, we can obtain the probability.
Qθ(y ∗|x) =
n∏
i=1
Qθ(yi|x)
where Qθ(yi|x) is defined as Pθ(yi|x) if yi is observed, otherwise Qθ(yi|x) = 1.
The final model parameters can be optimized by minimizing the following objective:
L(θ) = − logQθ(y∗|x)
For the pure labeled data D = {(x(i),y(i))}Ni=1, we direct set y∗ = yi and obtain the loss function.
L(θ) = − ∑
(x(i),y(i))∈D logQθ(y
∗ = y(i)|x(i))

heading: 2.2 NER with Propagated Unlabeled Data
text: Building Propagating Graph Compared to labeled data, large-scale unlabeled natural texts can be acquired much more easily. We attempt to utilize these natural texts for augmentation by building a graph between the labeled data nodes and the unlabeled text nodes according to their textual similarity. Given a labeled sample (x(i),y(i)), we retrieve its corresponding augmented sentences {x′(i,j)}mj=1 via a search engine. For common NER datasets, the search engine is built on the Wikipedia corpus with one of the two methods we explore: sparse retrieval based on BM252 or dense retrieval3 based on L2 similarity. The top related sentences will be treated connected to the original labeled sentence in the graph.
2Sparse retrieval is implemented with Elastic Search 3Dense retrieval is implemented with ColBERT
Label Propagation While building the graph, label propagation is conducted from labeled data (x(i),y(i)) to unlabeled data {x′(i,j)}mj=1 to generate partially annotated {(x′(i,j),y′(i,j))}mj=1. To strengthen the precision, propagation will not happen unless the anchor text in Wikipedia matches the labeled entity. By graph propagation, we obtain the augmented data D′ = {(x′(j),y′(j))}Mj=1 sharing the same entities but with more diverse contexts. Along with the original labeled data D, we train the NER model following the same objective in Section 2.1:
L(θ) = − ∑
(x(i),y(i))∈D∪D′ logQθ(y
∗ = y(i)|x(i))

heading: 2.3 NER with Explored Entity Annotations
text: To make the most efficient utilization of the explored annotations in D′, we adopt consistencyrestricted self-training. A well-trained model from Section 2.2 will be utilized to re-annotate the partially labeled augmented data under consistency restriction. Particularly, an augmented sample (x′(j),y′(j)) will be re-annotated to (x′(j), ŷ(j)). Now we have D̂ = {(x′(j), ŷ(j))}Mj=1. Along with the original labeled data D, we train a better NER model following the objective in Section 2.1:
L(θ) = − ∑
(x(i),y(i))∈D∪D̂ logQθ(y
∗ = y(i)|x(i))

heading: 3 Experiments

heading: 3.1 Dataset
text: We conduct experiments on the CrossNER (Liu et al., 2020) dataset of 5 genres (AI, Literature, Music, Politics, Science) and an anonymous multi-lingual E-commerce query NER dataset (Ecom) consisting of 3 languages (English, Spanish, French) Detailed statistics about these two datasets is provided in the Table 2.
For CrossNER, the search engine is manually built on the Wikipedia corpus. While for Ecom, an off-the-shelf E-commerce search engine is utilized to build the augmentation graph.

heading: 3.2 Results and Analysis
text: Low-resource NER Tasks As illustrated in Table 1, the proposed GPDA consistently achieves the best F1 scores across the five genres of CrossNER and gains an average improvement of 2.2% over the baseline BERT-CRF model. It also outperforms other data augmentation methods, demonstrating its effectiveness on multi-domain low-resource NER.
Furthermore, GPDA with Explored Entity Annotation (EEA) strategy achieves 1.1% higher F1 than GPDA without EEA, suggesting that it is also crucial to extend unique entities rather then only diversifying entity contexts in data augmentation.
It can be noticed that GPDA with dense retrieval performs worse than with sparse retrieval, which is not intuitive. This may be attributed to dense retrieval requires careful supervised training in the target domain, but our pre-trained matching model is not finetuned. We will leave this part for future work.
Real-world Low-resource NER Scenarios Table 3 shows the F1 results on three languages from the real-world Ecom dataset. The augmented data generated by GPDA improves model performances for multilingual NER. For specific domain datasets where high-quality knowledge or texts can be fetched easily, GPDA are indeed helpful.
Size of Gold Samples We study the impact of GDPA on different size of gold samples in Fig. 2. On the low-resource settings where 10%-25% gold samples are available, the improvement is striking which outperforms the baseline model by at most 37%.
Case Study Taking a closer look at the augmented cases in Fig. 3, we notice that GPDA generates different contexts concerning the entity "Adobe Creative Suite". The augmented data generated by GPDA introduces more diversity to help reduce overfitting. Different from synthetic data, these generated data are all from natural texts so that there is no need to worry about the coherence in syntax or semantics.

heading: 4 Discussion
text: Retrieving relevant texts from databases has been widely used in NLP tasks. RaNER (Wang et al., 2021) retrieves context using a search system to enhance the token representation for NER tasks. To help entity disambiguation in domain-specific NER, Zhang et al. (2022) retrieves the domainspecific database to find the correlated sample. In order to leverage the extensive information about entities in Wikipedia and Wikidata, Wang et al. (2022) and Tan et al. (2023) construct databases and retrieve context to enhance model performance. In this study, we propose the utilization of retrieval techniques for data augmentation in low-resource settings. Furthermore, while they perform retrieval on both the training and testing datasets, we only use the small seed training dataset for retrieval. It’s noteworthy that our approach can also be combined with theirs to further enhance the performance of NER in low-resource settings.

heading: 5 Conclusion
text: We present GPDA as a data augmentation framework for low-resource NER, which utilizes graph propagation with natural texts for augmentation. To make the most efficient utilization of the explored partially labeled text, we adopt consistencyrestricted self-training. Experiment results show
that our proposed GPDA achieves substantial improvements over previous data augmentation methods on multiple low-resource NER datasets.

heading: Acknowledgements
text: This work was supported by the National Natural Science Foundation of China (61976139) and by Alibaba Group through Alibaba Innovative Research Program.

heading: 6 Limitations
text: There are some limitations in the use of GPDA.
• The label propagation procedure requires anchor matching in the light of annotation precision, which limits the unlabeled data source. However, Wikipedia is a open-domain easyto-fetch corpus with anchor links, which can somehow mitigate the issue.
• Augmented Data generated by GPDA provide more diversity. But for some datasets, simple modifications (NERDA) on the original words performs better. We are investigating a hybrid approach to apply GPDA and NERDA in the same framework.

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section 5
3 A2. Did you discuss any potential risks of your work? Section 5
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
B1. Did you cite the creators of artifacts you used? No response.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? No response.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No response.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? No response.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? No response.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. No response.
C 3 Did you run computational experiments? Section 3
7 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Due to the limitation of page, we didn’t report these.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
7 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Due to the limitation of page, we didn’t report these.
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 3
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 2
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We will leave this part for future work. Real-world Low-resource NER Scenarios Table 3 shows the F1 results on three languages from the real-world Ecom dataset. The augmented data generated by GPDA improves model performances for multilingual NER. For specific domain datasets where high-quality knowledge or texts can be fetched easily, GPDA are indeed helpful. Size of Gold Samples We study the impact of GDPA on different size of gold samples in Fig. 2. On the low-resource settings where 10%-25% gold samples are available, the improvement is striking which outperforms the baseline model by at most 37%. Case Study Taking a closer look at the augmented cases in Fig. 3, we notice that GPDA generates different contexts concerning the entity "Adobe Creative Suite". The augmented data generated by GPDA introduces more diversity to help reduce overfitting. Different from synthetic data, these generated data are all from natural texts so that there is no need to worry about the coherence in syntax or semantics