A Unified Knowledge Graph Augmentation Service for Boosting Domain-specific NLP Tasks
Year: 2023

Abstract:
By focusing the pre-training process on domain-specific corpora, some domain-specific pre-trained language models (PLMs) have achieved state-of-the-art results. However, it is under-investigated to design a unified paradigm to inject domain knowledge in the PLM finetuning stage. We propose KnowledgeDA, a unified domain language model development service to enhance the task-specific training procedure with domain knowledge graphs. Given domain-specific task texts input, KnowledgeDA can automatically generate a domain-specific language model following three steps: (i) localize domain knowledge entities in texts via an embedding-similarity approach; (ii) generate augmented samples by retrieving replaceable domain entity pairs from two views of both knowledge graph and training data; (iii) select high-quality augmented samples for fine-tuning via confidence-based assessment. We implement a prototype of KnowledgeDA to learn language models for two domains, healthcare and software development. Experiments on domainspecific text classification and QA tasks verify the effectiveness and generalizability of KnowledgeDA.

heading: 1 Introduction
text: Although general NLP models such as GPT-3 (Brown et al., 2020) have demonstrated great potential, they may not consistently perform well in domain-specific tasks like healthcare (Kwon et al., 2019) and programming (Liu et al., 2019b). This is because most pre-trained language models are trained on general-domain corpora, e.g., OpenWebText (Radford et al., 2019) and C4 (Raffel et al., 2022). However, the words or knowledge entities frequently used in a specific domain are typically different from those in a general domain. For instance, scientific texts use different words than general texts, with only a 42% overlap (Beltagy et al.,
∗*Corresponding authors
2019). Consequently, general PLMs struggle to capture many important domain entities that rarely appear in general corpora. Therefore, it is necessary to develop a suitable training mechanism for domain-specific NLP tasks.
In general, two steps are needed for domainspecific NLP model development: (i) language model pretraining and (ii) task-specific model training (Gu et al., 2021). Most existing studies focus on pretraining. In particular, to learn domainspecific word embeddings, they retrain PLMs with domain-specific corpora, including ClinicalBERT (Alsentzer et al., 2019), BioBERT (Lee et al., 2020), SciBERT (Beltagy et al., 2019), etc. In contrast, how to improve the second step (i.e., task-specific training) is under-investigated. A common practice is directly fine-tuning the task-specific model with annotated data (Gu et al., 2021). However, it is difficult to obtain abundant annotated data for a domain-specific task, as labeling often requires domain experts’ knowledge (Yue et al., 2020); without sufficient data, direct fine-tuning may not lead to a satisfactory performance due to overfitting (Si et al., 2020). Some studies propose task-dependent methods to train task-specific models by introducing some types of domain knowledge (Zhu et al., 2022), but they are hard to be generalized to other tasks (Tushev et al., 2022).
Then, a research question appears: can we introduce domain knowledge to task-specific model training in a unified way? To answer the question, two main issues need to be addressed: (i) where to find a unified format of domain knowledge? (ii) how to improve the task-specific training of various domains’ models in a unified way?
On one hand, the domain knowledge graph (KG) is an effective and standardized knowledge base for a specific domain (Abu-Salih, 2021). KGs have been constructed for various domains such as cybersecurity (Jia et al., 2018), social-impact funding (Li et al., 2020b), and healthcare (Li et al.,
353
2020a; Zhang et al., 2020), which emphasizes the wide availability of domain KGs. Hence, domain KG could be a feasible source for unified domain knowledge. On the other hand, data augmentation (DA) is a data-space approach to enrich training data to avoid overfitting regardless of the task-specific model structure. They are often taskagnostic (Longpre et al., 2020), i.e., not specified to any particular task. This property inspires us that it may be possible to design a unified DA process to introduce domain knowledge to task-specific model training. However, current DA methods in NLP are mostly proposed for general texts (Wei and Zou, 2019), and the performance on domain-specific tasks is limited (Feng et al., 2021). In general, domain-specific DA is still an under-researched direction (Feng et al., 2021).
To fill this research gap, by exploiting domain KGs, we propose KnowledgeDA, a novel and unified three-step procedure to perform domainspecific DA: (i) domain knowledge localization to map phrases in the text to entities in the domain KG; (ii) domain knowledge augmentation to fully utilize the KG and the training data to achieve domain-specific augmentation; and (iii) augmentation quality assessment to single out highquality augmented data for fine-tuning the taskspecific model. Specifically: (i) To the best of our knowledge, this is one of the pioneering efforts toward proposing a unified development process for domain-specific NLP models, especially focusing on task-specific model training. (ii) KnowledgeDA consists of three core steps, domain knowledge localization, domain knowledge augmentation, and augmentation quality assessment. We implement a prototype of KnowledgeDA, which can automatically learn domain-specific models given domainspecific texts, especially in healthcare domain. (iii) Experiments are run on text classification and QA tasks (English and Chinese) mainly in healthcare. Results show that KnowledgeDA can obtain ∼ 4% improvement compared to direct fine-tuning, and significantly outperform existing DA methods (Wei and Zou, 2019; Yue and Zhou, 2020). The source codes are available1.

heading: 2 Related Work
text: Domain-specific Knowledge-augmented NLP Methods. To improve domain-specific NLP model development, a general strategy is introducing do-
1https://github.com/RuiqingDing/KnowledgeDA
main knowledge (Zhu et al., 2022). For zero and few-shot text classification tasks, KPT (Hu et al., 2022) incorporates external knowledge into the projection between a label space and a label word space. For text generation, KG-BART (Liu et al., 2021) proposes a novel knowledge graph augmented pre-trained language generation model to promote the ability of commonsense reasoning. For question answering and dialogue, some work use external knowledge bases to inject commonsense, like KaFSP (Li and Xiong, 2022), KG-FiD (Yu et al., 2022), etc. Besides task-dependent methods, there are also some unified training strategies to incorporate knowledge (domain-specific corpora) into PLMs, leading to domain-specific PLMs such as BioBERT (Lee et al., 2020), SciBERT (Beltagy et al., 2019), ClinicalBERT (Alsentzer et al., 2019), and UmlsBERT (Michalopoulos et al., 2021). Also, there are three primary techniques to integrate knowledge graphs and PLMs: (i) pre-training a PLM from scratch by using KG or other structural knowledge/texts (Feng et al., 2022; Huang et al., 2022); (ii) adapting a given PLM to incorporate KG information with new network layers in task-specific training/fine-turning (Zhang et al., 2022b; Yasunaga et al., 2022; Kang et al., 2022); (iii) augmenting training data with KGs during taskspecific training/fine-tuning, e.g., PHICON (Yue and Zhou, 2020). Our work also attempts to improve the domain-specific NLP model development in a unified manner. Different from PLM, we focus on task-specific NLP model fine-tuning (Gu et al., 2021). Hence, our proposed KnowledgeDA can be used with domain-specific PLMs together to construct NLP models.
Text Data Augmentation (DA). DA has received increasing interest, especially low-resource situations(Feng et al., 2021). In general, there are three types of text DA methods: (i) Rulebased techniques, e.g., EDA (Wei and Zou, 2019), adopt token-level random perturbation operations including random insertion, deletion, and swap; (ii) Interpolation-based techniques, pioneered by MIXUP (Zhang et al., 2018), interpolate the inputs and labels of two or more real examples. Followups include SwitchOut (Wang et al., 2018), MixText (Chen et al., 2020), etc; (iii) Generator-based techniques, e.g., LAMBADA (Anaby-Tavor et al., 2020) and GPT3Mix (Yoo et al., 2021), learn generators by fine-tuning the large language generation models (e.g., GPT) on the training data to generate
new samples. Basically, three types of methods can be used together as they augment data from diverse perspectives. However, regardless of the type, most existing studies do not explicitly introduce domain knowledge. PHICON (Yue and Zhou, 2020) attempts to use the domain-entity dictionary for text DA, which replaces an entity mention in a sentence with another same-category entity. Compared to PHICON, KnowledgeDA further considers relationships in the domain KG; besides, KnowledgeDA introduces other newly-designed components, e.g., augmentation quality assessment, to ensure highquality augmentation.
3 The KnowledgeDA Framework
3.1 Workflow of KnowledgeDA
To facilitate the development of domain-specific NLP models, we propose a unified domain KG service, KnowledgeDA, which can achieve explicit domain knowledge injection by domain-specific DA. Challenges to be addressed include:
C1. How to discover domain knowledge in texts? Detecting entities in a text is the first step to link the text with a knowledge base. A domain entity may have multiple expressions, e.g., lungs and pulmonary share a similar meaning in the healthcare domain. It is important to deal with synonyms.
C2. How to ensure that the augmented texts retain the domain information and are semantically correct? We aim to achieve interpretable data augmentation through explicit domain knowledge injection. The domain information and the semantic correctness of augmented samples are desirable to be kept after data augmentation.
C3. How to ensure the quality of augmented texts? As PLMs grow larger, simple DA method becomes less beneficial (Feng et al., 2021). It is essential to select beneficial samples from all the augmented samples for efficient fine-tuning.
To address the above challenges, we design corresponding modules in KnowledgeDA (shown in Figure 1): (i) domain knowledge localization, which locates the mentions of domain KG entities in texts; (ii) domain knowledge augmentation, which incorporates a dual-view DA strategy by considering both domain KG and training data; (iii) augmentation quality assessment, which retains beneficial augmented samples for fine-tuning using a confidence-based strategy. When the task data and the PLM (e.g., BERT) are given, KnowledgeDA can automatically conduct data augmentation based
on built-in domain KGs and output the final domain task-specific model.

heading: 3.2 Module 1: Domain Knowledge Localization
text: Detecting entities in texts can identify domainspecific objects and the relations between them. Considering that an entity may correspond to multiple mentions (Florian et al., 2004), exact string matching will lead to a low matching rate. Although there are some open entity detection tools, like TAGME (Ferragina and Scaiella, 2010) and BLINK (Wu et al., 2020), and some studies achieve supervised non-exact matching of entities and mentions (Hu et al., 2019), the performance on domainspecific entities can not be guaranteed. Then, we use an annotation-free string-similarity-based strategy (Bunescu and Pasca, 2006; Karadeniz and Özgür, 2019) to discover non-exact but correct mappings between mentions in the text and entities in the KG. Specifically, we calculate the inner product of word embeddings as string similarity (Wu et al., 2020).
As seen in Figure 1, we follow the NLP preprocessing pipeline and match the processed text with KG. During preprocessing, we add the entity strings in KG to the dictionary of tokenizer to avoid word segmentation errors, e.g., ‘cerebral embolis’ should be treated as a medical term rather than being splitted into two words. Also, we use POS Tagger and Lemmatizer to convert each token to the canonical form (lemma), e.g., the lemma for ‘coughed’ is ‘cough’. While knowledge localization, we extract the entities’ embeddings and the mentions’ embeddings from the PLM, and then calculate the similarity between them. We consider the pair of a mention and the most similar KG entity as a match if the similarity score is larger than a threshold λ (0.9 in our implementation).
An example in healthcare is illustrated in Figure 2. Without similarity match, we will ignore that scour and diarrhea are analogous. Through localization, some relations between entities can be also constructed, e.g., fever and scour are symptoms of pneumonia and respiratory syndrome. These will be used in the next module for data augmentation.

heading: 3.3 Module 2: Dual-view Domain Knowledge Augmentation
text: After locating the domain knowledge, i.e., entity mentions in the text, the next step is to replace these mentions with other relevant entity words for
domain-specific data augmentation. Here, we propose a dual-view strategy to conduct the relevant entity retrieval by considering both KG and the training text data.
View 1: KG-based Entity Retrieval (KGER) A direct strategy for domain KG-based DA is to replace the entity with another same-category entity, e.g., replacing ‘William’ with ‘Mike’ as both are person names (Yue and Zhou, 2020). However, it may suffer from two pitfalls: (i) Although the original and replaced entities are in the same category, they can be totally different, such as pneumonia and fracture (both are diseases), which may negatively impact the downstream tasks, e.g., classifying a medical transcription to the relevant department2; (ii) When two or more entities appear in a text, they may have certain valuable relationships (e.g., disease and symptom), but replacing these entities separately would ignore this information.
To address the above issues, we propose two principles for KGER: (i) entity relevance, refers
2https://www.kaggle.com/datasets/tboyle10/medicaltranscriptions
to ensuring that the retrieved entity is similar to the original entity, not just with the same category; (ii) relation consistency, means keeping the relationships unchanged between multiple replaced entities in one text. We formulate a domain KG as G = {E,R, T,C}, where E, R, T , and C are the sets of entities, relations, triples, and entities’ categories, respectively. Specifically, T = TR ⋃ TC , where TR = {(h, r, t)|h, t ∈ E, r ∈ R} and TC = {(e,BelongTo, c)|e ∈ E, c ∈ C}.
Given an entity e, we can get its category c, involved triples Te = {(e, r, t) ∈ TR} ⋃{(h, r, e) ∈ TR}, and the adjacent entities Ee = {e′|(e, r, e′) ∈ Te, (e, r, e
′) ∈ Te}. To obtain more same-category entities, we further retrieve the involved triples of Ee, named Te2, and put Te and Te2 together as the candidate triples (i.e., 2-hop triples around e).
That is: (1) If only one entity exists, or multiple entities exist but do not have direct KG relations in the text, we randomly select a same-category entity e′ from the candidate triples to replace each original entity e. Note that e′ must be within 2-hop around e, ensuring the entity relevance. (2) If there exist certain pairs of entities with KG relations, we would seek the same relation-type triple from the candidate triples for replacing the pair of entities together, following the relation consistency.
For instance, ‘I have a fever and scour. Could it be pneumonia?’ (shown in Figure 3), fever and scour are the symptoms of pneumonia. So we need to search for suitable triples to satisfy relation consistency. For instance, diarrhea and sore throat are the symptoms of respiratory syndrome, so the augmented text could be ‘I have a diarrhea and sore throat. Could it be respiratory syndrome?’.
View 2: Training Data-based Entity Retrieval (TrainER)
In View 1, we mainly retrieve relevant entities that are close in the KG. However, entity pairs far away in the KG may be helpful for the specific task if being replaced with each other. For example, for the task to detect the medical query intent, ‘blood routine examination’ and ‘CT’ is the entity pair that could be replaced with each other for augmentation because they most probably appear in the queries about diagnosis and cause analysis, but they are distant from each other in the medical KG, like CMedicalKG 3.
To find such task-specific valuable replacement entity pairs which may not be near in the KG, we design a new View 2, Training Data-based Entity Retrieval (TrainER), to retrieve task-specific entity pairs from training data. REINA (Wang et al., 2022a) has verified that retrieving from training data to enrich model inputs (concatenating the original input and retrieved training data) may generate significant gains. Inspired by this idea, TrainER aims to extract gainful entity pairs from the training data for augmentation.
In general, a good entity pair for replacement may satisfy at least two properties: (i) label consistency, indicates that the two entities in the pair should be contained in two training texts with the same task label; (ii) expression diversity, means that the two texts containing the two entities should have different expression patterns, so as to enrich the training data diversity. Specifically, to reach label consistency, for an entity e in a text t, we
3https://github.com/liuhuanyong/QASystemOnMedicalKG
would retrieve a same-category entity e′ from another text t′ if t and t′ have the same label. To achieve expression diversity, we first cluster all the training texts into different clusters with diverse expression patterns. Then, for an entity e in a text t, the replaced entity e′ will be retrieved from t′ only if t and t′ are not in the same cluster. Figure 3 elaborates on the process of TrainER.
To conduct training data clustering to differentiate expression patterns, we first mask entities with their categories to extract the expression templates for each training text. For instance, ‘what is pneumonia?’ and ‘what is fracture’ share the same expression template ‘what is [disease]?’, as both sentences have the same pattern regardless of the specific entity (i.e., disease). Then, we run a clustering algorithm on the masked texts, i.e., expression templates, to identify diverse expression patterns. The K-means clustering (Arthur and Vassilvitskii, 2007) is applied due to its high efficiency and effectiveness in empirical experiments; the feature of a masked text is represented by TF-IDF vectorization (Jones, 2004). Same as the relation consistency principle in KGER, if there are certain entity pair with KG relations in the original text, we will retrieve the entity pair with the same relation from other training texts.

heading: 3.4 Module 3: Augmentation Quality Assessment
text: After Module 1 & 2, we obtain a set of augmented texts. A straightforward way is to fine-tune task-specific models with these texts like most prior studies (Zhang et al., 2015; Wei and Zou, 2019).
Recent work (Zhou et al., 2022) has found that not all the augmented texts are equivalently effective; thus, selecting high-quality ones may further improve the model performance.
Inspired by this finding, KnowledgeDA includes a quality assessment module to justify the quality of each augmented text. Prior work (Anaby-Tavor et al., 2020; Zhou et al., 2022) uses the prediction confidence as the quality metric and selects topK high-confidence augmented samples for finetuning, because this ensures the label correctness of augmented texts. However, we argue that it may not significantly improve the model performance since high confidence means that the pattern inside the augmented sample has already been encoded in the original model (without augmentation).
Hence, we first fine-tune PLM (e.g., BERT) on the task texts; then use this plain fine-tuned model M to predict the augmented texts and obtain the confidence scores. Instead of selecting top-K confident samples, we pick K augmented samples whose confidence is close to a predefined threshold δ. Note that δ should not be a too small number, as we still want to ensure the correctness of the training labels for augmented texts; meanwhile, δ should not be too large, as a very high-confident sample would contribute little new knowledge to the model. Based on this idea, we design a novel confidence-based data filtering strategy to retain gainful augmented samples.
The task data D = {(xi, yi)}ni=1 and the plain fine-tuned model M (without augmentation) are known, where xi is a string of text, and the label yi ∈ {1, 2, · · · , q} is the label of xi among a set of q labels. Through KGER and TrainER, we can generate the augmented samples Daugi = {x1i , x2i , · · · , xmi } for the i-th sample, xi. The prediction confidence (probability) of Daugi can be calculated as P augi = {p j i}mj=1, where p j i = prob(M(xji ) = yi). We propose a confidence threshold δ to adjust sample selection criteria. Given δ, the sampling weights of Daugi can be calculated by
w1i , w 2 i , · · · , wmi = softmax(ξ1i , ξ2i , · · · , ξmi ) (1)
where ξji = 1−|δ−p j i |. If p j i is closer to δ (0.75 in
our implementation), we have a higher probability to select this sample. With this confidence-based sampling strategy, we can select augmented samples to further fine-tune the task model M. In general, the selected samples would be relatively
confident but not too highly-confident, thus ensuring both label correctness and new knowledge.

heading: 4 Empirical Evaluation

heading: 4.1 Text Classification

heading: 4.1.1 Setup
text: Datasets. We conduct experiments on four datasets in healthcare: CMID4 and KUAKE-QIC (Zhang et al., 2022a)are in Chinese; TRANS5 and ABS6 are in English. The basic information is enumerated in Table 1. For Chinese, we use an opensource medical KG, CMedicalKG7; for English, we adopt the Unified Medical Language System (UMLS) (Bodenreider, 2004) as KG. The preprocessing of KGs can be found in Appendix A.3.
Baselines. As KnowledgeDA focuses on explicit knowledge injection during DA by domain entity replacement, we mainly compare with state-of-theart rule-based DA methods: SR (Vijayaraghavan et al., 2016) uses token-level replacement with synonyms; EDA (Wei and Zou, 2019) uses token-level random perturbation operations including random insertion, deletion, and swap; PHICON (Yue and Zhou, 2020) uses entity-level replacement with other entities belonging to the same category. For each DA method, we scale up the training data to 5 times the original size and select the best model on the validation set for evaluation. All the methods are based on the same text classifier with the same hyper-parameters (in Appendix A.1). In the main experiments, the base classifier is BERT-base (hereinafter referred to as BERT). We also experiment with domain-specific PLMs as stronger classifiers, discussed in the later part. And the experiment in the software development domain is shown in Appendix C.

heading: 4.1.2 Results in Healthcare
text: Main Results. Table 2 shows the results of different DA methods. We can observe
4https://github.com/ishine/CMID 5https://www.kaggle.com/datasets/tboyle10/medicaltranscriptions 6https://github.com/PoojaR24/Medical-Text-Classification 7https://github.com/liuhuanyong/QASystemOnMedicalKG

heading: Method CMID KUAKE-QIC TRANS ABS
text: that KnowledgeDA achieves the best performance among all the methods in both accuracy and F1 score on four datasets. At the same time, PHICON also outperforms SR and EDA in most cases, verifying the effectiveness of domain-specific knowledge. Specifically, on two Chinese datasets, CMID and KUAKE-QIC, KnowledgeDA improves the accuracy by 3.03% and 3.62%, respectively, over the fine-tuned model without augmentation. Moreover, compared to the best baseline, PHICON, KnowledgeDA’s improvements on accuracy are still statistically significant. Similar results are also observed on two English datasets. In a nutshell, the results suggest that domain-specific entity replacement can facilitate text classification in healthcare. Compared to PHICON which only considers entity categories, KnowledgeDA selects entities from dual views and accounts for the KG relations between them, which further improves the quality of the augmented text and thus achieves a better performance. To further quantitatively verify that KnowledgeDA can introduce more domain knowledge, following Wang et al. (2022b), we calculate Novel Entity Coverage, the percentage of the novel entities in the test data covered by augmented texts (novel means
not appearing in the training data). As illustrated in Table 3, KnowledgeDA has the highest coverage, which also explains the effectiveness.

heading: Domain-specific PLMs as the Base Classifiers.
text: Domain-specific PLMs contain domain knowledge by pre-training with domain corpus. To confirm that KnowledgeDA is still beneficial with a domainspecific PLM, we use eHealth (Wang et al., 2021) and ClinicalBERT (Alsentzer et al., 2019) as the PLMs for Chinese and English datasets, respectively. According to Table 4, the improvement brought by the domain-specific PLMs is evident (comparing with the results of BERT in Table 2). Consistent with the survey (Feng et al., 2021), we discover that when using the domain-specific PLMs, baseline DA methods may not generate an obvious performance improvement and even have a negative effect compared to no-augmentation. For instance, on TRANS, EDA improves the performance over BERT (increasing F1 score from 71.50 to 73.71); while EDA worsens the performance when a domain-specific PLM is used (reducing F1 score from 75.22 to 74.91). However, even with domain-specific PLMs, KnowledgeDA can still improve the domain NLP task performance consistently. Note that for TRANS, KnowledgeDA is the only DA method with positive improvement (and this improvement is also statistically significant).
Larger PLMs as the Base Classifiers. In addi-
tion to using domain-specific PLMs with the same parameter size as BERT-base (110 million parameters), we further take RoBERTa-large (Liu et al., 2019a) and DeBERTa-large (He et al., 2021) with more than 350 million parameters as base classifiers on the TRANS dataset. As shown in Figure 4, with increasing parameters, RoBERTa-large and DeBERTa-large achieve better accuracy than BERT without DA. However, there are still notable improvements of 2.78% and 2.08% in accuracy with KnowledgeDA on RoBERTa-large and DeBERTa-large, demonstrating the generalizability of KnowledgeDA.
Ablation Study. To validate the effectiveness of each module of KnowledgeDA, we design corresponding ablation experiments: KnowledgeDA w.o. SimMatch removes the similarity-based non-exact matching and only uses exact string matching in Module 1 (Sec. 3.2); KnowledgeDA w.o. KGER removes the KGER (view 1) in Module 2 (Sec. 3.3); KnowledgeDA w.o. TrainER removes the TrainER (view 2) in Module 2 (Sec. 3.3); KnowledgeDA w.o. Assess removes the quality assessment module, i.e., Module 3 (Sec. 3.4). Table 5 shows the results. KnowledgeDA outperforms all the other methods that remove certain components. This verifies the validity of each module of KnowledgeDA.
Time Consumption. Table 6 reports the time consumption of all DA methods on CMID and TRANS. The time required for fine-tuning without augmentation is short (∼ 5 minutes). As PHICON and KnowledgeDA need to retrieve the entity mentions and then replace them, time consumption is increased.In particular, KnowledgeDA takes more time because it considers the relations between entities in the KG. In general, the learning process can be completed in about half an hour for
Figure 5: Performance on CMID under n% CMedicalKG disturbance.
Figure 6: Combining with LAMBADA and GeniusAug (TRANS).
KnowledgeDA (also not much longer than competitive baselines like PHICON).
Impact of KG Errors. Considering that the KG quality may affect the quality of the augmented texts (Kang et al., 2022), we randomly change the categories of n% entities and the relation types of n% triples in CMedicalKG, and test the performance of KnowledgeDA in CMID dataset. As shown in Figure 5, when we adjust n from 0 to 10, the accuracy is between 71.0 and 72.5, with a slight decline. When n ≥ 4, SR, the KG-independent DA method, performs better. This illustrates the importance to ensure the KG quality for KG-based DA methods, which is consistent with the findings of other KG-based applications (Hu et al., 2022). In the future, we will explore how to identify potential KG errors so as to improve the robustness of KnowledgeDA.
Different Strategies for Augmented Data Quality Assessment and Selection. We compare two strategies for augmented data quality assessment and selection: δ-K is proposed in Sec. 3.4; Top-K (Anaby-Tavor et al., 2020; Zhou et al., 2022) selects the top K augmented samples with the highest confidence for each original sample. Table 7 shows the results of different strategies, as well as the results of KnowledgeDA without quality assessment. δ-K and Top-K both outperform KnowledgeDA without assessment, verifying the necessity to select high-quality samples for augmentation. And δK performs better than Top-K. This empirically validates our intuition that an augmented sample with a not-too-high confidence may bring more new knowledge to the NLP model, as discussed in Sec. 3.4.
Combine with Generator-based Augmentation Techniques. KnowledgeDA provides a unified framework for domain-specific knowledge augmentation, which may be combined with other DA tech-
KnowledgeDA
niques. Here, we use generator-based augmentation methods as an example. Specifically, we generate augmented samples with two methods, LAMBADA (Anaby-Tavor et al., 2020) and GeniusAug (Guo et al., 2022); based on these augmented samples, we leverage KnowledgeDA to acquire more augmented samples. Since generator-based methods are mostly applied to few-shot tasks (AnabyTavor et al., 2020), we randomly select 50 to 200 samples for each task label in the TRANS dataset. LAMBADA and GeniusAug both generate 200 more samples for each label. Figure 6 shows the results. As expected, the performance of each method goes up as the number of labeled samples increases. More importantly, combining KnowledgeDA with LAMBADA or GeniusAug both can achieve higher accuracy. This demonstrates the general utility of KnowledgeDA to combine with generator-based DA methods to improve the few-shot NLP tasks.
Compare with GPT-3.5. Recently, ChatGPT has shown powerful text generation capabilities. To explore the performance of this large language model on domain-specific tasks, we use the OpenAI API 8 to query text-davinci-003 (the most powerful GPT-3.5) by the prompt, ‘decide which label the following text belongs to, {label names}: \n Text:{sentence} \n Label: ’. It can be seen as a zero-shot manner to response directly. For TRANS(English), the test accuracy is 66.67% (∼ 10% lower than KnowledgeDA with BERT). It performs even worse on CMID(Chinese) with an accuracy of only 32.32%, perhaps due to the limited exposure to relevant texts and knowledge. Therefore, more effective prompt engineering or fine-tuning of GPT (especially for non-English languages) is still necessary for domain-specific tasks, which may be potential future work.

heading: 4.2 QA Tasks
text: Setup. The CMedQA(Chinese) (Zhang et al., 2017) and PubMedQA(English) (Jin et al., 2019) are used for the QA task. Both datasets give the la-
8https://platform.openai.com
bel of each question-answer pair (i.e., match or mismatch). For CMedQA, we sample 1000 questionanswer pairs from the original dataset. For PubMedQA, we keep the original data size (429 samples). In KnowledgeDA, we take the question and answer pair as input and retrieve the entity mentions together. While fine-tuning, we feed questions and answers, separated by the special [SEP] token to BERT (Jin et al., 2019). The KGs and other settings are the same as classification tasks.
Results. Table 8 compares the performance of different DA methods based on BERT. It is obvious that using any data augmentation strategy can make the performance more stable under different seeds (i.e. smaller standard deviation). Also, KnowledgeDA outperforms all the baselines.

heading: 5 Conclusions
text: In this paper, we present KnowledgeDA, a unified knowledge graph service to boost domain-specific NLP tasks. The intrinsic technical novelty is a three-step framework of task-specific data augmentation process based on domain KGs. The experiments on healthcare-related texts both in English and Chinese verify the effectiveness and generality of KnowledgeDA. We also confirm that it can be flexible and effective to incorporate other generatorbased DA methods on few-shot tasks. In the future, we can further investigate how to better combine KnowledgeDA and generator-based DA methods and add KG quality inspection methods to avoid the negative impact of errors in KG.

heading: Limitations
text: Domain KGs are the premise of KnowledgeDA, while open and high-quality domain KGs may be rare in some domains. Therefore, the method will be limited in the domains without suitable KGs. Besides, we use a similarity-based method to map entity mentions in the text to the corresponding entities in the KG. Although this method performs efficiently, it ignores the problem of entity ambigu-
ity (Vretinaris et al., 2021). For instance, the abbreviation, CAT, can stand for ‘catalase’ or ‘COPD Assessment Test’ in healthcare. To address this problem, it is necessary to use contextual information to clarify the specific meaning of the mention (Phan et al., 2017; Orr et al., 2021; Vretinaris et al., 2021). Last but not least, KnowledgeDA may be not good at tasks of paragraph-level texts and the efficiency will reduce. Because long texts probably contain more entity mentions and have more complex syntax, it is more difficult to retrieve the entities and acquire their relations from the KG.

heading: Ethics Statement
text: This paper proposes a unified framework, KnowledgeDA, for text augmentation based on domain KGs for domain-specific NLP tasks. All the experimental datasets and KGs are publicly available, and the related papers and links have been listed in the paper. Also, though PLM and KG are publicly available, there may still be several ethical considerations to bear in mind when applying KnowledgeDA in real-world scenarios. For instance, it is crucial to check whether KG contains biases concerning race, gender, and other demographic attributes.

heading: Acknowledgements
text: This research was supported by National Key R&D Program of China (2020AAA0109401) and NSFC Grants no. 61972008, 72071125, and 72031001.

heading: A.1 Experiment Platform & Settings
text: Our experiment platform is a server with AMD Ryzen 9 3900X 12-Core Processor, 64 GB RAM and GeForce RTX 3090. We use Python 3.6 with pytorch 1.8 on Ubuntu 20.04 for algorithm implementation.
For the text classification task, we feed the [CLS] representation into the output layer when BERTbase as the classifier (Devlin et al., 2019). We split the dataset into training set, validation set, and test set as 8:1:1. When fine-tuning PLMs, we set batch size to 32, learning rate to 1e-5, and training epoch to 10. It will early stop if the loss of the validation set does not decrease in 500 iterations. Accuracy and micro-F1 are used as the metrics in text classification and QA tasks. We repeat each experiment 5 times and record the average results.

heading: A.2 Algorithms
text: In this part, we summarize the detailed implementations of domain knowledge localization (i.e., Module 1) and domain knowledge augmentation & augmentation quality assessment in Algorithm 1 and Algorithm 2, respectively.
Algorithm 1: Domain Knowledge Localization Input: A text x, the entities list E, words
embeddings dictionary Embeds, and similarity threshold θ
Output: A matched pair list Matchs of mentions in x and entities in E 1 Initialize Matchs as an empty list ; 2 Preprocess x with NLP preprocessing pipeline to get
word list words ; 3 Construct entity embedding matrix Eemb and
embedding matrix Wemb by searching for E and words from Embeds ;
4 Compute similarity matrix Sim = Wemb × Eemb.T ; 5 Query the maximum similarity sim_values between each word and entity; 6 if sim_value ⩾ θ then 7 Find the index of sim_values in Sim ; 8 Get the pair (word, entity) according to index ; 9 Add (word, entity) to Matchs ;
10 end 11 Return Matchs ;

heading: A.3 KGs Preprocessing
text: Healthcare is a field with rich professional knowledge. There are also publicly available knowledge graphs, e.g., the Unified Medical Language System (UMLS) (Bodenreider, 2004). We take such open medical KGs for healthcare
Algorithm 2: Domain Knowledge Augmentation & Augmentation Quality Assessment
Input: Train Data D = {(xi, yi)ni=1}; the KG G = {E,R, T, C}; the pre-trained language model PLM ; a confidence threshold δ.
Output: The selected augmented samples Daug 1 Fine-tune without augmentation M =
fine-tune(PLM,D) ; 2 for xi in {xi}ni=1 do 3 Get the Matchesi in xi by Algorithm 1; 4 Generate augmented samples Daugi with G following the steps in Figure 3; 5 Initialize the prediction probabilities of Daugi as P augi ; 6 for xji in D aug i do 7 Calculate the prediction probability pji = prob(M(xji ) = yi); 8 Add pji to P aug i ;
9 end 10 Calculate the sampling weights of Daugi according to Eq. 1 ; 11 Sample 5 samples from Daugi by weights and add them to Daug; 12 end 13 return Daug;
KnowledgeDA. UMLS Metathesaurus is a compendium of many biomedical terminologies with the associated information, including synonyms, categories, and relationships. It groups semantically equivalent or similar words into the same concept, for example, the words ‘flu’, ‘syndrome flu’ and ‘influenza’ are mapped to the same concept unique identifier (CUI) C0021400, which belongs to the category, disease or syndrome. There are 127 semantic types in biology, chemistry, and medicine, consisting of 4,441,326 CUIs (16,132,273 terminologies) in the UMLS 2021AA version. Since the size of the KG is too large to affect the speed of retrieval, we only screen out entities that belong to the type of medicine (e.g., body part, organ, or organ component, disease or syndrome, etc.). At the same time, we also delete non-English strings. Finally, we keep 1,145,062 CUIs (16 semantic types), 502 types of relationships and 4,884,494 triples. Although there are Chinese medical terminologies in UMLS, the number is limited. Hence, we use an open-source Chinese medical KG, CMedicalKG,9 which includes 44,111 entities (7 categories), 10 types of relationships, and 294,149 triples.

heading: B Case Study
text: Fig. 7 shows the examples in English and Chinese with various DA methods. We can observe
9https://github.com/liuhuanyong/QASystemOnMedicalKG
that the sentence augmented by KnowledgeDA has a high quality as it can introduce more domain entities and the whole sentence has a good semantic meaning.

heading: C Text Classification in Software Development

heading: C.1 Dataset
text: We use a open data, SO-PLC10, which is a Stack Overflow dataset for 4 programming language classification: python, C#, java, and javascript.

heading: C.2 KG for Software Development
text: There is little research on building KGs for software development NLP tasks, and thus we decide to build one from scratch.
To build the KG, we refer to the software developer forum Stack Overflow to obtain raw text data.11 Stack Overflow is one of the biggest forums for professional and enthusiastic software developers. Various technical questions are covered on the platform and marked with appropriate tags. These tags are usually programming-specific terminologies and can be beneficial to learn about tech ecosystems and the relationships between technologies (Singh et al., 2020). To build a KG from tags,
10http://storage.googleapis.com/download.tensorflow.org/data/stack_ overflow_16k.tar.gz
11https://stackoverflow.com/
we follow the existing KG construction process (Li et al., 2020a):
Step 1. Data Collection: We use programming languages (e.g., python, C#, java, and javascript) as keywords to search for related questions on Stack Overflow, and sort them according to ‘most frequency’; then crawl the tags that appeared in the top 7,500 related questions (i.e., the first 150 pages).
Step 2. Entity Recognition: A tag is a word or phrase that mainly describes the key information of the question, which is usually a programmingspecific terminology (Singh et al., 2020). Hence, we directly treat tags as the entity names in the KG.
Step 3. Relation Formation: There is usually more than one tag in one question. When multiple tags co-appear at the same question, we link them in the KG. Afterward, there is still a lack of entity types and edge types, and we use the community detection algorithm, Louvain (Blondel et al., 2008), to automatically classify tags, and the edge type is defined by the types of the two connected entities.
Finally, we get TagKG, which includes 6,126 entities (11 categories), 56 types of relationships, and 41,227 triples.

heading: C.3 Result
text: As illustrated in Table 9, there are almost no improvements or even slight decreases with EDA and SR, meaning these general DA methods are not suitable for the texts in software development forums. With the help of our constructed TagKG, PHICON achieves some performance gains by replacing same-category programming entities; this indicates that the category identified by the community detection algorithm in TagKG is effective for understanding software development related texts. By leveraging TagKG more comprehensively, KnowledgeDA works even better and improves accuracy and F1 score by 2.42% and 2.58%, respectively, compared with no-augmentation. It also implies that the construction of TagKG is valid.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Limitations

heading: 3 A2. Did you discuss any potential risks of your work?
text: Limitations and Ethics Statement

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Abstract and Sec. 1 (Introduction)
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Sec. 4

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Sec. 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.

heading: 3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided
text: that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Sec. 4
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Sec 4.1, Sec 4.2, Appendix A.1
C 3 Did you run computational experiments? Sec. 4.1.2

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Sec. 4.1.2
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Sec. 4.1.1, Sec. 4.2, Appendix A.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Sec. 4
C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Not applicable. Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants?

heading: Left blank.
text: D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: In the future, we can further investigate how to better combine KnowledgeDA and generator-based DA methods and add KG quality inspection methods to avoid the negative impact of errors in KG