KGAST: From Knowledge Graphs to Annotated Synthetic Texts (2024)

Abstract: In recent years, the use of synthetic data, either as a complement or a substitute for original data, has emerged as a solution to challenges such as data scarcity and security risks. This paper is an initial attempt to automatically generate such data for Information Extraction tasks. We accomplished this by developing a novel synthetic data generation framework called KGAST, which leverages Knowledge Graphs and Large Language Models. In our preliminary study, we conducted simple experiments to generate synthetic versions of two datasets—a French security defense dataset and an English general domain dataset, after which we evaluated them both intrinsically and extrinsically. The results indicated that synthetic data can effectively complement original data, improving the performance of models on classes with limited training samples. This highlights KGAST’s potential as a tool for generating synthetic data for Information Extraction tasks.

heading: 1 Introduction
text: Information Extraction (IE) models serve as crucial components across various domains, enabling us to make informed decisions based on complex data. However, the effectiveness of these models is dependent on the availability and quality of training data. In this context, we encounter two critical challenges: 1) Data Scarcity: Frequently, despite having complex modeling techniques, researchers deal with datasets that are either insufficient in size or entirely unavailable. Without a sufficient number of labeled examples, IE models struggle to generalize effectively, compromising their predictive capabilities. 2) Privacy and Compliance Concerns: In an era of heightened data privacy regulations, organizations must navigate the balance between model performance and safeguarding sensitive information. Certain datasets whether due to privacy risks or legal constraints cannot be openly shared, which further complicates training effective IE models. To address these issues, researchers often resort to manual data augmentation. This labor-intensive process involves collecting additional data points and meticulously annotating them. While effective, it is time-consuming and resource-intensive, making it less feasible for small organizations operating on limited budgets. Numerous studies have explored the expansion of training data by introducing additional synthetic data (Kobayashi, 2018; Wei and Zou, 2019; Zhang et al., 2020). These studies presented straightforward strategies, such as substituting certain words with their equivalent terms. These equivalents can be retrieved from external sources like WordNet (Miller, 1995), DBnary (Sérasset, 2015), or they can be calculated using word embedding models such as Word2Vec (Mikolov et al., 2013), FastText (Bojanowski et al., 2016), and Glove (Pennington et al., 2014). Although these techniques can indeed augment the initial training dataset, they fail to generate adequate diversity for the models to generalize effectively in subsequent tasks, owing to the minimal semantic variations from the original data. Back-translation is another recognized technique for augmenting the initial training data. With Machine Translation models (Xie et al., 2017; Fabbri et al., 2020), paraphrases of each sentence can be obtained through the back-translation process. While back-translation effectively amplifies the dataset size twofold, it introduces a notable challenge in terms of annotation. The text derived from back-translation diverges from the original annotation. Thus, either careful manual annotation or sophisticated annotation algorithms are required to update the annotations in alignment with the back-translated text, ensuring the precision of the dataset. To address these issues of low data diversity and misalignment of the original annotation, we propose a novel synthetic data generation
43
framework called KGAST. This framework leverages Knowledge Graph to automatically generate Annotated Synthetic Texts that can be used for training IE models. In this study, we sought to answer questions similar to the ones raised in this paper (Claveau et al., 2021):
• Can synthetic data serve as supplementary data to improve the performance of classes with limited training samples?
• Can synthetic data be a viable alternative to gold standard data?

heading: 2 Related Works

heading: 2.1 Data Augmentation
text: Given the resource-intensive nature of manual data creation and annotation, a variety of data augmentation strategies have been employed to address the issue of data scarcity. One of the well-known approaches is the rule-based (Kobayashi, 2018; Wei and Zou, 2019; Zhang et al., 2020) word replacement. This method requires a heuristic for selecting and replacing words within a sentence. On the other hand, some research has approached this at sentence level by leveraging dependency tree (Coulombe, 2018; Dehouck and Gómez-Rodríguez, 2020). For example: "John did the math exercises." is replaced with "The math exercise was done by John". The data augmented through these methods often conveys information very similar to the original, thereby limiting semantic diversity. Backtranslation (Xie et al., 2017; Fabbri et al., 2020) is another method to augment the original dataset. This straightforward technique involves translating text from the original language to another language, and then translating it back to the original language to produce a new version. However, this method presents its own challenges, as labels from the original text may no longer align with the new text due to changes in syntax or semantics.

heading: 2.2 Distant Supervision
text: Automatically generating new supervised data is a compelling alternative to manual annotation, especially when creating large-scale datasets for natural language processing tasks. One such approach is Distant Supervision (Roller et al., 2015; Deng and Sun, 2019), which leverages existing knowledge bases to construct and label new training samples. The core assumption of this method is that if two entities share a relation in the knowledge base, any
sentence containing those two entities might express that relation. However, the automated annotation process introduces errors such as incorrectly assumed entity types or the relations between entity pairs.

heading: 2.3 In context Learning
text: In-context learning (ICL) has recently emerged as a new paradigm in the field of natural language processing. This approach allows Large Language Models (LLMs) to make predictions based solely on contexts that are augmented with a select number of examples. Often, ICL aids in refining the output of an LLM, enhancing the accuracy of the output even in the absence of fine-tuning. With ICL, the performance achieved by LLMs can rival that of previous supervised learning methods (Brown et al., 2020; Shin et al., 2021; Wan et al., 2023). This can be achieved by carefully crafting clear instructional prompts along with high-quality task-specific k-shot examples (Zhao et al., 2021; Liu et al., 2022).

heading: 3 Method

heading: 3.1 Overview
text: Our proposed methodology is built upon two primary elements: LLMs and ICL. Contrary to previous research that modified the original texts of the dataset, our approach involves the generation of new synthetic texts and their corresponding annotations using LLMs. A well-known reasoning prompt method, Chain-of-Thoughts (Wei et al., 2022), enables us to create complex ICL prompt templates to instruct LLM models for such tasks. The intuition behind this approach stems from the concept of distant supervision, where we make a naive assumption that if a pair of entities (ehead, etail) is present in both the text and the Knowledge Graph (KG), these two entities maintain the same relation as the one in the KG.

heading: 3.2 Task Formulation
text: We formalize the task of synthesizing annotated data as a natural language generation task. Consider a given gold text t ∈ G, where G = {t1, . . . , tn} represents the set of gold standard texts, At = {a1, . . . , an} is the set of label annotations, Rt = {r1, . . . , rn} is the set of relations, Kt is the KG and At, Rt ⊆ Kt. The intuition is to construct a text generation prompt p by utilizing the KG as an input to get our intended output synthetic
text t′. The KG Kt of a prompt p is constructed by extracting all the annotated relations Rt from the text t. Once the synthetic text t′ is produced, we proceed to extract the set of label annotations At′ and the set of relations Rt′ by simply filtering out any triples of the KG Kt where either the head or the tail is not present in the text t′. This can be formulated as At′ , Rt′ ⊆ Kt′ , where:
Kt′ = naive_filter(Kt, t′) (1)

heading: 3.3 Prompt Construction
text: Our prompt template is divided into three main sections for ease of understanding.
Instruction Serves as a clear directive for the LLM, which outlines its role and task. We clearly specify the role and task for which we want to generate output. For instance, in the case of text generation:
"You are a creative text writer. Write me a text using the provided Knowledge Graph. Your objective is to write a coherent text that incorporates all the given triples (head, relation, tail) of the Knowledge Graph. You have the right to make the text creative and informative, but you must make sure that the text reflects the given Knowledge Graph."
For text annotation, we draw inspiration from Tree-of-thought (Yao et al., 2023) prompting and construct the instruction as follows:
"You are a text annotator. Your objectives are to: 1/ Analyze the given text in detail, 2/ Annotate possible entities based on these entity types: person(PER), location(LOC), organization(ORG), time(TIME), numbers(NUM), and miscellaneous entity names(MISC). Response with the annotation in this format: "Possible Entities: e_1 e_2,...., e_n", where e_1 to e_n are the extracted entities."
k-shots Examples Similar to the RetrievalAugmented Generation (RAG) (Lewis et al., 2020), where all documents are embedded into latent space, we did the same for all examples retrieved from G. We then use top-k retrieval to find examples that are close to the input KG Kt as k-shot examples for prompting.
Test Input We used KG Kt as an input in the form of a list of triples in natural language to prompt the LLM model. The structure of these input triples is as follows:
("Head":Type, "Relation", "Tail":Type)
The goal is to provide the LLM with as much information as possible so that it can generate a coherent text that corresponds to the input triples.

heading: 3.4 The Framework: KGAST
text: The process outlined in Section 3.2 yields both the text and its corresponding annotation. However, we identified two primary issues:
Incomplete Text Annotation Despite having annotations, we found that it is often incomplete as seen in Figure 2. The method’s effectiveness is heavily reliant on the performance of the LLM used. Consequently, the likelihood of generating a text, t′, that includes all the input triples of Kt is contingent on the LLM’s performance for our given tasks.
Text Coherence and Validity Without a validation heuristic, the texts generated by our method may contain nonsensical phrases. This is because LLMs are known to produce hallucinations, such as incoherent texts with their input KGs, repetitive tokens, inclusion of parts of the prompt, and texts in different languages.
To address these shortcomings, we integrated Self-consistency (Wang et al., 2022) into our framework, aiming to mimic real human annotation processes as closely as possible. The idea behind selfconsistency is to prompt the LLM to generate a set of n outputs and select the most consistent one. For text generation, we prompted LLM to generate n = 3 outputs. We used the 3 output texts as input to prompt the LLM to vote n = 5 times, evaluating each based on creativity, coherence, and the text’s capacity to include all the input triples
of the KG. We then select the best text t′ with the highest voting score. A similar approach is also applied for extracting annotations from text t′. We prompted the same model to generate n = 5 outputs and merged the most consistent annotation with a threshold of 0.5. This means that if an annotation appears in at least 50% of the n outputs, it is extracted. Subsequently, this annotation is merged with the annotation from our naive heuristic. The entire procedure is described in Figure 1.

heading: 4 Experiment Setup

heading: 4.1 Datasets
text: For simplicity, we will refer to gold standard data as G and synthetic data as S. DocRED (Yao et al., 2019), is a documentlevel relation extraction dataset constructed from Wikipedia and Wikidata. This dataset contains a total of 96 relations and 6 entity types for English general domain. Each relation is annotated along with its supporting evidence.
French Security and Defense for which we will refer as FRSD, is a document-level relation extraction dataset that covers the annotation for Event Extraction, Entity Recognition, Attribute Extraction, and Relation Extraction tasks for French intelligence service. FRSD contains 2,000 French documents, of which 800 are manually written and annotated by humans. It consists of 35 entity types, 20 attributes, and 49 relations.

heading: 4.2 Synthetic Data Generation
text: In this preliminary study of generating synthetic data, we did simple generation experiments by using the training set of G as a reference to generate the synthetic version S . For DocRED, this resulted in a total of 3023 new documents along with their annotations. The LLM model used in the framework for this dataset was Zephyr-7B1, a fine-tuned model of Mistral-7B (Jiang et al., 2023). The same approach was applied to FRSD but on 400 (train) documents. In FRSD, we observed a significant data imbalance among Event classes. With this in mind, we manually selected the top 10 event classes with the fewest samples and used their texts as a reference to generate 1200 new documents (Oversampling). We used Vigostral-7B2, a chat-
1https://huggingface.co/HuggingFaceH4/ zephyr-7b-beta
2https://huggingface.co/bofenghuang/ vigostral-7b-chat
based model that has been fine-tuned on Mistral-7B (Jiang et al., 2023) for this dataset. Supporting evidence for each relation of S on both datasets was automatically extracted using a simple heuristic. This heuristic identifies the sentence index where the head or the tail entity appears and uses it as supporting evidence.

heading: 4.3 Tasks
text: To evaluate the effectiveness of our proposed framework, two types of evaluations were conducted:
Intrinsic Evaluation This evaluation aims to understand the accuracy of our framework’s annotations. We evaluated the annotation accuracy of DocRED’s synthetic documents. For Named Entity, this was done by using BERT-NER3 as an inference model to predict the set of synthetic texts, and then comparing the output prediction with our framework’s annotations. As for Relations, we used the best model of DREEAM (Ma et al., 2023) (RoBERTa)4 as the inference model and followed the same procedure.
Extrinsic Evaluation The goal of this evaluation is to assess how S impacts the performance of downstream tasks. For DocRED, we used S to train on two tasks: Named Entity Recognition (NER), and Relation Extraction (RE). NER task was trained on the Flair framework5 which used BiLSTM with flair embeddings. RE was trained us-
3https://huggingface.co/dslim/bert-base-NER 4https://github.com/YoumiMa/dreeam 5https://huggingface.co/flair/ner-english
ing DREEAM (teacher) model, which is based on BERT with λ = 0.05. We trained a total of 5 times with different seeds, evaluated the models against the original test set, and computed the average to get the final results. For FRSD, we conducted experiments on two models: Boundary Smoothing (BS) (Zhu and Li, 2022): which is used for Event, Entity, and Attribution recognition tasks. Unified Model (UM) (Prieur et al., 2024): this model approaches the tasks jointly for NER and RE tasks. The architecture of this model includes a module for detecting entity spans and a second for predicting their interactions. In these experiments, we follow the same as the experiment conducted with DocRED and report the results in Section 5.3.

heading: 5 Experiment Results

heading: 5.1 Dataset Characteristic
text: A descriptive statistic of both datasets can be seen in Table 3. We observed that G tends to contain longer documents and possesses a greater number of unique entities and labeled tokens, indicating a higher semantic quality and more robust representation of specific objects, people, places, etc. We computed the Self-BLEU (trigram) for each dataset, revealing lower Self-BLEU scores for G. The scores suggest that the gold datasets are more diverse and less prone to repetitive use of the same tokens/entities in the text. On the other hand, S can be seen to have a larger number of entity pairs (triples) due to the repetition of entities used in the texts. While this increases the raw count of entity
pairs, it may not necessarily enhance the diversity or quality of these pairs.
In addition to the descriptive statistics of the datasets, we also studied the semantic and lexical difference between the set of gold texts t and synthetic text t′ using Cosine Similarity as a measure. The sentence encoders, SimCSE6 (for English) Camembert-large7 (for French), were used for analyzing the semantic difference, while Bag-ofWords with TF-IDF was used for the lexical difference. Figure 3 shows the distribution of the score. Since t and t′ describe the same knowledge graph Kt, despite their different writing styles, higher semantic similarity scores are expected. Lower lexical similarity scores indicate that different lexical properties and grammatical structures were used, even though both t and t′ describe roughly the same Kt. A more in-depth study for producing more diverse texts needs to be done whether through parameter control, reworking the prompt template, or filtering out texts that will increase the Self-BLEU score.

heading: 5.2 Intrinsic Evaluation
text: The performance on NER task can be observed in Table 4. We achieved high F1-scores of 0.93 and 0.92, demonstrating the effectiveness of our framework’s annotation capacity. Among all the tags, we noticed that MISC was the only tag that scored the lowest. As for RE tasks, we considered two types of annotation accuracy: 1) Relation and 2) Evidence. We need to take into account that, originally the best DREEAM model only achieved a 67.53 F1-
6https://github.com/princeton-nlp/SimCSE 7https://huggingface.co/dangvantuan/
sentence-camembert-large
score on DocRED test set, thus the results reported might not be very accurate. Table 5 presents the accuracy results, showing that our naive assumption heuristic achieved a 0.63 F1-score for Relation and a 0.37 F1-score for Evidence. As S’s evidence was solely based on a very naive heuristic, an overprediction of the evidence is to be expected, leading to a low precision score and a high recall score.

heading: 5.3 Extrinsic Evaluation
text: While generating a large volume of new annotated synthetic texts can be accomplished with relative ease, the challenge lies in optimally utilizing this synthetic data. We conducted a series of experiments in order to address this.
DocRED In NER task, we conducted training under different scenarios. We trained the models separately on 1) the original training set G, 2) the synthetic set S, 3) a combination of G + S, 4) a subset of G by sampling documents that have ≥ 20% labeled tokens which left us with 1564 documents. As can be seen from Table 6, using only G generally yields better results. Although training solely on S produces acceptable results (79.14 on Weighted-F1), there is a significant performance gap between G and S. Similar training strategies were applied for the RE task, except for scenario
4, in which the model was trained using a random sample of 20% of the training set. We observed the same trend based on the scores shown in Table 7. From the evaluation results, it can be inferred that both tasks might benefit from more diverse training sets with higher semantic differences to generalize better and produce more robust performance.
FRSD For our first model BS, we randomly split S into three and used them as complement data to train the BS model. Results in Table 8 show the interest of using S as a complement training data. Notably, as shown in Tables 9 and 10, S enhances the performance of classes with fewer samples for Event and Attribute extraction tasks. As outlined in Section 3.4, annotations from S carry the risk of introducing a lot of noise due to incomplete annotation or wrongly assumed relations. Furthermore, the annotations also heavily rely on the capacity of the LLMs used. To address this issue, we tried to improve annotations by implementing a Teacher-Student learning strategy. This solution consists of training a Teacher model on G. The Teacher model is then used to make predictions on S . These predictions are used as annotations to pretrain a second model, the Student model. Finally, the training of the Student model is refined on the G. For this experiment, we used only the batch of 400 texts from S. Four training scenarios were explored: no pre-training, pre-training with S’s annotation, pre-training on synthetic texts with annotations produced by the Teacher model, and finally, a pre-training with S’s annotation together with those of the Teacher model. We discovered that there is an increase in the performance for classes with low samples, except for Attributes. The second observation is that using S’s annotations alone
is useful for low sample classes for Events and Entities. This significance grows when the annotations are combined with predictions from the Teacher model. The results are shown in Tables 1, 2.

heading: 6 Conclusion
text: In this paper, we introduced a novel framework that leverages Knowledge Graphs and Large Language Models to generate annotated synthetic data for Information Extraction tasks. Our preliminary experiments demonstrated that while the data generated by this framework can enhance the performance of classes with limited training samples, it cannot yet serve as a substitute for the original data. Theoretically, within this framework, data anonymization and bias mitigation can be easily accomplished by modifying the input Knowledge Graphs. However,
further research and experimentation are required to fully realize and validate these possibilities.

heading: 7 Limitation
text: One of the limitations of this study is that we only generated new data based on the original data’s Knowledge Graphs, which led to low diversity in the dataset. Future work could involve experimenting with modified Knowledge Graphs to enhance diversity. We acknowledge that the annotations produced by our framework are far from perfect and require further enhancements. One potential im-
provement could be the use of a dependency tree to identify co-references and annotate them. It could also be used to extract relations between entities. Another path for improvement could be the use of attention weights from the generated texts. This could help identify the evidence of relations by pinpointing where the head and tail entities most attentively interact within the texts.

heading: A Example Appendix
text: Examples of text comparison between G and S are provided in Table 11 for English and Table 12 for French. Figure 4 illustrates a sample of Knowledge Graph Kt along with its corresponding gold text
t and synthetic text t′. An example of an entity extraction prompt can be seen in Figure 5.

heading: Future Work
text: Future work could involve experimenting with modified Knowledge Graphs to enhance diversity. We acknowledge that the annotations produced by our framework are far from perfect and require further enhancements. One potential im-
provement could be the use of a dependency tree to identify co-references and annotate them. It could also be used to extract relations between entities. Another path for improvement could be the use of attention weights from the generated texts. This could help identify the evidence of relations by pinpointing where the head and tail entities most attentively interact within the texts