A Few-Shot Approach for Relation Extraction Domain Adaptation using Large Language Models

Authors: vanni  zavarella
juan  carlos
sergio  consoli
Venue: KDD_org_2024_Workshop_DL4KG
Published: 2024-06-10 20:12:13

Abstract: Knowledge graphs (KGs) have been successfully applied to the analysis of complex scientific and technological domains, with automatic KG generation methods typically building upon relation extraction models capturing fine-grained relations between domain entities in text. While these relations are fully applicable across scientific areas, existing models are trained on few domain-specific datasets such as SciERC and do not perform well on new target domains. In this paper, we experiment with leveraging in-context learning capabilities of Large Language Models to perform schema-constrained data annotation, collecting in-domain training instances for a Transformer-based relation extraction model deployed on titles and abstracts of research papers in the Architecture, Construction, Engineering and Operations
(AECO) domain. By assessing the performance gain with respect to a baseline Deep Learning architecture trained on off-domain data, we show that by using a few-shot learning strategy with structured prompts and only minimal expert annotation the presented approach can potentially support domain adaptation of a science KG generation model.

Main paper: 
 introduction: 
 Knowledge graphs (KGs) [1] have proved effective for representing research knowledge discussed in scientific papers and patents across several different domains [2, 3, 4] . New generation "scientific KGs" have moved from representing purely bibliographic information of research publications to support the construction of extensive networks of machine-readable information about entities and relationships pertaining to a certain domain, enabling fine-grained semantic queries over large scientific text collections such as: "retrieve all methods that are used for Indoor Air Remediation in the time range 𝑇".Therefore, they can support downstream analytical services like technology trend analysis. For example, [5] uses the statistics of relation triples of type <Method;Used-for;Task> automatically extracted from paper abstracts to reconstruct historical trends of the top applications of target methods such as "neural networks" in different areas like speech recognition and computer vision.Workshop at KDD 2024 on Deep Learning and Large Language Models for Knowledge Graphs, 25-29 August, Barcelona. * Corresponding author. Envelope vanni.zavarella@unica.it (V. Zavarella); jgamero@unav.es (J. C. Gamero-Salinas); sergio.consoli@ec.europa.eu (S. Consoli) Methods for automatic generation of scientific KGs typically build upon training supervised Relation Extraction (RE) models for capturing fine-grained relations between scientific entities in text [6] . In the scholarly domain, the entity and relation specifications for this task are defined by the SciERC initiative [5] 1 and are fully applicable across scientific areas. As part of an ongoing endeavour on innovation intelligence analytics for the Architecture, Engineering, Construction and Operation (AECO) industry, we applied SciERC guidelines to annotate domainspecific instances of scientific entities (e.g.Task, Method, Metrics) and their relations (e.g. Usedfor,Evaluate-for). In Figure1 we show a few sentence-level annotations of SciERC entity and relation types respectively from an NLP paper abstract from the SCIERC dataset (a) and from an AECO research paper annotated via the same tagging schema (b).One can see that the semantic schema is perfectly portable to the AECO domain. However, state-of-the-art models for scholarly Relation Extraction have been trained on the SciERC dataset comprising AI/ML articles and do not perform well on new target domains such as AECO [8, 9] . Moreover, manually annotating training data for this domain is a costly and time-consuming strategy that does not scale well.Therefore, we experiment on an empirical solution leveraging in-context learning capabilities of Large Language Models (LLMs) [10, 11] to perform schema-constrained data annotation, generating in-domain training instances for a baseline Relation Extraction architecture from only a few manually designed sentence annotation examples, together with explicit task instructions. Then, we test different training configurations of the model on a small test set of titles and abstracts of AECO research papers and compare the performance with a baseline trained on out-of-domain data.Research on techniques for distilling knowledge from pre-trained LLMs to downstream NLP tasks is currently highly active [12] . Prompt tuning approaches, that translate the target downstream tasks to a masked language modeling problem have been applied to Relation Classification [13, 14, 15] . Similarly to [15] , we combine few-shot examples prompt with rich schema information to elicit LLMs' comprehension of the RE task. However, we explicitly formulate the task as data annotation to the LLM.Finally, the presented approach is in line with the view from [12] that leveraging few-shot learning capabilities of LLMs for optimizing local, lower-sized models is a more cost-effective strategy than relying on direct use of LLMs for inference in a production setting, which typically faces recurrent API usage costs or requires extensive high-end computational infrastructures for fine-tuning. 
data: 
 The source data used in this experiment comprise titles and abstracts from a large collection of around 476k research articles in the AECO area published in the time range 2010-2023, retrieved from the OpenAlex2 open scientific graph database [16] using a set of platform-specific topic filtering tags.We sampled a test set of around 50 abstracts, pre-processed and sentence split them using Spacy's English transformer pipeline en_core_web_trf-3.6.1 3 and finally had them independently annotated by two domain experts using the Brat annotation tool [17] , resulting in a total of 314 sentences, 448 entities, 132 relations instances. The inter-annotator positive specific agreement on entity detection ([18]4 ) reached a mean F1 score of 0.73, indicating an overall satisfying agreement between the human annotators, although some marginal ambiguity is encountered for such a complex task. We publicly share the current version of the test dataset (called SCIERC-AECO) in the github: https://github.com/zavavan/sperty/blob/main/datasets/scierc_ aec/scierc_aec_test.json and plan to release extended versions in the future.We used two random samples of respectively 3 and 10 sentences as example annotations for the few-shot LLM prompts described below. 
experimental setups: 
 The full-stack Relation Extraction task consists of generating, for an input token sequence 𝑋 = 𝑥 0 , ..., 𝑥 𝑛 : a) a set of tuples 𝐸 = < (𝑥 𝑖 , ..𝑥 𝑗 ), 𝑇 𝑒 > of typed token sub-sequences of 𝑋, with 0≤𝑖, 𝑗≤𝑛 and 𝑇 𝑒 ∈ 𝑇 𝐸 being a label belonging to the set 𝑇 𝐸 of entity labels; b) a set 𝑅 of triples We re-trained SpERT on SciERC training set (1861 sentences) using SciBERT (cased) embeddings [20] . When tested over out-of-domain SCIERC-AECO data though, SpERT performance degrades drastically. First row in Table 1 shows Micro-average F1 scores on SCIERC-AECO for entity extraction (NER), relation detection without argument entity classification (RE) and relation detection considering entity classification (RE_w/NEC), respectively.In order to test few-shot learning capability of LLMs for training data generation, we experiment on schema-constrained instruction prompts sent to the Chat Completion endpoint of the OpenAI gpt-3.5-turbo-0125 (ChatGPT) API [21] . The context length of the model is approximately 4096 tokens. We test a few variations of this Prompt schema. Namely, we optionally add plain text descriptions of the Entity and Relation types to increase LLM comprehension of the annotation task. Then, we increment the number of prompt examples to 𝐾 = {3, 10}. Full prompt instructions and examples can be found at https://github.com/zavavan/sperty/tree/main/prompts.All ChatGPT API calls were run with temperature parameter set to 0, top_p token probability mass sampling set to the default value 1, token frequency_penalty and presence_penalty set to 0 default value.For each prompt configuration, we query ChatGPT on 3373 sentences from AECO abstracts, parse ChatGPT response messages into a training dataset and train SpERT with SciBERT (cased) embeddings 6 on it. Finally, we evaluate each model on the SCIERC-AECO test set. We also test for training SpERT with a merge of ChatGPT-generated data with original (out-of-domain) SciERC dataset (3800 sentences). 
results and discussion: 
 Table 1 reports Micro-average F1 scores when training with ChatGPT-generated data using the basic instruction Prompt (Schema Prompt) and the prompt enriched with schema description (Schema/Descr Prompt), for different values of 𝐾 prompt examples. The last two rows report performance when training on merged ChatGPT-generated and SciERC data.At a first glance the LLM seems to fully comply with the structural requirements of the annotation task, consistently generating schema-based output. In some cases, it "semantically" manipulates the input text (average 2% occurrence) so as to make the output annotations not directly usable for sequence labeling. For example, from the sentence "The carbon emissions throughout the entire life cycle of the building have been reduced by 20.99%. " it generates an entity not anchored in text(T1;Task;Carbon emissions reduction) 7 .Overall, the performance level is not outstanding across all configurations, considering that the same model architecture is achieving a F1 measure of 2-3 factors higher when trained on in-domain manually curated data (SciERC) of comparable size. This may be due to the model degrading its generalization performance by learning from noisy labels [22] , which is confirmed by observing that the best results are obtained by adding ChatGPT generated labels to curated out-of-domain SciERC labels. By considering only LLM-generated data, most configurations slightly outperform the baseline for NER while only one does it for RE, indicating that this is an harder task for LLM few-shot learning with respect to NER.Adding explicit Task definitions and increasing the number of few-shot examples both consistently raise the performance with respect to all metrics, with the second finding seemingly in contrast with what reported in [15] . The results indicate that the quality of the LLM-generated annotations may not be sufficient to support domain customization of a RE model from ground up. However, when combined with curated out-of-domain labels it can boost the performance on the new domain significantly. 
conclusions: 
 Overall, the research highlights the potential of using ChaptGPT for optimizing local, lowersized models, which can be a more cost-effective strategy than relying on direct use of LLMs for inference in production settings.Future work might include expanding the test set and conducting more extensive tests to further validate the approach, also considering other domains than AECO. In addition, in the future we plan to experiment with GPT-4 for data generation rather that ChatGPT, leveraging its powerful capabilities to improve the quality of synthetic data, such as in the powerful LLaVA multi-modal model [23] .Object of further investigation and experiments will involve also the use of the latest advances in open-source LLMs, such as by employing open-source models like Mistral-7B-OpenOrca8 , Nous Hermes Mixtral9 , and Llama-3-70B 10 , to explore their potential in relation extraction tasks. This could involve comparing the performance of these models directly with the current approach, as well as exploring their capabilities in generating high-quality synthetic data for fine-tuning smaller models like SpERT. 
 
 Appendix: https://nlp.cs.washington.edu/sciIE/annotation_guideline.pdfhttps://docs.openalex.org/https://github.com/explosion/spacy-models/releases/tag/en_core_web_trf-3.6.1This corresponds to classical Cohen κ inter-rater agreement, in tasks like NER where the number of negative cases is undefined.Due to OpenAI API maximum request tokens limit, these are sent in batches of 10 sentences each.We trained for 20 epochs with 0.1 dropout rate. Experiments were run on NVIDIA A100-SXM4-40GB GPU device.In a few other cases, annotation labels out of the schema are assigned to Entity and Relations.Mistral-7B-OpenOrca, https://huggingface.co/Open-Orca/Mistral-7B-OpenOrcaNous Hermes Mixtral, https://huggingface.co/NousResearch/Nous-Hermes-2-Mixtral-8x7B-DPOLlama-3-70B, https://huggingface.co/meta-llama/Meta-Llama-3-70B 



Reviews: review: title: title: The research lacks some solid justification
main_review: main_review: The main problem reading the paper I have is the following: the authors use a LLM to generate training instances that will later be used to train a classifier that performs relation extraction.
I don't understand (or I could not find any justification about it) why the training instances are needed to be generated from a LLM.
If the ultimate goal is to perform relation extraction, why don't the authors use the LLM to directly tackle that task?
Results and discussions of the paper are therefore built on top of this problem. If the authors can find a justification of why they need a training set generated from a LLM with respect to use directly a LLM to solve the relation extraction task then I will be happy to reconsider my score.
title: title: Interesting use case, would benefit from a deeper analysis and discussion
main_review: main_review: This short paper explores the utilization of in-context learning capabilities of Large Language Models (LLMs) to perform schema-constrained data annotation for generating training instances. These instances are used to train a Transformer-based relation extraction model, specifically applied to titles and abstracts in the Architecture, Construction, Engineering, and Operations (AECO) domain. The study demonstrates the potential for domain adaptation of a science KG generation model using a few-shot learning strategy with minimal expert annotation and evaluates against a baseline using SpERT, a model for joint entity and relation extraction that takes as input pre-trained embeddings. 

The paper would benefit from a more detailed explanation of the approach. This includes a clearer step-by-step description of how the schema-constrained data annotation process is conducted, more details on the structured prompts that include Relation and Entity type text definitions, and the exact setup of the few-shot learning strategy. It is also likely that other prompts were tested, but did not perform as well. A brief discussion on this would be interesting.

I understand there is supplementary material covering some of this, but considering the paper is not at the length limit (and that some aspects can be summarized, see below) there should be sufficient space to directly address these issues in the main text.

It would be interesting to understand whether the better performance seen when combining ChatGPT-generated data with original (out-of-domain) SciERC dataset is due to a larger number of training instances (it appears that the size is more than doubled with the merge), or due to some synergistic effect between in-domain and out-of-domain data.

The “Results and Discussion” section is currently underdeveloped. Expanding this section to include detailed reflections on the challenges faced, how they were addressed, and insights gained during the experimentation would provide valuable insight for future research. There is a good potential to present this work as a “lessons learned” use case that may guide others that face similar challenges in other domains.
title: title: A Few-Shot Approach for Relation Extraction Domain Adaptation using Large Language Models
main_review: main_review: The paper presents a novel approach to domain adaptation in relation extraction using large language models (LLMs). The authors propose leveraging the in-context learning capabilities of LLMs to perform schema-constrained data annotation, collecting in-domain training instances for a Transformer-based relation extraction model deployed on titles and abstracts of research papers in the Architecture, Construction, Engineering, and Operations (AECO) domain.

One of the key strengths of this paper is its focus on a specific and relevant problem in knowledge graph generation: the need for domain-specific models that can accurately extract relations from text. The authors provide a detailed description of their approach, including the use of structured prompts and minimal expert annotation. This makes the approach more accessible and applicable to other researchers.

Authors use a large dataset of 476k research articles in the AECO area published between 2010 and 2023, retrieved from OpenAlex. This provides a solid foundation for evaluating the performance of the proposed method. However, it would be helpful to see more details about the preprocessing steps and how the data was prepared for training. Authors do send github link but it would help to see the details.

However, there are some areas where the paper could be improved. For example, the authors state that the method is effective for in-domain manually curated data, but they do not provide enough information about how they evaluated the performance of their method compared to existing approaches. It would be useful to see a more detailed comparison with other methods.

Overall, this paper presents an interesting and promising approach to domain adaptation for relation extraction. While there are some limitations in terms of evaluation and comparison, the idea of the paper for data augmentation and schema-constrained annotation is a valuable contribution to the field. Further work is needed to fully evaluate the effectiveness of the approach and compare it with other methods.