How to Unleash the Power of Large Language Models for Few-shot Relation Extraction?
Year: 2023

Abstract:
Scaling language models have revolutionized widespread NLP tasks, yet little comprehensively explored few-shot relation extraction with large language models. In this paper, we investigate principal methodologies, in-context learning and data generation, for few-shot relation extraction via GPT-3.5 through exhaustive experiments. To enhance few-shot performance, we further propose task-related instructions and schema-constrained data generation. We observe that in-context learning can achieve performance on par with previous prompt learning approaches, and data generation with the large language model can boost previous solutions to obtain new state-of-the-art few-shot results on four widely-studied relation extraction datasets. We hope our work can inspire future research for the capabilities of large language models in few-shot relation extraction1.

text: Proceedings of The Fourth Workshop on Simple and Efficient Natural Language Processing (SustaiNLP), pages 190–200 July 13, 2023 ©2023 Association for Computational Linguistics

heading: 1 Introduction
text: Few-shot Relation Extraction (RE) appeals to many researchers in Natural Language Processing (NLP) due to the capability to extract textual information where only a few support examples are given (Han et al., 2018; Yang et al., 2021; Han et al., 2021a; Brody et al., 2021; Ma et al., 2023). Most previous works focus on fine-tuning (Soares et al., 2019; Ye et al., 2022) or prompt-tuning (Chen et al., 2022; Han et al., 2021b) with relatively small language models, e.g., RoBERTa (Liu et al., 2019). Recently, with the scaling of model size and corpus size, large language models (LLMs) such as ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023a) have demonstrated powerful abilities by demonstrating only a few instances, a.k.a In-Context Learning (Dong et al., 2023). Although LLMs have achieved remarkable results in many NLP tasks, their po-
∗ Corresponding author. 1Code is available in https://github.com/zjunlp/
DeepKE/tree/main/example/llm.
tential in few-shot relation extraction has not been fully explored yet.
In this paper, we take GPT-3.5 (OpenAI, 2023b) as an exemplary LLM to investigate how to maximize the utilization of LLMs for the few-shot relation extraction task with in-context learning and data generation. Different from text classification, the relation extraction task contains rich predefined schemas (e.g., entity and relation type constraints) and a relatively large and complex classification space with noisy data. We further design two simple-yet-effective strategies to unleash the power of large language models better: task-related instructions and schema-constrained data generation. We conduct exhaustive experiments on four well-known relation extraction datasets. Empirical results indicate that LLMs can potentially be advantageous to few-shot relation extraction and boost previous prompt learning performance.

heading: 2 Background

heading: 2.1 Few-shot Relation Extraction
text: The relation extraction task aims to extract the relationship between head and tail entities within a plain context. Specifically, one instance for the relation extraction task consists of a context x = {x1, x2, ..., h, ..., t, ..., x|x|}, head and tail entity mentions h and t, entity types th and tt, and the relation y ∈ Y between h and t, where Y is the set of candidate relations. RE systems will predict y given x,h, t, th and tt. For few-shot relation extraction, fine-tuning pre-trained language models (PLMs) is a direct solution (Han et al., 2019; Yamada et al., 2020; Joshi et al., 2020; Lyu and Chen, 2021; Zhou and Chen, 2022). To alleviate the gap between pre-training objectives and downstream applications, prompt tuning has recently been applied to relation extraction, especially for low-resource scenarios (Chen et al., 2022; Han et al., 2021b, 2022). Most of those approaches
190
utilize relatively small language models (RoBERTa (Liu et al., 2019), GPT2 (Radford et al., 2019)), demonstrating empirical success regarding fewshot relation extraction performance. To date, large language models have demonstrated powerful abilities by prompting a few instances without tuning (Ding et al., 2022); however, the power of LLMs for few-shot relation extraction is little known.

heading: 2.2 Large Language Models
text: Large language models, trained with exceedingly large corpora and often with a great number of parameters (≥10B), have achieved excellent performance in numerous downstream NLP tasks (Taylor et al., 2022; Zhang et al., 2022; Zeng et al., 2022; Chowdhery et al., 2022; Ouyang et al., 2022). Compared to relatively small language models (SLMs), LLMs are usually not open-source and can not be fine-tuned, which is challenging for downstream task adaptation. Therefore, in-context learning (Brown et al., 2020) is proposed to utilize prompts with a few demonstrations for few-shot learning. Previous studies (Yoo et al., 2021; Wang et al., 2021) have investigated using LLMs for text classification and generation. In this work, we take the first step to study few-shot RE with large language models, which brings new challenges and insights.

heading: 3 LLMs for Few-shot Relation Extraction
text: In this section, we introduce two strategies to utilize LLMs for relation extraction: 1) in-context learning (§3.1); 2) data generation (§3.2) with LLMs, as shown in Figure 1.

heading: 3.1 In-Context Learning with LLMs
text: The first strategy applies in-context learning (ICL) by providing LLMs with demonstrations in the
prompt to elicit comprehension of the relation extraction task from LLMs. To this end, specific and compelling prompts for RE with demonstrations are manually constructed and designed to instruct LLMs to understand the relation extraction task and how to execute relation extraction. Considering aspects and characteristics of the relation extraction task, including task definition, candidate relation (label) words, entity types (schemas) and so on, we design prompts of different articulation and complexity to investigate how prompts help LLMs release the power of few-shot RE. First, TEXT PROMPT only contains essential elements for RE, including relation categories, contexts, and corresponding head and tail entities. Inspired by the fantastic performance of InstructGPT (Ouyang et al., 2022) and ChatGPT (OpenAI, 2022), we design the task-related instruction describing the relation extraction task and add it to the prompt, which is named INSTRUCT PROMPT. Meanwhile, according to previous few-shot RE works (Zhou and Chen, 2022), entity types (schemas) are helpful; therefore, we also explore the effectiveness of schemas in prompts.

heading: 3.2 Data Generation with LLMs
text: To complement the scarcity of labeled data, we introduce another strategy: data generation via LLMs. Specifically, we utilize specific prompts with descriptions of data forms to guide LLMs to generate more in-domain labeled data autonomously, which is subsequently employed to fine-tune a relatively small language model with existing few-shot labeled training data. We design the prompt to tell the essential components (x,h, t, th, tt and y) of one RE training instance and show few-shot instances as demonstrations to teach LLMs to comprehend
features of labeled RE data. Note that schemas, such as types of relations and entities, are significant structural information in RE data. Therefore, we propose schema-constrained data generation by adding entity types as schema guidance to the prompt (in Figure 1) to boost performance. Then, the prompt is utilized to guide LLMs to create augmented relation extraction data that are converted into the expected format for future usage.

heading: 4 Experimental Setups

heading: 4.1 Methods and Datasets
text: GPT-3.5 is utilized via OpenAI API2 as the large language model in our experiments. We implement experiments on four relation extraction datasets, including TACRED (Zhang et al., 2017), TACREV (Alt et al., 2020), RE-TACRED (Stoica et al., 2021) and SciERC (Luan et al., 2018). Compared with the LLM, six baselines methods are conducted via relatively small models (details in Appendix A).

heading: 4.2 Few-shot Settings
text: K instances per relation (K-shot) are sampled for training and validation. For all baselines, we use randomly sampled 8-shot and 16-shot datasets
2https://platform.openai.com/docs/models/ gpt-3-5
for training and validation. As for in-context learning, because GPT-3.5 has the limitation of maximum request tokens (4097 tokens) and the series of TACRED datasets have more than 40 relations, one-shot demonstrations can only be used, and the one-shot performance is reported in Table 1. For the same reason, to generate more labeled data for each relation independently, only three demonstrations for the relation are added to the prompts.
In-context learning is implemented on the four whole test sets. Different demonstrations are randomly sampled from the shuffled training set every time to avoid effects from permutations of demonstrations (Lu et al., 2021). As for data generation, generated data from GPT-3.5 and original few-shot training data are combined to fine-tune two baselines, TYP Marker (Zhou and Chen, 2022) and KnowPrompt (Chen et al., 2022). Using different shots of generated data will lead to different results. Therefore, we increasingly add generated k-shot (k ∈ {8, 16, 32, 48}) data to the original 8-shot and 16-shot training data respectively and report the best performance over k in Tabel 1. More details are shown in Appendix A.3.

heading: 5 Results and Discussion

heading: 5.1 Main Findings for Relation Extraction
text: In-context learning on LLMs can achieve comparable performance for RE with tuning relatively small PLMs. From Table 1, we notice that ICL with only one-shot demonstrations can obtain competitive performance with full parameter tuning-based prompt learning baselines. Using LLMs via ICL does not necessitate any parameter updates, which contains the potential value of mak-
ing models scenario-adaptable, unlike supervised learning requiring parameter optimization.
Data generation with LLMs can boost previous solutions to obtain new state-of-the-art few-shot RE results. We find that previous baselines can significantly improve with 10.7% for 16-shot in SciERC and 6.6% for 16-shot in RE-TACRED by simply using generated data from GPT-3.5 in Table 1. To be noted, data generation is a simple yet effective approach to elicit the power from the LLM to previous methods, and we demonstrate that using schema-constrained generation with LLMs can benefit all previous approaches with SLMs.

heading: 5.2 Prompts in In-context Learning with LLMs
text: Instructions and schemas play an essential role in in-context learning for RE with LLMs. From Table 2, we notice that the model with INSTRUCT PROMPT obtains better performance than TEXT PROMPT in most cases, indicating taskrelated information indeed helps to unlock more ability of LLMs for RE. Aberrant results are shown in TACRED and TACREV because incorrectly labeled demonstrations from the two datasets will violate the correct instruction fed into LLMs, which confuses LLMs and results on worse performance than ICL without the instruction. Moreover, adding schema information obtains much better performance, exhibiting the importance of pre-defined structural information for relation extraction.
More demonstrations, counter-intuitively, may not lead to performance improvement for RE with LLMs. We find performance will not improve even drop and the gap between INSTRUCT PROMPT and TEXT PROMPT becomes relatively smaller as the number of in-context demonstrations increases from Figure 2. We argue that there may be two reasons: 1) it is challenging to select rep-
resentative demonstrations; 2) it is non-trivial for LLMs to understand structure prediction tasks with more large output (relation) space. More case studies for GPT-3.5 can be found in Appendix B.1.

heading: 5.3 Utility of Generated Data from LLMs
text: Combining data generated from LLMs with original training data can yield better RE performance than from traditional data augmentation approaches. We compare data generation through the LLM with previous widely used data generation approaches, such as substituting words in training sets with WordNet’s synonyms and contextual word embedding in Figure 3 (details in Appendix A.3). Data generation with LLMs can obtain better performance than all others, indicating guiding LLMs to generate data is an effective method to compensate for the lack of labeled data.
Using more and more generated data from LLMs can only boost RE performance to a certain extent, not continuously better. From Figure 4, we observe that with more generated data, the result climbs up first and then declines, and is always higher than without generated data. We think low-quality generated data introduces much noise in the training course, according to the analysis on generated data in Appendix B.2, and LMs may have an anti-noise capacity (Song et al., 2020).

heading: 6 Discussion and Conclusion
text: In this paper, we take the first step to investigate how to utilize the large language model for few-shot relation extraction. We observe that task-related information, including instructions or
schemas, helps to elicit the capability of LLMs and boost few-shot relation extraction performance. At this stage, using LLMs to generate data may be a simple yet effective solution to enhance the power of foundation models (relatively small PLMs) for practical applications. We hope this work can deliver the benefits of using LLMs for the NLP community. Note that LLMs can make predictions only based on contexts combined with a few training examples as demonstrations. We argue that it has the potential to design sophisticated human-readable prompts for scenario-adaptable (e.g., low-shot and any domains) relation extraction.

heading: Acknowledgment
text: We would like to express gratitude to the anonymous reviewers for their kind comments. This work was supported by the National Natural Science Foundation of China (No.62206246), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Ningbo Natural Science Foundation (2021J190), and Yongjiang Talent Introduction Programme (2021A-156-G), CAAIHuawei MindSpore Open Fund.
Limitations
Despite our best efforts, there may still be some limitations remaining in this paper.
LLMs: Due to the limited budgets, we can not afford all kinds of LLMs, so we only evaluate GPT-3.5 (text-davinci-003). We will try to investigate relation extraction with more LLMs, such as OPT (Zhang et al., 2022), GLM-130B (Zeng et al., 2022), or code language models (Bi et al., 2023) like Codex.
Other Methods to utilize LLMs: There are several other techniques to leverage LLMs, such
as black-box optimization (Sun et al., 2022) and feature-based learning (Lang et al., 2022); however, we find that most of those approaches cannot directly be applied to relation extraction due to the large label space and complex schema structures. We leave these for future work to leverage other methods with LLMs for relation extraction.
Datasets: We only evaluate four relation extraction datasets and will try to investigate relation extraction performance with LLMs on more diverse datasets across different domains and languages.

heading: A Experimental Details
text: A.1 Datasets TACRED3 is a widely used RE dataset. It has 42 relation labels, including no_relation, meaning no relation is found. TACREV4 includes the same training set and relabeled development and test sets from TACRED. RE-TACRED5 is a re-annotated version of TACRED with 40 relations. SciERC6 has seven relation categories and is constructed in the scientific domain. All datasets are derived from their official webs without modification, including contents and train/test/dev splits.
A.2 Baselines We compare LLMs with recent baseline methods using relatively small models. 1) Normal finetuning methods: SpanBERT (Joshi et al., 2020), a span-based PLM; LUKE (Yamada et al., 2020), pre-trained contextualized representations of words and entities based on the bidirectional transformer; GDPNet, a gaussian dynamic time warping pooling net able to select important words for relation prediction; TYP Marker (Zhou and Chen, 2022), fine-tuning with entity typed markers. 2) Generative method: TANL (Paolini et al., 2021), framing a structured prediction language task as a translation task between augmented natural languages. 3) Prompt-tuning methods: KnowPrompt, knowledge-aware continuous prompt-based tuning with synergistic optimization.
A.3 Implementation Details Generated data with existing training data is then evaluated on KnowPrompt. Data augmentation
3https://nlp.stanford.edu/projects/tacred/ 4https://github.com/DFKI-NLP/tacrev 5https://github.com/gstoica27/Re-TACRED 6http://nlp.cs.washington.edu/sciIE/
methods with Word-Net’s synonyms and contextual word embedding are achieved by nlpaug7. The parameter temperature in OpenAI API is set to 0 for precision in ICL and 1 for generating diverse RE data. One NVIDIA GeForce RTX 3090 GPU with 24GB memory is employed to run all experiments. We rerun the official code of baselines with their original settings except on the SciERC dataset. Due to the vertical domain of SciERC, SciBERT (Beltagy et al., 2019) is used in TYP Marker and KnowPrompt for fairness. And for another three datasets, RoBERTa-large is utilized in TYP Marker and KnowPrompt.

heading: B Case Analysis
text: B.1 Wrong Cases from ICL From Table 4, we notice that some RE instances are challenging for LLMs, and there are several limitations with LLMs: 1) LLMs are not good at clearly distinguishing the order between head and tail entities. 2) The same mention of head and tail entities will confuse LLMs. 4) If the distance between head and tail entities in the context is long, it is difficult for LLMs to decide the relation correctly. 5) Semantically-similar relation label words and entity mentions will puzzle LLMs because their embeddings are similar. 6) LLMs cannot afford very long instances since there is a large label space for relation extraction. 7) LLMs may mostly fail to extract those ambitious or wrongly labeled relations; those are also challenging for humans. More highquality demonstrations may help mitigate these issues. And we think it is necessary to develop step-by-step (Chat-style) approaches with LLMs to extract limited relations in one stage.
B.2 Generated Data from LLMs There are some cases for generated data from GPT3.5 in Table 5. Through human checks on 100 generated samples per dataset, about 78% generated data are corrected labeled and of a high quality (85% for TACRED, 82.5% for TACREV, 72% for RE-TACRED, 75% for SciERC). Meanwhile, we add generated data and original gold training data respectively to 8-shot datasets and fine-tune KnowPrompt, we evaluate the quality of generated data as shown in Table 3. We observe that labeled data generated by GPT-3.5 are mostly correct. As for TACRED and TACREV, generated data achieve more improvements than gold labeled data. Since
7https://github.com/makcedward/nlpaug
there are many incorrect labeled data in TACRED and TACREV (Zhang et al., 2017; Alt et al., 2020), we think better performance results from GPT-3.5’s help. However, we also find that Some generated data from GPT-3.5 are of less quality than gold data. As for RE-TACRED and SciERC, using more gold data perform better than generated data. Through human checks, some generated samples are too short and concatenated by some semantically irrelevant sentences. Meanwhile, big performance’s difference on SciERC shows GPT-3.5 is not good at vertical domains such as science.

heading: Future Work
text: We leave these for future work to leverage other methods with LLMs for relation extraction. Datasets: We only evaluate four relation extraction datasets and will try to investigate relation extraction performance with LLMs on more diverse datasets across different domains and languages