Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors
Year: 2023

Abstract:
Recent work has shown that fine-tuning large language models (LLMs) on large-scale instruction-following datasets substantially improves their performance on a wide range of NLP tasks, especially in the zero-shot setting. However, even advanced instructiontuned LLMs still fail to outperform small LMs on relation extraction (RE), a fundamental information extraction task. We hypothesize that instruction-tuning has been unable to elicit strong RE capabilities in LLMs due to RE’s low incidence in instruction-tuning datasets, making up less than 1% of all tasks (Wang et al., 2022). To address this limitation, we propose QA4RE, a framework that aligns RE with question answering (QA), a predominant task in instruction-tuning datasets. Comprehensive zero-shot RE experiments over four datasets with two series of instruction-tuned LLMs (six LLMs in total) demonstrate that our QA4RE framework consistently improves LLM performance, strongly verifying our hypothesis and enabling LLMs to outperform strong zero-shot baselines by a large margin. Additionally, we provide thorough experiments and discussions to show the robustness, few-shot effectiveness, and strong transferability of our QA4RE framework. This work illustrates a promising way of adapting LLMs to challenging and underrepresented tasks by aligning these tasks with more common instruction-tuning tasks like QA.1

heading: 1 Introduction
text: Large language models (LLMs) (Brown et al., 2020; Chowdhery et al., 2022; Zhang et al., 2022) have been shown to achieve impressive performance on many NLP tasks. Using the in-context learning paradigm, without any parameter updating, LLMs are able to achieve comparable performance with small language models (LMs) fine-tuned on thousands of examples (Liu et al., 2022; Min et al.,
1Code and data are available at https://github.com/OSUNLP-Group/QA4RE.
2022a; Liang et al., 2022).2 More recently, finetuning LLMs on datasets containing thousands of downstream tasks transformed into an instruction following format (i.e., instruction-tuning) has been shown to improve LLMs considerably across the board, especially in zero-shot setting (Iyer et al., 2022; Ouyang et al., 2022; Chung et al., 2022).
We examine the capability of LLMs in identifying the relationship between entities in a sentence, i.e., relation extraction (RE), a fundamental task in information extraction. Recent work (Jimenez Gutierrez et al., 2022) has found that LLMs underperform fine-tuned small LMs for RE in the biomedical domain. Our results on general domain RE in Fig. 1 reveal that even two of the most advanced instruction-tuned LLMs, FLANT5 XXL (Chung et al., 2022) and text-davinci-003 (Ouyang et al., 2022), fail to outperform the stateof-the-art (SoTA) zero-shot RE method based on small LMs (Sainz et al., 2021).
We hypothesize that the limited relation extraction capability of instruction-tuned LLMs could be
2We regard LMs with less than 1B params as small.
794
a byproduct of the low incidence of RE tasks in instruction-tuning datasets (Ouyang et al., 2022; Sanh et al., 2022; Chung et al., 2022; Wang et al., 2022).3 To address the low incidence issue, we propose the QA4RE framework, which aligns RE with multiple-choice question answering (QA), a task that appears much more frequently in most instruction-tuning datasets—around 12-15% of all the tasks in both Wang et al. (2022) and Ouyang et al. (2022). Specifically, by casting the input sentence as a question and possible relation types as multiple-choice options (Fig. 2), LLMs are able to perform RE by selecting the option representing the correct relation type.
Thorough evaluations on four real-world relation extraction datasets and six instruction-tuned models from two different series (OpenAI GPT3.5 and FLAN-T5 (Chung et al., 2022)) show that QA4RE brings significant gains over the standard RE formulation on, validating its effectiveness and our hypothesis concerning the low incidence of RE. More specifically, our framework enables textdavinci-003 and FLAN-T5-XXLarge to achieve an average of 8.2% and 8.6% absolute improvements in F1, respectively. For the first time, an LLM is able to outperform prior small-LM-based SoTA in the zero-shot setting by a large margin. In-depth analyses further demonstrate the robustness and few-shot effectiveness of QA4RE. More importantly, our framework has been proven to be effectively transferable on instruction-tuned models with various sizes, ranging from 80M to 175B. Our contributions are summarized as follows: (1) We systematically investigate instruction-tuned LLMs on four real-world relation extraction datasets and note that their limited performance on RE might stem from the low incidence of RE tasks in instruction-tuning datasets. (2) We reformulate RE as multiple-choice QA in an effort to appropriately leverage QA’s much higher prevalence in instruction-tuning datasets and achieve significant improvements on six recent instruction-tuned LLMs, significantly outperforming previous SoTA zero-shot RE methods based on small LM for the first time. (3) In addition, we demonstrate our QA4RE method’s robustness to diverse prompt designs as well as its promising results in the few-shot setting. (4) Finally, we show the effectiveness of QA4RE
3RE-like tasks are <0.5% of the largest available instruction dataset (Wang et al., 2022); see Appendix A for details.
framework is transferable and consistent on various instruction-tuned models with different sizes from 80M to 175B. Our study illustrates the potential of aligning infrequent and challenging tasks with frequent instruction-tuning tasks and can guide others in exploring this direction.

heading: 2 Related Work
text: Instruction Tuning. Large language models originally obtained impressive zero and few-shot performance by leveraging self-supervised next token prediction at massive scales. More recently, supervised fine-tuning on a large number of downstream tasks has been shown to improve LLM accuracy, robustness, fairness, and generalization to unseen tasks (Ouyang et al., 2022; Iyer et al., 2022; Wei et al., 2022a; Chung et al., 2022; Sanh et al., 2022). Several strategies have been developed to align LLMs to human instructions including Reinforcement Learning from Human Feedback (RLHF) (Ouyang et al., 2022) as well as the more standard language modeling objective, used to finetune LLMs on a wide range of tasks reformulated as instruction following tasks (Iyer et al., 2022; Wei et al., 2022a; Chung et al., 2022; Sanh et al., 2022).
Eliciting LLM Abilities. The high cost and increasingly private nature of LLM pre-training make it quite challenging to conclusively determine how different pre-training techniques bring about different LLM capabilities. Many factors involved in pre-training such as simple self-supervised scaling, code or multi-lingual text pre-training (Chowdhery et al., 2022; Chen et al., 2021; Chung et al., 2022) as well as the distinct versions of instruction-tuning mentioned above (Ouyang et al., 2022; Iyer et al., 2022; Wei et al., 2022a; Chung et al., 2022), can interact in a wide variety of ways to unleash the abilities LLMs display. Nonetheless, Fu and Khot (2022) hypothesize that the use of code during pretraining seems to improve an LM’s reasoning ability, evidenced by the improved ability to leverage Chain-of-Thought prompting (Wei et al., 2022b) by models trained partially on code such as PaLM (Chowdhery et al., 2022), code-davinci-002 (Chen et al., 2021), and text-davinci-002/003 (Ouyang et al., 2022), compared to text-only models like text-davinci-001 and OPT-175B (Zhang et al., 2022). Additionally, instruction-tuning on a large set of tasks has been shown to improve generalization to unseen tasks, reduce the need for few-shot examples and improve accuracy and robustness
across many language tasks (Ouyang et al., 2022; Iyer et al., 2022; Chung et al., 2022).
Low-Resource Relation Extraction. Several reformulations of standard RE have enabled small LMs to achieve fairly strong performance in the zero and few-shot settings. Sainz et al. (2021) utilize small LMs fine-tuned on natural language inference (NLI) datasets to perform zero-shot RE by selecting the entity-filled relation template which is mostly entailed by the test sentence. Lu et al. (2022) frame RE as a summarization task and leverage generative models to summarize the relation between target entities in the sentence. Other lowresource RE methods augment prompt-tuning by using logical rules to create complex prompts from sub-prompts (Han et al., 2022) and injecting knowledge about entity types using learnable virtual tokens (Chen et al., 2022). Our current work uses several relation templates designed in these studies.
LLMs for Relation Extraction. In terms of exploring the RE capabilities of LLMs, most previous work has focused on investigating biomedical RE. Jimenez Gutierrez et al. (2022) report that LLMs underperform standard small LMs fine-tuning in the few-shot setting on a comprehensive set of biomedical RE datasets and show evidence that the poor handling of the none-of-the-above (NoTA) relation category is one of the major culprits. Furthermore, although a few RE-like tasks were included in Super Natural Instruction (Wang et al., 2022), these tasks constitute about 0.5% of the dataset and none of them were selected for model evaluation.

heading: 3 Methodology
text: In this section, we formally define the relation extraction problem and describe our multi-choice QA approach for the problem in detail.

heading: 3.1 Problem Statement
text: Relation extraction (RE) aims to extract the relationship between two given entities based on a specific sentence. More concretely, one relation example contains a sentence S as well as a head entity Eh and a tail entity Et within S. Given a relation example (S,Eh, Et), models are required to identify the relation between Eh and Et expressed in the S from a set of pre-defined relation types.

heading: 3.2 Relation Templates
text: Recent low-resource RE approaches (Sainz et al., 2021; Lu et al., 2022; Han et al., 2022) utilize relation-entailed templates as label verbalization (e.g., “per:city_of_birth” -> “{Eh} was born in the city {Et}”). As illustrated in Fig. 2 (left), the current SoTA method for low-resource RE (Sainz et al., 2021) utilizes manually constructed relation templates to reformulate the RE task as a natural language inference (NLI) task.
To ensure a fair comparison, we utilize the same templates developed in previous studies (Sainz et al., 2021; Lu et al., 2022) to generate answer options within our QA4RE framework. Furthermore, in Sec. 6.2 we discuss the possibility of directly applying the NLI formulation for RE in LLMs.

heading: 3.3 QA4RE Framework
text: As shown in Fig. 2 (right), we reformulate the relation extraction task as a multi-choice QA problem. By integrating the given head and tail RE entities (Eh and Et) into the relation templates and using them as multiple-choice options, LLMs are able to leverage extensive QA instruction fine-tuning which has dramatically improved recent models. Additionally, our method allows LLM to generate only an answer index instead of the verbalized relation as in previous work (Jimenez Gutierrez et al., 2022), also shown in Fig. 2 (center).
Type-Constrained Answer Construction. To transform RE into a multiple-choice question, for a given relation example (S,Eh, Et), we utilize sentence S as the context in standard QA and create options composed of pre-defined templates filled with Eh and Et entities. To fairly compare with previous work, we apply type constraints (when applicable) to eliminate options for relation types that are not compatible with the entity types of the head and tail entities. For instance, if the type of Eh is PERSON, the relation “org:country_of_headquarters” would be deemed invalid given that a person does not have headquarters.

heading: 4 Experiment Setup

heading: 4.1 Datasets
text: We evaluate our methods on four RE datasets: (1) TACRED (Zhang et al., 2017), (2) RETACRED (Stoica et al., 2021), (3) TACREV (Alt et al., 2020), and (4) SemEval 2010 Task 8 (SemEval for brevity) (Hendrickx et al., 2010). Following previous work (Sainz et al., 2021; Lu et al., 2022; Han et al., 2022; Chen et al., 2022), we report the micro averaged F1 with the none-of-the-above relation excluded. To keep OpenAI API costs under control, we randomly sample 1,000 examples from each dataset’s test split as our test set.

heading: 4.2 Baselines
text: Zero-Shot. For small LM-based models, we evaluate two low-resource SoTA RE baselines: (1) As shown in Fig. 2 (left), NLI (Sainz et al., 2021) reformulates RE as a natural language inference task and leverages several LMs fine-tuned on the MNLI dataset (Williams et al., 2018): BARTLarge (Lewis et al., 2020), RoBERTa-Large (Liu et al., 2019), and DeBERTa-XLarge (He et al., 2021). This method holds the SoTA perfor-
mance on both zero and few-shot RE. (2) Besides, SuRE (Lu et al., 2022) frames RE as a summarization task and utilizes generative LMs such as BART-Large (Lewis et al., 2020) and PEGASUSLarge (Zhang et al., 2020), achieving competitive results in few-shot and fully-supervised settings.
For the NLI approach (Sainz et al., 2021), we report performance using their own templates on TACRED and TACREV. As this method does not have templates for RETACRED and SemEval, we use the templates from the follow-up work, SuRE (Lu et al., 2022), on these two datasets instead. All the zero-shot methods, including those on LLMs, apply entity type constraints to reduce the relation label space. Since SemEval does not provide entity types, the above methods use all possible relations in every instance as the label space.
Few-Shot. Though our main experiments focus on zero-shot RE, we further explore our method’s capabilities by comparing their few-shot performance against several competitive small LM-based methods on the TACRED dataset.
The NLI baseline can be easily extended to the few-shot setting.4 Furthermore, we add (1) standard Fine-Tuning (Jimenez Gutierrez et al., 2022), (2) PTR (Han et al., 2022) using prompt-tuning with logical rules, and (3) KnowPrompt (Chen et al., 2022) using entity type knowledge via learning virtual tokens, all of which are initialized with RoBERTa-Large (Liu et al., 2019). For hyperparameter details, please refer to Appendix B.1.

heading: 4.3 QA4RE Implementation Details
text: Our QA4RE framework utilizes the same templates and type constraints developed by prior work (Sainz et al., 2021; Lu et al., 2022). In particular, we use SuRE (Lu et al., 2022) templates for our QA4RE approach on all 4 datasets since NLI (Sainz et al., 2021) templates were only designed for TACRED. For prompt engineering, we explore prompt formats and task instructions for vanilla RE and QA4RE in pilot experiments, using text-davinci-002 on a 250-example subset of the TACRED dev set. We then use the same task instructions and prompt format for all four datasets and LLMs. Please refer to Appendix B.2 and B.3 for prompt format and relation verbalization template details, respectively.
4SuRE can also be extended to the few-shot setting but we were unable to replicate their results with the code provided.
To systematically compare our QA4RE framework with the vanilla RE formulation, we evaluate them on two series of LLMs, resulting in seven models in total. In GPT-3.5 series LLMs, for LLMs accessible via Text Completion API (code-davinci002, text-davinci-002, and text-davinci-003), we follow previous work (Jimenez Gutierrez et al., 2022) and use the logit bias option to constrain token generation to relation labels for vanilla RE and option indices for QA4RE. Due to the fewer available control options for LLMs in Chat Completion API (gpt-3.5-turbo-0301), we only set the temperature as 0 and use the default system prompt.
We also examine open-sourced FLAN-T5 series LLMs (Chung et al., 2022) that are trained on a mixture of tasks (Sanh et al., 2022; Wei et al., 2022a; Wang et al., 2022). The 1,836 tasks utilized in training include less than 0.5% of RE-similar tasks, making FLAN-T5 series LLMs the ideal models for verifying our hypothesis. Specifically, we use XLarge (3B) and XXLarge (11B) models and adopt the same prompts and greedy decoding strategy as GPT-3.5 series LLMs to ensure a fair comparison.

heading: 5 Results

heading: 5.1 Zero-Shot Results
text: Our main experimental results on four relation extraction datasets can be found in Tab. 1. We have the following observations from our results: (1) By reformulating RE as QA, our framework
improves upon the vanilla RE formulation on all the LLMs and most datasets, making them much stronger zero-shot relation extractors. In particular, text-davinci-003 and FLAN-T5 XL and XXL are able to outperform the prior SoTA, NLIDeBERTa, by a large margin. One thing worth noting is that QA4RE brings the largest gain on the best LLM in each series (text-davinci-003 and FLAN-T5 XXL), showing that stronger LLMs may benefit more from our framework.
(2) The two FLAN-T5 LLMs in Tab. 1 benefit significantly from our QA4RE framework. Moreover, consistent and substantial improvements can also be observed in other FLAN-T5 models and the full test set, as discussed in Sec. 6.3 and Appendix C. Considering that relation extraction tasks account for less than 0.5% of the instruction tasks used to train FLAN-T5 models, these findings strongly support our hypothesis that aligning underrepresented tasks with more common instruction-tuning tasks, such as QA, unlocks LLMs’ ability to solve low-frequency tasks.
(3) The SemEval dataset poses a significant challenge for all baselines given its lack of typeconstraints, particularly for SuRE (Lu et al., 2022). With such a large search space, generative LMs without fine-tuning tend to summarize all examples into NoTA relation, resulting in its systematic failure. It should be noted that without type constraints, the RE problem becomes a 19-choice
question answering task in our QA4RE framework. Despite this, our method still demonstrates substantial improvements for LLMs, particularly for text-davinci-003 and FLAN-T5 XXL.

heading: 5.2 Robustness to Verbalization Templates
text: For our experiments, we utilize manually written relation templates from previous work (Sainz et al., 2021; Lu et al., 2022). However, Lu et al. (2022) note that model performance may vary significantly with template design. Thus, to investigate the robustness of models to different templates, thorough experiments are conducted with four different templates, described in detail in Appendix B.3, across all zero-shot methods on the TACRED dataset. Tab. 2 shows results comparing these four templates on all methods used in our main experiments, including vanilla RE as a template-free reference.
From Tab. 2, we observe the following: (1) Our method consistently outperforms small LM baselines and the vanilla RE framework, regardless of the template. It is worth noting that even with templates that are constructed with label name information only and no expert knowledge (TEMP3 and TEMP4), our QA framework still performs better than vanilla RE, indicating the effectiveness and consistency of our QA framework. (2) NLI and SuRE performance is largely template dependent. When using carefully crafted highquality templates (TEMP1 and TEMP2), several LM-based NLI methods outperform text-davinci003 with vanilla RE. However, when equipped with templates created without expert knowledge (TEMP3 and TEMP4), the performance of both NLI and SuRE deteriorates dramatically. In contrast, QA4RE is more robust to variation in verbalization templates, reducing trial-and-error development efforts as well as making it more readily transferred to settings where obtaining quality templates may
be limited due to the high cost of expert annotations, such as the biomedical or financial domains.

heading: 5.3 None-of-the-Above Relation Evaluation
text: The none-of-the-above (NoTA) relation (Gao et al., 2019; Sabo et al., 2021; Jimenez Gutierrez et al., 2022) is defined as the case where no relation of interest exists between the given entities. Jimenez Gutierrez et al. (2022) demonstrate that the earlier inferior performance of LLMs on RE tasks can be largely attributed to their inability to handle the NoTA relation. To evaluate the efficacy of zero-shot methods on NoTA relation, following previous work (Fei and Liu, 2016; Shu et al., 2017; Sainz et al., 2021), we apply NoTA-included macro F1 metric as well as micro and macro P vs. N (all positive relations vs. NoTA relation as binary classification) F1 metrics.
From Tab. 3, we observe that, when enhanced by our QA framework, text-davinci-003 achieves significant improvement in NoTA-included metrics, outperforming the small LM-based NLI methods. This further demonstrates the effectiveness of our framework, even in handling the challenging NoTA relation. It is worth noting that these superior results are achieved by simply adding an entity-filled NoTA relation template as an answer option for QA, without the additional thresholding requirements of previous methods (Sainz et al., 2021; Lu et al., 2022). This eliminates the need for additional hyperparameter searching, which can be tricky for low-resource settings.

heading: 5.4 Few-Shot Results
text: While zero-shot RE is our main focus, we also evaluate our method under the few-shot setting. Results are shown in Tab. 4. Due to budget limitations, we restrict our case study to the 4-shot scenario (i.e., 4 labeled examples per relation) with
the best-performing LLM in the zero-shot setting (text-davinci-003). After determining the optimal number of in-context examples searched on the dev set, we randomly select the examples with the same entity type constraints from the given train set.
Interestingly, vanilla RE is unable to obtain any improvement from labeled examples, suggesting that it is also limited in the few-shot setting. The limited performance shown by vanilla RE indicates that few-shot demonstrations might bias the model towards incorrect relations in the context rather than helping it perform the task more accurately.
Even employing our QA4RE framework, the few-shot text-davinci-003 does not outperform the DeBERTa-based NLI method (Sainz et al., 2021) when using their own templates (TEMP1). However, fine-tuning the NLI model on RE data can be brittle even with careful hyperparameter tuning, as evidenced by the unstable gains seen as more data is added for both TEMP1 and TEMP2. Furthermore, we find that few-shot NLI results when using TEMP2 drop substantially from TEMP1, suggesting that this approach also lacks robustness to templates in the few-shot setting. Thus, considering that our QA approach enables LLMs to obtain few-shot improvements over zero-shot results using random in-context learning example selection, obtains only around 2% lower performance than the best NLI model, and is robust to different template designs, our approach is competitive on few-shot RE and has the potential to achieve even stronger performance with more exploration. We leave further investigation on how to improve LLMs for few-shot RE to future work.

heading: 6 Discussions

heading: 6.1 Are Relation Templates All LLMs Need?
text: We conduct an ablation study to better understand how relation templates contribute to the performance improvement obtained by QA4RE. As illustrated in Fig. 3, we fill the relation verbalization templates with markers Entity 1 and Entity 2 as relation explanations, thereby presenting the expert knowledge from the templates to the LLM. Using the same templates and type constraints, we compare this framework (termed Vanilla+TEMP) with vanilla RE and QA4RE on the TACRED dataset and GPT-3.5 series LLMs.
As shown in Tab. 5, introducing relation explanations using the same templates does not result in consistent or significant performance improvement. In fact, adding extra information to the task instruction might make it more challenging for the LLM to understand the task. In contrast, using our QA4RE framework, we do not need to separately specify the entities of interest or relation explanations; they are both naturally embedded in the answer options. These ablation results show that the gains from QA4RE mainly come from the QA reformulation, not simply from the relation explanations/templates.

heading: 6.2 QA4RE vs. NLI4RE
text: Given the strong performance obtained by small LMs using the NLI reformulation of RE, we leverage this same formulation (Sainz et al., 2021) for LLMs (termed NLI4RE).5 More concretely, for each example, we use the LLM to predict whether
5We follow the NLI format from ANLI (Wang et al., 2022).
the given sentence (the premise) entails each answer option from the QA4RE formulation (the hypothesis). We allow the LLM to generate entailment, neutral, or contradiction for each sentencerelation pair. If the maximum probability of entailment among all possible positive relations is below the threshold of 0.5, the example will be classified as NoTA, as done by Sainz et al. (2021).
As shown in Tab. 6, when using the NLI formulation, text-davinci-003 surprisingly underperforms the vanilla RE formulation. The reason for its poor performance is two-fold: (1) The heuristically predefined threshold 0.5 is not ideal for LLMs and thus many positive predictions are classified as NoTA. However, it is also difficult to find a good threshold under the zero-shot setting. (2) Under NLI4RE, unlike vanilla RE or QA4RE, an LLM is not seeing the full relation space but assigning probabilities to each candidate hypothesis individually. The final prediction is thus more sensitive to the LLM’s bias over different relations.
NLI4RE also requires multiple inference runs for each relation example to evaluate all the candidate relations, incurring a significantly higher cost.

heading: 6.3 QA4RE & Model Size
text: To verify the effectiveness and transferability of our QA4RE framework on smaller instruction-tuned models, we further evaluate the FLAN-T5 Small
(80M), Base (250M), and Large (780M) on the full test set over four RE datasets. Tab. 7 shows our QA4RE framework can still bring considerable gains to instruction-tuned models with various sizes, even for the smallest one (80M). This demonstrates the effectiveness of QA4RE is transferable across various model sizes from 80M to 175B, considering the consistent improvements of QA4RE on several GPT-3.5 models.
In the FLAN-T5 series, larger models benefit more from our framework. However, we note that this trend does not continue when scaling up to much larger GPT-3.5 models. In fact, all GPT3.5 models except for text-davinci-003 benefit less from QA4RE than FLAN-T5 models. The smaller improvements of QA4RE on these models make their overall RE performance only comparable with models that are approximately 20 and 50 times smaller. This indicates that the wide variety of alignment strategies used by the GPT-3.5 series models discussed in Sec. 2 might not be universally more effective than standard instruction-tuning for improving model generalization on low-incidence tasks even when aligned to high incidence ones. Nevertheless, the strong improvement observed in the strongest models tested, text-davinci-003 and FLAN-T5-XXL, demonstrates the potential for QA4RE’s effectiveness to continue as models become even more capable in the future.

heading: 7 Conclusions and Future Work
text: In this work, we first show that even the most recent instruction-tuned LLMs underperform fine-tuned small LMs on the relation extraction (RE) task. To address this limitation, we reformulate RE into multiple-choice question answering (QA) with the purpose of leveraging a task that is widely cov-
ered in instruction-tuning datasets like QA, instead of RE, which is barely present in these datasets. Comprehensive experiments demonstrate that our QA4RE framework unlocks the power of LLMs as zero-shot relation extractors, especially for two recent LLMs (text-davinci-003 and FLAN-T5 XXL). We also conduct thorough experiments to explore the robustness and few-shot effectiveness of our method as well as study in what LLM training scenarios it is most effective.
In future work, we hope to explore additional underrepresented tasks in instruction-tuning that might be challenging for LLMs and could be successfully aligned with more widely adopted instruction-tuning tasks like QA. Additionally, we plan to continue exploring this line of work by leveraging our QA4RE framework for other LLMs such as the OPT-series (Zhang et al., 2022; Iyer et al., 2022) and PaLM (Chowdhery et al., 2022), which are not included in this work due to the limited computational resources and/or access.

heading: 8 Limitations
text: Even though our method helps unleash the power of six recent strong LLMs as zero-shot relation extractors, earlier LLMs without strong instruction tuning such as text-davinci-001 saw no improvements from our framework. Additionally, although we carry out comprehensive experiments on the zero-shot RE setting, our few-shot exploration is more limited. It is still unclear from our investigation whether including even more training examples can improve LLM’s RE performance and to what extent the same trends seen across GPT3 models in the zero-shot setting hold steady in the few-shot setting. We leave answering these questions for future work.

heading: 9 Ethics Statement
text: In this work, we propose a method to improve LLM performance on the important and fundamental task of relation extraction. We do not anticipate any ethical issues regarding the topics of this research.

heading: Acknowledgements
text: The authors would like to thank Renze Lou, colleagues from the OSU NLP group, and the anonymous reviewers for their valuable feedback. The authors would also like to thank Keming Lu for discussions and guidance on reproducing SuRE. This research was supported in part by NSF OAC
2112606, NIH R01LM014199, and Ohio Supercomputer Center (Center, 1987).

heading: B Experimental Details
text: B.1 Hyperparameters for Few-Shot Methods
In the few-shot setting, for each K, we randomly sample 3 times to obtain different training subsets, each of which will be used as in-context demonstrations for LLMs or used to train the small language models in baselines. Report results are averaged over the three subsets. To avoid over-estimating few-shot performance with too many dev examples (Perez et al., 2021), we use 100 randomly selected examples of dev set for all the hyperparameter searching.
For LLMs, we use the dev set to search for the optimal number of in-context examples as a hyperparameter from {1, 2, 5}. Then we randomly select the same type-constrained in-context examples from the given train set.
For all small LM-based baselines, we use their publicly available code and hyper-parameters for training. According to the original papers of NLI (Sainz et al., 2021) and SuRE (Lu et al., 2022), we use the checkpoints available online and hyperparameters reported for model training. Unfortunately, we were unable to reproduce SuRE results with default hyperparameters. For standard FineTuning (Jimenez Gutierrez et al., 2022), PTR (Han et al., 2022), and KnowPrompt (Chen et al., 2022), we perform a grid search over hyperparameters on dev with the range shown in Tab. 10.
We use 8 NVIDIA GeForce RTX 2080 Ti and 2 NVIDIA RTX A6000 to conduct all the experiments. The total GPU hours used and the cost for OpenAI API are listed in Tab. 11.
B.2 Prompts for LLMs As shown in Tab. 12, we list all templates used in this paper including vanilla + TEMP in Tab. 5, NLI4RE in Tab. 6, and vanilla as well as QA4RE in all experiments.
B.3 Relation Verbalization Templates In the relation verbalization template robustness experiment shown in Tab. 2, the differences between four templates are described below using the org:top_members/employees relation from TACRED benchmark as an example:
1. Concrete Examples: {Eh} is a chairman/ president/director of {Et}
2. Semantic Relationship: {Eh} is a high level member of {Et}
3. Straightforward: The relation between {Eh} and {Et} is top members or employees
4. Word Translation: {Eh} organization top members or employees {Et}
The first set of templates was written by Sainz et al. (2021), while the remaining three were explored by Lu et al. (2022). We use the templates from their official GitHub repositories.6 In addition, we further list relation verbalization templates used by all LLMs in our paper in Tab. 13, Tab. 14, and Tab. 15.

heading: C Full Test Results on FLAN-T5
text: We present the full test set results of all four RE datasets in Tab. 8. Our observations align with the findings from experiments on 1,000 test examples: (1) Our QA4RE framework can bring consistent and significant improvements over all FLAN-T5 series models on the averaged results. Additionally, larger models benefit more from our framework. These two signals strongly demonstrate the effectiveness of QA4RE. (2) We notice that our QA4RE does not improve smaller versions of FLAN-T5 on SemEval, a 19- choice QA task. This may be due that these models have difficulties in understanding long input fed by QA4RE.
6Templates for Robustness Experiments: TEMP1: https://github.com/osainz59/Ask2Transformers/blob/ master/resources/predefined_configs/tacred.relation.config.json TEMP3: https://github.com/luka-group/SuRE/blob/main/data templates/tacred/rel2temp_forward.json TEMP4: https://github.com/luka-group/SuRE/blob/main/data /templates/tacred/rel2temp_raw_relation.json
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8.
A2. Did you discuss any potential risks of your work? Not applicable. Our work helps LLM solve the relation extraction tasks, we don’t anticipate any risks.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Before Section 1 (abstract) and Section 1 (introduction).

heading: 3 A4. Have you used AI writing assistants when working on this paper?
text: Grammarly. Grammar check for sections 1-8.
B 3 Did you use or create scientific artifacts? Section 3, Section 4.

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? For artifacts we used in the paper, they have licenses in the public GitHub repos.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? For artifacts we used in the paper, they have licenses in the public GitHub repos. We are following the standard use of these artifacts. Our code will be released under the same license.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? These datasets are widely used as relation extraction benchmarks in the research field and as far as we know, no previous work has reported offensive or sensitive content in these datasets.
7 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Our code focuses on using OpenAI API for a specific task, relation extraction, and we only have tested our code on standard English benchmarks for the relation extraction task.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 4.1
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Sections 5 and 6.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix B.1

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Section 4.2 and Appendix B.1

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section 4.2 Experimental Setup and Section 5.4. Averaged results over multiple runs are reported.
7 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? No, we use the standard data format without extra processing and we use official GitHub repos for baseline comparison.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We leave answering these questions for future work