GPT-RE: In-context Learning for Relation Extraction using Large Language Models
Authors: Zhen Wan
Fei Cheng
Zhuoyuan Mao
Qianying Liu
Haiyue Song
Jiwei Li
Sadao Kurohashi
Venue: EMNLP 2023 Main (2023)
Keywords: Large language models
relation extraction
in-context learning
TL;DR: Substantial improvements of in-context learning on relation extraction

Abstract: In spite of the potential for ground-breaking achievements offered by large language models (LLMs) (e.g., GPT-3) via in-context learning (ICL), they still lag significantly behind fully-supervised baselines (e.g., fine-tuned BERT) in relation extraction (RE). This is due to the two major shortcomings of ICL for RE: (1) low relevance regarding entity and relation in existing sentence-level demonstration retrieval approaches for ICL; and (2) the lack of explaining input-label mappings of demonstrations leading to poor ICL effectiveness. 
In this paper, we propose GPT-RE to successfully address the aforementioned issues by (1) incorporating task-aware representations in demonstration retrieval; and (2) enriching the demonstrations with gold label-induced reasoning logic. We evaluate GPT-RE on four widely-used RE datasets, and observe that GPT-RE  achieves improvements over not only existing GPT-3 baselines, but also fully-supervised baselines as in Figure 1. Specifically, GPT-RE achieves SOTA performances on the Semeval and SciERC datasets, and competitive performances on the TACRED and ACE05 datasets. Additionally, a critical issue of LLMs revealed by previous work, the strong inclination to wrongly classify NULL examples into other pre-defined labels, is substantially alleviated by our method. We show an empirical analysis.

GPT-RE: In-context Learning for Relation Extraction using Large Language Models
Zhen Wan
Fei Cheng Zhuoyuan Mao
Qianying Liu'
Haiyue Song' Jiwei Li
Sadao Kurohashi'
Kyoto University, Japan
2 Zhejiang University, China
(zhenwan, zhuoyuanmao, ying, song]enla ist. kww.o-y.oac.... {feicheng, krroojeikyottou..a. jp. [jiwei_lijezju.co cn
Abstract
Semeval
TACRED
In spite of the potential for ground-breaking achievements offered by large language models (LLMMs) (e.g., GPT-3) via in-context learning (ICL), they still lag significantly behind fully- supervised baselines (e.g., fine-tumed BERT) in relation extraction (RE). This is due to the two major shortcomings of ICL for RE: (1) low relevance regarding entity and relation in exist- ing sentence-level demonstration retrieval ap- proaches for ICL, and (2) lack of explaining input-label mappings of demonstrations lead- ing to poor ICL. effectiveness.
89.9
70.04
32.49
33.45
Fine-tuning Baseline (PURE)
GPT-Random
GPT-Sent
GPT-RE (ours)
Figure I: Míero F1 performances two RE datasets. Previous GPT baselines (GPT-Random: randomly se- lected demonstrations and GPT-Sent: sentence-level demonstration retrieval) largely underperform fine- tuning baseline PURE while our GPT-RE substantially outperforms all baselines.
this paper, we propose GPT-RE to success- fully address the aforementioned issues by (I) incorporating task-aware representations demonstration retrieval; and (2) enriching the demonstrations with gold label-induced rea- soning logic. We evaluate GPT-RE four widely-used datasets and observe GPT- not only exist- ing GPT-3 baselines, but also fully-supervised baselines Figure Specifically, GPT-RE achieves SOTA performances the Semeval and SciERC datasets, and competitive perfor- mances the TACRED and ACE05 datasets.
task-specific dataset in fully-supervised man- ner, LLMs employ a new paradigm known as in- context learning (ICL) (Brown et al., 2020; Min et al., 2022a) which formulates an NLP task under the paradigm of language generation and makes predictions by learning from few demonstra- tions. Under the framework of ICL, LLMs achieve remarkable performance rivaling previous fully- supervised methods even with only a limited num- ber of demonstrations provided in various tasks such as solving math problems, commonsense rea- soning, text classification, fact retrieval, natural language inference, and semantic parsing (Brown et al., 2020: Min al., 2022b; Zhao et al., 2021; Liu al., 2022b; Shin al., 2021).
RE achieves improvements
Additionally, critical issue of LLMs revealed by previous work, the strong inclination to wrongly classify NULL examples into other pre- defined labels, is substantially alleviated by method. We show an empirical analysis.'
1 Introduction
The emergence of large language models (LLMs)
Despite the overall promising performance LLMs, the utilization of ICL for relation extraction (RE) is still suboptimal. RE is the central task for knowledge retrieval requiring deep understanding of natural language, which seeks to identify pre- defined relation between specific entity pair men- tioned in the input sentence or NULL if relation found. Given test input, ICL. for RE prompts the input LLMs with the task instruction, few demonstrations retrieved from the training data,
such as GPT-3 (Brown et al., 2020; Thoppilan et al.,
2022; Chowdhery et al., 2022; Rae al., 2021; Hoffmann et al., 2022) represents significant ad- vancement in natural language processing (NIP).
Instead of following a pretraining -ettraiiing-and---ae aid-finetuning pipeline (Devlin et al., 2019; Beltagy et al., 2019; Raffel et al., 2019; Lan et al., 2019; Zhuang et al., 2021), which finetunes pre-trained model on
Coodss will be released after the anonymous period.

---

BERT Fine-tuning
GPT-3 In-context Learning
Microsoft a technology company located the U.S.
Retrieval
Training Data
NOLL
1.0
by Bill Gates
Relation Triplet: [Microsoft, Gates, by)
True
CE
CE
Freeshot Demorstration
Microsoft is famous computer technology
company
history Triplet: [Microsaft, U.S.
PP
PP
in)
0.0
NULL CE I Predicted
NULL CE
PP
Predicted
Retrieval without considering the lask-aware triplet results in noisy demonstrations.
Figure 2:
Figure 3: Confusion matrix on Semeval dataset with three selected relation labels. The NULL examples are overpredicted other relations by GPT-3. CE: Cause- Effect, IA: Instrument-Agency, PP: Product-Producer.
and the test input itself. Then LLMs generate the corresponding relation. Recent research (Gutiér- rez et al., 2022) has sought to apply GPT-3 ICL to biomedical RE, but the results are relatively nega- tive and suggest that GPT-3 ICL still significantly underperforms fine-tuned models.
relations. Both methods contain more RE-specific information than sentence semantics, thus effec- tively addressing the problem of low relevance.
The reasons that cause the pitfall of GPT-3 ICL in RE are two folds: (1) The low relevance re- garding entity and relation in the retrieved demon- strations for ICL. Demonstrations are selected ran- domly via k-nearest neighbor (kNN) search based on sentence embedding (Liu et al., 2022b; Gutiérrez al., 2022). Regrettably, kNN-retrieval based on sentence embedding more concerned with the relevance of the overall sentence seman- tics and not as much with the specific entities and relations it contains, which leads to tow-quality demonstrations. As shown in Figure 2, the test in- put retrieves a semantically similar sentence but is not desired in terms of entities and relations.
For (2) gold label-induced reasoning, we pro- pose to inject the reasoning logic into the demon- stration to provide more evidence to align an in- put and the label, a strategy akin to the Chain-of- Thought (CoT) research (Wei et al., 2022; Wang et al., 2022b; Kojima et al., 2022). But different from previous work, we allow LLMs to elicit the reasoning process explain not only why a given sentence should be classified under a particular la- bel but also why a NULL example should not be assigned to any of the pre-defined categories. This process significantly improves the ability of LLMs to align the relations with diverse expression forms.
(2) The lack of explaining input-label mappings in demonstrations leads to poor ICL effectiveness: A vanilla form of ICL lists all demonstrations as input-label pairs without any explanations. This may mislead LLMs learn shallow clues from surface words, while relation can be presented in diverse forms due to language complexity. Es- pecially when ICL has a maximal input length, optimizing the learning efficiency of each single demonstration becomes extremely important.
Recent work reveals another crucial problem named "overpredicting" as shown in Figure 3: we observe that LLMs have the strong inclination to wrongly classify NULL examples into other pre- defined labels A similar phenomenon has also been observed in other tasks such as NER (Gutiér- rez et al., 2022; Blevins et al., 2022). In this paper, show that this issue can be alleviated if the rep- resentations for retrieval can be supervised with the whole set of NULL in the training data.
To this end, we propose GPT-RE for the RE task. GPT-RE employs two strategies to resolve the issues above: (1) task-aware retrieval and (2) gold label-induced reasoning. For (1) task-aware retrieval, its core is to use representations that de- liberately encode and emphasize entity and relation information rather than sentence embedding for ANN search. We achieve this by two different re- trieval approaches: (a) entity-prompted sentence embedding; (b) fine-tuned relation representation, which naturally places emphasis on entities and
We evaluate our proposed method on three popu- lar general domain RE datasets: Semeval 2010 task 8, TACRED and ACE05, and one scientific domain dataset SciERC. We observe that GPT-RE achieves improvements over only existing GPT-3 base- lines, but also fully-supervised baselines. Specifi- cally, GPT-RE achieves SOTA performances on the Semeval and SciERC datasets, and competitive per- formances on the TACRED and ACE0S datasets.

---

Retrieved Demonstrations
Task Description
Context: ideas for continuous development Label-induced Reasoning (Sec. 2.5)
predict the relation The pre-defined relations are CAUSE AND SFFECT, COMPONENT AND WHOLE,
given the contest.
umerged the students' collaborative
Relation: PRODUCT AND PRODUCER
NULL the relation does belong
Retrieval
Training
ICL Demonstrations (k shott)
ideas contiruous development emerged
students' collaborative efforts. Given the conteat, the relation between ideas and students ARODUCT AND ARODUCER
Neemtence
Piswaj kuau
Reason: The "collsborative working together create something;
students
SimCSE
Fine- tuned RE Model
"emerged"
Raconstructed relation between 'idea' "dsughter" contoxt: My year old daughter came up with extremely good ides.
Tokenss: [C1.S] My 8 year-old [OB]]] caughter [/OB/) came
Test Inpur
an
Context: My 8 year old daughter came up with an extremely good idea.
extremely good [SUB] idea [/SUB]
Given the context, the relation between idea and doughter is
Prompt Construction [Set. 2.3)
Tasti aware Demonstration
Retreval (Sec.
Test Output
Content: year-old daughter came
PRODUCT AND PRODUCER
Test input
good idea.
Relation:
Figure 4: An illustration of GPT-RE. Given a test input, we first leverage two different task-aware retrieval methods to search for highly relevant demonstrations from the training set, and then incorporate the gold abel-induced reasoning for each demonstration. Above contents will then be included in the prompt construction to make the prediction.
2 Methodology: GPT-RE
ICL Demonstrations D We first leverage task- aware retriever to acquire k-shot demonstration set, then enrich each demonstration (24,, 9:) with the gold label-induced reasoning r; build a new set of (Ii, Vi, Ti) as D.
2.1 Task Definition
Let C denote the input context and Csub E C.
Cobj E C denote the pair of subject and object entity. Given set of pre-defined relation classes IR, rela- tion extraction aims to predict the relation y E IR between the pair of entities (eaah; Pobj) within the context C, or if there no pre-defined relation be- tween them, predict y = NULL.
Test Input Ttost Similar to the demonstrations, offer the test input Ttest: and GPT-3 is expected to generate the corresponding relation ylest. In summary, GPT-RE can be formulated as:
2.2 Overview
P (Vtest E R (UNLULL)]Z Ztest)
(1)
We will first introduce the prompt construction formalize RE language generation task in Sec. 2.3. Then improve the ICL frame- work for RE, we will introduce two modules: (1) task-aware demonstration retrieval to select higher- quality demonstrations (Sec. 2.4); (2) gold label- induced reasoming to enrich each demonstration with explanations (Sec. 2.5). In Figure 4, we show the concrete workflow of processing test input.
2.4 Task-aware Demonstration Retrieval
Since ICL demonstrations closer to the test sample in the embedding space result in more consistent and robust performance (Liu et al., 2022b). Re- cent work (Gutiérrez et al., 2022; Liu et al., 2022b) employs the &NN to retrieve the most similar amples in the training set as the few-shot demon- strations for each test input. As &NN relies on the choice the embedding space to encode both test input and examples in the training set, they propose to obtain sentence embedding using pre-trained language models, or other improved sentence em- bedding.
2.3 Prompt Construction
We construct prompt for each given test example, which fed to the GPT-3 model. Each prompt consists of the following components:
However, using sentence embedding for ANN retrieval has severe drawback: relation extrac- tion focuses on pair-wise entities, which diverge from the semantic meaning of the entire sentence, leading to an ambiguous retrieval using sentence embedding. In this study, we propose two novel
Instructions I We provide succinet overview of the RE task description and the set of pre-defined classes R. The model is explicitly asked to out- put the relation, which belongs to the pre-defined classes. Otherwise, the model will output NULL.

---

methods to provide more robust representations for better retrieval quality: (1) a naive entity-ppre prommped sentence embedding in Sec. 2.4.1; (2) an advanced fine-tuned relation representation in Sec. 2.4.2.
ICL Demostration
Context: Ideas continuous development also emerged from the students' collaborative efforts. Given the context, the relation between
ideas and students is PRODUCT AND PRODUCER.
Query
What the clues lead relation between "ideas" and "students" be PRODUCT AND PRODUCER in the sentence "Ideas for continuous development also emerged from the students' collaborative efforts."?
2.4.1 Entity-Prompted Sentence Embedding
Given the discrepancy between sentence embed- ding and relation extraction, the original context is insufficient for demonstration retrieval. Consider- ing the importance of entity information in RE, we propose reconstructing the context by incorporat- ing entity pair information. For example, given the context "Le has a sister Lisa," the reconstructed context with the entity prompted will be "The re- lation between 'He' and 'Lisa' the context: has a sister Lisa." This approach preserves both the semantic meaning of the sentence and the en- tity pair-centered information during retrieval. In the paper, we employ the latest robust model Sim- CSE (Gao et al., 2021) for computing sentence embedding-basse -baed similarity.
Reason
The phrase "collaborative efforts" suggests that students are warking together to create or produce something; The use the
is because:
"emerged" implies product of the students' efforts. Therefore, the relation between students and ideas PRODUCT AND PRODUCER.
ideas
or
Figure An illustration of adding reasoning.
Datnset
# Relation
* Train
Dev
# Test (# Subset)
(U)..(9)
Semeral
6.507
1.493
2,717 (2,717)
17.40%
TACRED
41
68,124
22,631
15,509 (1.600)
79.40%
SeIRRC ACBO5
16,872
2,033
4,088 (4,088)
90. 16%
27,597
(2,443)
95
Table 1: Statistics of datasets.
demonstrations, the fine-tuning process is unbun- dled and can be done on the whole train data. It has two subsequent merits. First, the relation repre- sentations are directly fine-tuned to fit the RE task, which could significantly boost the overall retrieval quality. Second, the overpredicting NULI issue will be substantially alleviated because the similar NULL demonstrated can be accurately recognized by the fine-tuned model.
2.4.2 Fine-tuned Relation Representation
Compared prompt entity information into con- text sentences, more straightforward solution is to extract the relation representation from a fine-tuned RE model for retrieving demonstrations.
Current BERT-based fine-tuning methods for RE (Baldini Soares et al., 2019; Zhong and Chen, 2021; Wan et al., 2022) attempts to capture both the context information and the entity information by adding extra marker tokens to highlight the sub- ject and object entities and their types. Specifically, given an example: "He has a sister Lisu.", the in- put tokens are "[CLS] [ER] He [SUB_PERR has a sister [OBJ__PR Lisa [/OR]PEER]. [SEP]" where "PER" is the entity type if provided. De- note the n-th hidden representation of the BERT encoder as h Assuming i and j are the indices of two beginning entity markers (SUB PER] and [OBI_PER], define the relation representation as Rel h + hy where @ stands for concatenation of representations in the first dimension. Subse- quently, this representation fed into a feedfor- ward network for predicting the relation probability p(y E R U [NULL}} Rel).
2.5 Gold Label-induced Reasoning
Recent CoT work has reported significant progress in the commonsense and numerical reasoning tasks by automatically eliciting the reasoning steps for solving a question. While in the RE task, two entities can possibly hold multiple relations, e.g., "Joe Biden" can be either the president of or lives in "U.S.". The reasoning generation could be out of focus if it lacks interaction with the gold label. In this section, we propose to let GPT-3 induce the reasoning logic for each demonstration by the corresponding gold relation label. As shown Figure 5, given selected demonstration, we first generate query prompt "What are the clues that lead to the relation between [entity|] 1] and [entity2] to be [relation] in the sentence [context]?" based on the demonstration and subsequently ask GPT-3 to generate clues "It is because: ... on the labeled relation between the pair of entities in the context. Finally, we augment the demonstration by incorpo- rating the generated clues induced by GPT-3.
The entity markers have wwweeeippiiit explicitly encoded sub- ject and object entities and the relation represen- tation Rel is naturally enriched with the entity in- formation. We believe this approach can poten- tially compensate for the limitations of GPT-3 in RE. While GPT-3 ICL has a constraint of limited

---

Methods
Retriever
Semeval
TACRED
SciERC
ACE05
GPT-3 Baselines (Best k-shot)
GPT-Random
70.04 (30)
32.49 (15)
17.92 (25)
9.04 (25)
GPT-Sent
SimCSE
79.94 (30)
33.45 (15)
20.96 (25)
6.31 (25)
Ours (Best k-shot)
OPT-RE_SimCSS
SimCSE
81.02 (30)
37.44 (15)
26.46 (25)
8.67 (25)
GPT-RE *
SimCSE
77.49 (15)
31.58 (10)
Reasoning
SimCSE
79.88 (15)
33.18 (10)
GPT-RE_FT
PURE
91.90 (25)
72.14 (15)
69.00 (30)
68.73 (25)
GPT-RE_FT F
PURE
91.11 (15)
70.38 (10)
+ Reasoning
PURE
91.82 (15)
70.97 (10)
Fine-tuned RE Baselines
Cohen et al. (2020)
91.90
Wang et al. (2022a)
76.80
PURE (Zhong and Chen, 2021)
89.90
69.72
68.45
70.09
Table Main Results on four RE datasets. All results given by Micro-FI. denotes the k-shot for the comparison with + Reasoning. Due to the costly GPT-3 expense, we conducted Reasoning experiments on the two relatively smaller datasets Semeval TACRED. denotes that this performance not comparable evaluates the entire test set. The underline denotes the results outperforming the fine-tuning baseline PURE.
3 Experiment Setup
prompt construction (Sec. 2.3) via OpenAl API. We implement two categories of GPT-3 baselines:
3.1 Datasets
We evaluate on three popular general domain RE datasets and one scientific domain dataset. Due to the cost of running the model in the API with GPT-3, in our main results, sample subset (See Appendix from the original test for datasets: ACE05 and TACRED shown in Table
(1) GPT-Random Instead of randomly selecting few-shot demonstrations from the training data for each test input, add extra constraints make the label distribution of selected demonstrations more uniform. Our preliminary experiments suggest that this is a stronger baseline than the vanilla random.
(2) GPT-Sent Previous work attempts various sen- tence embedding in retrieval. In this work, our im- plementation adopted SimCSE (Gao al., 2021). which has been demonstrated to be the state-of-the the art method for sentence similarity tasks.
Semeval 2010 task 8 Hendrickx et al. (2010) fo- cuses on semantic relations between pairs of nomi- nals collected from general domain resources.
Zhang et al. (2017) is a large-scale relation extraction dataset with 106,264 examples built over newswire and web text.
TACRED
SciERC Luan (2018) collects AI paper abstracts and annotated relations, especially for scientific knowledge graph construction.
Fine-tuned RE Models In our experiment, we choose PURE (Zhong and Chen, 2021), an en- tity marker-based fine-tuned model mentioned in Sec. 2.4.2 to obtain the representations for retrieval. Meanwhile, PURE performs as a directly compara- ble baseline. We also compare with corresponding SOTA fine-tuned baselines on Semeval Cohen et al. (2020) (reformulate RE as the question answering task) and TACRED Wang al. (2022a) (extra pre- training to capture RE structure) datasets.
ACE0S contains the entity, relation, and event annotations collected from domains including newswire, broadcast, discussion forums, etc.
3.2 Baseline Methods
GPT-3 baselines For GPT-3 baselines and our methods, we select "text-davinci--09" with
maximal 4,097 input tokens and use the identical
All implementation details are in Appendix A.

---

MicroFF2 F2
Micro-F3
40/40
5005
Bassline
81,25
OPI-handom
80
72.22
700
70
75/4
73.02
60
68,96
66
63
50
64
Migm
Grisent
oPria PT
40
10
15
25
4300
#200
#400
#03e 130%
#1300 120% #3998 (30%)
shot demonstrations
Number (percentage) of training data
(a) The comparison on retrieval modules
Figure 7: Low-resource Scenario Semeval. We limit the percentage of training data for both fine-tuning and retrieval GPT-RE.
Micro-F1
100
GPT-RE, SimcsE OPT-RE Reasoning
95
85
of demonstrations. Meanwhile, the high-quality demonsirations obtained by GPT-RE FT offset the effort of enriching reasoning into demonstrations, which shows relatively trivial improvements. Since reasoning aims enriching demonstrations, this feature potentially works better with fewer demon- strations, as shown in Section 4.3.
75
70
65
60
k shot demonstratooss
(h) Reasoning with fewer demonstrations.
4.2 Ablation Study on Task-aware Retrieval
Figure 6: Ablation study on the retrieval and reason- ing components on Semeval. We sampled a subset from the test data with 300 examples. We show the "w/o reasoning' results with je 30 for comparison.
We first implement the ablation experiments of the retrieval component with the setting of increasing k-shot demonstrations (Figure 6a). We find that: (1) compared to GPT-Random, all the retrieval- based models have higher F1 scores and large gra- dients of the performance curves. It means that GPT-3 can learn from high-quality demonstrations more effectively; (2) after adding entity informa- tion the SimCSE retrieval. ESCSE achieves better performance throughout all K shots, indicating that lask-aware sentence embedding can capture the feature of RE and provide more proper demonstrations; (3) finally, the fine-tuned relation representation retriever GPT-RE FT significantly outperforms all retrieval-based methods and beats the fine-tuning baseline when & > 15. Note that with k: 5 demonstrations, GPT-RE_P FT still works better than GPT-RE SimCSE with &: 30 (80.30 (3.433++3.13),, which indicates that the quality of demonstrations shows much more impor- tant than the number of demonstrations.
4
Experimental Results
4.1 Main Results
We compare our main experiment results with pre- vious methods Table 2. PETT-E_SSimSS) de- notes our untity-prompted sentence embedding for retrieval and GPT-RE_ FT denotes our fine-tuned relation representation for retrieval. From the table, observe that: (1) both STT-RE_SimCSE and GPT-RE FT outperform the retrieval-based GPT- Sent, indicating that it is necessary to inject the task- specific information into sentence embedding for selecting proper demonstrations; (2) GPT-RE_ succeeds to outperform the fine-tuning baseline PURE on three datasets by +2.00, +2.42, +0.55 Micro-F1. It suggests that GPT-3 has the poten- tial to beat fine-tuning when the retriever has prior task knowledge. GPT-RE_FT eventually achieves SOTA results on Semeval and SciERC. (3) reason- ing module improves SPT-RE__SmCCS, by around 2% Micro-F1, indicating that gold label-induced reasoning successfully enriches the knowledge
GPT-RE_SimCSZ
4.3 Ablation Study on Reasoning Enhancing
We then check the influence of our proposed reasoning-enhannn
demonstration, as shown in Figure 6b. Due to the limited amount of input to- kens of GPT-3, we have to set the & S 15 for the

---

Semeval
SciERC
100
GPTSem
OPT kundas
GPTAC, SinCSS
Beseline
80
Raseline
60
20
20
(FX)
w/o NULL (F1)
w/ NULL (F1)
Recall on NULL
(a) General domain with 17.4% NULL examples
(b) Scientific domain with 90.16% NULL examples
Figure 8: Analysis the effects of NULL examples. w/o NULL refers the classification setting that NULL examples are excluded from the train and test data. w/ NULL refers to the original extraction setting. We use the full test set for the evaluation.
tokens of reasoning, leading to a trade-off between adding reasoning and adding more demonstrations. From the result, we find that: (1) with reasoning- enhanced demonstrations, GPT-3 always achieves better scores across all the k-shot settings of both PPT-RE_SimCSS and GPT-RE_FT, indicating that the reasoning induced from ground truth relation labels can effectively unlock the reasoning ability GPT-3 and improve the ICL with a deeper under- standing of demonstrations. Specifically, for GPT- RE_FT, the performance improvement becomes less significant when more demonstrations are pro- vided, which is feasible with more high-quality demonstrations available, GPT-3 can already learn the internal reasoming behind each demonstration; (2) since the reasoning enhancement works better with fewer demonstrations, we expect this method can be an effective solution to low-shot relation extraction (Han et al., 201 8; Geng et al., 2020; Liu et al., 2022a), which aims at recognizing novel re- lations with very few examples, and we leave this for future work.
could limit the effects of retrieval; (3) GPT-RE_FT achieves an upper bound performance in all set- tings, even when the fine-tuned model shows poor performance with hundreds of training data (from #100 to #400). This emphasizes the impressive effectiveness of fine-tuned relation representations for capturing higher-quality demonstrations. The observation in the low-resource setting is very dif- ferent from Gutiérrez et al. (2022). We assume the difference could be caused by the domain and NULL proportion of the task.
Analysis
5.1
The Issue of "Overpredicting"
To analyze the influence of NULL class, com- pare the effectiveness of each method for allevi- ating this issue on two datasets: general domain Semeval with 17.4% NULL examples and scientific domain SciERC with 90.16% NULL examples. As shown in Figure 8, (1) by comparing the perfor- mance Semeval and SciERC, larger percentage of NULL examples results in more significant per- formance drop showing the negative influence of overpredicting NULL examples; (2) by comparing w/o NULL and w/ NULL, our GPT-RE FT shows the most robustness to the influence of NULII ex- amples, indicating that the RE fine-tuned represen- tations in retrieval can release the overpredicting issue of GPT-3 by providing higher-quality demon- strations; (3) however, even with task-aware repre- sentations, all GPT-3 methods still underperform the fine-tuning baseline on NULL examples, this is due the confusing definition of NULL,, in many cases, there certain relation between entities in the context, but out of the distribution of pre-
4.4 Low-resource Scenario
We conduct the experiment for observing the low- resource performance in the general domain Se- meval task. As shown in Figure 7, we observe that: (1) all the GPT-3 based results work bet- ter than fine-tuning in when the training examples are less than # 650 (10%). It indicates that in the general domain RE, GPT-3 benefits from its abun- dant prior knowledge to understand the relations; (2) GPT-RF_SSmee SimCSEE starts show substantial difference to GPT-Sent after the training size sur- passes 30%. We believe fewer training candidates

---

GPT-Sent Demonstration
6 Related Work
Context: among contents vessel were carpenter
tools, several large storage jars, ceramic utensils, rope
In-context Learning Recent work shows that ICL of GPT-3 (Brown et al., 2020) perform numerous tasks when provided a few examples in a natural language prompt. Existing work focuses on various aspects to effectively utilize the advantages of GPT-3, from prompt design (Perez et al., 2021) for proper input to coherence calibration (Malkin et al., 2022) for tackling the diverse generated out- put. Another research path locates in the demon- stration part, including ordered prompts (Lu et al., 2022) and retrieval-baset demonstrations (Rubin et al.. 2022; Liu al., 2022b; Shin et al., 2021). To the best of our knowledge, there is no pre- vious work exploring the potential of GPT-3 on general domain RE tasks. A recent work attempts to leverage GPT-3 in biomedical information ex- traction (NER and RE). and reveals issues of ICL that may be detrimental to IE tasks in general. Our work succeeds in overcoming these issues to some extent and confirms the potential of GPT-3 in both general and the scientific domain RE.
Given the context, the relation between tools and vessel is CONTENT
AND CONTAINER.
SimCSE Demonstration
Context: local fisherman heated some of their catch cooked over
coals in a scuttle.
Given the context, the relation between catch and scuttle is
CONTENT AND CONTAINER.
GPT-RE FT Demonstration
Context: september normally marks arrival earllest run of october
fish into lake tributaries, and peak
Given the context, the relation between run and fish is NULL.
Test Input
Context: preserve its catch of fish, each boat loads between 2000 and 3000 kilos of ice before it goes out to sea
Given the context, the relation between catch and fish is (NULL)
Butricnal
Went
GPT-RE SimCSE
GPT-RE
GPT-5 Dulput
CONTENT AND
CON WANTAINER X
PRODUCT AND
PRODUCER X
NJLL
Figure 9: case study of demonstration quality on Semeval. [NULL] is the gold label here.
defined classes. In these cases, GPT-3 tends to overpredict as the relation information may be cov- ered in its prior knowledge. We think this ability of GPT-3 can be useful in more open fields, such open RE (Banko and Etzioni, 2008) which has no pre-defined relation classes.
Retrieval-based Demonstrations Several stud- ies have demonstrated that dynamically selecting few-shot demonstrations for each test example, in- stead of utilizing a fixed set, leads to significant improvement in GPT-3 ICL (Liu et al., 2022b; Shin et al., 2021; Rubin et al.. 2022). They also show that nearest neighbor in-context examples yield much better results than the farthest ones. This leads to the significance of better retrieval modules for demonstrations, Existing attempts rely on sentence embedding in retrieval, including the sentence encoders of PLMs such as BERT (De- vlin et al., 2019), RoBERTa (Zhuang et al., 2021) KATE (Liu et al., 2022b) SimCSE (Gao et al., 2021), Sentence-BERT (Reimers and Gurevych, 2019; Wolf al., 2020). Unlike these sentence embeddings, we propose to fine-tune PLMs on our target RE tasks to produce more task-specific and robust representations for retrieval.
5.2 Case Study of Demonstration Quality
We select one typical test example to better illus- trate the amendment of our task-aware demonstra- tion retrieval. As shown in Figure 9, given the NULL Example, we show the most similar demon- stration in retrieval based on three methods. The GPT-Sent retrieved demonstration focuses on the semantic meaning of "CONTENT AND CON- TAINER" which is shared in the test context, but not revealed in the target entity pair. This mismatch confirms the problem of lacking entity information in retrieval. Instead, GPT-RE_Sinteeeee much more relevant demonstration that shows the same semantic relation between "catch" and "fish" but still faces a minor mismatch as the gold label is between "catch" and "scuttle." " Finally, GPT. RE_FT demonstration shares a similar structure with the test input regarding the pair of entities, which is the key clue for predicting the relation between entities. This result shows a level-by- level enhancement with more entity information provided in retrieval. We also show some other case examples in Appendix B.
retrieves a
Conclusions
This work explores the potential of GPT-3 ICL RE for bridging the performance gap to the fine-tuning baselines via two strategies: (1) task- aware demonstration retrieval emphasizes entity and relation information for improving the accu- racy of searching demonstrations; (2) gold label- induced reasoning enriches the reasoning evidence

---

of each demonstration. To the best of our knowl- edge, GPT-RE is the first GPT-3 ICL rescarch that significantly outperforms the fine-tuning baseline on three datasets and achieves SOTA on Semeval and SciERC. We implement detailed studies to ex- plore how GPT-3 overcomes the difficulties such NULL example influence.
In Proceedings of the 2015 Conference on Empiri- cal Methods in Natural Language Processing, pages 632-642, Lishon, Portugal. Association for Compu- tational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Iyya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Limitations
Despite the overall positive results, GPT-RE still faces two shortcomings: (1) the issue of overpre- dicting has been significantly alleviated but not completely solved, and the NULL recall still lags behind full-supervised baselines, especially on the datasets containing a large proportion of NULL ex- amples such as ACE05 ("95.60%"); (2) Though the task-aware retriever optimizes the representa- tions of PLMs such as SimCSE and BERT, it is widely considered that LLMS can generate more ro- bust representations than small PLMs. Future work can replace representations generated by smaller PLMs with GPT-3 itself. However, due to the ac- cess limitation to the representations of GPT-3, we can nearly confirm this proposal up to now.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi. Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao. Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstem, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR, abs/2204.00311.
References
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 2895- 2905, Florence, Italy. Association for Computational Linguistics.
Michele Banko and Oren Etzioni. 2008 The tradeoff's between open and traditional relation extraction. In Proceedings of ACL-08: HLT, pages 28-36, Colum- bus, Ohio. Association for Computational Linguis- tics.
Amir D. N. Cohen, Shachar Rosenman, and Yoav Gold- berg. 2020. Relation extraction as two-way span- prediction. CoRR, abs/2010.04829.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long and Short Papers), pages 417 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB- ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 3615- 3620, Hong Kong, China. Association for Computa- tional Linguistics.
Terra Blevins, Hila Gonen, and Luke Zettlemoyer: 2022, Prompting language models for linguistic structure, CoRR, abs/2211..7292 1.07830.
Tianyu Gao, Xingeheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large anno- tated corpus for learning natural language inference.

---

Xiaoqing Geng, Xiwen Chen, Kenny Q. Zhu, Libin Shen, and Yinggong Zhao. 2020. MICK: A meta- learning framework for few-shot relation classifica- tion with small training data. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Viriual Event, Ireland, October 19-23, 2020, pages 415-424. ACM.
pages 100-114, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Bernal Jiménez Gutiérrez, Nikolas McNeal, Clay Wash- ington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 in-context learning for biomedical ie? think again. CoRR, abs/2203.08410.
Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 3219-3232. Brussels, Belgium. Association for Computational Linguistics.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang. Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of of-the--ttheeee art evaluation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4803-4809, Brussels. Belgium. Association for Computational Linguistics.
Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. 2022. Coherence boosting: When your pretrained language model is not paying enough attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (
Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowiez. 2010. SemEval-2010 task 8: Multi- way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33-38. Up- psala, Sweden. Association for Computational Lin- guistics.
(Volume I: Long Papers), pages 5214-8236, Dublin, Ireland. Association for Computational Linguistics.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022a. Rethinking the role of demonstrations: What makes in-context learning work?
Jordan Hoffmann. Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai. Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark. Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. 2022. Training compute-optinal large language models.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettle- moyer. 2022b. Rethinking the role of demonstra- tions: What makes in-context learning work? CoRR. abs/2202.12837.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. In Ad- vances in Neural Information Processing Systems 34: Annual Conference on Neural Information Process- ing Systems 2021, NeuriPS 2021, December 6-14, 2021, virtual, pages 11050-1107..
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. CoRR, abs/2205. 19166.
Jack W. Rac, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides. Sarah Henderson, Roman Ring. Susan- nah Young, Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Pa- gamini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro. Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou. Arthur Mensch, Jean-Baptiss Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong. Daniel Toyama, Cyprien de Masson d' Autume, Yujia
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel. Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations.
Fangchao Liu, Hongyu Lin. Xianpei Han, Boxi Cao, and Le Sun. 2022a. Pre-training to match for unified low- shot relation extraction. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5785- 5795, Dublin, Ireland. Association for Computational Linguistics.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022b. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022); The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures,

---

Li, Tayfun Terzi. Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger. Jason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukcuoglu, and Geoffrey Irving. 2021. Scaling language models: Methods, analysis &amp; insights from training gopher.
Zhen Wan, Qianying Liu, Zhuoyuan Mao, Fei Cheng, Sadao Kurohashi, and Jiwei Li. 2022. Rescue im- plicit and long-tail cases: Nearest neighbor relation extraction. CoRR, abs/2210.1 1222100.110000 11800.
Chenguang Wang. Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. 2022a. DeepStruct: Pre- training of language models for structure prediction. In Findings of the Association for Computational Lin- guistics: ACL 2022, pages 803-823, Dublin, Ireland. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with unified text-to-text trans- former.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe V. Lc, Ed H. Chi, and Denny Zhou. 2022b. Self- consistency improves chain of thought reasoning in language models. CoRR, abs/2203. 1171.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoe Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. CoRR, abs/2201.11903.
Nils Reimers and Iryna Gurevych. 2019. Sentence-
BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 3982 3992, Hong Kong, China, Association for Com- putational Linguistics.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 11 12-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies. pages 2655-2671, Seattle, United States. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Remi Louf, Morgan Funtow- icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite. Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Trans- formers: State-of-the-an natural language processing, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Richard Shin, Christopher Lin, Sam Thomson, Charles Chen, Subhro Roy, Emmanouil Antonios Platanios. Adam Pauls, Dan Klein, Jason Eisner, and Benjamin Van Durme. 2021. Constrained language models yield few-shot semantic parsers. In Proceedings of the 2021 Conference on Empirical Methods in Natu- ral Language Processing, pages 7699-7715, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin. Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung- Ching Chang, Igor Krivokon, Will Rusch, Mare Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-HHelsterr, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny So- raker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Ale- jandra Molina, Erin Hoffman-John, Josh Lec, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm. Viktoriya Kuzmina, Joe Fenton, Aaron Co- hen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera- Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoe Le. 2022. Lamda: Language models for dialog appli- cations.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35-45, Copenhagen. Denmark. Association for Com- putational Linguistics.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In Proceedings of the 38th International Conference on Machine Leaningg ICMI. 2021, 18-24 July 2021, Vir- rual Event, volume 139 of Proceedings of Machine Learning Research, pages -27066 PMLR
Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies.

---

pages 50-61, Online. Association for Computational Linguistics.
Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. 2021. robustly optimized BERT pre-training approach with post-traiming. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1218-1227, Huhhot, China. Chinese Informa- tion Processing Society of China.

---

Hyperparameterr
In Experiment
Engine
lext-davinci
003
Temperature
0.0
Max tokkens
256
Tup__p
Frequency ponalty
0.0
Tresence_pemally
0.0
Besi
Logprob
Table 3: GPT-3 Hyperparamters.
Gen-Sent Demonstration
Context: this paper describes set of principlas designed to help archives position themselves address the management
Dataset
Lower bound
Upper hound
Given the context, the relation between principles and MEMBER AND COLLECTION.
Semoval
5
30
TACRED
5
GPT-RE SimCSE Demonstration
SciERC
5
30
screen
using just
newspapers
ACEOS
5
25
but displays the ink particles electronically.
Given the context, the relation between ink and screen
Table 4: Search range for each dataset.
COMPONENT
GPT-RE FT Demonstration
Context: computer mouse
input device choice
for a long time now in the computer world
A
Hyperparameters
Given the context, the relation between mouse and computer Is
Test Input
A.1 GPT-3 Hyperparameters
diagrams
computer
they are carefully designed to match the grid of pixels on the screen.
We use the GPT-3 API during the experiments and set the hyperparameters as in Table 3. Since the "Temperature" is set to be 0.0, denoting the stable output of GPT-3, report the result of the single run for all experiments. Due the input length lim- itation of GPT-3 and the various average lengths of contexts from each dataset, we different search ranges for the number of demonstrations of each datasct as shown in Table 4.
Given the context, the relation between screen and computer [COMPONENT AND WHOLE]
Retrieual
GFT-Sent
GPT-EE Simest
GPTRE FT
MEMBER AND
COMPONENT WHOLE
COMPONENT WHOLE
COLLECTION
(a) ICOMPONENT AND WHOLEJ denotes the gold label
GPT-Sent Demonstration
Context: a woman diagnosed with breast cancer today joins huge survivors ready Given the context, the relation between survivors and sisterhoad is MEMBER AND COLLECTION.
sisterhood
along the way .....
GPT-RE SimCSE Demonstration
A.2 Fine-tuning Baseline PURE
organs several patients who have been waiting for donated organs. Given the context, the relation between organs and patients is
night accident
We follow their single-sentence setup to keep con- sistency among datasets as Semeval and TACRED are both sentence-level RE datasets. For the PL.Ms we also follow PURE by using scibert-scivocab- uncased (Beltagy et al., 2019) as the base en- coder for SciERC and bert-base-uncasee (Devlin et al., 2019) for the remaining three general domain datasets. We follow hyperparameters in their paper. We used 2 NVIDIA RTX3090 for training.
DESTINATION
GPT-RE FT Demonstration
Context: operation homefront and partners delivered toys military children.
Given the context, the relation between toys and children is ENTITY
Test Input
wheelchair foundation donated wheelchairs people
with physical problems in hundred countries. Given the context, the relation between wheelchairs and people
[NNTTTT AND DESTINATION].
Retrioval
GFT Sent
GPTRE SinCSE
GPTRE FT
GPI-
MEMBER AND COLLECTION
ENTITY AND DESTINATION
ENTITY AND DESTINATION
X
A.3 Sentence Embedding Methods
(b) [NNTITY AND DESTINATION denotes the gold label,
Gutiérrez et al. (2022) uses the [CLS] of RoBERTa- large as the representation in retrieval, Liu et al. (2022b) fine-tunes RoBERTa-large on two natural language inference (NLI) datasets: SNLI (Bow- man et al., 2015) and MultiNLI (Williams al., 2018) enhance the quality of sentence embed- ding. For the sentence embedding method Sim- CSE in our experiment, we utilize the version: sup-simcse-bers baaesiinnse--brt--b-----sssses base-unncased.
Figure More casees.

---

Label
# Num
PHYS
28
GEN-AFF
12
PER-SOC
11
GEN-AFF
33
PART-WHOLE
13
Label
# Num
ART
19
Per:title
40
NULL
2329
PFR:city_off ddd_hheahhhhhh
Org:shareholders
Table 5: ACE05
Perorigin
12
Org:top_ members/employees
36
Orgicity of headquarters
B Case Study
Per:religion
Per:city_oo Seacityyofbbbe _a__ff_bbrtt birth
To verify the effectiveness of our task-aware demonstration retricval, we provide more cases. For Figure 10a, GPT-Sent retrieves a demonstra- tion that shares the same semantic meaning of "de- sign" with the test input. However, the entity pair is irrelevant to the concept "design" resulting in noisy demonstration. Instead, OPT-RE_SimCCS, retrieves
Persemployee_n of
27
Per:data_oo of death
frrother_family Org:website
Per:cause _f death
Org:subsidiaries
Org:stateorproovine. of headquarters
relative demonstration with closer pair of entities sharing the same relation label. Fur- thermore, GPT-RE_ FT retrieves the demonstration containing both the closing entity pair and the same linguistic structure between entities. This case em- phasizes level-by-level improvement using our pro- posed methods. Figure 10b shows similar phe- nomenon.
Per.countries_ sotiarrie_oo_rsiddnnee
10
Per:siblings
orrssaatorpoovines___eesssoe rf_ddeence
11
Org:alternate names
27
Per:spouse
Per:parents
Org:country.coe of _e_aduuuuuuarrterrrss
Per:age
Per:date of birth
Per:country_ _ftttryy_o_dahhhhhheee
C Subset
Per:schools attended
Org:member_of
The number of sampled examples is not only re- lated to the size of the training data itself. A more important factor is the proportion of NULL. We have to maintain the original label distribution in datasets with high proportion of NULL.. Thus, the rule sample the subset is to keep the proportion of each relation label consistent with the original set. Table 5 are label distributions of two subsets.
Per:children
Org:parents
Per:citics_of re_siddeeeeeeeeeeeeeeeeeeeeeeee_
24
Per:stateorprovince of brith
Per:charges
12
Org:founded
Org:country_foomee founded by
Persstateorrrovinc__.. of_death
Org:members
Per:countryoo__ie _f__birthhhhhhh
GPT-RE_FT on TACRED surpasses the super- vised baseline in the current subset. As we show above, some labels in TACRED are indeed not well presented (only example), since TACRED dataset contains some long-tail tail labels. We decided add additional results of GPT-RE_FT by enlarging our sampled set 3200 (2 times the current version), and the performance of GPT-RE_ FT (k 15) is 73.16 while the performance PURE is 70.48.
Per:alternate names
Org.number_no rng:uumberoo__empplee httpaeeoooeeclllloeeemmeettrr
Orgadissolved
Ong:politica//eeliioosstttiiiiin affiliaatiooiiiiiiii
NULL
1271
Table 6: TACRED


Reviews:
[{"review_id": "JRebPsRo6Q", "reviewer": "Reviewer_UZWY", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable advice for improving this paper. \nTo deepen the understanding of both the shortcomings of ICL and the mechanism behind our proposal to improve ICL, we think it necessary to get deeper access to the GPT-3 models (e.g., embedding layers, parameters), which however, is limited. One possible solution could be conducting a deeper analysis with open-source LLMs (e.g., LLaMA). Meanwhile, to figure out the mechanism behind ICL, conducting analysis on one single task (RE) may be insufficient since other NLP tasks also face the challenges of ICL. Thus, we find that another aspect could be to extend our proposed method to solve shortcomings of ICL in other NLP tasks, which can further improve our understanding of ICL. We will discuss the above issues in our future work section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper adeptly highlighted a potential concern regarding GPT-3's utilization of in-context learning (ICL) and effectively provided a resolution. The observed performance enhancements are indeed promising, as they notably elevate the performance beyond that of GPT-3 alone. This achievement suggests that the strategies introduced in the paper hold the potential to offer benefits to a wide array of large language models (LLMs).", "reasons_to_reject": "I think the paper could further strengthen its argument by providing additional evidence to substantiate the identified shortcomings of in-context learning (ICL). By delving deeper into the reasons behind these shortcomings, the paper could enhance its overall credibility and contribute to a more comprehensive understanding of the challenges associated with ICL.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper underscores potential limitations associated with in-context learning (ICL) and suggests that these limitations might be the contributing factor to the underperformance of large language models (LLMs) with ICL when compared to fully supervised baselines in the context of relation extraction (RE). The identified shortcomings encompass two key aspects \"(1) low relevance regarding entity and relation in existing sentence-level demonstration retrieval approaches for ICL; and (2) the lack of explaining input-label mappings of demonstrations leading to poor ICL effectiveness.\"\nWithin the scope of this paper, the authors introduce their GPT-RE model as a solution to these challenges. Notably, the paper also highlights a pitfall observed in applying GPT-3's ICL approach to RE. To counter these issues, GPT-RE employs two strategic approaches: (1) task-aware retrieval and (2) gold label-induced reasoning. Moreover, the paper delves into addressing a newly identified concern termed \"overpredicting.\"\nIn the experimental phase, the authors demonstrate that the GPT-RE approach not only surpasses GPT-3 but also outperforms fully-supervised baselines. This highlights the efficacy of their model in enhancing RE performance.\n\n\n\n\n\n\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "2jyjIez6vi", "reviewer": "Reviewer_Ucne", "summary": "", "questions": "In section 2.4.2, we define the relation representation as Rel = hi + hj. Does the relation representation include the context in between, or does it include only the representations of two entities? If it is the former, it is not surprising that the GPT-RE_FT model achieves good performance, the way the relation representation is obtained is similar to previous works, but instead of putting through a neural network, now it is put into a retriever. The GPT-RE_SimCSE has the novelty in the entity prompts, which is still limited.\n\nHowever, if it is the latter, as stated in section 2.4.2 \"representation Rel is naturally enriched with the entity information.\", it is kind of weird that the GPT-RE_FT can achieves state-of-the-art with only entity information and without context.\n\nIn section 3.1, \"Due to the cost of running the model in the API with GPT-3, in our main results, we sample a subset\". What are the rules to take the proportion for the subset? It seems the subset size is not proportional to either the size of training or test set.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable advice for improving this paper.\n\n**Question**: “In section 2.4.2, we define the relation representation as Rel = hi + hj. Does the relation representation include the context in between, or does it include only the representations of two entities? If it is the former, it is not surprising that the GPT-RE_FT model achieves good performance, the way the relation representation is obtained is similar to previous works, but instead of putting it through a neural network, now it is put into a retriever. The GPT-RE_SimCSE has the novelty in the entity prompts, which is still limited.\nHowever, if it is the latter, as stated in section 2.4.2 \"representation Rel is naturally enriched with the entity information.\", it is kind of weird that the GPT-RE_FT can achieve state-of-the-art with only entity information and without context.\n\n**Answer**: In format, “Rel = hi + hj” is the concatenation of two marker vectors regarding two entities and does not include the surrounding vectors in between. However,  transformer-based pre-trained models like BERT naturally learn contextualized token representations that aggregate context information by the self-attention mechanism.  During the fine-tuning of BERT-based RE models, the representation “Rel” has already been trained to capture the relevant contextual information to predict the target label, and existing work (refer [1][2]) has proved the SOTA performance of marker-based methods in RE. Thus, semantically we could say that “Rel” includes the context between two entities.\n\n[1]: A Frustratingly Easy Approach for Entity and Relation Extraction. (Zhong +, NAACL 2021)\n\n[2]: Matching the Blanks: Distributional Similarity for Relation Learning. (Soares +, ACL 2019)\n\n**Question**: In section 3.1, \"Due to the cost of running the model in the API with GPT-3, in our main results, we sample a subset\". What are the rules to take the proportion for the subset? It seems the subset size is not proportional to either the size of training or test set.\n\n**Answer**: The number of sampled examples is not only related to the size of the training data itself. A more important factor is the proportion of NULL. In datasets with a high proportion of NULL, we have to maintain the original label distribution. Thus, the rule to sample the subset is to keep the proportion of each relation label consistent with the original test set. \nBelow are label distributions of two subsets.\n\nFor ACE05, \n| Label |  #num| \n| ----------- | ----------- |\n| PHYS |  26| \n| GEN-AFF |  12| \n| PER-SOC |  11| \n| ORG-AFF |  33| \n| PART-WHOLE |  13| \n| ART |  19| \n| NULL |  2329| \n\nFor TACRED, \n| Label |  #num| \n| ----------- | ----------- |\n|Per:title | 40|\n|Per:city_of_death | 1|\n|Org”shareholders | 2|\n|Per:origin | 12|\n|Org:top_members/employees | 36|\n|Org:city_of_headquarters | 11|\n|Per:religion | 4|\n|Per:city_of_birth | 1|\n|Per:employee_of | 27|\n|Per:data_of_death | 3|\n|Per:other_family | 5|\n|Org:website | 6|\n|Per:cause_of_death | 3|\n|Org:subsidiaries | 4|\n|Org:stateorprovince_of_headquarters | 5|\n|Per:countries_of_residence | 10|\n|Per:siblings | 5|\n|Per:stateorprovinces_of_residence | 11|\n|Org:alternate_names | 27|\n|Per:spouse | 4|\n|Per:parents | 7|\n|Org:country_of_headquarters | 9|\n|Per:age | 21|\n|Per:date_of_birth | 1|\n|Per:country_of_death | 1|\n|Per:schools_attended | 4|\n|Org:member_of | 3|\n|Per:children | 5|\n|Org:parents | 7|\n|Per:cities_of_residence | 24|\n|Per:stateorprovince_of_brith | 1|\n|Per:charges | 12|\n|Org:founded | 2|\n|Org:country_founded_by | 5|\n|Per:stateorprovince_of_death | 1|\n|Org:members | 4|\n|Per:country_of_birth | 1|\n|Per:alternate_names | 1|\n|Org:number_of_employees/members | 1|\n|Org:dissolved | 1|\n|Org:political/religious_affiliation | 1|\n|NULL | 1271|\n\nWe also think it important to choose a reasonable sample strategy for these two datasets.\n- For comparison with previous GPT-3 baselines, since the performance gap is too large between previous GPT-3 baselines and our GPT-RE, we think that the current subset is sufficient to show the effectiveness. Also note that some previous GPT3-based work uses much smaller subsets in their experiment (e.g., #1000 examples for each dataset in refer [1], and we have tried our best to choose as large as possible proportion considering our allowing.\n- For comparison with the SOTA supervised baselines, \n    - GPT-RE_FT on ACE05 achieves competitive performance since we keep the same proportion of NULL in our subset, otherwise, the GPT-RE may outperform the supervised baseline. We did experiments by removing all NULL examples in the test set, leading to the improvement of all GPT-based methods, which is consistent with Figure 8.\n    - GPT-RE_FT on TACRED surpasses the supervised baseline in the current subset. As we show above, some labels in TACRED are indeed not well presented (only 1 example), since TACRED dataset contains some long-tail labels. We decided to add additional results of GPT-RE_FT by enlarging our sampled set to # 3200 (2 times the current version). We already finished the experiment, and the performance of GPT-RE_FT (k = 15) is 73.16 while the performance of PURE is 70.48. \n\nWe will add statistics of subsets above, and the additional experiment results in our camera-ready version.\n\n[1]: Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again. (Gutiérrez +, EMNLP 2022 Findings)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written with good survey of related works.\n\nThe authors showed that with task-aware representations, all the retrieval-based models have higher performance than the GPT-Random, while the gold label-induced reasoning helps to improve the performance.\n\nThe approach can be a good resource for future research of LLMs to resolve the RE problems.", "reasons_to_reject": "In section 2.4.2, we define the relation representation as Rel = hi + hj. Does the relation representation include the context in between, or does it include only the representations of two entities? If it is the former, it is not surprising that the GPT-RE_FT model achieves good performance, the way the relation representation is obtained is similar to previous works, but instead of putting through a neural network, now it is put into a retriever. The GPT-RE_SimCSE has the novelty in the entity prompts, which is still limited.\n\nHowever, if it is the latter, as stated in section 2.4.2 \"representation Rel is naturally enriched with the entity information.\", it is kind of weird that the GPT-RE_FT can achieves state-of-the-art with only entity information and without context.", "questions_to_authors": "In section 2.4.2, we define the relation representation as Rel = hi + hj. Does the relation representation include the context in between, or does it include only the representations of two entities? If it is the former, it is not surprising that the GPT-RE_FT model achieves good performance, the way the relation representation is obtained is similar to previous works, but instead of putting through a neural network, now it is put into a retriever. The GPT-RE_SimCSE has the novelty in the entity prompts, which is still limited.\n\nHowever, if it is the latter, as stated in section 2.4.2 \"representation Rel is naturally enriched with the entity information.\", it is kind of weird that the GPT-RE_FT can achieves state-of-the-art with only entity information and without context.\n\nIn section 3.1, \"Due to the cost of running the model in the API with GPT-3, in our main results, we sample a subset\". What are the rules to take the proportion for the subset? It seems the subset size is not proportional to either the size of training or test set.", "paper_topic_and_main_contributions": "In this paper, the authors proposed an approach for relation extraction. Grounded on large language models (LLMs) via in-context learning (ICL), it prompts LLMs with the task instruction and demonstration data to generate the corresponding relation. They proposed two methods for representation, task-aware representations and  gold label-induced reasoning. The first includes Entity-Prompted Sentence Embedding, which reconstructs the context for entity information and extracting relation representation from a fine-tuned RE model and use these representations for retrieval. Experiments were done on four datasets: Semeval, TACRED, SciERC, and ACE05. The proposed model achieves state-of-the-art performance on the Semeval and SciERC datasets.\n\nThe paper is well written with good survey of related works.\n\nThe authors showed that with task-aware representations, all the retrieval-based models have higher performance than the GPT-Random, while the gold label-induced reasoning helps to improve the performance.\n\nThe approach can be a good resource for future research of LLMs to resolve the RE problems.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZHgq3xoJuR", "reviewer": "Reviewer_wnQk", "summary": "", "questions": "Question A: Can you provide more information on the distribution of the relations in test subsets ?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for pointing out the typos, and we appreciate your valuable advice for improving this paper.\n\n**Question**: \"Can you provide more information on the distribution of the relations in test subsets ?\"\n\n**Answer**: The number of sampled examples is not only related to the size of the training data itself. A more important factor is the proportion of NULL. In datasets with a high proportion of NULL, we have to maintain the original label distribution. Thus, the rule to sample the subset is to keep the proportion of each relation label consistent with the original test set. \nBelow are label distributions of two subsets.\n\nFor ACE05, \n| Label |  #num| \n| ----------- | ----------- |\n| PHYS |  26| \n| GEN-AFF |  12| \n| PER-SOC |  11| \n| ORG-AFF |  33| \n| PART-WHOLE |  13| \n| ART |  19| \n| NULL |  2329| \n\nFor TACRED, \n| Label |  #num| \n| ----------- | ----------- |\n|Per:title | 40|\n|Per:city_of_death | 1|\n|Org”shareholders | 2|\n|Per:origin | 12|\n|Org:top_members/employees | 36|\n|Org:city_of_headquarters | 11|\n|Per:religion | 4|\n|Per:city_of_birth | 1|\n|Per:employee_of | 27|\n|Per:data_of_death | 3|\n|Per:other_family | 5|\n|Org:website | 6|\n|Per:cause_of_death | 3|\n|Org:subsidiaries | 4|\n|Org:stateorprovince_of_headquarters | 5|\n|Per:countries_of_residence | 10|\n|Per:siblings | 5|\n|Per:stateorprovinces_of_residence | 11|\n|Org:alternate_names | 27|\n|Per:spouse | 4|\n|Per:parents | 7|\n|Org:country_of_headquarters | 9|\n|Per:age | 21|\n|Per:date_of_birth | 1|\n|Per:country_of_death | 1|\n|Per:schools_attended | 4|\n|Org:member_of | 3|\n|Per:children | 5|\n|Org:parents | 7|\n|Per:cities_of_residence | 24|\n|Per:stateorprovince_of_brith | 1|\n|Per:charges | 12|\n|Org:founded | 2|\n|Org:country_founded_by | 5|\n|Per:stateorprovince_of_death | 1|\n|Org:members | 4|\n|Per:country_of_birth | 1|\n|Per:alternate_names | 1|\n|Org:number_of_employees/members | 1|\n|Org:dissolved | 1|\n|Org:political/religious_affiliation | 1|\n|NULL | 1271|\n\nWe also think it important to choose a reasonable sample strategy for these two datasets.\n- For comparison with previous GPT-3 baselines, since the performance gap is too large between previous GPT-3 baselines and our GPT-RE, we think that the current subset is sufficient to show the effectiveness. Also note that some previous GPT3-based work uses much smaller subsets in their experiment (e.g., #1000 examples for each dataset in refer [1], and we have tried our best to choose as large as possible proportion considering our allowing.\n- For comparison with the SOTA supervised baselines, \n    - GPT-RE_FT on ACE05 achieves competitive performance since we keep the same proportion of NULL in our subset, otherwise, the GPT-RE may outperform the supervised baseline. We did experiments by removing all NULL examples in the test set, leading to the improvement of all GPT-based methods, which is consistent with Figure 8.\n    - GPT-RE_FT on TACRED surpasses the supervised baseline in the current subset. As we show above, some labels in TACRED are indeed not well presented (only 1 example), since TACRED dataset contains some long-tail labels. We decided to add additional results of GPT-RE_FT by enlarging our sampled set to # 3200 (2 times the current version). We already finished the experiment, and the performance of GPT-RE_FT (k = 15) is 73.16 while the performance of PURE is 70.48. \n\nWe will add statistics of subsets above, and the additional experiment results in our camera-ready version.\n\n[1]: Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again. (Gutiérrez +, EMNLP 2022 Findings)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is well organized and mostly easy to follow.\n- The study addresses a performance gap between fine-tuned models and in-context learning for relation extraction, and it proposes a solution which achieve SoTA or at least competitive results on multiple, widely used RE datasets.\n- In additional ablation studies a deeper analysis of the performance demonstrates the potential of the approach in different settings.\n- Overall the study is an interesting contribution to the relation extraction task", "reasons_to_reject": "Regarding the experiments done on TACRED and ACE05: due to the cost of running the model the authors sampled test subsets from that data, which is larger then SEMEVAL and SciERC. The approach is understandable. However, strangely the sampled subsets are smaller than the whole test data of the other two datasets; I would expect the subsets of TACRED and ACE05 to have at least the same size or even a bigger size as the other two datasets in order to reflect the whole test data better. It does hamper the comparison between the models. This is especially to relevant wrt TACRED, which contains a much larger number of relations than other datasets, and  many of them are probably not well represented in the sampled subset. The authors plan to release the subsets, however the paper contains no information about the distribution of the relations in the test subsets.", "questions_to_authors": "Question A: Can you provide more information on the distribution of the relations in test subsets ?", "paper_topic_and_main_contributions": "The authors present an approach for the relation extraction task (RE) using large language models (LLM) and via in-context learning (ICL). They address the current shortcomings of ICL for RE, which according to the authors are the reason for poor results of LLMs used for RE. The authors try to solve the problem by (1) incorporating task-aware representations in demonstration retrieval (i.e. representations emphasizing entity and relation instead of kNN search based on sentence embeddings) and (2) enriching the demonstrations with gold label-induced reasoning logic. Additionally the problem of overpredicting of NULL examples is analyzed.\n\nThe approach is tested on four RE datasets (Semeval, TACRED, SciERC, ACE05). The approach is compared to two GPT-3 baselines (one using randomly selecting the demonstrations and another using sentence embedding retrieval) as well as fully-supervised models. The new model GPT-RE achieves SOTA performances on the Semeval and SciERC datasets, and competitive performance on TACRED and ACE05 datasets.\n\nThe authors analyze the results on multiple levels: (1) An ablation study regarding the retrieval of an increasing number of demonstrations shows that task-aware retrieval beats a random selection as well as the sentence embedding approach even for the lowest k demonstrations which indicates that the quality of demonstrations is more important than the number of demonstrations. (2) An ablation study on reasoning enhancing shows that inducing reasoning improves the performance, especially with fewer demonstrations, which might be an effective solution for low-shot relation extraction (3) A comparison of the performance in a setting without NULL examples shows that GPT-RE_FT (GPT with fine-tuned relation representation for retrieval) is most robust to the influence of a number of NULL examples, however it underperforms the fine-tuning baseline on NULL examples.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- l. 333: retrieval"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper calls out the limitations of LLMs with in-context learning in performing the task of relation extraction, that they still lag behind supervised baselines such as fine-tuned BERT. It underlines a very valid concern regarding LLMs for this important problem. It outlines the limitations of these models and proposes an approach to mitigate them. The paper is very well-written and easy to follow. It provides a comprehensive survey of the related works in this space on four-widely used RE datasets. The community can benefit from this work for furthering the research in relation extraction using LLMs. The author responses are succinct and clarify most of the concerns of the reviews.
Decision: Accept-Main