Revisiting Large Language Models as Zero-shot Relation Extractors
Authors: Guozheng Li
Peng Wang
Wenjun Ke
Venue: EMNLP 2023 Findings (2023)
Keywords: Relation Extraction
Large Language Models
Zero-shot Learning
TL;DR: Exploring LLMs such as ChatGPT as zero-shot relation extractors.

Abstract: Relation extraction (RE) consistently involves a certain degree of labeled or unlabeled data even if under zero-shot setting. Recent studies have shown that large language models (LLMs) transfer well to new tasks out-of-the-box simply given a natural language prompt, which provides the possibility of extracting relations from text without any data and parameter tuning. This work focuses on the study of exploring LLMs, such as ChatGPT, as zero-shot relation extractors. On the one hand, we analyze the drawbacks of existing RE prompts and attempt to incorporate recent prompt techniques such as chain-of-thought (CoT) to improve zero-shot RE. We propose the summarize-and-ask (\textsc{SumAsk}) prompting, a simple prompt recursively using LLMs to transform RE inputs to the effective question answering (QA) format. On the other hand, we conduct comprehensive experiments on various benchmarks and settings to investigate the capabilities of LLMs on zero-shot RE. Specifically, we have the following findings: (i) \textsc{SumAsk} consistently and significantly improves LLMs performance on different model sizes, benchmarks and settings; (ii) Zero-shot prompting with ChatGPT achieves competitive or superior results compared with zero-shot and fully supervised methods; (iii) LLMs deliver promising performance in extracting overlapping relations; (iv) The performance varies greatly regarding different relations. Different from small language models, LLMs are effective in handling challenge none-of-the-above (NoTA) relation.

Revisiting Large Language Models as Zero-shot Relation Extractors
Guozheng Li and Peng Wang and Wenjun K
School of Computer Science and Engineering, Southeast University, China Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China { {liguozheng, pwang, kewenjun)@seu. edu. cn
Abstract
from texts using zero or few-shot methodologies has garnered significant scholarly attention (Han et al., 2018; Chen and Li, 2021).
Relation extraction (RE) consistently involves a certain degree of labeled or unlaheled data even if under zero-shot setting. Recent studies have shown that large language models (L.LMs) transfer well ply given natural language prompt, which pro- vides the possibility extracting relations from text without any data and parameter tuning. This work focuses on the study of exploring LLMs, such as ChatGPT, as zero-shot relation extractors. On the one hand, we analyze the drawbacks of existing RE prompts and attempt incorporate recent prompt techniques such as chain-of-thought (CoT) to improve zero- shot RE. We propose the summarizz-and-ask (SUMASK) prompting. simple prompt recur- sively using L.LMs to transform RE inputs to the effective question answering (QA) format. On the other hand, we conduct comprehensive experiments on various benchmarks and set- tings investigate the capabilities of LLMs on zero-shot RE. Specifically, we have the following findings: (i) SUM ASK consistently and significantly improves L.MSs performance on different model sizes, benchmarks and set- tings; (ii) Zero-shot prompting with ChatGPT achieves competitive superior results com- pared with zero-shot and fully supervised meth- ods: (iii) LLMs deliver promising performance in extracting overlapping relations; (iv) The performance varies greatly regarding different relations. Different from small language mod- els, LLMs are effective in handling challenge none-of-the-abov (NoTA) relation.
Recent studies (Wei et al., 2022: Wang et al., 2023b) on large-scale pre-trained language mod- els (LLMs), such as GPT-3 (Brown et al., 2020), demonstrate that LLMs perform well in various downstream tasks without any training fine- tuning but only with few examples instruc- tions, which is called in-context learning. However, there is currently no consensus on whether LLMs are good few-shot information extractors (Agrawal et al., 2022; Jimenez Gutierrez et al., 2022). Dif- ferent with some other tasks, RE more challeng- ing for LLMs because the structured data contain- ing multiple dependent elements are difficult to extract directly and accurately. Although recent studies (Wang et al., 2023a) indicate that some conventional fine-tuning models still outperform LLMs in few-shot RE tasks, we still want to ex- plore whether LLMs can achieve competitive per- formance compared to fine-tuning models.
tasks out-of-the-box sim-
Similar to few-shot learning, LLMs also show promising performance on zero-shot settings (Ko- jima et al., 2022). Recent work for zero-shot RE via prompting LLMs has achieved remarkable progress. QA4RE (Zhang et al., 2023) is a multiple- choice question answering prompt format, in which each relation transformed into template and LLMs are expected predict only single let- ter. This prompt is simple but requires manually crafted templates and unable to deal with overlap- ping relations, which motivates us to find more general and effective prompts. ChatlE (Wei et al., 2023) transforms the zero-shot IE task into a multi- turn question answering problem with two-stage framework, even surpasses some full shot models on several datasets. Nonetheless, ChatlE still per- forms worse than the state-of-the-art and is only evaluated on limited benchmarks. Thus is still unclear how to improve extracting performance by
1 Introduction
Relation extraction (RE) aims to identify the re- lationships between entities in texts, and plays important role in information extraction (IE). Most of existing RE methods (Zeng et al.. 2014; dos San- tos et al., 2015) require large amounts of labeled training data which is labor-intensive and time- consuming in practice. Hence, extracting relations

---

designing effective prompts and whether LLMs are good zero-shot relation extractors. To this end, we revisit and investigate the potential of LLMs in zero-shot RE on the following research questions:
in Fl-score compared VANILLA prompting w.r.t. diverse experimental settings.
We comprehensively evaluate the capabilities of LLMs on zero-shot relation classification. LLMs achieve competitive superior results compared with stte-of-thheaat zero-shot and fully supervised methods. Notably, ChatGPT with SUMASK prompting outperforms the state-of-the-an art fully supervised method on TACRED by an average of 2.8% micro-F1.
(RQ1) How does LLMs perform on RE incor- porating existing prompt techniques?
(RQ2) How does LLMs perform on zero-shot relation classification?
(RQ3) How does LLMs perform on zero-shot overlapping relation extraction?
We investigate the capabilities of LLMs in tracting overlapping relations. LLMs deliver consistently promising performance encoun- tering different number of triples and various overlapping patterns.
Previous work (Ma et al., 2023; Wang et al., 2023a) fails to achieve promising results on RE since black box LLMs such ChatGPT are dif- ficult ensure the reliability of outputs. To an- swer the first question, we investigate the feasibil- ity of incorporating recent prompt techniques to improve the reliability of extracted results. For example, chain-of-thoughi (CoT) prompting (Wei et al., 2022) improves the reliability of model out- put by providing intermediate reasoning steps. Ac- tive prompting (Diao al., 2023) is an uncertainty- based active learning method quantify the un- certainty so as to select the most uncertain outputs. Specifically, we propose the summarize-and (SUMAsK) prompting, which decomposed RE into two subtasks: text summarization (Liu and Lap- ata, 2019) and question answering (Chen et al., 2017a). We further introduce an uncertainty esti- mation method to approximately characterize out- put probabilities of LLMs, which yields substantial improvements compared to VANILLA prompting. To answer the last two questions, we evaluate LLMs on both zero-shot relation classification and overlapping relation extraction. And six RE bench- marks are used for evaluation: (1) FewRel (Han et al., 2018) and Wiki-ZSL (Chen and Li, 2021) for comparision with zero-shot RE methods; (2) TACRED (Zhang al., 2017), TACREV (Alt et al., 2020) and Re-TACRED (Stoica et al., 2021) for comparision with fully supervised methods; (3) NYT (Riedel et al., 2010) for evaluating over- lapping relation extraction. Experimental results demonstrate that LLMs achieve promising results in all experimental settings. In summary, the con- tributions of this work are three-fold:
Related Work
Few and Zero-shot Relation Extraction Few- shot RE (Han et al., 2018) aims to predict novel relations by exploring few labeled instances. Pro- totypical networks (Snell et al., 2017) are widely used and combined with pre-trained language mod- els (Devlin al., 2019) in few-shot settings achieve impressive results. To be capable of ex- tracting relations that were not specified in advance, zero-shot RE (Levy et al., 2017) is proposed to in- vent new models to predict new relations. How- ever, existing zero-shot methods (Chen and Li, 2021) still requires much labeled data. Recent stud- ies (Zhang et al., 2023; Wei et al., 2023) leverage the LLMs with zero-shot prompting extract re- lations from texts without any labeled samples advance. But it is still unclear whether LLMs are good zero-shot relation extractors by carefully de- signed prompts. Thus this work aims to investigate the capabilities of LLMs in zero-shot RE.
Large Language Models and Prompting Be- sides the "pre-train and fine-tune" paradigm (Liu et al., 2023), pre-trained LLMs possess character- istics that are advantageous for few-shot (Brown et al., 2020) and zero-shot (Kojima et al., 2022) learning, whereby appropriate prompts are used to effectively guide the model towards generat- ing desired task outputs, thus beginning era of "pre-train and prompt" (Liu et al., 2021). Prior works (Zhao et al., 2021; 1; Liu et al., 2021) note the sensitivity of prompting under slight modifications. Empirical results demonstrate that answering the restrictive prompts is challenging due to biases ac- quired during pre-training (Zhao et al., 2021; 1; Arora
We propose the SUMASK prompting and evaluate the effectiveness of this method. SUMASK consistently and significantly im- proves LLMs performance by 5.2% 48.3%

---

al., 2023). In this study, we evaluate different prompt formats tailored to the particularities of RE and propose SUMASK prompting which outper- forms VANILLA prompting by a large margin.
Given the possible relations: [member field of work, work location, ..... father, sibling). What the relations between the subject entity and the object cntity expressed by the sentence? Sentence: Savi was born in Pisa, son of Gactano Savi, professor of Botany at the University of Pisa.
3 Problem Definition
Subject: Gactano Savi
Previous zero-shot RE (Chen and Li, 2021) only involves single relation classification. We extend this zero-shot setting to multiple entities and re- lations. Given the pre-defined relation set R = (ri, r,,,.,,,NN' and the sentence S containing the entity set E = {e1, ....... ) we aim to extract all the relations between these entities composing the relational triples set Z {(Ei, Tky (er,)))))))))))) E;) where N denotes the number of relations, M represents the number of entities, and ei, ej E & r E R.
Object: Botany
Relation: field of work
Figure I: Illustration of the VANILLA prompting. The output LLMs is highlighted in color
processes mentioned above, we design the prompt illustrated in Figure 2 as three steps: (i) Summa- rize the relations between subject and object given the [INPUT] between the two and obtain the intermediate re- sult [SUMMARIZATION]; (ii) Ask LLMs to generate the yes/no questions based on possible triples so as to transform the abstract relation labels to the natural relation descriptions and obtain the interme- diate result [QUESTION]; (iii) Ask LLMs to answer the [QUESTION] based on the [SUMMARIZATION] to match the relation semantics between the enti- ties and relations. Then we get the final [ANSWER].
extract the relation semantics
4 Prompt Design
4.1
VANILLA Prompting
Previous work (Ma et al., 2023; Wang et al., 2023a) claims that LLMs achieve poor results on IE tasks such as RE. We argue that one important reason why LLMs underperform the statc-of-the-ar the poor prompt design, as different prompts towards same tasks can cause large variations in the model predictions (Zhao et al., 2021; ; Arora et al., 2023). Figure 1 illustrates the most direct and common prompt strategy which directly asks LLMs to ex- tract relation labels from text through instructions. However, we empirically find that this approach is ineffective because it makes LLMs to accom- plish three no-trivial reasoning processes in only one step: (i) Extracting the relation semantics be- tween the subject and object in the sentence; (ii) Understanding the semantics of each relation la- bel; (iii) Matching the relation semantics between the entities and the given relation labels. Consis- tent with existing findings (Jimenez Gutierrez et al., 2022; Ma et al., 2023; Wang et al., 2023a), LLMSs using VANILLA prompting are unable to achieve satisfactory performance on zero-shot RE.
Uncertainty Estimation Generally, relation clas- sification assumes that only one correct relation is extracted. However, SUMASK prompt possibly obtains multiple "yes" while querying all the rela- tions in R. Therefore, the final predicted relation is required to select from multiple candidates. Given the sentence S, the subject ea and object co, the predicted relation is obtained by:
r arg max (riilS,e,,e)) NOER
(1)
Then we aim to transform the multi classifica- tion form into multiple binary classification forms. Hence, we define random variable F as follows:
(1. 0, ... 0) r=
4.2 SUMASK Prompting
(0. 1. 0) r r2
Due to the difficulty of LLMs in completing three reasoning processes in one step, we leverage the idea of CoT (Wei et al., 2022) and suggest decom- posing this step artificially guide LLMs in under- standing and reasoning. To classify the relations between the subject and object, simple method is to sequentially ask LLMs whether each relation exists between two entities. For the three reasoning
(2)
(0. 1) r= FN
Here we make an assumption that only one positive label exists among N binary classification. We denote the intermediate results summarization and
question as 8, and qi corresponding to relation r;-

---

[INPUT]
Savi born in Pisa, of Gaetano Savi, professor of Rotany the University of Pisa.
Summarize the relations brtwern SURJECT and ORJECT from contest.
Rewrit triple question. Triple: (SURJCC ORJFCT) Question: [QUESTION)
Answer the qurstion from context with yrchus.
Contest:: BuNiNNSAIEATION]
Questio:: [QUESTION]
Summaricction: [SUMMARIZATION) (n.
(ANSWER]
p p( (o,,
(F.
St, 9c)
dieitano Sest professor Botary at the University
[1] dastano Sast sperifically engaged in the his work?
Wes. No.
Majurity
Yes
University Picz, Goetano Sast serves as professor specialiing in Botary.
Does Gretano professional occupation musiow anound Botsmy?
holds Botany at University of Piia.
professorin
place
Botany?
Vote
[5] No.
[K] Gzetano Savl a protessor Botany at the University of Piza
Is Gzetano Sout's residncce associated with the field of Botanyy
No.
U (a: e,e,, Fi.
U [g.. fo.
p (ficid of work)) (f) Finiss (orre) (q (,, Tf monn)) Usthurrrstatiun 0 Priatd of 1) (/iind = {5.9)
p (x (5, 5,e4, Tyeas work) (q (5, For of p (Tpoos of work (3.4)
1/x Ú (s | S, Tyote wora) U (y (,,,, Ty-Teii of work) U (Freed
P (residence) = (s 18, Ferestonc)) y (q Pg. Farker retation
0 Frestaesce 1) Fresiaesce = ( =
Figure 2: Illustration of the SUMASK prompting. The outputs of LLMs are highlighted color. The probability of relation "residence" is 0 because the system answers "no" via majority vote. To estimate the uncertainty of relation "field of work", we generate & [ [SUMMARIZATIONN [QUESTION], [ANSWER] representations, respectively. Then we calculate the dispersion degree among these representations to approximate the uncertainty.
Then the probability of relation is obtained by:
probabilities. Finding the relation that satisfies equation 5 is equivalent to:
p (r.) (r (r|S, C.. Co, Ti)
(p (Fl sis 9:) (s;, 9: |S, es ri)
r arg min U (F; 11 1 T.ER U (gi (S Cos Co: r;) U (8; ca.
-p =p (F- F, = (F; = 1 Si, 9i)
(6)
p (q: S, e. Co, r:) (s: S, e. Coy ri))
(3)
where 1 indicates that LLMs answer "yes" based on the summarization and question of re- lation i. Based on the above positive label assumption, have:
where U(}{Y represents the uncertainty of the random variable X under the known random vari- able Y. Therefore, the relation with the smallest uncertainty selected as final prediction.
Inspired by Diao et al. (2023), we consider mea- suring the uncertainty using the dispersion degree among k generated answers A = {ar,... ax} as shown in Figure 2. Specifically, we feed an- swers A into pre-trained Sentence-BERY en- coder (Reimers and Gurevych, 2019) to generate the answer representations Z (21, 3 Then the uncertainty is calculated by:
F
p P 0 F=
(4)
r:
The final predicted relation is selected from all the candidate relations with the max probability:
r' = arg max p (Fr = 8i, q:)
TanCR
p (q: (i es Co. rz) p (s: S, r.)
Unfortumately, it is difficult get the conditional probability of each step in LLMs. For instance, the "ept-3.5-turbo" model only provides the final natural text output without any logit or probability. To this end, we introduce an uncertainty estimation method to approximately characterize conditional
k
(Zis
2i
(7)
1:
where d(.) function measures the distance between two representations. After obtaining the uncer-

---

tainty of each step, we select the relation via:
relations given the entities in its test set. We use micro-F1 for evaluation. NYT is only evaluated on LLMs as existing baselines have not considered this multiple entities and relations zero-shot setting. To keep OpenAT API costs under control, we randomly select 1,000 samples in the correspond- ing test set according to the proportion of samples in each relation class. Specifically, the number of samples corresponding to each relation in FewRel is the same. Note that 78.56% of samples in TA- CRED test set belongs to NoTA relation, while it is 57.91% in Re-TACRED. We provide the statistics of the datasets in Appendix A.
arg min 22 u3
(8)
T.MER
01 x U (st] S, Ca ri)
(9)
u2 U (q: S, Es: Co. ri)
(10)
ug U (r: 1 Si, qi)
(11)
We adopt the majority vote (Wang et al., 2023b) to determine the yes/no answer in last step. If the system answers "no" with every relation r; C R. the prediction NoTA. For overlapping relations, we simply consider all relations that answer with "yes" as predictions.
5.2 Baselines
Entity-Relation Mapping Obviously, asking LLMs for each relation is inefficient. Inspired by Li et al. (2022), we adopt the entity-relation mapping mechanism to deal with relation redun- dancy. Specifically, when the entity type is de- termined, the relations that possibly related to it are also determined, so most impossible relations are discarded in advance. Note that the VANILLA prompting also adopts this simple strategy. This simple mechanism not only improves efficiency but also benefits overall performance.
Zero-shot Baselines For FewRel and Wiki- ZSL, we choose R-BERV (Wu and He, 2019), ESIM (Chen et al.. 2017b), CIM (Rocktäschel et al., 2016) and ZS-BERT (Chen and Li, 2021) as the zero-shot RE baselines. Note that Relation- Prompt (Chia et al., 2022) uses the seq models to generate pseudo data of unseen relations to fine-tune the model. And recent method RE- Matching (Zhao et al., 2023) requires elaborated re- lation descriptions achieve superior performance but the open source code is not yet available. Thus the two methods are not discussed in this study.
5 Experiments
5.1 Datasets
Supervised Baselines For TACRED, TACREV and Re-TACRED, fully supervised models such as PA-LSTM (Zhang et al., 2017), C- GCN (Zhang et al., 2018), SpanBERT (Joshi et al., 2020), LUKE (Yamada et al., 2020), NLI-DeBERTa (Sainz et al., 2021), SuRE- PEGASUS (Lu et al., 2022) and DeepStruct (Wang et al., 2022) are selected to compare with our zero-shot prompt based methods. We also test NLI-DeBERTa, SURE-PEGASUS, DeepStruct and QA4RE (Zhang et al., 2023) under zero-shot setting to investigate the NoTA relation impact.
Simple Relation Classification We evaluate LLMs in zero-shot simple relation classification on FewRel (Han al., 2018), Wiki-ZSL (Chen and Li, 2021), TACRED (Zhang et al., 2017), TACREV (Alt et al., 2020) and Re-TACRED (Sto- ica et al., 2021). For FewRel and Wiki-ZSI, we follow the previous work (Chen and Li, 2021) and randomly select m = 5, 10, 15 unseen rela- tions. For TACRED, TACREV and Re-TACRED, we evaluate the LLMs on its test set. Note that there is no entity type information provided in FewRel and Wiki-ZSL while entity types become available in TACRED, TACREV and Re-TACRED. We adopt the precision, recall and macro-F1 for FewRel and Wiki-ZSL (Chen and Li, 2021), while micro-Fi is used for TACRED, TACREV and Re- TACRED (Zhou and Chen, 2021).
LLMs Baselines We investigate open source LLMs such as GPT-J (Wang and Komatsuzaki, 2021), BLOOM (Scao et al., 2022) and TO (Sanh et al., 2022) with SUMASK prompting with other state-of-the- models in zero-shot settings. For the parameter scale, we choose GPT-J-6B 1, BLOOM-7.1B and TOpp- 11B for experiments. For ChatGPT (Ouyang et al., 2022), we use the
Overlapping Relation Extraction We adopt the NYT (Riedel et al., 2010) to test the ability of ex- tracting overlapping relations. For NYT, we as- sume the entities and their types in the sentence are available and models only extract the overlapping
http://hgggnggeee..oeeeeeeeeaaaa httpss//wuggnngfeeeecooggieeeeee cobbigsciencc/hlomm https//hhugginnfacccooocgciiioip

---

Datasets
FewRel
Wiki-ZSL
Datasets
TACRED
TACREV
Re-TACRED
m=5
m=5
FI
PA-LSTM
63.1
73.3
79.41
C-GCN
66.3
74.6
80,31
R-BERT
42.19
48.61
45.17
39.22
43.27
41.15
SpanBERT
70.8
78.0"
85.3
ESIM
56.27
58.44
57.33
48.58
47.74
48.16
LUKE
72.7
80.6
90.3
CIM
58.05
61.92
59.92
49.63
48.81
49.22
NLI-DeBERTa
73.9
7S ZS-BERT
76.96
78.86
77.90
71.54
72.39
71.96
SuRE-PEGASUS
75.1
GPT-J
40.75
45.63
43.05
40.23
46.94
43.33
DeepStruct
76.8
BLOOM
43.62
48.15
45.77
41.97
45.38
43.61
GPT-J
44.4
40.7
38.3
TO
43.05
54.97
48.29
42.16
53,92
47.32
BLOOM
46.5
40.8
VANILLA
67.41
72.97
70.08
64.47
70.83
67.50
TO
59.0
57.5
55.5
SUMASK
78.27
72.55
75.30
75.64
70.96
73.23
VANILLA 31.3
30.4
28.0
m=10
m=10
SUMASK
79.6
73.8
P
R
F1
R
F1
R-BERT
25.52
33.02
28.20
26.18
29.69
27.82
Table 2: Micro-FI score on TACRED, TACREV and Re- TACRED. marks re-implemented results from Alt al. (2020). i marks re-implemented results from Stoica et al.
ESIM
42.89
44.17
43.52
44.12
45.46
44.78
CIM
47.39
49.11
48.23
46.54
47.90
45.57
7S-BERT
56.92
57.59
57.25
60.51
60.98
60.74
(2021). t marks re-implemented results from Zhou and
28.37
32.27
30.19
27.13
32.76
29.68
Chen (2021). Others are retrieved from original papers.
BLOOM
29.28
33.81
31.38
29.45
34.19
31.64
TO
29.87
34.26
31.91
30.18
35.48
32.62
VANILLA
42.48
46.26
26
44.29
41.83
46.22
43.92
Methods
Micro-P P Miero-R R Miero-FI Macro-F1
SUMASK
64.77
60.94
62.80
62.31
61.08
61.69
SURE-PEGASUS
13.8
51.7
21.8
14.9
m=15
m=15
NI.-DeBERRTa
42.9
76.9
55.1
55.0
P
F1
DeepStructi
32.7
40.6
36.2
32.8
QA4RE 47.7
78.6
59.4
58.9
R-BERT
16.95
19.37
18.08
17.31
18.82
18.03
ESIM
29.15
31.59
30.32
27.31
29.62
28.42
VANILLA SUMASK
33.8
39.2
36.3
27.4
CIM
31.83
33.06
32.43
29.17
30.58
29.86
62.2
53.8
57.7
57.9
7S- ZS-BERT
35.54
38.19
36.82
34.12
34.38
34.25
41-class micro-F1 and NoTA included 42-class macro F1 on TACRED, marks our re- implementation results. The rest results of the baselines are retrieved from Zhang et al. (2023).
Table 3:
GPT-J
20.36
35.00
25.74
20.83
34.37
25.94
BLOOM
22.62
36.45
27.92
22.37
34,26
27.07
TO
24.05
36.83
29.09
23.16
34.90
27.84
VANILLA
25.71
27.77
26.70
23.17
27.82
25.28
SUMASK
44.76
41.13
42.87
43.55
40.27
41.85
Table Main results on FewRel and Wiki-ZSL. In order to reduce the effect of experimental noise, the un- seen label selection process repeated for five different random seeds produce the test set. The results of the baselines are retrieved from Chen and Li (2021).
tions. Specially, the proposed SUMASK prompting makes ChatGPT deliver superior results compared to ZS AS-BERT in most cases. As increases, it is straightforward that models are difficult predict the right relation since the possible choices have increased. The superiority of SUMASK gets more significant when the number of unseen relations increases while VANIILA suffers grave declines. Such results not only validate the effectiveness of proposed prompting. but indicate SUMASK is less sensitive to the number of relations compared to baselines. Moreover, GPT-J-6B, BLO0MM7.11B and TO-11B with SUMASK exceed GPT-335-175B with VANILLA, and match the performance of pre- vious text entailment models ESIM and CIM.
10t-3.5-turbo-0301"" which is the most capable GPT-3.5 model and optimized for chat. We denote the combination of ChatGPT and two prompts as VANILLA and SUMASK for brevity. Similar to Ko- jima al. (2022), after the model outputs a text, our method picks up only the part of the answer text that first satisfics the answer format. The im- plementation details are provided in Appendix B.
5.3 Relation Classification Results
The main results on TACRED, TACREV and Re- TeCRED are shown in Table 2. Compared fully supervised methods, zero-shot prompting with LLMs still show competitive results. Notably, ChatGPT with SUMASK prompting outperforms the state-of of- the-art fully supervised method Deep- Struet on TACRED by an average of 2.8% micro-
Main Results The results by varying unseen relations on FewRel and Wiki-ZSL summa- rized in Table 1. Generally, LLMs with zero-shot prompting achieve competitive results compared to existing zero-shot RE methods over two datasests when targeting at different numbers of unseen rela-

---

Best performance
Accuracy
Worst performance
Accuracy
voice type
97.4
language of work
13.0
occupation
97.1
tributary
22.7
contains administrative territorial entity
95.4
residence
31.6
participant
95.1
mouth of the scrcenwriter
33.3
95.0
42.4
located in the administrative territorial entity
94.9
performer
44.0
league
head govermment
44.3
constellation
93.9
father
45.3
competition class
distributor
48.4
heritage designation
located on terrain feature
49.6
Table 4: Top- 10 relations with best (left) and worst (right) performance.
Fl. The interesting finding is that the performance of LLMs decreases TACREV and Re-TACRED while fine-tuned models steadily improves. The rea- son might be that the high proportion of NoTA in TACRED makes zero-shot prompting with LLMs surpass fully supervised methods. It difficult for conventional models to form good NoTA repre- sentation (Han et al., 2020). Jimenez Gutierrez et al. (2022) also demonstrate that the earlier in- ferior performance of LLMs RE tasks can be largely attributed to their inability handle the NoTA relation. To this end, we provide an evalua- tion of zero-shot methods on NoTA relation. Fol- lowing previous work (Sainz et al., 2021; Zhang et al., 2023), we report the NoTA-excluded micro- Fl and MoTA-included macro-F1 to investigate the extracting ability of normal and NoTA relation.
0004
Mitm-1
G713
Yaril
Performance comparison between five similar, random and dissimilar relations.
Figure 3:
best and worst performance, as shown in Figure 4. Surprisingly, the accuracy difference between the best ("vviice type") and the worst ("language of work name") relation is 84.4%. We provide the detailed analysis in Appendix C.
The semantic similarity between relations in the embedding space greatly impacts the zero-shot RE performance. Following Chen and Li (2021), we select five semantically distant relations and the other five relations that possess similar semantics evaluate on our baselines, illustrated in Figure 3. Obviously, dissimilar relations lead better sults. First, when enhanced by SUMASK prompt- ing, LLMs delivers more stable results because of the smaller performance gap between three set- tings. Second, the text-cntailment based methods are less affected by similar relations compared to embedding- based models such R-BERT and ZS- BERT. Because the predictions by text entailment based methods ESIM, CIM and SUMASK prompt- ing do not resort similarity search.
The NoTA relation results are shown in Table 3. First, SUMASK prompting is not prominent in semantic relations, which demonstrate the high micro-FI score is mainly due to the high portion of NoTA relation. Second, SUMASK also achieves significant improvement like QA4RE in NoTA- included metrics compared to the small LM-based NLI methods. For VANILLA, excluding NoTA rela- tion brings better results. This further demonstrates the sensitivity of prompting and the effectiveness of proposed prompting.
Relation Specific Analysis Due to biases ac- quired during pre-training, LLMs have different abilities to understand different relations, which leads to varying levels of extraction results. We analyze the performance differences through ex- periments on 80-relation dataset FewRel. Specifi- cally, under the SUMASK framework, we ask the LLMs whether the answer of question generated by golden triple is "yes". Then we adopt the ac- curacy metric to evaluate the performance of each relation. Finally, we select 10 relations with the
Prompt Strategy Analysis We study the effec- tiveness of proposed SUMASK prompting. We con- duct the ablation study about summarization gener- ation, question generation and uncertainty estima- tion. Specifically, we omit the summarization pro- cess, replace the LLMs generated questions with pre-defined question templates (Appendix D) and

---

Datasets
FewRel m=
TACRED NoTA w/o NoTA
Methods
F1
N=1
N=2
N=3
N=4
N==5
m=10 m=15
GPT-J
31.4
53,6
39.6
39.5
36.9
39.3
39.8
30.3
SUMASK
75.3
62.8
42.9
79.6
51.6
BLOOM
35.2
56.9
43.5
39.2
38.4
42.1
44.6
33,6
SUMASK w/o Sum. 67.6
58.1
40.4
76.4
48.5
TO
39.6
57.3
46.8
44.3
41.9
47.2
48.3
35.7
SUMASK w/o Ask. 71.4
56.8
37.7
78.7
54.3
VANILLA
20.4
16.5
18.2
31.3
22.7
16.3
12.8
7.7
SUMASK w/o Unc. 38.8 29.6
23.0
76.3
25.2
SUMASK
55.7
78.3
65.1
65.6
62.5
66.7
70.8
59.5
Table 5: Results of ablation study.
Table 6: Main results on NYT.
Visce
2
Gooed
SEP
NEO
SBD
Fên
Figure 4: The correlations between uncertainty estima- tion and ground truth.
Figure 5: Fl-score of extracting overlapping relations from sentences with different overlapping patterns.
randomly select the relation from candidates with- out uncertainty estimation, respectively. Table 5 shows the ablation study results. Summarization consistently improves the overall performance un- der different settings, which indicates that incor- porating reasoning steps before predicting relation is reasonable. Compared to pre-defined templates,
regarding all the candidate relations as predictions brings many false positives. Setting thresholds for uncertainty estimation might be feasible solution. To further study the capability of LLMs in ex- tracting overlapping relations, conduct experi- ments on different types of sentences. We split the sentences into five classes and use N denote the number of relational triples in a sentence. Again, the SUMASK prompting achieves good perfor- mance over all five classes. Overlapping relational triples are summarized into four patterns: SEP (Sin- gleEntiitPair), NEO (NoEntityOverlap), SEO (Sin- gleEnnityOverlap) and EPO (EntityPairOverlap), Figure 5 shows that the performance of most base- lines four patterns presents decreasing trend, reflecting the increasing difficulty of extracting re- lational triples from sentences with different over- lapping patterns, encouraging more consistent and effective methods in the future research.
LLMs generated questions may not necessarily be the best choice. Manually designed template en- ables the semantic description of relations more accurate, but our simple method is convenient and requires no external interference. Uncertainty es- timation shows the significant impact on perfor- mance. Note that SUMASK still achieves 76.3% Fl on TACRED, because the uncertainty estima- tion has no impact on NoTA relation theoretically. To understand the rationality of uncertainty esti- mation, we select 500 samples from each datasets to illustrate the correlations between uncertainty estimation and ground truth. Intuitively, golden relations have relatively low uncertainty. Figure shows that most golden relations correspond to low uncertainty, while only a few correspond to large uncertainty, which is consistent with our intuition.
Conclusion
This work provides a comprehensive study on zero- shot RE with compting-bassed LLMs. Besides the VANILLA prompting, we introduce a novel SUMASK prompting to fully explore the power of LLMs. Our experiments on six benchmarks demonstrate the capability of LLMs in zero-shot RF. Furthermore, we are able to answer the three aforementioned questions. Recent prompt tech- niques such as CoT significantly improve zero-shot RE prompting. Properly instructed LLMs not only deliver competitive or superior results compared to state-of- of-ec-ofthee--o tithhhart art relation classification models, but also are promising for zero-shot overlapping RE.
5.4 Overlapping Relation Extraction Results
The overlapping relation extraction results are illus- trated in Table 6. VANILLA prompting is difficult to handle the overlapping relations as LLMs al- ways tend output only one relation. In contrast, SUMASK prompting is transferable and consistent on L.MSs with different sizes. Note that different from the relation classification results, the recall of SUMASK is higher than its precision. Because

---

Limitations
Qian Chen, Xiaodan Zhu, Zhen-Hus Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017b. Enhanced LSTM for natural language inference. In ACI.
We only carry out comprehensive experiments on zero-shot RE without few-shot and exploration. It is still unclear what are the capabili- ties of LLMs on domain-specific datasets and how much performance could be improved by few-shot prompting. Our limited budget also restricted our study small set of prompt styles. It possible that having larger prompt design search space could narrow the gap between models fine-tuning and LLMs in in-context learning.
Yew Ken Chia, Lidong Bing, Soujanya Poria, and Luo Si. 2022. Relationprompt: Leveraging prompts to generate synthetie data for zero-shot relation triplet extraction. In Findings of ACL.
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing. In NAACL-HLTT.
Shizhe Diao, Pengcheng Wang, Yong Lin, and Tong Zhang. 2023. Active prompting with chain-of- thought for large language models. arXiv preprint arXiv:2302.
Ethics Statement
In this work, we investigate the capability of LLMs the important and fundamental task of zero-shot relation extraction. We do not anticipate any ethical issues regarding the topies of this research.
Cicero dos Santos, Bing Xiang, and Bowen Zhou. 2015. Classifying relations by ranking with convolutional neural networks. ACL.
Acknowledgement
Xu Han, Tianyu Gao, Yankai Lin, Hao Peng, Yaoliang Yang, Chaojun Xiao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. More data, more rela- tions, more context and more openness: A review and outlook for relation extraction. arXiv preprint MXXi:2000403188
This work was supported by National Science Foun- dation of China (Grant 6os.62376057), a special- ized intelligent science project (2023-1 (033110 XX Technology for Single and Group Intelligent, a specialized intelligent science project Key entity recognition and analysis technology in XX field.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang. Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: A large-scale supervised few-shot relation classification dataset with state-of-the-an evaluation. In EMNLP.
References
Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 in-context learn ing for biomedical IE? think again. In Findings of EMNLP.
Monica Agrawal, Stefan Hegselmann, Hunter Lang. Yoon Kim, and David Sontag. 2022. Large language models are few-shot clinical information extractors. In EMNLP.
Christoph Alt, Aleksandra Gabryszak. and Leonhard Hennig. 2020. TACRED revisited: thorough eval- uation of the TACRED relation extraction task. In ACL.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predict- ing spans. Transactions of the Association for Com- putational Linguistics, 8:64-77.
Simran Arora, Avanika Narayan, Mayee F Chen, Lau- rel J Orr, Neel Guha, Kush Bhatia, Ines Chami, Fred- eric Sala, and Christopher Ré. 2023. Ask me any- thing: A simple strategy for prompting language mod- els. In ICLR.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models zero-shot reasoners. In NeurIPS.
Omer Levy, Minjoon Sco, Eunsol Choi, and Luke Zettle- moyer. 2017. Zero-shot relation extraction via read- ing comprehension. In CoNLL.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In NeurIPS.
Guozheng Li, Xu Chen, Peng Wang. Jiafeng Xie, and Qiqing Luo. 2022. Fastre: Towards fast relation extraction with convolutional encoder and improved cascade binary tagging framework. In IJCAL.
Chih-Yao Chen and Cheng-Te Te 2021. ZS-BERT: Towards zero-shot relation extraction with attribute representation learning. In NATCL-HLT.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples gpt-3? arXiv preprint arXiv:2101.06W:
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017a. Reading wikipedia answer open- domain questions. In ACL.

---

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang. Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods natural language processing. ACM Computing Surveys, 55(9):1-35.
Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong. Jie Tang, and Dawn Song. 2022. Deepstruct: Pre- training of language models for structure prediction. In Findings of ACL.
Xiao Wang, Weikang 7hou, Can Zu, Han Xia, Tianze
Yang Liu and Mirella Lapata. 2019. Text summarization with pretrained encoders. In EMNLP.
Chen, Yuansen Zhang. Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al. 2023a. Instructuie: Multi- task instruction tuning for unified information extrac- tion. arXiv preprint arXiv:2304.08083
Keming Lu, I Hsu, Wenxuan Zhou, Mingyu Derek Ma,
Muhao Chen, et al. 2022. Summarivation as indirect supervision for relation extraction. In Findings of EMNLP.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. 2023b. Self-consistency improves chain of thought reasoning in language models. In ICLR.
Yubo Ma. Yixin Cao. YongChing Hong, and Aixin Sun. 2023. Large language model is not good few-shot information extractor, but good reranker for hard samples! arXiv preprint arXiv:2303.cm 085559.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. In NewrIPS.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models follow instruc- tions with human feedback. In NeurIPS.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang. Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, et al. 2023. Zero- shot information extraction via chatting with chatgpt. arXiv preprint arXiv:2302. 10205.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert: Sentence embeddings using siamese bert-networks.
Shanchan Wu and Yifan He. 2019. Enriching pre- trained language model with entity information for relation classification. In CIKM.
EMNLP.
Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. ECML PKDD.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. Luke: Deep con- textualized entity representations with entity-aware self-attention. In EMNLP.
Tim Rocktäschel, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiský, and Phil Blunsom. 2016. Reasoning about entailment with neural attention. In ICLR.
Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via con- volutional deep neural network. In COLING.
Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, An- der Barrena, and Encko Agire. 2021. Label verbal- ization and entailment for effective zero-and few-shot relation extraction. EMNLP.
Kai Zhang, Bernal Jiménez Gutiérrez, and Yu Su. 2023. Aligning instruction tasks unlocks large language models as zero-shot relation extractors. In Findings ACL.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, 7aid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022. Multitask prompted training enables zero-shot task generalization. In ICLR.
Yuhao Zhang, Peng Qi, and Christopher D Manning. 2018. Graph convolution over pruned dependency trees improves relation extraction. EMNLP.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-awes aware attention and supervised data improve slot filling. EMNLP.
Teven Le Scao, Angela Fan. Christopher Akiki, EI- lie Pavlick. Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprini urXiv:2211.05100
Jun Zhao, Wenyu Zhan, Xin Zhao, Qi Zhang, Tao Gui, Zhongyu Wei, Junzhe Wang. Minlong Peng, and Mingming Sun. 2023. Re-matching: A fine-grained semantic matching method for zero-shot relation ex- traction. In ACL.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning. In NIPS.
Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In ICML.
George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. 2021. Re-tacred: Addressing short- comings of the tacred dataset. In AAAI
Wenxuan Zhou and Muhao Chen. 2021. An improved baseline for sentence- level relation extraction. arXiv preprint arX.::2102.01333
Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A billion parameter autoregressive language model.

---

A Dataset Statisties
female name but ChatGPT treats the name as a person (Case 2), which also indicates the draw- back of this method that the generated questions might be unexpected without providing the context or template. This also highlights the importance of incorporating entity types into relation extrac- tion. On the other hand, prompting is brittle process wherein small modifications to the prompt can cause large variations in the model predictions. For example, we use "Answer the question from context" rather "Answer the question from con- text with yes/no" to expect that ChatGPT could not only give the "yes/no" answer but also provide the reason for its judgment. We achieve the expected results in most relations. However, the answers corresponding to the relation "language of work or name" frequently do not contain "yes/no" while express positive (Case 3), which makes automatic evaluation difficult. Therefore, the specific form of the statement answer important, otherwise it may lead to unreasonable evaluation. Moreover, the annotation errors (Case 4) of this relation also lead to biased and unreliable evaluation.
The statistics of the datasets are shown Table and Table 8.
Dataset
# instances
entities
relations
FewRel
56,000 94.383
72,954
Wiki-ZSL
77,623
Table 7: Statistics of RewRel and Wiki-ZSL..
Dataset
train
dev
# test relations
TACRED
12.124
22,631
15,509
42
TACREV
68.124
22,631
15,509
42
Re-TACRED
58,465
19,584
13,418
40
NYT
56,196
5,000
5,000
24
Table Statistics of TACRED, TACREV, Re-TACRED and NYT.
B Implementation Details
For the hyper-parameters of SUMASK prompting, we set the number of generated answers l as 5, and version
we the peer-largenli-mmen--okeee of Sentence- BERT encoder to generate the answer representations. For open source LLMs GPT-J, BLOOM and TO, we set the max generated length as 128 and the temperature as 0.3. Note that the results of VANILLA prompting with open source LLMs are not discussed in our paper because they all achieve near-0 performance. Due to the high pick up
"tributary" and "mouth of the watercourse" ChatGPT performs poorly in both two relations. First, these two relations have very similar se- mantics because they are reciprocal in FewRel, as shown in Case 1. Unfortunately, ChatGPT fre- quently reverses the subject and object correspond- ing to these two relations during question genera- tion step. Specifically, ChatGPT treats the triple (lisava river, tributary, natra river) (lisava river, tributary of, natra river). This phenomenon high- lights the advantage of manually crafted question templates and persuades us to provide few-shot demonstrations to generate reliable questions. Sec- ond, we also find plenty of annotation errors of two relations because of distant supervision, shown in Case 2. We can see that ChatGPT provides the correct judgments and reasons for this case, which makes it possible for LLMs to become reliable an- notation inspectors.
noise content in the output of LLMs, only the part of the answer text that first satisfies the answer format to alleviate the unexpected be- haviors. For 8pt-3.5-turbo-0300
we set the max length as 256 and the temperature as 0.7 according to official default setting. We treat the outputs of ChatGPT as valid results without post post-processing.
C Relation Specific Analysis
We provide accuracy results for all relations in FewRel. The results are summarized Table 9, And we provide several case studies to analyze the performance of ChatGPT on different relations.
D Question Templates
"language of work or name" We observe two important reasons for the poor performance of this relation, shown in Table 10. On the one hand, ChatGPT sometimes misunderstand the semantics of entities or relations, which leads to generated questions deviating from the original meaning ex- pressed. For example, Elizabeth is an English
In ablation study, we replace the LLMs generated questions with pre-defined question templates. For FewRel, we simply define template that forms with "The relation between 'subject" and 'object' is 'relation name". Yes or No?". For TACRED, we follow Zhang et al. (2023) to use the templates shown in Table 12.

---

E Discussions
in datast FewRel is hard to describe by a single template, while this relation can not only describe language versions of some literary works, but also describe what language a name belongs to. Con- sider the following two templates: (1) The lan- guage of subject is object. and (2) subject is a name in object. The triple (Elizabeth, language of work name, English) satisfy the second template but deliver confusions in the first template, while the triple (The Lord of the Rings, language of work or name, English) only satisfy the first template. SUMASK does not require any sort of prompt engineering or template writing to start using.
The empirical validation of SUMASK prompting The underlying operational mechanisms intrinsic to SUMASK prompting rely on the strong capabil- ities of LLMs zero-shot reasoners. Following the prompt instructions, LLMs pay attention to the entities of interest, infer and summarize the rela- tions between them. To the best of our knowledge, the logical reasoning faculties of LLMs are not explicitly utilized in previous work of zero-shot re- lation extraction. From raw text inputs to extracted relation labels, this process lacks intermediate rea- soning steps. We decompose the zero-shot relation extraction into three steps to make LLMs sensitize to the semantic understanding and logical reason- ing. With the proper instruction for summarization, LLMs are able to perform logical reasoning on specific entities and obtain relations between them. The LLMs can automatically do this via prompting, but the small fine-tuned model cannot. SUMASK elicits the logical reasoning ability inside LLMs for relation deduction. The ablation results also show that without the summarization step, the over- all performance drops 2.5% 7.7% F1 FewRel and TACRED under different settings. The exper- imental results on overlapping relation extraction also demonstrate the superior of SUMASK prompt- ing over VANIILL prompting.
The complexity of SUMASK prompting Typi- cally, the SUMASK prompting method suffers from relatively high inference complexity as it needs enumerate all possible triples to obtain summariza- tions, questions, and answers, for times. Suppose we have n samples to be extracted and r candidate relations, the total complexity is O(k x r x n). First, using entity types to discard the most inrele- vant relations is useful method, which achieves less complexity O(k x f X n) where ř represents the maximum value of the mapped relation can- didate set and much less than r. Second, we can certainly ask multiple samples (not exceeding the model maximum length) to LLMs at one time to im- prove inference speed. Suppose we concatenate h samples in a prompt, then the complexity becomes Of x n). More efficient zero-shot prompting for relation extraction is worth exploring in the future.
Here we provide case study on NYT. The in- put prompt is: Summarize the relations between "Cambodia" and "Penh" from context. Context: Homage to Cambodia was performed at Chakto- muk Conference Hall in Phnom Penh on Oct. 21 attended by the king. Summarization:
Uncertainty estimation in all LLMs Uncer- tainty estimation based on an assumption that the outputs of LLMs would be stabler with the predictions equivalent to the ground truth. The effi- ciency of using logits to generate the probabilities of intermediate results is relatively low, because we are required to obtain the probability of tokens at each position. Moreover, it is highly susceptible to extreme values. For example, if token with low probability is sampled during sampling pro- cess, will affect the probability value of the entire sentence. In addition, due to the generated long text sequence, the difference of probability values between generated sentences is not obvious, mak- ing it difficult to choose the relation with the high- est probability. Using SUMASK with uncertainty estimation brings better outcomes and we show this technique is suitable for both white box (e.g.. BLOOM) and black box (e.g., ChatGPT) models.
Then the response from ChatGPT is "Phnom Penh" " is the capital city of "Cambodia," " where the event "Homage to Cambodia" took place at the "Chaktomuk Conference Hall" on October 21. Obviously, the first sentence "Phnom Penh" " is the capital city of "Cambodia." generated by LLMS clearly elucidates the relationship between two entities, facilitating the subsequent processes.
The advantages of SUMASK prompting Gen- erally, the SUMASK prompting does not require any sort of prompt engincering or template writing start using. And this is one of our contribution points. Specially, not all relations can be accurately described by templates because the description of relations may vary across different entities. For instance, the relation language of work or name

---

Relation Name
Accuracy Relation Name
Accuracy
voice type
97.4
after a work by
77.4
occupation
97.1
mountain range
76.6
contains administrative territorial entity
95.4
composer
76.3
participant of
95.1
operating system
76.1
crosses
95.0
notable work
76.0
located in the administrative territorial entity
94.9
sibling
75.6
league
94.4
developer
73.3
constellation
93.9
located in or next to body of water
71.1
competition class
93.1
record label
69.7
heritage designation
93.0
follows
69.0
position held
92.7
original language of film or TV show
68.7
member of political party
92.7
operator
67.9
location
92.0
location of formation
67.4
field of work part of has part
92.0
country of origin
66.4
91.6
successful candidate
66.1
91.0
country of citizenship
65.9
military branch
91.0
subsidiary
65.9
instance of sport
90.1
licensed to broadcast to
64.1
88.6
main subject
63.6
applies to jurisdiction member of participating team taxon rank
88.1
owned by
62.6
88.1
mother
62.3
88.1
occupant
61.9
87.9
position played team speciality
60.9
characters
87.6
followed by
59.4
genre
86.1
said to be the same as
59.3
instrument
86.0
place served by transport hub
59.1
director
85.9
sports season of league or competition
58.7
participant
85.6
child
57.9
manufacturer
85.1
headquarters location
56.6
country
84.3
work location
53.0
movement
84.3
located terrain feature
49.6
architect
84.0
distributor
48.4
winner
83.9
father
45.3
original broadcaster
82.9
head of government
44.3
religion
82.1
performer
44.0
nominated for
81.4
screenwriter
42,4
platform military rank publisher
80.3
mouth of the watercourse residence
33.3
79.9
31.6
79.7
tributary
22.7
spouse
79.6
language of work or name
13.0
Table 9: Accuracy of each relation in FewRel.

---

Case
Summarize the relations between "brut brenhinedd" and "welsh" from context. Context: Walter 's name is attached to the brut tysilio a variant of the welsh chronicle brut y brenhinedd Summarization: Walter is associated with the brut tysilio, which is welsh chronicle known as brut y brenhinedd.
Rewrite the triple as a yes/no question.
Triple: (brut brenhinedd, language of work or name, welsh) Question: Is the language of work or name of "brut y brenhinedd" welsh?
Answer the question from context.
Context: Walter is associated with the brut tysilio, which is welsh chronicle known as brut brenhinedd.
Question: Is the language of work or name of "brut y brenhinedd" welsh?
Answer: Yes, "brut y brenhinedd" is a welsh chronicle.
Case
Summarize the relations between "elizabeth" and "english" from context.
Context: Annabeth is a female english given name created from a combination of the names anna and elizabeth. Summarization: The name annabeth is an english female given name that is created from combining the names anna and clizabeth.
Rewrite the triple a yes/no question. Triple: (elizabeth, language of work or name, english)
Question: Does elizabeth's language of work or name include english?
Answer the question from context.
Context: The name amnabeth is an english female given name that is created from combining the names anna and elizabeth.
Question: Does clizabeth's language of work name include english?
Answer: It is not clear from the given context whether elizabeth 's language of work or name includes english.
Case
Summarize the relations between "luxembourgish wikipedia" and "luxemhourgish language" from context.
Context: The luxembourgish wikipedia is the version of wikipedia that is written in the luxembourgish language. Summarization: The luxembourgish wikipedia is the version of wikipedia that is written in the luxembourgish language.
Rewrite the triple yes/no question.
Triple: (luxembourgish wikipedia, language of work or name, luxembourgish language) Question: Is luxembourgish the language of work name for the luxembourgish wikipedia?
Answer the question from context.
Context: The luxembourgish wikipedia is the version of wikipedia that is written in the luxembourgish language. Question: Is luxembourgish the language of work name for the luxembourgish wikipedia?
Answer: Luxembourgish is the language of the luxembourgish wikipedia.
Case 4
Summarize the relations between "kuranty" and "russian" from context.
Context: Peter the great replaced the "kuranty" with the first printed newspaper in russian, the "vedomosti". Summarization: Kuranty was replaced by the first printed newspaper in russian called "vedomosti" by peter the great.
Rewrite the triple yes/no question.
Triple: (kuranty, language of work or name, russian)
Question: Is russian the language of work or name for kuranty?
Answer the question from context.
Context: Kuranty replaced by the first printed newspaper in russian called "vedomosti" by peter the great.
Question: Is russian the language of work or name for kuranty?
Answer: Russian is the language of the first printed newspaper called "vedomosti", not "kuranty".
Table 10: Case study regarding relation "language of work or name".

---

Casc
Summarize the relations between "lişava river" and "natra river" from context.
Context: The
tributary of the lişava river romania. Summarization: The river smaller river that flows into the lişava river romania.
Rewrite the triple a yes/no question. Triple: (lişava river, tributary, natra river)
Question: Is lişava river a tributary the natura river?
Answer question from context.
Context: The natra river a smaller river flows into the lisava river romania.
Question: Is the lişava river a tributary the natura river? Answer: No, the natra river flows into the lişava river, so the lişava river not a tributary of the natra river.
Summarize the relations between "natra river" and "lisava river" from context. Context: The natra river a tributary of the lisava river in romania. Summarization: The natra river is a tributary of the lisava river located in romania.
Rewrite the triple as a yes/no question. Triple: (natra river, mouth of the waterenurse, lişava river)
Question: Is the mouth of the lişava river located at the natra river watercourse?
Answer the question from context.
Context: The natra river is a tributary of the lisava river located in romania.
Question: Is the mouth of the lisava river located at the natra river watercourse?
Answer: The context does not provide information to answer this question.
Case 2
Summarize the relations between "danube" and "lech" from context. Context: His jurisdiction extended at that time from the iller castward over the lech, north of the danuhe to the alb, and south to the spurs of the alps.
Summarization: The text mentions that the jurisdiction in question encompassed an area that extended from iller towards east over lech, then further towards north over danube unto alb, and also southwards towards the alps.
Rewrite the triple yes/no question. Triple: (danube, tributary, lech) Question: Is the lech tributary of the danube?
Answer the question from context.
Context: The text mentions that the jurisdiction in question encompassed urea that extended from iller towards east over lech, then further towards north over danube unto alb, and also southwards towards the alps.
Question: Is the lech tributary of the danube?
Answer: The text implies that the lech is located between the iller to the west and the danube to the north, but it does not explicitly state whether or not the lech is tributary of the danube.
Summarize the relations between "blackberry run" and "east branch fishing creek" from context. Context: the restoration blackberry the third phase the east branch fishing creck restoration plan. Summarization: The restoration of blackberry part of the third phase of the east branch fishing creek restoration plan.
Rewrite the triple a yes/no question.
Triple: (blackberry run, mouth of the watercourse, cast branch fishing creek)
Question: Does the mouth the watercourse located the east branch the fishing creek have blackberry run?
Answer question from context.
Context: The restoration blackberry run part the third phase of east branch fishing creek restoration plan.
Question: Does the mouth the wateronurse located the cast branch the fishing creek have blackberry run? Answer: The context does not provide information about whether the mouth of watercourse located in the cast branch of the fishing creek have blackberry run.
Table 11: Case study regarding relation "tributary" and "mouth of the watercourse".

---

Relation
Template
perstattorrroovicee of death per:tille
subject died in the state or province object, Yes or No? subject is a object, Yes or No? subject is the member of object, Yes or No?
org.member of
per:other_famil family
subject is the other family member of object, Yes or No?
subject has headquarter in the country object, Yes or No?
org-parents
subject has the parent company object, Yes or No?
perrstateorprovinee__.. birth per:spouse
subject was born in the state or province object, Yes or No?
subject is the spouse of object. Yes or No?
pertorigin
subject has the nationality object, Yes No?
per:date of birth
subject has birthday object. Yes or No? subject studied in object, Yes or No? subject has the member object, Yes or No?
per:scholls_atee
org:members
org:founded
subject was founded in object, Yes or No?
perrstateorprovinces_. residence
subject lives in the state province object, Yes or No? subject died in the date object, Yes or No?
per:date_or _f__cathh
orgssharcholders
subject has shares hold in object. Yes No?
org:website
subject has the website object. Yes No? subject subject convicted of object, Yes No?
org:subsidiaries
object. Yes or No?
per:charges
org:dissolved
subject dissolved in object, Yes No?
orgzstatcorprovince of headquarters
subject has headquarter the state province object, Yes No?
per:country_ood barplry_oo__r___eee birth
subject was horn the country object, Yes No?
per:siblings
subject the siblings of object, Yes No?
org.top_memberseeee members/employces
subject has the high level memher object, Yes or No?
per:eause_dd catlss____daahhhh
subject died because object,
No?
per:alteraale names
subject has the alternate name object, Yes No?
org:numerr_ov cettuubber_o_empllyye wtt.ooooeeoeeeeeeeebeeeee
subject the number employees object, subject lives the city object. or No?
per:cities_ of residence
orgacity headquarters
subject has a headquarter in the city object, Yes or No?
per:children
subject is the purent of object, Yes or No?
per:employee_g __f
subject is the employee of object, Yes No?
aggpoliticalretiioossammi aafiliiationi
subject has political affiliation with object, Yes or No?
per:parents
subject has the parent object, Yes No?
per:eity_of_ jercittyof_birth
subject was born the city object, Yes or No?
per:age
subject has the age object. Yes or No?
per:countries of Presidennce
subject lives in the country object, Yes or No?
org:alternate names
subject is also known as object, Yes or No?
perreligion
subject has the religion object. Yes or No? subject died in the city object, Yes or No? subject died in the country object, Yes or No? subject was founded by object, Yes or No?
per:city of death
per:country of death
org:founded_by
Table 12: Templates for TACRED.


Reviews:
[{"review_id": "6VO91didWy", "reviewer": "Reviewer_ry3f", "summary": "", "questions": "You can refer to the reviewer's comments, which include feedback on the paper as well as my queries regarding certain aspects of the paper.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments.\n\n### 1. Specific contributions of uncertainty estimation\nThe uncertainty estimation method is utilized to improve the reliability of extracted results in black box LLMs such as ChatGPT. This method is important for multi-class classification tasks like relation classification. Because we assume only one valid relation exists in a sentence. And we highlight the substantial role of this method in line 111-114 of the paper **\"We further introduce an uncertainty estimation method to approximately characterize output probabilities of LLMs, which yields substantial improvements compared to VANILLA prompting.\"**\n\nThe ablation results demonstrate the pivotal role of uncertainty estimation compared to summarization generation and question generation. Specifically, this technique improves the absolute F1 scores by 19.9% - 36.5% on FewRel, which highlights the importance of uncertainty estimation utilized in relation classification via SUMASK prompting. The specific contributions are not included in the contribution summary, and we would provide some case studies and emphasize this in the revision. \n\n\n### 2. The empirical validation of SUMASK prompting\n Thanks for your suggestions. The underlying operational mechanisms intrinsic to SUMASK prompting rely on the strong capabilities of LLMs as zero-shot reasoners [1]. Following the prompt instructions, LLMs pay attention to the entities of interest, infer and summarize the relations between them.\n\nTo the best of our knowledge, the logical reasoning faculties of LLMs are not explicitly utilized in previous work of zero-shot relation extraction. From raw text inputs to extracted relation labels, this process lacks intermediate reasoning steps. We decompose the zero-shot relation extraction into three steps to make LLMs sensitize to the semantic understanding and logical reasoning. With the proper instruction for summarization, LLMs are able to perform logical reasoning on specific entities and obtain relations between them. The LLMs can automatically do this via prompting, but the small fine-tuned model cannot. \n\nTherefore, the accurate statement is not that SUMASK \"enhances the logical reasoning faculties of extensive language models\", but that **\"SUMASK elicits the logical reasoning ability inside LLMs for relation deduction\"**. The ablation results also show that without the summarization step, the overall performance drops 2.5% - 7.7% F1 on FewRel and TACRED under different settings. The experimental results on overlapping relation extraction also demonstrate the superior of SUMASK prompting over VANILLA prompting. \n\nHere we provide a case study on NYT. The input prompt is: \n\n*Summarize the relations between “Cambodia” and “Penh” from context.*\n\n*Context: Homage to Cambodia was performed at Chaktomuk Conference Hall in Phnom Penh on Oct. 21 , attended by the king.*\n\n*Summarization:*\n\nThen the response from ChatGPT is \n\n*\"Phnom Penh\" is the capital city of \"Cambodia,\" where the event \"Homage to Cambodia\" took place at the \"Chaktomuk Conference Hall\" on October 21.* \n\nObviously, the first sentence *\"Phnom Penh\" is the capital city of \"Cambodia,\"* generated by LLMs clearly elucidates the relationship between two entities, facilitating the subsequent relation prediction including */location/country/administrative_divisions*, */location/location/contains*, */location/administrative_division/country* and */location/country/capital*. We would augment the empirical validation of SUMASK prompting in the revision. Thanks for your instructive suggestions. \n\n\n### References\n[1] Large Language Models are Zero-shot Reasoners. Kojima et al., 2022. In NeurIPS.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The introduction of the novel SUMASK prompting method presents a fresh perspective on addressing the issue of zero-shot relation extraction.This approach transforms the relation extraction task into a multi-stage question-answering process, thereby offering a novel avenue for applications in similar downstream tasks.\n2.  Through meticulous experimental design and extensive validation, the paper effectively demonstrates the impact of the SUMASK prompting method in the realm of zero-shot relation extraction. This approach exhibits outstanding performance across multiple benchmark datasets, underscoring its practicality and scalability.", "reasons_to_reject": "While the SUMASK technique proposed by the authors showcases impressive state-of-the-art performance in relation extraction tasks, \nthe paper does raise two noteworthy concerns that merit further scrutiny：\n1.  The ablation experiments presented in Table 5 strongly emphasize the pivotal role of uncertainty estimation methods.Paradoxically, the authors do not place adequate emphasis on explicating the specific contributions stemming from uncertainty estimation—an omission that requires clarification. \n2.  As exemplified by the sentence \"Savi was born in Pisa, son of Gaetano Savi, professor of Botany at the University of Pisa,\" it becomes evident that the accurate delineation of intricate relationships among entities—Savi, Pisa, Gaetano Savi, Botany, and University of Pisa—necessitates not just semantic comprehension, but also a degree of logical deduction within the purview of the relation extraction model.Given the discernible efficacy of SUMASK Prompting vis-à-vis VANILLA Prompting, it would be prudent to augment the empirical validation, elucidating how SUMASK Prompting has the potential to enhance the logical reasoning faculties of extensive language models, or to furnish an elucidation of the underlying operational mechanisms intrinsic to SUMASK prompting.", "questions_to_authors": "You can refer to the reviewer's comments, which include feedback on the paper as well as my queries regarding certain aspects of the paper.", "paper_topic_and_main_contributions": "This paper explores the approach of using a large language model (ChatGPT) to address the zero-shot relation extraction problem. The authors introduce a novel prompting technique (Such as Chain of Thought (CoT)) to enhance zero-shot relation extraction, and propose the SUMASK prompting method. This method comprises three steps: Summarization, Question and Answer, and guiding the model for relation extraction through prompts. Through experiments conducted on various benchmark and configuration datasets, the paper thoroughly demonstrates the effectiveness of SUMASK prompting in zero-shot relation extraction.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dDw3Lz5hKp", "reviewer": "Reviewer_PmA9", "summary": "", "questions": "The uncertainty estimation method is indeed fancy and useful. I am curious about the cases with high uncertainty. Figure 4 indicates that many ground truth types have an uncertainty of more than 200. I wonder what level of uncertainty of the other types.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments.\n\n### 1. Level of uncertainty\nThank you for your interest in the uncertainty experiments. Figure 4 indicates the correlations between uncertainty estimation and ground truth. The uncertainty values corresponding to different relation samples vary greatly. Specifically, the minimum value of uncertainty is actually 0.0, which indicates that the LLMs are quite confident with its prediction on this ground truth. Take the FewRel as the example, approximately 79.4% of the uncertainty values are between 0 and 1, while 14.7% of the values are between 1 and 200. Only 5.9% of the uncertainty values are greater than 200. Specially, the cases with high uncertainty are due to poor extraction performance of LLMs for certain relations as we discussed in Relation Specific Analysis section and Appendix C.\n\nFor other types, the corresponding uncertainty values is mostly relatively high, with only a few being low. Although we make the assumption that LLMs have the relatively low uncertainty regarding to ground truth, other relation types may also corresponding to low uncertainty and cause confusions. Concretely, on all benchmarks, around 82.7% of uncertainty values corresponding to other relation types are greater than 1. There are also 17.3% of uncertainty values are between 0 and 1, indicating LLMs are also confident in treating some other relations as correct labels. Note that the uncertainty values counted here are all based on positive candidate relations (i.e., the LLMs answer with \"yes\" via the majority vote). \n\n### 2. Inference complexity\nTypically, the SUMASK prompting method suffers from relatively high inference complexity as it needs to enumerate all possible triples to obtain summarizations, questions, and answers, for $k$ times. Suppose we have $n$ samples to be extracted and $r$ candidate relations, the total complexity is $O(k \\times r \\times n)$.\nFirst, using entity types to discard the most irrelevant relations is a useful method, which achieves less complexity $O(k \\times \\hat{r} \\times n)$ where $\\hat{r}$ represents the maximum value of the mapped relation candidate set and much less than $r$. Second, we can certainly ask multiple samples (not exceeding the model maximum length) to LLMs at one time to improve inference speed. Suppose we concatenate $k$ samples in a prompt, then the complexity becomes $O(\\hat{r} \\times n)$.\nMore efficient zero-shot prompting for relation extraction is worth exploring in the future. Although the complexity of SUMASK is not advantageous, using SUMASK prompting does not necessitate any form of prompt engineering or template crafting compared to previous work like QA4RE. Thanks for your suggestions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The writing of this paper is clear and easy to understand. \n2. Their method is effective, achieving competitive zero-shot performance, especially compared to vanilla prompts.\n3. The proposed uncertainty estimation method is highly beneficial and contributes significantly to the final success of the model, to some extent.", "reasons_to_reject": "While they adopt the entity-relation mapping mechanism to handle relation redundancy, my concern remains with the inference complexity of this method. This is due to the fact that it needs to enumerate all possible triples to obtain summarizations, questions, and answers, for k times.", "questions_to_authors": "The uncertainty estimation method is indeed fancy and useful. I am curious about the cases with high uncertainty. Figure 4 indicates that many ground truth types have an uncertainty of more than 200. I wonder what level of uncertainty of the other types.", "paper_topic_and_main_contributions": "This paper introduces a novel prompt method to enhance Language Model (LLM) capabilities in solving Relation Extraction (RE) tasks. The authors decompose RE into two subtasks: text summarization and question answering, employing a chain-of-thought process to address the task effectively.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "F7Wf7CsAbj", "reviewer": "Reviewer_83on", "summary": "", "questions": "-  It seems like the most interesting and strongest motivation for this paper is that it does not require any sort of prompt engineering or template writing to start using. Please correct me if I’m wrong. If this is the case, is there a reason this was not more explicitly discussed in the paper?\n   - It is indeed a very important contribution which should be published but proper baselines need to be used.\n- Is there a reason for using the uncertainty estimation technique which uses “dispersion degree” for LLMs which do output logits?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the constructive comments.\n\n### 1. Absence of several relevant baselines\nThank you for taking our work so seriously. The reasons for the absence of several relevant baselines from the main experiments are as follows.\n\n- **NLI-DeBERTa and QA4RE are not in Table 1** There are two reasons why we did not summarize the experimental results on FewRel and Wiki-ZSL for these two baselines NLI-DeBERTa and QA4RE. **First**, NLI-DeBERTa is only evaluated in TACRED dataset so we could not directly retrieve the other results from its original paper [1]. And QA4RE [2] follows the NLI-DeBERTa experimental settings. **Second**, both NLI-DeBERTa and QA4RE require the verbalized description of a relation as hypothesis. While the authors of NLI-DeBERTa manually create templates and valid arguments for the 42 relations in TACRED, there are no available crafted verbalized descriptions for FewRel and Wiki-ZSL. And it is a non-trivial task to manually label high-quality and reasonable templates for so many relations. If we use a stiff template like \"The relation between *subject* and *object* is *relation*\", it actually does not comply with the grammar rules and confuse the relative positional property of the subjects and objects because we have not resolved the specific semantics of the relation label. Therefore, we do not conduct experiments on FewRel and Wiki-ZSL with NLI-DeBERTa and QA4RE. But we would specify the used stiff templates of two baselines and add the re-implementation results in the revision.\n\n- **Zero-shot NLI-DeBERTa is not in Table 2** NLI-DeBERTa is indeed also a zero-shot baseline that can be compared with other methods. We did not put it in Table 2 for two reasons. **First**, the intention of Table 2 is to show the superior performance of SUMASK prompting even compared to the fully supervised methods. Thus the zero-shot methods are not summarized in Table 2. **Second**, we have demonstrated the fully supervised performance of NLI-DeBERTa in Table 2, which is definitely better than zero-shot results. Hence we argue that it is not necessary to put the zero-shot results here compared to many state-of-the-art supervised baselines. Nonetheless, we would add the missing zero-shot results in the Appendix of the revision.\n\n- **QA4RE is not in Table 2** There are several reasons that make the experimental results of QA4RE are not included in Table 2. **First**, the experimental settings of QA4RE and SUMASK is different. QA4RE adopts the micro-F1 as evaluation metrics but does not consider the none-of-the-above (NoTA) relation. This setting is different with other supervised baselines and SUMASK. Thus the results of QA4RE is only evaluated in Table 3 measuring the NoTA-excluded 41-class micro-F1 and NoTA-included 42-class macro F1 on TACRED following [2]. **Second**, the paper of QA4RE is published as a conference paper at ACL 2023 in June. And the source code is uploaded to Github last month. At that time, as the deadline for EMNLP was approaching, we unfortunately did not have enough time to reproduce their method. Now we have obtained the micro-F1 for QA4RE on TACRED using gpt-3.5-turbo and SUMASK surpasses it by around 4.8% absolute F1 score, which indicates that QA4RE is also effective for handling NoTA relation. However, we should note that even though QA4RE is competitive to SUMASK, it requires the verbalized description of relations like NLI-DeBERTa, which is inconvenient and requiring too much human intervention. **Specially**, not all relations can be accurately described by templates because the description of relations may vary across different entities. For instance, the relation *language of work or name* in datast FewRel is hard to describe by a single template, while this relation can not only describe language versions of some literary works, but also describe what language a name belongs to. Consider the following two templates: (1) The language of *subject* is *object*. and (2) *subject* is a name in *object*. The triple (*Elizabeth*, *language of work or name*, *English*) satisfy the second template but deliver confusions in the first template, while the triple (*The Lord of the Rings*, *language of work or name*, *English*) only satisfy the first template. Like you said, SUMASK does not require any sort of prompt engineering or template writing to start using.\n\nWe truly appreciate your instructive suggestions that really help we polish the revision.\n\n### 2. Explanation of Figure 4\nWe apologize for the unclear explanations. The Figure 4 is to illustrate that our intuition is consistent with the empirical results. The uncertainty estimation technique is based on an assumption that the outputs of LLMs would be stabler with the predictions equivalent to the ground truth. In other words, if a LLM regards a relation as ground truth in a sentence, then its response to this relation is always \"yes\" (ie., the uncertainty is low). For the ground truth relations that are not predicted by LLMs, their uncertainty values are relatively high. From Figure 4, it is obviously that most ground truth relations correspond to low uncertainty (< 1), while a fewer correspond to high uncertainty (> 200). Note that Figure 4 indicates that the LLMs consider most gold relations with low uncertainty, making SUMASK has a better chance of choosing ground truth from multiple candidate relations with lowest uncertainty values. Reviewer PmA9 also asks some interesting questions about the uncertainty estimation, you can refer to our replies for better understanding. And we would explain more clearly about this part in the revision.\n\n### 3. Without prompt engineering or template writing\nGenerally, the SUMASK prompting does not require any sort of prompt engineering or template writing to start using. And this is one of our contribution points. Nonetheless, the original purpose of this paper is to show the huge potential of LLMs as zero-shot relation extractors because some recent studies is not optimistic that the performance of prompting LLMs can surpass the performance of fine-tuning small language models. And we design this simple SUMASK prompting without additional human intervention that achieves competitive or superior performance compared to existing zero-shot and fully supervised methods. The decomposition of zero-shot relation extraction and incorporation of uncertainty estimation consistently suggest that proper prompting techniques are beneficial for relation understanding and reasoning.\n\n### 4. Uncertainty estimation in all LLMs\nThe efficiency of using logits to generate the probabilities of intermediate results ([SUMMARIZATION] and [QUESTION]) is relatively low, because we are required to obtain the probability of tokens at each position. Moreover, this method is highly susceptible to extreme values. For example, if a token with a low probability is sampled during the sampling process, it will affect the probability value of the entire sentence. In addition, due to the generated long text sequence, the difference of probability values between generated sentences is not obvious, making it difficult to choose the relation with the highest probability. Using SUMASK with uncertainty estimation brings better outcomes and we show this technique is suitable for both white box (e.g., BLOOM) and black box (e.g., ChatGPT) models.\n\n### 5. Typos Grammar Style And Presentation Improvements \nThanks for your suggestions. \n- **“How does LLMs perform on RE incorporating existing prompt techniques?”** The meaning of this sentence is that we are investigating whether some recent techniques such as chain-of-thought or active prompting can bring improvements to LLMs on RE.\n- **“But it is still unclear whether LLMs are good zero-shot relation extractors by carefully designed prompts. Thus this work aims to investigate the capabilities of LLMs in zero-shot RE”** Some previous studies [3][4] claim that LLMs achieve inferior results on RE compared to traditional fine-tuned models. However, they do not design a suitable prompt for RE, but instead adopt the most direct approach like VANILLA prompting. Therefore, we aim to investigate the capabilities of LLMs in zero-shot RE equipping with advanced prompting techniques. And the conclusion is LLMs can become good zero-shot relation extractors with proper prompting techniques.\n\n### References\n\n[1] Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction. Sainz et al., 2021. In EMNLP.\n\n[2] Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors. Zhang et al., 2023. In Findings of ACL.\n\n[3] Thinking About GPT-3 In-Context Learning for Biomedical IE? Think Again. Jimenez Gutierrez et al., 2022. In Findings of EMNLP.\n\n[4] Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples! Ma et al., 2023. arXiv preprint arXiv:2303.08559", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Interesting methodology for leveraging multiple steps of LLM prompting while accounting for the uncertainty in each of the predictions.\n- Strong evidence that the current methodology is effective over vanilla prompting in different LLMs.\n- Choice of datasets and tasks is thorough.", "reasons_to_reject": "- The most striking concern with this work is the absence of several relevant baselines from the main experiments even though these baselines are mentioned elsewhere. \n  - NLI-DeBERTa and QA4RE are mentioned and compared to in a secondary experiment measuring the NoTA performance but are notably absent from Table 1 without an explanation. \n  - NLI-DeBERTa is shown in Table 2 but only as a supervised baseline when it is a zero and few-shot method which could be compared to directly. \n  - QA4RE is not mentioned in Table 2 at all even though it is competitive and much simpler than the presented technique. The motivation for not including such baselines is unclear.\n-  I was unable to understand Figure 4 from the explanations given, please explain it further.", "questions_to_authors": "-  It seems like the most interesting and strongest motivation for this paper is that it does not require any sort of prompt engineering or template writing to start using. Please correct me if I’m wrong. If this is the case, is there a reason this was not more explicitly discussed in the paper?\n   - It is indeed a very important contribution which should be published but proper baselines need to be used.\n- Is there a reason for using the uncertainty estimation technique which uses “dispersion degree” for LLMs which do output logits?", "paper_topic_and_main_contributions": "In this paper, the authors propose a new prompting scheme for zero-shot relation extraction. This framework is based on a four step process in which the model is prompted to 1) summarize the context, 2) rewrite the RE triple as a natural question, 3) answer the question based on the simplified context and 4) approximate the probability of each relation uncertainty based on a combination of the probability of each summarized context, generated question and final relation probability. This methodology, termed SUMASK, sharply improves the performance of LLMs over vanilla RE prompting.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- “How does LLMs perform on RE incorporating existing prompt techniques?” L088\n- “But it is still unclear whether LLMs are good zero-shot relation extractors by carefully designed prompts. Thus this work aims to investigate the capabilities of LLMs in zero-shot RE” (not sure what this sentence means) L164"}}]

Author rebuttal:

Meta-review: This paper tackles zero-shot relation extraction and proposes an interesting summarize-and-ask strategy to use LLMs for this challenging task (reviewer 83on gave a good summary of the proposed method). Extensive experiments show fairly strong zero-shot relation extraction performance on multiple standard datasets.

Strengths:
- The main idea is intuitively plausible and may generalizable to other tasks
- The empirical performance, as far as I could tell, is fairly strong for zero-shot RE
- The proposed method doesn't need manually written templates, which is a nice and unique property
- The writing is clear and easy to follow

Weakness
- Added cost for calling LLMs many more times
- There's an inadequate discussion on the uncertainty part, which is quite important. the authors are encouraged to strengthen this part in the revision.
Decision: Accept-Findings