Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction
Authors: Xilai Ma
Jing Li
Min Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: Relation Extraction
In-context Learning
Few Shot Learning

Abstract: Few-shot relation extraction involves identifying the type of relationship between two specific entities within a text, using a limited number of annotated samples. A variety of solutions to this problem have emerged by applying meta-learning and neural graph techniques which typically necessitate a training process for adaptation. Recently, the strategy of in-context learning has been demonstrating notable results without the need of training. Few studies have already utilized in-context learning for zero-shot information extraction. Unfortunately, the evidence for inference is either not considered or implicitly modeled during the construction of chain-of-thought prompts. In this paper, we propose a novel approach for few-shot relation extraction using large language models, named CoT-ER, chain-of-thought with explicit evidence reasoning. In particular, CoT-ER first induces large language models to generate evidences using task-specific and concept-level knowledge. Then these evidences are explicitly incorporated into chain-of-thought prompting for relation extraction. Experimental results demonstrate that our CoT-ER approach (with 0% training data) achieves competitive performance compared to the fully-supervised (with 100% training data) state-of-the-art approach on the FewRel1.0 and FewRel2.0 datasets.

Chain of Thought with Explicit Evidence Reasoning for Few-shot Relation Extraction
Xilai Ma, Jing Li and Min Zhang
Harbin Institute of Technology, Shenzhen, China
com {li. jing, zhangmin2021 jChit edu. cn
maxilai.m maxilai.hourr@reee.e
Abstract
Few-shot relation extraction involves identify- ing the type of relationship between spe- cific entities within text, using a limited num- ber annotated samples. variety of solu- tions to this problem have emerged by apply- ing meta-learning and neural graph techniques which typically necessitate a training process for adaptation. Recently, the strategy of in- context learning has been demonstrating no- table results without training. Few studies have already utilized in-context learning for zero- shot information extraction. Unfortunately, the evidence for inference is either not considered or implicitly modeled during the construction of chain-of-thought prompts. this paper, we propose novel approach for few-shot rela- tion extraction using large language models, named CoT-ER, chain-of tf-thught with explicit evidence reasoning. In particular, CoT-ER first induces large language models to generate ev- idence using task-specific and concept-level knowledge. Then this evidence is explicitly incorporated into chain-of-thought prompting for relation extraction. Experimental results demonstrate that our CoT-ER approach (with 0% training data) achieves competitive perfor- mance compared to the suply-sue (with 100% training data) state-of-the-ar approach the FewRel 0. and FewRel2.0 datasets.
prompt
predicted
CoT-ER
head
brompt
label
tal Inference
Figure The comparison between Auto-CoT and CoT- ER (ours) prompting methods. Specifically, CoT-ER leverages side information induce LLMs to generate explicit evidence for relation reasoning.
researchers have tackled this problem by employ- ing meta-learning and neural graph techniques (Fangchao et al., 2021; Dou al., 2022; Li and Qian, 2022; Zhang et al., 2021; Li et al., 2023). These methods have achieved satisfying results by meta-training the model on large dataset or incor- porating external knowledge.
More recently, pre-trained Large Language Mod- els (LLMs) such as GPT-series models, have ex- hibited significant in-context learning capabilities (Brown et al., 2020; Min et al., 2022), achieving promising results across many NLP tasks. These findings suggest that LLMs can effectively per- form various tasks without the need for parame- ter optimization, a concept known as In-context Learning (Dong et al., 2022). Within the paradigm of in-context learning, LLMs demonstrate com- petitive performance compared to standard fully- supervised methods across many NLP tasks, even with just a few examples provided few-shot demonstrations in the prompt (Wang et al., 2022).
1 Introduction
Relation extraction (RE) aims at identifying the relation between two given entities based on con- textual semantic information (Cardie, 1997; Bach and Badaskar, 2007; Pawar al., 2017). How- ever, the performance of RE models often degrades significantly when the labeled data is insufficient. The few-shot relation extraction (FSRE) task needs to addressed with a limited amount of anno- tated training data (Han et al., 2018; Gao et al., 2019b; Brody et al., 2021). Recently, numerous
Furthermore, the chain-of-thought prompting method (Wei al., 2022: Kojima al., 2022: Zhang al., 2022) elicits an impressive reason- ing capability from the LLM in mathematical prob- lems and commonsense reasoning. While the
Corresponning author.

---

RE task, there may exist a reasoning process that guides the LLM in determining the relation label. However, there is a lack of research to fill this gap. Though GPT-RE (Wan et al., 2023) introduces a golden label-induced reasoning method by prompt- ing the LLM to generate suitable reasoning pro- cess solely based on the given golden label. The performance improvement from the auto- auto-generated reasoning process is marginal compared metic- ulously designed approach for few-shot demonstra- tion retrieval.
foundation for RE-specific reasoning. In the third step, CoT-ER prompts the LLM to extract the rel- evant contextual spans as evidence that explicitly establishes a specific relationship between these two entities. By combining the head entity, tail en- tity, and relation label to form coherent sentence, LLMs can determine the relation label between two given entities more semantically, addressing the issue of semantic ambiguity of relation labels in prompting methods. Figure demonstrates the difference between Auto-CoT and our CoT-ER.
We argue that the one-step reasoning process generated by LLM does not fully unleash the po- tential of LLM: (1) Previous studies and our ex- periments indicate that the one-step auto-generated reasoning process by LLM does not emphasize the higher-level abstraction of entity types, specifically the concept-level entities, which has been proven to be beneficial for FSRE task (Hao al., 2019; Zhang al., 2021). For instance, consider the fol- lowing simple example: the relation between two location entities should not be categorized under the relation type between human beings. (2) Due to the huge amount of pre-training data, the LLM has already possessed a considerable knowledge base (Petroni et al., 2019; Roberts et al., 2020), which can be beneficial when the LLM encounters FSRE task. (3) The quality of semantic rep- resentation of the relation label is not crucial in the fully-supervised setting, but in-context learn- ing sensitive to the relation label. For instance, given the relation labels= (mother, child, sport) in FewRel 1.0 (Han et al., 2018), relation labels "mother" and "child" would confuse the LLM without appropriate prompt designing, the pri- mary distinction between these two relations is the positioning of the parent entity as either the head or tail entity. Moreover, the word "sport" barely contains enough relation information for the LLM to perform the RE task. We call this issue the se- mantic ambiguity of relation labels.
Related Work
Few-shot Relation Extraction. Few-shot relation extraction aims at predicting semantic relations between head and tail entities indicated in a given instance based on limited amount of annotated data. FewRel, large-scale dataset introduced by Han et al. (2018), was the first to explore few-shot learning relation extraction. Many approaches (Qu et al., 2020; Yang et al., 2021; Zhang et al., 2021) incorporate external knowledge to improve performance given the scarcity of training data. Another line of FSRE research (Gao et al., 2019a; Han et al., 2021; Liu et al., 2022b) relies solely the input text and provided relation description information, without incorporating external knowledge. Most of the previous methods usually adopt complicated designs of neural networks or introduce external knowledge, which can be labor-intensive in realistic scenarios.
In-context Learning. GPT-3 in-context learn- ing (ICL. (Brown et al., 2020; Dong et al., 2022) has emerged as a novel paradigm in NLP and demonstrates competitive performance across var- ious tasks when compared to fine-tuned models. It's much casier to introduce prior knowledge into LLMs by incorporating relevant text information into prompt (Liu et al., 2022a; Lu et al., 2022; Wei et al., 2022). Furthermore, ICI. is training-free approach by directly prompting the LLMs, which means it's ready-to-use method and can be easily applied to various tasks with a few demonstrations in the prompt.
To this end, this paper presents a novel chain- of-thought prompting method for the FSRE task: sfuin-of-thuughe with Explicit Evidence Reason- ing, achieving competitive results compared to the state-of-the-an result on FewRel 1.0 and FewRel 2.0. Our method employs 3-step reasoning ap- proach to address the aforementioned issues. In the first and second steps, CoT-ER requires the L.LM to output the concept-level entities corresponding to the head and tail entities, which serve as the
Recently, most researchers focus on the demon- stration designing of ICL to improve the perfor- mance in NLP tasks and gradually developed into two categories (Dong al., 2022). The first line of demonstration designing tries to seek an optimal arrangement of the few-shot demonstrations in the

---

Human-Instructed Reasoning Module
Inference Module
Similarity-aase KNN Retrieval
Human annotated data
Suppori instance with evidence reasoning
Support instance without evidence reasoning
XM
Query instance
Candiclate Set
XN*K
N- N-way K-shot Task
Sample
Query output with reasoning
Dalasel
Support Set
Query
An illustration of CoT-ER for few-shot RE. Different colored lines indicate the flow of support and query instances from an N-way K-shot task. a) reasoning process b) Instances retrieval module ($3.4) selects the few-shot demonstrations from the candidate set for the ultimate prompt based on their similarity to the query instance. c) Inference module ($3.5) utilizes the ultimate prompt, which is composed of M support instances with their associated reasoning process, derive an evidence reasoning process for the query instance.
Figure 2:
reasoning module ($3.3) associates an evidence instance from support set by prompting L.LM with human-annotated demonstrations;
prompt by selecting instances from the dataset (Liu et al., 2022a; Valmeekam et al., 2022; Wu et al., 2022; Wan et al., 2023) and ordering the selected demonstration examples (Lu et al., 2022; Liu et al., 2022a; Ma et al., 2023). Another line of demonstra- tion design aims to discover effective prompting method to unleash the potential of LLMs. Several studies (Honovich et al., 2022: Zhou al., 2022) find that the LLMs can generate the task instruc- tion automatically. Furthermore, Wei et (2022)
True Few-Shot Learning. Perez al. (2021) ar- gues that prior research has achieved promising results by choosing prompts or tuning other hy- perparameters using a large development set, such as selecting few-shot demonstrations from large training set, which does not truly demonstrate the few-shot learning capability of LLMs. This setting has been adopted by many works (Logan IV al., 2022: Schick and Schütze, 2022; Lu al., 2022; Jimenez Gutierrez al., 2022) to get a more aceu- rate result of few-shot performance. We will also adopt the setting in this paper.
revealed the reasoning ability of LLM by adding in- termediate reasoning steps manually before giving the answer, which is called chain-of-though! (CoT). Additionally, Kojima et al. (2022) shows that by simply adding "Let's think step by step" before each answer, LLM can do the zero-shot reasoning without manually annotated data. Based this discovery, Zhang et (2022) proposed Auto-CoT, replacing the manually written reasoning process in CoT with the automatically generated reasoning process by LLM.
3 CoT-ER
3.1 Problem Formulation
Definition of Relation Extraction. Let æ denote the input sentence and Csud Coby denote the pair of subject and object entities in the given sentence. The RE task aims to identify the relation label r between the marked entities in sentence æ. Here R represents predefined set of relations, and r is an element of R.
Despite the CoT prompting method achieving promising results in many NLP tasks, it still lacks relevant exploration for RE. Hence, in this paper. we propose novel CoT prompting method called CoT-ER to fill this gap.
Definition of Few-shot Relation Extraction. Given an N-way K-shot RE task, the goal is solve this problem for each instance in the query set based on the support set. The relation label

---

set R. consists of N type of relations. For each r E R. the support set S, includes K instances, represented as {s], sf, sy....,,6, 3. The query set Q comprises the test input instance for each r E R. Since the N and K are usually quite small, pre- dicting relations in query instances with limited labeled data presents a significant challenge. Previ- ous studies tackled this problem by training data- efficient network, specifically the meta-learning- based method. In the subsequent section, we will discuss a training-free method address this prob- lem by leveraging the reasoning ability of the LLM.
relation label "crosses". To further illustrate our 3-step reasoning process, few-shot demonstrations in Figure demonstrate the template of this reasoning process.
CoT-ER Generating, We annotated one CoT-ER reasoning example for each relation class in the dataset to be seed examples. Then we design an appropriate prompt using the annotated example as the few-shot demonstration to instruct the LLM in generating a similar reasoning step for each sup- port instance. Each support instance with the CoT- ER reasoning steps will be appended to the candi- date set. Figure 3 shows a similar prompt designed for the Human Instructed Reasoning Module.
3.2 Overview
An overview of our proposed CoT-ER shown in Figure 2, which consists of 3 components: (1) The Human-Instruckes Instructed Reasoning Module, which aims to associate a reasoning process with each in- stance from the support set by prompting LLM with human-annotatee data. (2) A Similarity Based KNN Retrieval Module will select instances with the reasoning process from the support set based on the similarity to query instance, which are con- sidered as few-shot demonstrations in the ultimate prompt. (3) The Inference Module predicts the relation label of a query instance by instructing the LLM through the ultimate prompt, which concate- nates the task instruction, few-shot demonstrations, and question about the instance.
3,4 Instance Retrieval Module
Several studies (Liu et al., 2022a; Lu et al., 2022; Wan et al.. 2023) suggest that selecting few-shot demonstrations based on similarity yields strong improvements in in-context learning. Wan et al. (2023) achieved promising performance in RE by employing task-specific fine-tuned model the encoder, which means this approach does not fit the true few-shot setting mentioned in $2. Moreover, this advantage diminishes rapidly as the number of candidates decreases.
Because of the limited input tokens of LLM, single prompt may not hold all support instances given an N-Way K-Shot task. For instance, in the case of the 10-way 5-shot task, having a total of 50 candidate samples leads to the inability to append them all in one single prompt. In this paper, we fol- low the similarity-basee method for selecting few- shot demonstrations. To obtain relation-specific similarity representation, we first reconstruct the input text as "Context: [text] Given the context, what is the relation between "[head entity]" and "[tail entity]"'' by incorporating entity-level infor- mation. Then, we utilize the GPT series model www.emmeddnngad------e- as the encoder to get the semantic embedding. Subsequently, we compute the Euclidean distance between each instance in the candidate set and the query instance. Finally, M instances from the candidate set are selected few-shot demonstrations based their lower Eu- clidean distance the query instance. Intuitively, we aim to provide as much information as possible for LLM, so we follow the principle of filling the
3.3 Human-Instrueted Reasoning Module
Since the LLM has the ability of in-context learning (Brown al., 2020), we propose a human-instructed approach to guide the LLM in performing accurate reasoning using a minimal amount of annotated data.
CoT-ER Designing. To fully leverage the knowl- edge stored in LLM and facilitate step-by-ster reasoning, we introduce novel 3-step reasoning framework with concept-level knowledge and explicit evidence. In Step 1, the LLM infers concept-level knowledge related to the head entity, while Step 2 does the same for the tail entity. Through these steps, the LLM can easily exclude options with incorrect concept entities. Step 3: To figure out which relation label fits this pair of entities most within the given context, explicitly highlight relevant text spans as evidence, and subsequently construct a coherent expression that combines the two entities and the relation label together. Table 6 shows an example with the
All seed examples are shown in Appendix B. 2 All prompts are shown in Appendix C.

---

Task Instruction
100 relation labels are divided into 64/16/20 splits for train//alidationntess set. FewRel 2.0 extends FewRel 1.0 by introducing additional validation and test sets from the medical domain, which in- cludes 10 relation labels with 1, 000 instances and 15 relation labels with 500 instances, respec- tively. Besides, note that FewRel 1.0 provides a description of each relation label in the dataset, but FewRel 2.0 does not. This difference is a crucial factor to consider when designing the seed CoT-ER reasoning process".
Please solve the Relation Extraction task.
Given the context, consider what's the most precise relation
between two entities belongs to the following possible relations. The relation to choose must be in these N possible relations:
[LABEL 1), [LABEL 2), ..., [LABEL N]
Few-shot Demoonstratios
Context: [TEXT]
Given the context, what is the relation between subject entity (head entity] and object entity (tail entityl? ?
1. Subject entity [head entity] is [head entity).
2. Object entity [tail entity) (tail entity].
3. According the context, [evidence] indicate that
Verbalize[[head entityylgolden [abil][tail entity]).
So, the relation between subject entity [head entity) and object
entity (tail entityy is [galden label].
4.2 Implementation Details
xM
Query
For LLM,, we select "text-davinci-003" and get the response from GPT by calling the open API of the OpenAl with the parameter temperature = 0. In a realistic scenario, it's reasonable to perform the RE task directly with fixed, manually annotated examples as few-shot demonstrations for each rela- tion label. To this end, we evaluate the performance by selecting the few-shot demonstrations from a predetermined human-annotated CoT-ER dataset (seed examples), which is denoted as Manual-CoT- ER. In this setting, the few-shot demonstrations are independent of the support set, meaning that LLM will perform the RE task using a smaller amount of annotated data. In contrast, Auto-CoT-ER utilizes the auto-generated CoT-ER reasoning process as the few-shot demonstrations based on the support set which is described in $3.3.
Context: [TEXT]
Given the context, what the relation between subject entity (head entity] and object entity [tail entityl?
Template of the ultimate prompt. .M repre- sents the number of few-shot demonstrations selected by the instance retrieval module, and V Verbalize() denotes a transformation function that combines the component into a coherent expression. The prompt used in human human-instructed reasoning module follows similar structure, but instead of few-shot demonstrations, it em- ploys annotated examples.
Figure 3:
context window to increase M as much possible.
3.5 Inference Module
To create the ultimate prompt, we simply concate- nate task instruction, few-shot demonstrations, and question that is tailored the query instance, using the support instances with CoT-ER reasoning as few-shot demonstrations. Figure 3 shows the framework of the ultimate prompt. It is worth not- ing that LLMs have a strong inclination to wrongly output NULL in a general setting (Wan et al., 2023; Jimenez Gutierrez et al., 2022). Here, we enforce the LLM to select one of the provided rela- tion labels, Above" scenario in the FewRel dataset (Han et al., 2018: Gao et al., 2019b).
Following the standard configuration of FewRel, we conducted experiments in these settings: 5- 5-Way I-shot, 5- Way 5-shot, 10- (0-Way 1-Shot, and 10- Way 5-Shot. Due to the high cost of running the CoT-ER the GPT-3 API and the golden labels of the test set are not publicly available, we evaluate all LLM- based methods by sampling 100 x N test queries for each N-way K-shot task from validation sets.
do not consider the "None-of the-
4.3 Compared Methods
We consider two categories of methods for the FSRE task. Methods with 100%-traiiing data: MTB (Baldini Soares et al., 2019), CP (Peng et al.. 2020), HCPR (Han et al., 2021), FAEA (Dou et al., 2022), GTPN (Fangchao et al., 2021), GM GEN (Li and Qian, 2022), KEFDA (Zhang et al., 2021). Generally, these methods train a model on FewRel 1.0 training set and evaluate their performance on FewRel 1.0, 2.0 validation and test sets.
4 Experimental Setups
4.1 Dataset For Few-shot Relation Extraction
There are two standard few-shot relation extrac- tion datasets: FewRel 1.0 (Han et al., 2018) and FewRel 2.0 (Gao et al.. 2019b)3. FewRel 1.0 is con- structed from Wikipedia, which consists of 70, 000 sentences annotated with 100 relation labels, these
MMore details are demonstrated Appendix B. htttps: /ttp:://llaffooo/ openai..ppa.....mmmmmmmmm watpnaai.ccooaaaa api- api-reference

---

Methods
5-Way 1-Shot
5-Way 5-Shot
10-Way 1-Shot
10-Way 5-Shot
Sully-Supervised (100% training data)
GTPN
/ 19.40
197.00
- 84.40
-99.88
CP
-/95.10
-/97.10
-/91.20
-/94.70
FAEA
90.81 / 195100
94.24 96.48
84.22 / 90.12
88.74 92.72
HCPR
94.10 / 96.42
96.05 / 97.96
89.13 / 99.97
93.10 96.46
GM_GEN
96.97 / 97.03
98.32 / 98.34
93.97 / 94.99
96.58 / 96.91
Training-free Baselines (0% Iraining data)
Bert-proto
29.32/-
35.18/-
17.84/-
23.88/-
GPT-proto
63.20/-
82.00/-
52.10/-
71,50/-
Vanilla-ICL
96.20(5)/-
97.00(25)/-
89.20(10y-
93.90(40y-
Auto-CoT
94.60(5)/-
95.80r20y-
87.40(10y-
91.40(20y-
+ rreasoning
95.40(5)/-
96.40(15y-
87.60(10y-
92.40(15y-
CoT-ER (0% training data)
Auto-CoT-ER
97.40(5V-
97.00(13y-
92. 10
94.70(13y-
Manual-CoT-ER
97.00(5)/-
92.60(10y-
Table Main Results FewRel 1.0 validation/test set. All results given by accuracy (%). (m)) means there instances selected as few-shot demonstrations. +reasoning denotes the demand of reasoning process generation.
Methods with 0%-trainnn data: To the best of our knowledge, no relevant evaluation has been conducted under the N-Way K-Shot setting on the FSRE dataset FewRel using the in-context learning approach. Thus we applied Vanilla-ICL (Brown et al., 2020) and Auto-CoT (Zhang et al., 2022; Wan et al., 2023) as the baseline prompt formatting methods. These methods utilize few examples as demonstrations and prompt the LLM to perform an NLP task. Vanilla-ICL designs template that directly combines the texts and relation label, such as "Centext:(text), Given the context, the relation between [head entity] and [tail entity] is [relation label]". Auto-CoT extends the Vanilla-ICL with auto-ge generateerreddd reasoning steps. Throughout the experiment, noticed that whether to require the LLM perform reasoning in the final answering stage can lead to inconsistent results, thus we report both of the results in Table 1 and Table 2. Addition- ally, we utilize the pre-trained BERT- base mode16 and the GPT series model text-embedding-aat -aaaggg--d-O000 002 as the encoder to directly obtain representation of the input text. For each N-way K-shot task, we obtain prototype of each class by averaging K instances that belong to this class. Then the pre- dicted label of the query instance is assigned the class whose prototype has the closest Euclidean distance to the query instance. We denote these two methods as Bert-proto and GPT-proto.
Results and Discussion
5.1 Main Results
We present our main experiment results with pre- vious methods Table and Table From the table, we can observe that:
First, Auto-CoT does not demonstrate significant improvement compared to Vanilla-ICL. in the few- shot scenario. This could be attributed to the low quality of the reasoning process and the reduced number of instances in few-shot demonstrations due to the maximum tokens limitation. Further- more, When it comes to generating a reasoning process in the ultimate answer, Auto-CoT with rea- soning outperforms the version of directly gener- ating a relation label on FewRel 1.0. However, an opposite conclusion is reached on FewRel 2.0. We try to provide an explanation for this: FewRel 1.0 draws instances from Wikipedia and often requires common sense for reasoning, whereas FewRel 2.0 necessitates medical-related expertise and consti- tutes a smaller portion the pre-training corpus compared to common sense. Consequently, the LLM encounters difficulties in performing reason- ing tasks in the medical domain.
Second, Both Manual-CoT-EE and Auto-CoT- ER outperform the training-free baselines with fewer instances used the few-shot demonstra- tions. Indicating the necessity of designing a spe- cific CoT prompting method tailored to the RE task in order to achieve better performance the

---

Methods
5-Way 1-Shot
5-Way 5-Shot
10-Way 1-Shot
10-Way 5-Shot
Fully-Supervised (100% training data)
MTB
-/74.70
-/87.90
-/62.50
-/81.10
HCPR
-/76.34
-/83.03
-/63.77
-/72.94
CP
-/79.70
-/84.90
-/68.10
-/79.80
FAEA
-/73.58
-/90.10
-/62.98
-/80.51
GTPN
82.8 / 80.0
91.4 92.6
71.0 69.25
86.0 / 86.9
KEFDA
86.18 / 87.81
94.38 95.00
79.46 / 81.84
90.77 / 90.63
Training-free Baselines (0% training data)
Bert-proto
27.30/-
33.22/-
15.80/-
20.65/-
GPT-proto
54.80/-
81,40/-
41,90/-
63.90/-
Vanilla-ICL
83,40(5)/-
89.80(25y-
68.00(10y-
81.40(30y-
Auto-CoT
81.34(5)/-
89.00(15y-
70.40(10y-
80.7(17y/
reasoning
78.34(5)-
89.80(15)/
66.00(10y-
73.20(13)-
CoT-ER (0% training data)
Auto-CoT-ER
85.40(5)/-
93.40(13)/
76. 10(10)-
86.4(13)/
Manual-CoT-ER
88.00(5)/-
82.60r10y-
Table Main Results FewRel 2.0 validation/test set. All results are given by accuracy (%). (m) means there are instances selected as few-shot demonstrations. +reasoning denotes the demand of reasoning process generation.
few-shot scenario.
abstraction of entity types, specifically the concept- level entities, are beneficial to the LLM performing RE task the few-shot scenario. (2) Despite the third step of CoT-ER pairing the support instance with simpler reasoning process compared to Auto- CoT, it achieves superior performance in certain challenging scenarios (10-Way 1-Shot and medi- cal domain $5.1). This finding indicates that the semantic information provided by the relation la- more beneficial to the LLM than low-quality reasoning information.
Third, CoT-ER prompting method achieves com- petitive performance compared to the state-of-the art fully-supervised method and surpasses the ma- jority of fully-supervised methods with minimal manual labor both on FewRel 1.0 and FewRel 2.0. This suggests that GPT series LLMs have the po- tential to beat previous fully-supervised methods when high-quality relation information and well- designed reasoning processes are provided.
5.2
A blation Study on CoT-ER
5.3 The Stability of CoT-ER
Does the incorporation of entity information sig- nificantly benefit the coT-ER? To this end, we con- ducted ablation experiments to demonstrate the ne- cessity of the 3-step reasoning process. In this experiment, removed the first and second steps and compared the performance with Auto-CoT- reasoning. For fairness concerns, we implemented this experiment using Auto-CoT-ER, which also employs an auto-generated reasoning process by LLM. Due to the limitation of maximum input and output tokens, we set the number of instances in the few-shot demonstrations to 13 for the ablation experiments. The results presented in Figure 4, We find that: (1) after removing the first and sec- ond steps, the performance of Auto-CoT-ER shows a significant decline with reductions 3.4, 2.2, 1.8, 2.9, and 5.2, 6, 5.3, 7.6 Accuracy on FewRel 1.0 and FewRel 2.0 respectively. It means hivher-level
Different Random Seeds for Task Sampling. Due to the high cost of the "text-davinci-003" we sample relatively small number of queries for testing, specifically 100 N for each N-Way K-Shot task. It may raise concerns that the results may not hold up when evaluated on the full test sets. To this end, we evaluated the CoT-ER and Vanilla-ICL using 8 random seeds for N-Way K-Shot task sampling. Table and Table 4 show experimental results with mean + standard devia- tion on FewRel2.0. Notably, CoT-ER consistently outperforms Vanilla-ICL across all N-way K-shot settings with a lower standard deviation.
Different Number of Few-shot Instances. To investigate how the selected number demonstra- tions contribute to the performance of CoT-ER,

---

Sits Lnd EaT-Ritth
Slie and
CoT-FRWthh fopt
futto-CTT
Autoo
02
140
89
S-Way L-hrtt
5-Jtay S-Snet
10/Wyy S-heet
10 S-Stet
S/NW 1-5hst
540b 5-thet
12Hee E-5tet
10A0ay 5-Bat
(a) FewRell.0
(b) FewRe12.0
Figure 4: Ablation study the first and second reasoning steps of CoT-ER. Auto-CoT refers to the "with reasoning generation" version.
Method
5-Way 1-Shot
5-Way 5-Shot
5.4 Case Study
Vanilla-ICL
80.75+1.70 85.64+1.28
88.1911.50
We select one typical reasoning example generated by LLM better demonstrate the superiority of our prompting method CoT-ER. As shown Table 6, this instance necessitates the LLM to correctly identify the relation label "crosses" between "Rail- way Bridge" and "Daugava". In FewRel 1.0, the relation label "crosses" is described as "obstacle (body of water, road, ...) which this bridge crosses this tunnel goes under".
CoT-ER
92.99+ 1.43
Table 5-way classification performance compari- son (mean + standard deviation) between CoT-ER and Vanilla-ICL on FewRel 2.0 across 8 different random seeds.
Method
10-Way 1-Shot
10-Way 5-Shot
Vanilla-ICL
18.21
80.92+1.00 1.05
However, using Auto-CoT prompting leads to a wrong prediction where the model incorrectly la- bels the relation with "located in or next to body of water", which pertains to the relationship between location entity and a water-related entity (river, lake, ...). The primary reason for the failure of Auto- CoT is the absence of higher-level abstraction of entities in reasoning, which is necessary compre- hend the entities involved in the relation. CoT-ER addresses this issue by incorporating concept-level information into the reasoning process through its first and second steps. Specifically, in this case, the LLM first reasons the subject entity corresponds to bridge and the object entity corresponds to river based its own knowledge base and con- textual information, thereby excluding the relation requiring a location entity and water-related enti- ties (as demonstrated by other examples in prompt). With this clue, the LLM can effectively perform subsequent reasoning steps.
CoT-ER
77.61+0.87
86.18+0.88
Table 4: 10-way classification performance compari- son (mean standard deviation) between CoT-ER and Vanilla-ICL on FewRel 2.0 across eight different ran- dom seeds.
we conducted experiments across different M un- der the 5- 5-W 5-Shot setting. A single prompt can hold 13 CoT-ER reasoning demonstrations at worst, whereas all the support instances (25) can be ap- pended to the prompt in Vanilla-ICL. The results are presented in Table 5.
We observe that both CoT-ER and Vanilla-ICL can benefit from more few-shot examples, which aligns with the same conclusion in previous work (Liu et al., 2022a). However, the performance of VanLlla-ICL deteriorates rapidly as the number of examples decreases. CoT-ER effectively lever- age the information from provided instances and maintain strong performance even with reduced number of instances. This demonstrates that CoT- ER exhibits greater stability compared to Vanilla- ICL when the number of fhw-shot instances varies.
Furthermore, the presence of both "crosses" and "located in next body of water" labels in an N-way K-shot task can indeed confuse the L.LM due to the lack of semantic information on these two phrases. CoT-ER addresses this issue by inte-

---

M
6
11
13
17
19
21
25
CoT-ER
88.40
88.80
90.80
91.60
91.20
89.20
89.20
91.80
94.20
Vanilla-ICL 73.40
78.00
78.80
82.90
81.40
82,39
81,20
82.79
84.20
87.60
89.0
90.40
90.80
90.60
89.80
Table 5: Performance comparison (given by accuracy) between CoT-ER and Vanilla-ICL on FewRel 2.0 across
different numbers of demonstrations in single prompt.
Prompting Method
Reasoning Process
Correct
The clues that lead to the relation between "Railway Bridge" and "Daugava" to be "locuted in
Auto-CoT
body water" in the sentence the words "crosses" and "river". These words indicate that the Railway Bridge is crossing Daugava River, which is a body water.
Subject entity "Railway Bridge" bridge name, which refers the entity bridge in the context.
Object entity "Daugava" is river name, which refers entity of river the context.
CoT-ER
3. According to the context, "The Railway Bridge is a bridge that crosses the Daugava river."
indicates that "Railway Bridge" emoses "Daugava".
Table We compare our prompting method CoT-ER with Auto-CoT for an example in FewRel 1.0. The context of this example is "The Ruilway Bridge is bridge that crosses the Daugava river in Riga, the cupital of Latvia.", and the relation between two highlight entities is "crosses". The key reasoning processes CoT-ER are highlighted with green.
grating both entities and the relation label into coherent expression, exemplified by this exam- ple "Railway Bridge" crosses "Daugava".
tial for future improvement. Our proposed method does not fully utilize all instances when handling larger support sets, such as 5-way 5-shot and 10- way 5-shot, due to the constraint of maximum length. Though we adopt a similarity-based KNN retrieval to select superior instances for few-shot demonstrations, we find not as effective the few-shot setting compared to other works that per- form well when there is large candidate set avail- able. Due the high cost of employing reasoning- required ICL via GPT-3 API, we have not evaluated the CoT-ER on superior LLM with longer maxi- mum input tokens and larger scale.
6 Conclusion
this paper, we explore the potential of LLM in- context learning on few-shot relation extraction. To enhance the general performance caused by low- quality auto-generated reasoning processes, we introduce CoT-ER, prompting method tailored to few-shot relation extraction. The core idea is prompt LLM to generate evidence using task- specific and concept-level knowledge stored in its pre-training stage. These pieces of evidence will be utilized by LLM during the RE task, and facili- tate the reasoning process. Additionally, we devise a label verbalizing technique by integrating both entities and the relation label into a coherent ex- pression. This technique addresses the semantic ambiguity of relation labels, which is common challenge encountered during relation extraction when utilizing in-context learning. The experimen- tal results on FewRel 1.0 and FewRel 2.0 outper- form all training-free baselines, demonstrating the effectiveness of our proposed approach. Moreover, achieving comparable results to the state-of-the the-art sully-sup method suggests that the paradigm of in-context learning holds promise novel so- lution for the few-shot relation extraction task.
Our limited budget also restricted the optimiza- tion for the construction seed examples. is possible to enhance the performance with more informative and appropriate design.
Ethics Statement
known that pre- trtirrined language models could capture the bias reflecting training data. Thus, our approach using LLMs can potentially generate of- fensive or biased content. We acknowledge this risk and suggest that practitioners should carefully examine the potential bias before deploying models in real-world applications.
References
Nguyen Bach and Sameer Badaskar. 2007. A review of relation extraction. Literature review for Language and Statistics II, 2:1-15.
Limitations
Although CoT-ER achieved decent results on FewRel 1.0 and FewRel 2.0, there is still poten-
Livio Baldini Soares, Nicholas FizzGerald, Jeffrey Ling.

---

and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In The 57th Annual Meeting of the Association for Compu- tational Linguistics (ACL), pages 2895-2905.
Junheng Hao, Muhao Chen, Wenchao Yu, Yizhou Sun, and Wei Wang. 2019. Universal representation learn ing of knowledge bases by jointly embedding in- stances and ontological concepts. In The 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (SIGKDD), pages 1709- 1719.
Sam Brody, Sichao Wu, and Adrian Benton, 2021. To- wards realistic few-shot relation extraction. In The 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5338-5345.
Or Honovich, Uri Shaham, Samuel Bowman, and Omer Levy. 2022. Instruction induction: From few examples to natural language task descriptions. arXiv preprint arXiv:2205. 10782.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems (NIPS), 1901.
Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 in-context learn- ing for biomedical IE? think again. In Findings of the Association for Computational Linguistics (EMNLP), pages 4497-4512.
Claire Cardie. 1997. Empirical methods in information extraction. Al magazine, 18(4))65-65.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large lan- guage models are zero-shot reasoners. arXiv preprint arXiv:2205.1 11916
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng. Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui, 2022. A survey for in-context learning. arXiv preprint 201.02330.00234.
Jing Li, Yequan Wang. Shuai Zhang, and Min Zhang, 2023. Rethinking document-level relation extraction: A reality check. In Findings of the Association for Computational Linguistics (ACL), pages 5715-5730.
Chunliu Dou, Shaojuan Wu, Xiaowang Zhang, Zhiyong Feng, and Kewen Wang. 2022. Function- words en- hanced attention networks for few-shot inverse rela- tion classification. arXiv preprint arXiv:2204. 12111.
Wanli Li and Tieyun Qian. 2022. Graph- based model generation for few-shot relation extraction. In The 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 62-71.
Liu Fangchao, Xiao Xinyan, Yan Lingyong, Lin Hongyu, Han Xianpei. Dai Dai, Wu Hua, and Sun Le. 2021. From learning- to- to-matc to learning-to. to- discriminate:global prototype learning for few-shot relation classification. In The 20th Chinese National Conference on Computational Linguistics (CCL). pages 1012-1023.
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022a. What makes good in-context examples for GPT-3? In Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 100-114.
Tianyu Gao, Xu Han, Zhiyuan Liu, and Maosong Sun. 2019a. Hybrid attention-based prototypical networks for noisy few-shot relation classification. In The AAAI conference on artificial intelligence (AAAT), volume 33, pages 6407-6414.
Yang Liu, Jinpeng Hu, Xiang Wan, and Tsung-Hul Chang. 2022b. A simple yet effective relation in- formation guided approach for few-shot relation ex- traction. In Findings of the Association for Computa- tional Linguistics (ACL), pages 757-763
Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2019b. FewRel 2.0: Towards more challenging few-shot relation classifi- cation. In The 2019 Conference Empirical Meth- ods in Natural Language Processing and the 9th In- ternational Joint Conference on Natural Language Processing (EMNLL//IJCNLP. P),, pages 6250-6255.
Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022. Cutting down prompts and parameters: Simple few-shot learning with language models. In Find- ings of the Association for Computational Linguistics (ACL), pages 2824-2835.
Jiale Han, Bo Cheng, and Wei Lu. 2021. Exploring task difficulty for few-shot relation extraction. In The 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2605-2616.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In The 60th Annual Meeting of the Association for Computational Lin- guistics ( (Volume I: Long Papers) (ACL), pages 8086- 8098.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang. Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. FewRel: large-scale supervised few-shot relation classifica- tion dataset with state-of-the-art evaluation. In The 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4809.
Huan Ma, Changqing Zhang. Yatao Bian, Lemao Liu, Zhirui Zhang, Peilin Zhao Shu Zhang, Huazhu Fu,

---

Qinghua Hu, and Bingzhe Wu. 2023, Fairness- guided seo- prompting for large language mod- els. arXiv preprint ar arXiv:2303.1 13217.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An- jana Arunkumar, David Stap, al. 2022. Super- naturalinsttuccioos:: Generalization via declarative instructions on 1600+ nlp tasks. In The 2022 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 5085-5109.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis. Hannanch Hajishirzi, and Luke Zettle- moyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In The 2022
Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 11048-10644
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoe Le, and Denny Zhou, 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11003
Sachin Pawar, Girish K Palshikar, and Pushpak Bhat- tacharyya. 2017. Relation extraction: A survey. arXiv preprini 1rXiv:1712.05191.
Zhiyong Wu, Yaoxiang Wang, Jiacheng Ye, and Ling- peng Kong. 2022. Self-adaptive in-context learning. arXiv preprint arXiv:2212.10375
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. In The 2020 Conference on Empirical Methods Natural Lan- guage Processing (EMNLP), pages 3661-3672.
Shan Yang. Yongfei Zhang, Guanglin Niu, Qinghua Zhao, and Shiliang Pu. 2021. Entity concept- enhanced few-shot relation extraction. In The 59th Annual Meeting of the Association for Computational Linguisties and the I 11th International Joint Confer- ence on Natural Language Processing (Volume 2: Short Papers) (ACL/UCNLP)), pages 987-991.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True few-shot learning with language models. Ad- vances neural information processing systems (NIPS), 34:1 1054-11000
Jiawen Zhang, Jiaqi Zhu, Yi Yang, Wandong Shi, Cong- cong Zhang, and Hongan Wang. 2021. Knowledge- enhanced domain adaptation in few-shot relation clas- sification. In The 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining (SIGKDD). pages 2183-2191.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis. Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In The 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP/JCCNLP) pages 2463-2473.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022. Automatic chain of thought prompt- ing in large language models. arXiv preprint arXiv:2210.03493.
Meng Qu, Tianyu Gao, Louis-Pascal Xhonneux, and Jian Tang. 2020. Few-shot relation extraction via Bayesian meta-learning on relation graphs. In The 37th International Conference on Machine Learning (ICML), volume 119, pages 7867-7876.
Yongchao Zhou, Andrei loan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-leve prompt engineers. arXiv preprint 11119.221..01910.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parame- ters of a language model? In The 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 5418-5426.
A Error Analysis
A.1 Error Analysis of Relation Type
To fully justify CoT-ER as superior approach to the baseline, we will provide a quantitative analy- sis comparing CoT-ER and Vanilla-ICL across all test classes. Taking the 5- 5-y 1-Shot setting as an example, Table and Table 8 display the accu- racy of these two methods across various relation classes within FewRel. Note that different numbers represent different relation types, and their order is based on the performance ranking of Vanilla-ICL.
Timo Schick and Hinrich Schüizee 2022. True few-shot learning with Prompts-4 A real-world perspective. Transactions of the Association for Compitational Linguistics (TACL.), 10:716-731.
Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can't 't plan (a benchmark for Ilms on plan- ning and reasoning about change). arXiv preprint arXiv:2206. 10498.
These accuracies are the average results obtained from multiple test runs, each performed with seven different random seeds.
Zhen Wan, Fei Cheng. Zhuoyuan Mao. Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023. Gpt-re: In-context learning for relation ex- traction using large language models. arXiv preprint arXiv:2305.02105.
Table 7 shows the experimental results on FewRel 1.0. can observe that CoT-ER surpasses

---

Relation Type
3
5
7
13
14
15
16
Vanilla-ICL
100
100
99.12
98.63
98.3
98.13
97.72
96.64
96.57
94.15
92.94
91.01
85.
84.63
83.71
82.47
CoT-FR
100
100
99.52
98.07
100
98.65
95.51
89.47
100
95.25
91.72
95.66
89.31
95.98
86.95
91.39
Difference
0
0.4
0.56
1.7
0.52
-2.11
77.17
3.43
1.1
-1222
4.65
4.07
11.35
3.24
8.92
Table Results of each label in FewRel 1.0. Definition of digital relation type:1 voice type, 2 position played on team/speciality, 3 original language of film or TV show, 4 constellation, 5 military rank, 6 competition class, 7
member of. 8 spouse, 9 located or next to body of water, 10 follows, 11 crosses, 12 sport, 13 main subject, 14 child, 15 part of, 16 mother.
Relation Type
2
9
10
Vanilla-ICI.
98.71
95.71
88.82
88.19
79.33
75.54
74.07
70.71
67.69
65.46
CoT-ER
97.84
95.95
89.29
87.3
71.32
93.01
71.32
89.84
80.41
81.16
Difference -0.87
0.24
0.47
-0.89
-8.01
17.47
-275
19.13
12.72
15.7
Table Results of each label in FewRel 2.0. Definition of digital relation type: inheritance type of, 2 ingredient of, 3 classified as, 4 gene found in organism, 5 is primary anatomic site of disease, causative agent of, biological process involves gene product, gene plays role process, 9 is normal tissue origin of discase, 10 occurs in.
Reatiio latel
Bessning Process
Corredt
Canoat:
hepacempas concriptiona sprogatetio oherval regrestorr poustrjary), C.T-ERt
Entky "trascriptiond" refers proces tratserihing inte which refers the easty biotogical process contest.
pote sate process
of game, woich eciars unity of poac soeaat 3. Accodda to lie conoo, "Irasciptiond sprepultices was olsevvd s lcee day pra-iijsayy, uas and pash serva days prosiartes" indicesa tha "mag" play rule in "tranectpthoree"
1. Eaiily "mag"
Su, fee vdlaiie boowess "mazacippisaa" and "maa" "acaz plase role procos".
Their matrral grodppoeees John Coort Holland Philigps Lrommbrrrg
precso
spoose
Subject codiy "Philiipo Loverbbburg" personal
entry
OPjeett raitt "John It, thelland" a persmna rome, ehnch ritrrs earty panson contrat
3. Acceedan to umraa "maacnal grasdpazzand sadicaze that Objes, catly "Sube II, Coust Hollesd sediiid Subjos enly "Phllipe Laseeitsuur"
So, noe notauee herwess "Philqua coverborrg" and Jote II, Court of Hoaline" "ybile"
Conteat:
other amas dets
majonty of
ofT
pricory
2. Ently "Ski"
organ,
estry plosioal locatin body part contest. other amas coar indicste ther "oos helr" occuss "etts"
3. Acceeding to contor 1oos
So, the relasies beoweca "loss of hail" and "aaa'' is "sccursia".
Table 9: The representative scenarios where CoT-ER makes mistakes for different relation class.
Vanilla-ICL 10 times, with 7 of them showing a rel- atively high improvement. However, in the 7th and 8th relations, CoT-ER still lags behind Vanilla-ICL. by a few percentage points.
Relation class gene plays role in process: In this case, CoT-ER not only recognize what "transcriptional" and "mean" mean in the context but also have concept-level knowledge. And the final prediction correct.
Table 8 shows the experimental results on FewRel 2.0. We can observe that CoT-ER surpasses Vanilla-ICL 6 times, with 4 of them showing sig- nificant improvement of over 10%. However, in the 5th and 7th relations, CoT-ER still lags behind Vanilla-ICL. by few percentage points.
Relation class spouse: this case, CoT-ER cor- rectly recognizes the entity type and precisely ex- tracts the crucial evidence "maternal grandparents". However, the LLM incorrectly interprets "mater- nal grandparents" as the relationship between these two entities, when they are actually a couple of "maternal grandparents". This demonstrates that the LLM with CoT-ER may overlook contextual information sometimes.
A.2 Error Analysis of Reasoning Process
CoT-ER shows significant improvements in scenarios when Vanilla-ICL performs poorly. Here, we present few cases illustrate the reason- ing process of CoT-ER in some representative scenarios in order to facilitate future research. Table 9 shows some correct and incorrect answers produced by CoT-ER.
Relation class primary anatomic site of dis- ease: In this case, CoT-ER can also recognize the entity well, and the conclusion is semantically right

---

("loss of hair" occurs in "skin"). However, the fi- nal prediction is incorrect, the predicted relation label "occurs in" while the ground truth label "is primary anatomic site of disease". The reason is that the label "occurs in" this dataset means "a condition occurs in period of the lifetime". This particular label should be matched with en- tity pairs like "condition disease" and "a period of person's lifetime (such congenital)". This issue indicates that the relation description would be a key component in such methods, but it's not included the FewRel 2.0 dataset.
B Seed Examples
In this section, will give more details about how we construct the seed examples and present all seed examples in Table 10, Table 11, Table 12, Table 13 and Table 14.
We first randomly select one instance from each relation class to serve as the seed example. Then we outline the three steps of CoT-ER in each in- stance. By considering the contextual information of the two entities involved, manually assign the entity types and identify relevant text spans evi- dence. Because FewRel 1.0 provides description each relation (such as "sport: sport in which the subject participates or belongs to") in separate file while FewRel 2.0 does not, this has implications for the design of seed examples. However, these descriptions are not directly used in the prompts. Furthermore, we haven't optimized the seed exam- ples, leaving room for further improvement.
C Prompts
All prompt templates used in this paper are pre- sented in Table 15.

---

mother
Context:
Anne de Bourbon Septemmeprr 1408) was a daughter of John I, Count of Marche and his wife Catherine of Vendome.
CoT-ER:
Subject entity "Anne de Bourbon" is personal name, which refers to the entity of son or daughter the context.
Object entity "Catherine of Vendôme" is a personal name, which refers to the entity female human beings in the context. According to the context, "was daughter of John Count La Marche" and "his wife" indicate that
object entity "Catherine of Vendôme" is subject entity "Anne de Bourbon"'s mother. So, the relation between subject entity "Anne de Bourbon" and object entity "Catherine of Vendôme" is "mother".
child
Context:
He was a son of Margrethe Rode, and a brother of writer Helge Rode.
CoT-ER:
Subject entity "Margrethe Rode" is a personal name, which refers to the entity of an adult in the context.
Object entity "Helge Rode" is personal name, which refers to the entity of son or daughter the context.
According to the context, "son Margrethe Rode" and "brother writer Helge Rode" indicate that Object entity "Helge Rode" is child of Subject entity "Margrethe Rode".
So, the relation between subject entity "Margrethe Rode" and object entity "Helge Rode" is "child".
spouse
Context:
Hanke's 's scemingly unstoppable ascent the coattails of Goehhels came sudden, alheit temporary, halt when he was drawn into the marital affairs Joseph Goebbels and his wife, Magda.
CoT-ER:
Subject entity "Magda" is a personal name, which refers to the eatity human beings in the context.
Object entity "Joseph Goebbels" is a personal name, which refers the entity human beings in context. According to the context, "marital affairs" and "his wife" indicate that "Magda" is "Joseph Goehbels"' "s wife, which means they are a married couple.
So, the relation between subject entity "Magda"" and object entity "Joseph Goebbels" is "spouse".
sport
Context:
Demetrius Rhaney (born June 1992) American football center for the Washington Redskins of the National Football League (NFL).
CoT-ER:
Subject entity "Demetrius Rhancy" personal name, which refers the entity of American foothall player in the context.
Object entity "American football" is a sport, which refers the entity sport the context. According to the context, "Demetrius Rhaney (born June 22, 1992) is an American football center" indicates that "Demetrius Rhaney" participates or belongs to "American football".
So, the relation between subject entity "Demetrius Rhaney" and object entity "American football" is "sport".
crosses
Context:
Wilton Bridge was major crossing of the River Wye and was protected by Wilton Castle.
CoT-ER:
Subject entity "Wilton Bridge" a bridge name, which refers the entity of bridge in the context.
Object entity "River Wye" is a river name, which refers the entity river in the context. According to the context, "Wilton Bridge was a major crossing of the River Wye" indicates that "Wilton Bridge" crosses "River Wye".
So, relation between subject entity "Wilton Bridge" and object eatity "River Wye" is "crosses".
original language of film or TV show
Context:
He composed song called "Zindagi aur bata tera irada kya hai" for the Hindi movie "Zindagi Aur Toofan" (1975).
CoT-ER:
Subject entity "Zindagi Aur Toofan" is movie name, which refers the entity of film or performance work in the context.
Object entity "Hindi" language, which refers the entity of language the context. According the context, "Hindi movie Zindagi Aur Toofan" indicates that the film "Zindagi Aur Toofan" was originally made in "Hindi". So, relation between subject entity "Zindagi Aur Toofan" and object entity "Hindi" "original language film TV show".
Table 10: Seed examples in FewRel 1.0.

---

competition class
Context:
Sasakul turned pro in 1991 and captured the WBC and lineal flyweight titles with win over Yuri Arbachakov 1997.
CoT-ER:
Subject entity "Yuri Arbachakov" is a personal name, which refers to the entity events, teams, participants, or equipment in the context.
Object entity "flyweight a competition type of boxing match, which refers to the entity of competition or organization in the context.
3. According to the context, "captured the WBC and lineal flyweight titles with a win over Yuri Arbachakov" indicates that object entity "Yuri Arbachakov" qualifies for inclusion in "flyweight".
So. the relation between subject entity "Yuri Arbachakov" and object entity "flyweight" is "competition class".
part
Context:
The observatory began operation around 1984 with the Isaac Newton Telescope, which was moved to La Palma from the Royal Greenwich Observatory site at Herstmonceux Castle Sussex, England.
CoT-ER:
1. Subject entity "Isaac Newton Telescope" a scientific instrument, which refers to the entity of components in the context.
Object entity "Royal Greenwich Observatory" is scientific institution, which refers to the entity of entirety in the context. According to the context, "which was moved to Palma from the Royal Greenwich Observatory" indicates that
"Isaac Newton Telescope" is part of "Royal Greenwich Observatory".
So, the relation between subject eatity "Isaac Newton Telescope" and object entity "Royal Greenwich Observatory" is "part of".
constellation
Context:
Tau2 Gruis, is a double star located in the constellation Grus.
CoT-ER:
Subject entity "Tau2 Gruis" is the name of a celestial body, which refers to the entity of a celestial body in the context.
Object entity "Grus" the name of constellation, which refers to entity of the constellation the context. According to the context, "located in the constellation Grus" indicates that
"Tau2 Gruis" is part of constellation "Grus".
So, the relation between subject entity "Tau2 Gruis" and object entity "Grus" is "constcllation".
position played on team/speciality
Context:
Haitham Simrrenn(born I January 1977) a retired Jordanian footballer Palestinian origin, who was Defender for Al- Wehdat and the Jordan national football team.
CoT-ER:
Subject entity "Haitham Simreen" is personal name, which refers to the entity of sports player the context. Object entity "Defender" a position in football, which refers to the entity of position or specialism the context. According to the context, "who was a Defender for Al- Wehdat and the Jordan national foothall team" indicates that "Haitham Simreen" plays as "Defender" position that team.
So, the relation between subject entity "Haitham Simreen" and ohject entity "defender" is "position played on tcam/spociality".
located in next body of water Context:
Both the Grand Canal, and the Royal Canal flow through Westmeath, and the River Shannon (Ireland's key tourism waterway) has modern inland harbour Athlone.
CoT-ER:
Subject entity "Athlone" is a place name, which refers to the entity of position in the context. Object entity "River Shannon" " the name of the river, which refers to the entity of river or water in the context. According to the context, "River Shannon (Ireland's key tourism waterway) has modern inland harbour Athlone"
indicates that "Athlone" is located in or next to body of water "River Shannon", So, the relation between subject entity "Athlone" and object entity "River Shannon" "located in or next to body of water".
voice type
Context:
Gabriella Gatti (July 1908- Octoher 22, 2003) was an Italian operatic soprano, primarily based Italy and associated with the Italian repertory.
CoT-ER:
Subject entity "Gahriella Gatti" personal name, which refers the entity of person in the context. sopraaoo indicates that
Object entity "soprano" is a tone, which refers to the entity of voice type in the context.
According to the context, "Italian operatic soprano"
"Gabriella Gatti" has voice type of "soprano".
So, the relation between subject entity "Gabriella Gatti" and object entity "soprano" is "voice type".
Table I: Seed examples in FewRel 1.0.

---

follows
Context:
Hot Dance Club Play chart, along with album tracks "Whammy Kiss" and "Song for a Future Generation".
CoT-ER:
1. Subject entity "Song for a Future Generation" is mention in the context, which refers to the entity of part of a series in the context.
2. Object entity "Whammy Kiss" is mention in the context. which refers to the entity of part of series in the context.
3. According to the context, "Whammy Kiss and Song for Future Generation" indicates that "Song for Future Generation" follows "Whammy Kiss".
So, the relation between subject entity "Song for Future Generation" and object entity "Whammy Kiss" is "follows".
military rank
Context:
The French under Jean de Vienne, Admiral of France joined forces with the Scots.
CoT-ER:
1. Subject entity "Jean de Vienne" is a personal name, which refers to the entity of a person in the context.
2. Object entity "Admiral of France" is name of military rank, which refers to the entity of military rank in the context.
3. According to the context, "Jean de Vienne, Admiral of France" indicates that
"Jean de Vienne" holds the military rank of "Admiral of France".
So. the relation between subject entity "Jean de Vienne" and object entity "Admiral of France" is "military rank".
member of
Context:
Neil Tennant and Chris Lowe of the Pet Shop Boys commented on this remix to journalist Mark Beaumont, writing for the "NME", in February 2017.
CoT-ER:
1. Subject entity "Chris Lowe" is personal name, which refers to the entity of a person in the context.
2. Object entity "Pet Shop Boys" is the name of the organization, which refers to the entity of the organization club in the context.
3. According to the context, "Chris Lowe of the Pet Shop Boys" indicates that "Chris Lowe" is member of "Pet Shop Boys"
So, the relation between subject entity "Chris Lowe" and object entity "Pet Shop Boys" is "member of".
main subject
Context:
The British Free Corps is featured in Jack Higgins' Worldd War Il thriller "The Eagle Has Landed".
CoT-ER:
1. Subject entity "The Eagle Has Landed" is an artistic work name, which refers to the entity of work in the context.
2. Object entity "World War 11" is a historical event, which refers to the entity of the main subject or theme in the context.
3. According to the context, "World War II thriller "The Eagle Has Landed indicates that "World War II' is the main subject of the work named "The Eagle Has Landed".
So, the relation between subject entity "The Eagle Has Landed" and object entity "World War II" is "main subject".
Table 12: Seed examples in FewRel 1.0.

---

biological process involves gene product
Context:
I have discussed various aspects the structure, regulation and function of the protein tyrosine phosphatase family, which hope will illustrate the fundamental importance of these enzymes the control signal transduction.
CoT-ER:
Entity "protein tyrosine phosphatase" is a type of enzyme, which refers to the entity of gene product in the context. Entity "signal transduction" the name of biological process, which refers to the entity of biological process in the context. According to the context, "the fundamental importance of these enzymes in the contral of signal transduction" indicates that "protein tyrosine phosphatase" is involved in "signal transduction". So, the relation between "protein tyrosine phosphatase" and "signal transduction") is
"biological process involves gene product".
gene plays role process
Context:
demonstrate here that critical step in establishing angiogenic capability of human tumor cells the repression a key secreted auti-angiogenic factor, thrombospondin- (Isp-1).
CoT-ER:
Entity "angiogenic" is the name of biological process, which refers to the entity of biological process in the context. Entity "tsp-1" is an abbreviation of thrombospondin-1, which refers to the entity of gene product the context. According to the context, "establishing the angiogenic capahility" and "the repression of a key secreted anti-anguu angioonni factor, thrombospondin- I indicate that "tsp- 1" plays role establishing the "angiogenic" capability, So, the relation between "angiogenic" and "Isp-1" is "gene plays role in process".
occurs in
Context:
determined whether this also true for the congenital muscle disease nemaline myopathy using our mouse model of this disease.
CoT-ER:
Entity "congenital" refers to condition or trait that present at or before birth, which refers to the entity of time interval the context.
Entity "nemaline myopathy" is the name of a discase, which refers to the entity of disease in the context.
3. According to the context, "congenital musele disease nemaline myopathy" indicates that "nemaline myopathy" occurs in "congenital" period.
So, the relation between "congenital" and "nemaline myopathy is "occurs in".
inheritance type
Context:
Here we are presenting two siblings, -e-year-old male and 13-year-old female with features of progeria, suggesting possible autosomal recessive inheritance.
CoT-ER:
Entity "progeria" is the name of disease, which refers the entity of the disease the context.
Entity "autosomal recessive inheritance" is the name of a specific pattern of inheritance, which refers to the entity of inheritance in the context.
According to the context, "suggesting a possible autosomal recessive inhoritance" indicates that "progeria" is type of "autosomal recessive inheritance" disease. So, the relation between "progeria" and "autosomal recessive inheritance" is "inheritance type of".
normal tissue origin of disease Context:
microcystic adnexal carcinoma (mac) is known as an infiltrating but non- non-metastasizing tumour the skin, that derives from sweat glands follicular epithelium.
CoT-ER:
Entity "microcystic adnexal carcinoma" the name of disease, which refers to entity of disease in the context. Entity "'pithelium" a type cell, which refers to the entity of biologicul tissue in the context. According to the context, "that derives from sweat glands or follicular epithelium" indicates that "cpithelium" is the normal tissue origin of "microcystic adnexal carcinoma". So, the relation between "microcystic adnexal carcinoma" and "opithelium" is "is normal tissue origin of disease".
causative agent of Context:
The pathogenesis of amochic dysentery is result of cytolysis of the colonic mucosa by parasitic protozoan entamoeba ytstal
CoT-ER:
Entity "amoebic dysentery" is name a disease, which refers to the entity disease in context.
Entity "entamoeba histolytica" a type of protozoa. which refers to the entity causative agent the context.
According to the context, "is a result of cytolysis the colonic mucosa by the purasitic protozoan entamoeba histolytica" indicates that "entamoeha histolytica" is the causative agent of "amoebic dysentery"..
So, the relation between "amoebic dysentery" and "entamocba histolytica" is "causative agent of".
Table I3: Seed examples in FewRel 2.0.

---

classified as
Context:
On the other hand, in the nonlethal p. yoelii 17xml infection, wt mice could control a primary infection with 1 x (5) parasitized erythrocytes.
CoT-ER:
1. Entity "infection" is the name of a biological process, which refers to the entity of abstractions of certain type of thing in the context.
2. Entity "primary infection" is the name of biological process, which refers to the entity of abstractions of certain type of thing in the context.
3. According the context, "in the nonlethal yoelii 17xnl infection" and
"control primary infection with x 10 (5) parasitized erythrocytes" indicate that "infection" and "primary infection" are related to each other in way that one is subset of the other two classes of something.
So, the relation between "infection" and "primary infection" is "classified as".
gene found in organism
Context:
Ustekinumab is fully human monoclonal immunoglohulin antibody that targets the interleukin (il) -12 and il-23 shared p40 subunit.
CoT-ER:
1. Entity "human" is type of organism, which refers to the entity of organism in the context.
2. Fintity "il-23" is the name of gene, which refers the entity of the gene in the context.
3. According the context, "ustekinumab is fully human monoclonal immunoglobulin antibody"
and "targets the interleukin (il) -12 and i1-23 shared p40 suhunit" indicate that "il-23" is found in "human" organism.
So, the relation between "human" and "il-23" is "gene found in organism".
ingredient of
Context:
Against a bla+ isolate, the combination of piperacillin with tazobactam with streptomycin resulted in synergistic effect relative to that of piperacillin with tazohactam; piperacillin plus streptomyein did not show synergism.
CoT-ER:
1. Entity "piperacillin with tazobactam" is combination of two entities, which refers to the entity of composite component in the context.
2. Entity "piperacillin" is the name of drug. which refers to the entity of individual components in the context.
3. According the context, "the combination of piperacillin with tazobactam" indicates that
"piperacillin" ingredient of "piperacillin with tazobactam",
So, the relation between "piperacillin with tazohactam" and "piperacillin" is "ingredient of".
is primary anatomic site of disease Context:
Twenty-five of 88 (28.4%) had endometrial carcinoma on tinal uterine pathology.
CoT-ER:
1. Entity "endometrial carcinoma" is the name of disease, which refers to medical condition in the context.
2. Entity "uterine" is the name of the body organ, which refers to the entity of physical location of body parts in the context.
3. According to the context, "endometrial carcinoma on final uterine pathology" indicates that "uterine" is primary anatomic site of "endometrial carcinoma".
So, the relation between "endometrial carcinoma" and "uterine" is "is primary anatomic site of discase"..
Table 14: Seed examples in FewRel 2.0.

---

Vanilla-ICL
Please solve the Relation Extraction task.
Given the context, consider what's the most precise relation between entities belonging the following [N]
possible relations.
The relation must be in these [N] possible relations: relation label ..., relation label
Context: [CONTEXT]
Given the context, the relation between [HEAD_ ENTITY] [NAIL_ENTITY] is IGOLDEN LABEL..
XM
Context: (CONTEXT)
Given the context, the relation between [HEAD ENTITY] and [TAIL ENTITY]
Auto-CoT
Please solve the Relation Extraction task.
Given the context, consider what's the most precise relation between entities belonging the following [N]
possible relations.
The relation
these [N] possible relations: relation label ... relation label
Context: [CONTEXT]
Given the context, what's the relation between [HEAD ENTITY] ENTITY? [Answer CoTI
relation between [ and (NIITY] is |GOLDEN LABEL). XMM
Context: [CONTEXT]
Given the context, what's the relation hetween [ENETTY and [TAIL,_ ENTITY]
CoT-generation
Please solve the Relation Extraction task.
Given the context, figure the reasoning steps that lead the relation between entities be the specific
Context: [CONTEXT]
Given the context, what's the relation between [HEAD_ENTITY Now, known relation [GOLDEN_ LABEL], [LAEEE)_LABEL), the reasoning steps are:
[TAIL_
[Seed example specific class] relation between [HEAD ENTITY) and [TAIL ENTITY] is (GOLDEN LABEL). x.M
Context: [CONTEXT]
Given the context, what's the relation hetween [HEAD ENTITY] ENTITYY Now, known the relation [GOLDEN, LABELJ, the reasoning steps are:
Table 15: Prompt templates in the paper.


Reviews:
[{"review_id": "Z5JW9nM9Y2", "reviewer": "Reviewer_cCve", "summary": "", "questions": "**Question A:**  In what scenarios does CoT-ER provide an improvement over Naive-ICL?\n\n**Question B:** How is the number of demonstrations selected from the entire few-shot samples determined?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We will explain your concerns point by point.\n\n1. **Comment**: Only 100 samples are used from each dataset for evaluation. Thus, the results may not hold up when evaluated on the full test datasets.\n\n   **Reply**: Thanks for your insightful suggestions. It is worth mentioning that 100 samples here mean 100  meta-tasks, which are randomly sampled from instances in the dataset. Each N-way K-shot meta-task contains N queries for each relation class, which means there are $100 \\times N \\times (K+1)$ instances in total. Due to limitations on the **text-davinci-003** API usage (such as access rate limit) and the higher cost of access ($0.02/1K tokens), we had to reduce the sampling number. \n\n   In addition, we have evaluated the CoT-ER and Naive-ICL **with 8 random seeds** (i.e., 8 independently sampling for the meta-task) in order to address this problem and will add these results to the final paper. The table below shows experimental results with **mean ± standard deviation** on FewRel2.0. We can observe that **CoT-ER consistently outperforms Naive-ICL in all N-way K-shot settings with a lower standard deviation**.\n\n   | Method       | 5-Way 1-Shot | 5-Way 5-Shot | 10-Way 1-Shot | 10-Way 5-Shot |\n   | ------------ | ------------ | ------------ | ------------- | ------------- |\n   | Naive-ICL    | 80.75±1.7    | 88.19±1.56   | 68.21±1.27    | 80.92±1.05    |\n   | CoT-ER(ours) | 85.64±1.28   | 92.99±1.43   | 77.61±0.87    | 86.18±0.88    |\n\n   \n\n2. **Comment**: No error analysis - The naive in context learning (Naive-ICL) baseline results are quite competitive with the best results from CoT-ER, the proposed approach. Some error analysis may help justify CoT-ER as an approach over the baseline.\n\n   Question A: In what scenarios does CoT-ER provide an improvement over Naive-ICL?\n\n   **Reply**: Here, we address the error analysis issue with a **quantitative** and **qualitative** analysis. We will add more analysis and discussion in the final paper.\n\n   **(1) Quantitative Analysis** \n\n   Taking 5-Way 1-Shot as an example, the table below shows the accuracy of these two methods on different relations of the FewRel. Note that different numbers represent different relation types, and their order is based on the performance ranking of Naive-ICL. These accuracies are the average of multiple test runs conducted with 7 different random seeds. \n\n   The table below shows the experimental results on **FewRel2.0**. We can observe that CoT-ER surpasses Naive-ICL 6 times, with 4 of them showing a significant improvement of over 10%. However, in the 5th and 7th relations, CoT-ER still lags behind Naive-ICL by a few percentage points.\n\n   | Relation Type | 1     | 2        | 3        | 4     | 5     | 6         | 7     | 8         | 9         | 10       |\n   | ------------- | ----- | -------- | -------- | ----- | ----- | --------- | ----- | --------- | --------- | -------- |\n   | Naive-ICL     | 98.71 | 95.71    | 88.82    | 88.19 | 79.33 | 75.54     | 74.07 | 70.71     | 67.69     | 65.46    |\n   | CoT-ER        | 97.84 | 95.95    | 89.29    | 87.3  | 71.32 | 93.01     | 71.32 | 89.84     | 80.41     | 81.16    |\n   | Difference    | -0.87 | **0.24** | **0.47** | -0.89 | -8.01 | **17.47** | -2.75 | **19.13** | **12.72** | **15.7** |\n\n   (Definition of digital relation type: 1 inheritance type of, 2 ingredient of, 3 classified as, 4 gene found in organism, 5 is primary anatomic site of disease, 6 causative agent of, 7 biological process involves gene product, 8 gene plays role in process, 9 is normal tissue origin of disease, 10 occurs in)\n\n   \n\n   The table below shows the experimental results on **FewRel1.0**. We can observe that CoT-ER surpasses Naive-ICL 10 times, with 7 of them showing a relatively high improvement. However, in the 7th and 8th relations, CoT-ER still lags behind Naive-ICL by a few percentage points.\n\n   | Relation Type | 1    | 2    | 3       | 4     | 5       | 6        | 7     | 8     | 9        | 10      | 11    | 12       | 13       | 14        | 15       | 16       |\n   | ------------- | ---- | ---- | ------- | ----- | ------- | -------- | ----- | ----- | -------- | ------- | ----- | -------- | -------- | --------- | -------- | -------- |\n   | Naive-ICL     | 100  | 100  | 99.12   | 98.63 | 98.3    | 98.13    | 97.72 | 96.64 | 96.57    | 94.15   | 92.94 | 91.01    | 85.24    | 84.63     | 83.71    | 82.47    |\n   | CoT-ER        | 100  | 100  | 99.52   | 98.07 | 100     | 98.65    | 95.51 | 89.47 | 100      | 95.25   | 91.72 | 95.66    | 89.31    | 95.98     | 86.95    | 91.39    |\n   | Difference    | 0    | 0    | **0.4** | -0.56 | **1.7** | **0.52** | -2.21 | -7.17 | **3.43** | **1.1** | -1.22 | **4.65** | **4.07** | **11.35** | **3.24** | **8.92** |\n\n   (Definition of digital relation type:1 voice type, 2 position played on team/speciality, 3 original language of film or TV show, 4 constellation, 5 military rank, 6 competition class, 7 member of, 8 spouse, 9 located in or next to body of water, 10 follows, 11 crosses, 12 sport, 13 main subject, 14 child, 15 part of, 16 mother)\n\n   \n\n   **(2) Qualitative Analysis** \n\n   In summary, CoT-ER shows significant improvements in scenarios when Naive-ICL performs poorly. Here, we present a few cases to illustrate the reasoning process of CoT-ER in some representative scenarios. More details will be added to the revised paper. We will showcase two query instances with the digital relation types 8 (the highest difference) and 5 (the lowest difference) in the FewRel2.0 above.\n\n   **(i) Positive Example**\n\n      - Context:\n\n        In the hippocampus, transcriptional upregulation was observed in nogo-a(one day post-injury), mag and pirb at seven days post-injury.\n\n      - Reasoning process of CoT-ER:\n\n        It can be inferred from the context that:  \n\n        - **Entity \"transcriptional\" refers to the process of transcribing DNA into RNA**, which refers to the entity of biological process in the context.\n        - **Entity \"mag\" is the name of a gene**, which refers to the entity of gene in the context.\n        - According to the context, \"transcriptional upregulation was observed in nogo-a (one day post-injury), mag and pirb at seven days post-injury\" indicates that \"mag\" plays role in \"transcriptional\".\n\n        So, the relation between \"transcriptional\" and \"mag\" is **\"gene plays role in process\"**.\n\n   In this case, CoT-ER can not only recognize what \"transcriptional\" and \"mag\" mean in the context but also have concept-level knowledge. And the final prediction is correct. \n\n   \n\n   **(ii) Negative Example**\n\n      - Context:\n\n        Loss of hair in the other areas of the skin is present in the majority of cases.\n\n      - Reasoning process of CoT-ER:\n\n        It can be inferred from the context that:  \n\n        - **Entity \"Loss of hair\" is the name of a condition**, which refers to the entity of condition in the context.\n\n        - **Entity \"skin\" is the name of a body organ**, which refers to the entity of physical location of body part in the context.\n\n        - According to the context, \"loss of hair in the other areas of the skin\" indicates that \"loss of hair\" occurs in \"skin\".\n\n        So, the relation between \"loss of hair\" and \"skin\" is `\"occurs in\"`.\n\n   In this case, CoT-ER effectively recognizes the entity and arrives at a semantically accurate conclusion (\"loss of hair\" occurs in \"skin\"). However, **the final prediction is incorrect**, as the predicted relation label is \"occurs in\" while the ground truth label is \"is primary anatomic site of disease\".\n\n   The possible reason is that the label \"occurs in\" in this dataset means \"a condition occurs in a period of the lifetime\". This particular label should be matched with entity pairs like \"condition or disease\" and \"a period of a person's lifetime (such as congenital)\". This issue indicates that the relation description would be a key component in such methods, but it's not included in the FewRel2.0 dataset. \n\n   \n\n3. **Comment**: Question B: How is the number of demonstrations selected from the entire few-shot samples determined?\n\n   **Reply**: Sorry for the unclear description. We will clarify this point in the final paper. An N-way K-shot meta-task contains $N\\times K$ support instances in total, while the **text-davinci-003** which has a context window of 4097 tokens could not hold all support instances. Intuitively, we aim to **provide as much information as possible for LLM**, so we follow the principle of filling the context window to select the number of demonstrations as much as possible.\n\n   \n\n4. **Comment**: The number of few-shot instances selected for reasoning vary a lot across the different N-way K-shot setups (5,1 0, 13, 15, 25, 40), and it is unclear how the selected numbers contribute towards the final performance.\n\n   **Reply**: There are two main reasons for this. First, the support set instances vary for each meta-task ($N\\times K$ instances for each N-way K-shot task). Second, the number of tokens in each instance varies depending on the method employed (~300 in CoT-ER and ~80 in Naive-ICL), but they are all determined by the intuition mentioned above.\n\n   | #few-shot instances | 5     | 6     | 7     | 8     | 9     | 10    | 11    | 12    | 13    |\n   | ------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n   | CoT-ER              | 88.40 | 88.80 | 90.80 | 91.60 | 91.20 | 89.20 | 89.20 | 91.80 | 94.20 |\n   | Naive-ICL           | 73.40 | 78.00 | 78.80 | 82.90 | 81.40 | 82.39 | 81.20 | 82.79 | 84.20 |\n\n   | #few-shot instances | 5     | 7     | 9     | 11    | 13    | 15    | 17   | 19    | 21    | 23    | 25    |\n   | ------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ---- | ----- | ----- | ----- | ----- |\n   | Naive-ICL           | 73.40 | 78.80 | 81.40 | 81.20 | 84.20 | 87.60 | 89.0 | 90.40 | 90.80 | 90.60 | 89.80 |\n\n   We can observe that CoT-ER can effectively leverage the information from the provided instances and still perform well with a smaller number of instances. However, the performance of Naive-ICL drops rapidly as the number of instances decreases. Demonstrating that CoT-ER has a more stable performance compared to Naive-ICL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A well-thought out and implemented approach that combines various ideas from prompting LLMs to for effective few-shot relation extraction.\n- Explanations for the setups involved in alternative prompting approaches few-shot  relation extraction.", "reasons_to_reject": "- No error analysis - The naive in context learning  (Naive-ICL) baseline results are quite competitive with the best results from CoT-ER, the proposed approach. Some error analysis may help justify CoT-ER as an approach over the baseline.\n- Unclear selection of few-shot subset: $M$, the number of few-shot instances selected for reasoning vary a lot across the different N-way K-shot setups (5,1 0, 13, 15, 25, 40), and it is unclear how the selected numbers contribute towards the final performance.\n- Evaluation on a sampled subset of the full data: Only 100 samples are used from each dataset for evaluation. Thus, the results may not hold up when evaluated on the full test datasets.", "questions_to_authors": "**Question A:**  In what scenarios does CoT-ER provide an improvement over Naive-ICL?\n\n**Question B:** How is the number of demonstrations selected from the entire few-shot samples determined?", "paper_topic_and_main_contributions": "In this paper, the authors propose an approach to effectively prompt LLMs for few-shot relation extraction. The approach - CoT-ER, utilizes in-context learning and chain-of-thought prompting techniques to create a three step process for relation extraction. First, a reasoning module is used to generate/manually extract concepts related to the head, tail, and relation, along with the relevant text (evidence) of the samples present in the few-shot samples support set. In the second step, a subset of the few-shot demonstrations are selected, using kNN of embeddings from the query sample. This step ensures that the selected subset do not exceed the token length limit of the LLM. The final step involves inferring the relation label for the query sample by concatenating the task description and the selected few-shot demonstrations with the query text. Experiments demonstrate that CoT-ER improves upon various baselines over a subset of test samples chosen from the FewRel 1.0 and FewRel 2.0 datasets. Further, ablations show the importance of the first two steps, and a case study of a single sample demonstrates the different steps involved in the CoT-ER approach.\n\n**Contributions:**\n\n- An new approach for effectively prompting  LLMs for the few-shot relation extraction task providing competitive performance against fully fine tuned baselines.\n- A comprehensive comparison of fully-trained and alternative prompting approaches for few-shot relation extraction.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yLA01RxBlD", "reviewer": "Reviewer_josS", "summary": "", "questions": "A: Could you provide more details about the meaning of the line 279-382, which states \"Due to the high cost of running the CoT-ER on the GPT-3 API, we evaluate all LLM-relevant methods by sampling 100 N-way K-shot tasks from each dataset\"?\n\nB: How significant are the improvements observed in Tab1 and Tab2 in the test results?\n\nC: It would be intriguing to investigate the types of questions that benefit from the reasoning process.\n\nD: Why were only the validation sets selected for Tab 1 and Tab 2? Line 346 - 355 mentions the difference in relation label descriptions between FewRel1.0 and FewRel2.0. Are these relation descriptions used in the prompts? If not, why including the test set would cause unfair comparision?\n\nE: Could you explain in detail how the Verbalize() function in Figure 3 operates?\n\nF: According to Figure 3, the instruction states that \"the relation to choose must be in these N possible relations.\" Does the model always follow this instruction and select a relation from the given labels?\n\nG: Have you conducted any error analysis on the CoT-ER? Considering the high performance of CoT-ER, it would be interesting to identify the types of cases that still cannot be solved.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We will explain your concerns point by point.\n\n1. **Comment**: Could you provide more details about the meaning of the line 279-382, which states \"Due to the high cost of running the CoT-ER on the GPT-3 API, we evaluate all LLM-relevant methods by sampling 100 N-way K-shot tasks from each dataset\"?\n\n   **Reply**: First, due to limitations on the *text-davinci-003* API usage (such as access rate limit) and the higher cost of access ($0.02/1K tokens), we had to reduce the sampling number. Second, we have evaluated the CoT-ER and Naive-ICL **with 8 random seeds** (i.e., 8 independent sampling for the meta-task) in order to address this problem and will add these results to the final paper. The Table below shows experimental results with **mean ± standard deviation** on FewRel2.0. We can observe that **CoT-ER consistently outperforms Naive-ICL in all N-way K-shot settings with a lower standard deviation**.\n\n   | Method       | 5-Way 1-Shot | 5-Way 5-Shot | 10-Way 1-Shot | 10-Way 5-Shot |\n   | ------------ | ------------ | ------------ | ------------- | ------------- |\n   | Naive-ICL    | 80.75±1.7    | 88.19±1.56   | 68.21±1.27    | 80.92±1.05    |\n   | CoT-ER(ours) | 85.64±1.28   | 92.99±1.43   | 77.61±0.87    | 86.18±0.88    |\n\n   \n\n2. **Comment**: How significant are the improvements observed in Tab1 and Tab2 in the test results?\n   Why were only the validation sets selected for Tab 1 and Tab 2? Line 346 - 355 mentions the difference in relation label descriptions between FewRel1.0 and FewRel2.0. Are these relation descriptions used in the prompts? If not, why including the test set would cause unfair comparision?\n\n   **Reply**: Both FewRel1.0 and FewRel2.0 have released their training and validation sets for public use. However,  the downloaded test set from the benchmark lacks semantic label names for each instance (such as \"sport\", \"member of\"). Instead, it employs class label IDs like 0, 1, 2, 3, 4. It means that the prompting method **cannot be deployed on these datasets**. More details can be found on the official FewRel Github repository. \n\n   FewRel1.0 provides a description of each relation (such as \"sport: sport in which the subject participates or belongs to\") in a separate file, while FewRel2.0 does not. This has an impact on the **Verbalize() function** mentioned in the paper, and we will talk about it next. However, these descriptions are not used in the prompts. \n\n   The term \"fairness\" in the paper refers to the exact same experimental settings of a comparison between our method and baseline methods (including fully supervised methods). However, since our method cannot be experimented with on the test set, we primarily contrast its performance with baseline methods on the validation set with the same settings. The performance on the test set is only shown as a reference.\n\n   \n\n3. **Comment**: Could you explain in detail how the Verbalize() function in Figure 3 operates?\n\n   **Reply**:  We apologize for the unclear description and will update our paper accordingly.\n   To ensure the semantic coherence of the reasoning process, we devised this label verbalization technique by integrating both entities and the semantic relation label into a coherent expression. Give a simple example, if the entity \"Demetrius Rhaney\" and \"American football\" holds a relation label \"sport\", then the output of **Verbalize(\"Demetrius Rhaney\", \"sport\", \"American football\")** is **\"Demetrius Rhaney participates or belongs to American football\"**.\n\n   \n\n4. **Comment**: According to Figure 3, the instruction states that \"the relation to choose must be in these N possible relations.\" Does the model always follow this instruction and select a relation from the given labels?\n\n   **Reply**: Yes, due to the uncertainty of the model's output, we have to prevent it from outputting a relation that is not in the meta-task. In addition, the metrics of FewRel1.0 and FewRel2.0 also require this.\n\n   \n\n5. **Comment**: It would be intriguing to investigate the types of questions that benefit from the reasoning process. Have you conducted any error analysis on the CoT-ER? Considering the high performance of CoT-ER, it would be interesting to identify the types of cases that still cannot be solved.\n\n   **Reply**: Here, we address the error anlaysis issue with a **quantitative** and **qualitative** analysis. We will add more analysis and discussion in the final paper.\n\n   **(1) Quantitative Analysis** \n\n   Taking 5-Way 1-Shot as an example, the table below shows the accuracy of these two methods on different relations of the FewRel. Note that different numbers represent different relation types, and their order is based on the performance ranking of Naive-ICL. These accuracies are the average of multiple test runs conducted with 7 different random seeds. \n\n   The table below shows the experimental results on **FewRel2.0**. We can observe that CoT-ER surpasses Naive-ICL 6 times, with 4 of them showing a significant improvement of over 10%. However, in the 5th and 7th relations, CoT-ER still lags behind Naive-ICL by a few percentage points.\n\n   | Relation Type | 1     | 2        | 3        | 4     | 5     | 6         | 7     | 8         | 9         | 10       |\n   | ------------- | ----- | -------- | -------- | ----- | ----- | --------- | ----- | --------- | --------- | -------- |\n   | Naive-ICL     | 98.71 | 95.71    | 88.82    | 88.19 | 79.33 | 75.54     | 74.07 | 70.71     | 67.69     | 65.46    |\n   | CoT-ER        | 97.84 | 95.95    | 89.29    | 87.3  | 71.32 | 93.01     | 71.32 | 89.84     | 80.41     | 81.16    |\n   | Difference    | -0.87 | **0.24** | **0.47** | -0.89 | -8.01 | **17.47** | -2.75 | **19.13** | **12.72** | **15.7** |\n\n   (Definition of digital relation type: 1 inheritance type of, 2 ingredient of, 3 classified as, 4 gene found in organism, 5 is primary anatomic site of disease, 6 causative agent of, 7 biological process involves gene product, 8 gene plays role in process, 9 is normal tissue origin of disease, 10 occurs in)\n\n   \n\n   The table below shows the experimental results on **FewRel1.0**. We can observe that CoT-ER surpasses Naive-ICL 10 times, with 7 of them showing a relatively high improvement. However, in the 7th and 8th relations, CoT-ER still lags behind Naive-ICL by a few percentage points.\n\n   | Relation Type | 1    | 2    | 3       | 4     | 5       | 6        | 7     | 8     | 9        | 10      | 11    | 12       | 13       | 14        | 15       | 16       |\n   | ------------- | ---- | ---- | ------- | ----- | ------- | -------- | ----- | ----- | -------- | ------- | ----- | -------- | -------- | --------- | -------- | -------- |\n   | Naive-ICL     | 100  | 100  | 99.12   | 98.63 | 98.3    | 98.13    | 97.72 | 96.64 | 96.57    | 94.15   | 92.94 | 91.01    | 85.24    | 84.63     | 83.71    | 82.47    |\n   | CoT-ER        | 100  | 100  | 99.52   | 98.07 | 100     | 98.65    | 95.51 | 89.47 | 100      | 95.25   | 91.72 | 95.66    | 89.31    | 95.98     | 86.95    | 91.39    |\n   | Difference    | 0    | 0    | **0.4** | -0.56 | **1.7** | **0.52** | -2.21 | -7.17 | **3.43** | **1.1** | -1.22 | **4.65** | **4.07** | **11.35** | **3.24** | **8.92** |\n\n   (Definition of digital relation type: 1 voice type, 2 position played on team/speciality, 3 original language of film or TV show, 4 constellation, 5 military rank, 6 competition class, 7 member of, 8 spouse, 9 located in or next to body of water, 10 follows, 11 crosses, 12 sport, 13 main subject, 14 child, 15 part of, 16 mother)\n\n   **(2) Qualitative Analysis** \n\n   In summary, CoT-ER shows significant improvements in scenarios when Naive-ICL performs poorly. Here, we present a few cases to illustrate the reasoning process of CoT-ER in some representative scenarios. More details will be added to the revised paper. We will showcase two query instances with the digital relation types 8 (the highest difference) and 5 (the lowest difference) in the FewRel2.0 above.\n\n   **(i) Positive Example**\n\n   - Context:\n\n     In the hippocampus, transcriptional upregulation was observed in nogo-a(one day post-injury), mag and pirb at seven days post-injury.\n\n   - Reasoning process of CoT-ER:\n\n     It can be inferred from the context that:  \n\n     - **Entity \"transcriptional\" refers to the process of transcribing DNA into RNA**, which refers to the entity of biological process in the context.\n     - **Entity \"mag\" is the name of a gene**, which refers to the entity of gene in the context.\n     - According to the context, \"transcriptional upregulation was observed in nogo-a (one day post-injury), mag and pirb at seven days post-injury\" indicates that \"mag\" plays role in \"transcriptional\".\n\n     So, the relation between \"transcriptional\" and \"mag\" is **\"gene plays role in process\"**.\n\n   In this case, CoT-ER can not only recognize what \"transcriptional\" and \"mag\" mean in the context but also have concept-level knowledge. And the final prediction is **correct**. \n\n   **(ii) Negative Example**\n\n   - Context:\n\n     Loss of hair in the other areas of the skin is present in the majority of cases.\n\n   - Reasoning process of CoT-ER:\n\n     It can be inferred from the context that:  \n\n     - **Entity \"Loss of hair\" is the name of a condition**, which refers to the entity of condition in the context.\n\n     - **Entity \"skin\" is the name of a body organ**, which refers to the entity of physical location of body part in the context.\n\n     - According to the context, \"loss of hair in the other areas of the skin\" indicates that \"loss of hair\" occurs in \"skin\".\n\n     So, the relation between \"loss of hair\" and \"skin\" is `\"occurs in\"`.\n\n   In this case, CoT-ER can also recognize the entity well, and the conclusion is semantically right (\"loss of hair\" occurs in \"skin\"). However, **the final prediction is incorrect**, as the predicted relation label is \"occurs in\" while the ground truth label is \"is primary anatomic site of disease\". \n\n   The possible reason is that the label \"occurs in\" in this dataset means \"a condition occurs in a period of the lifetime\". This particular label should be matched with entity pairs like \"condition or disease\" and \"a period of a person's lifetime (such as congenital)\". This issue indicates that the relation description would be a key component in such methods, but it's not included in the FewRel2.0 dataset.\n\n   \n\n6. **Comment**: Moreover, the similarity retrieval process employed in this paper not only incurs higher computational costs but also prolongs the prompting process, making it non-end-to-end. This limitation should be addressed to ensure a more efficient and streamlined approach.\n\n   **Reply**: \n\n   The computational costs of the retrieval module are not as high as commonly expected, because the scope of retrieval is limited to the support set within the meta-task. Additionally, the model used for computing semantic embedding vectors can be easily converted to other models that can be deployed locally, such as SimCSE (Gao, et al., EMNLP 2021). Consequently, **the additional computational cost brought by the retrieval module is at a relatively lower level** compared to getting responses from the LLM through network-accessed APIs.\n\n   We introduced this mainly considering **the inability to incorporate the entire support set instances into the prompt**, due to the limited context window of text-davinci-003 (4097 tokens). A relevant former study suggests that selecting few-shot demonstrations based on similarity yields improvements in in-context learning (Liu et al., DeeLIO 2022), while these relevant studies in order to select semantically close instances from a large training dataset, which is not a true few-shot learning (Perez et al., NIPS 2021). Many recent efforts are devoted to expanding the length of the context window. We believe that this issue will be addressed correspondingly in the near future.\n\n   Thanks again for the comments, we will provide a clearer elaboration on this in the revised paper.\n\n   References:\n\n   - What Makes Good In-Context Examples for GPT-3? (Liu et al., DeeLIO 2022)\n\n   - True few-shot learning with language models (Perez et al., NIPS 2021)\n\n   - SimCSE: Simple Contrastive Learning of Sentence Embeddings (Gao et al., EMNLP 2021)\n\n     \n\n7. **Comment**: Additionally, the absence of significance tests to determine the statistical significance of the improvements compared to existing approaches is a limitation. This raises uncertainty regarding the significance of the reported improvements.\n\n   **Reply**: Thanks a lot for your advice, We have conducted the experiment and more details will be updated to the final paper.\n\n   |      | 5-Way 1-Shot   | 5-Way 5-Shot   | 10-Way 1-Shot  | 10-Way 5-Shot  |\n   | ---- | -------------- | -------------- | -------------- | -------------- |\n   | 1    | $1.1×10^{-3}$  | $3.74×10^{-2}$ | $1.77×10^{-8}$ | $3.66×10^{-3}$ |\n   | 2    | $1.1×10^{-2}$  | $8.04×10^{-3}$ | $1.41×10^{-4}$ | $5.08×10^{-5}$ |\n   | 3    | $4.00×10^{-2}$ | $1.30×10^{-2}$ | $1.55×10^{-8}$ | $1.36×10^{-2}$ |\n   | 4    | $5.26×10^{-3}$ | $8.67×10^{-3}$ | $8.75×10^{-6}$ | $2.68×10^{-5}$ |\n   | 5    | $5.99×10^{-3}$ | $2.35×10^{-3}$ | $8.25×10^{-9}$ | $5.12×10^{-3}$ |\n\n   This table displays the **p-value** between Naivel-ICL and CoT-ER on FewRel2.0 across 5 different random seeds, which is obtained by sampling for each meta-test in all meta-tasks. **All the p-values are below 0.05, indicating significant differences between the two**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper contributes to the field of relation extraction by providing a baseline for in-context learning and an entity-aware prompting appraoch. It empirically demonstrates that incorporating entity information into prompts leads to further performance improvement in few-shot relation extraction.\n\nWhile the effectiveness of entity information in supervised relation learning has been previously shown, this paper focuses on incorporating entity type and explicit information from large-language models (LLMs) in prompting. This finding shows the importance of entity information in facilitating reasoning and strengthens the case for its inclusion in LLM-based approaches for training-free relation extraction.", "reasons_to_reject": "The primary weaknesses of this paper include incomplete presentation of results in Tab1 and Tab2. For a fair comparison, the paper should provide results for both the validation and test sets. Additionally, the absence of significance tests to determine the statistical significance of the improvements compared to existing approaches is a limitation. This raises uncertainty regarding the significance of the reported improvements.\n\nMoreover, the similarity retrieval process employed in this paper not only incurs higher computational costs but also prolongs the prompting process, making it non-end-to-end. This limitation should be addressed to ensure a more efficient and streamlined approach.", "questions_to_authors": "A: Could you provide more details about the meaning of the line 279-382, which states \"Due to the high cost of running the CoT-ER on the GPT-3 API, we evaluate all LLM-relevant methods by sampling 100 N-way K-shot tasks from each dataset\"?\n\nB: How significant are the improvements observed in Tab1 and Tab2 in the test results?\n\nC: It would be intriguing to investigate the types of questions that benefit from the reasoning process.\n\nD: Why were only the validation sets selected for Tab 1 and Tab 2? Line 346 - 355 mentions the difference in relation label descriptions between FewRel1.0 and FewRel2.0. Are these relation descriptions used in the prompts? If not, why including the test set would cause unfair comparision?\n\nE: Could you explain in detail how the Verbalize() function in Figure 3 operates?\n\nF: According to Figure 3, the instruction states that \"the relation to choose must be in these N possible relations.\" Does the model always follow this instruction and select a relation from the given labels?\n\nG: Have you conducted any error analysis on the CoT-ER? Considering the high performance of CoT-ER, it would be interesting to identify the types of cases that still cannot be solved.", "paper_topic_and_main_contributions": "This paper focuses on the issue of few-shot relation extraction, which involves identifying relations in a query set based on only a few training examples. While previous research has mainly focused on supervised learning methods, this paper aims to address the problem using training-free approaches that utilize large language models. \n\nThe main contributions of this paper are twofold. Firstly, it explores the potential of in-context learning in few-shot relation extraction, bridging the existing gap. Secondly, it proposes a 3-step typed chain-of-thought prompts, where the first two steps involve eliciting entity type information and then utilizing this information along with the context to identify relations. The experimental results demonstrate that the inclusion of entity type information leads to improvements in the FewRel 1.0 and FewRel 2.0 validation sets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. A space should be placed between the content and the citations, e.g., “learning(Brown et al., 2020)” should be corrected to “learning (Brown et al., 2020)”. There are many similar instances, please double-check."}}, {"review_id": "xTvILJ1dqB", "reviewer": "Reviewer_6ESd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. We will explain your concerns point by point.\n\n1. **Comment**: The paper does not provide a clear explanation of how the prompts used in CoT-ER are selected. The authors should provide more details on how different prompts were evaluated and how they affected the experimental results. It would be better to give a more thorough analysis of the impact of different prompts on the performance of the proposed method.\n\n   The paper does not provide a detailed analysis of the impact of the k value on the performance of the proposed method. k-NN search is a key component, a more thorough analysis of the impact of the k value on the experimental results is needed.\n\n   **Reply**: There are two perspectives to consider different prompts. **One is the wording, another is the in-context examples (the k value)**. For the wording of the prompt, we adhere to the principle of **extreme simplicity** in crafting prompts for all methods in the paper. It means we do not introduce additional sentences with task-irrelevant descriptions, aiming to avoid prompt engineering.\n\n   For the selection of in-context examples (the k value), we evaluate **the effect of the selected number** of instances in prompt and have conducted an ablation experiment. We show the scenario of 5-Way 5-Shot on FewRel2.0 here, where the CoT-ER method can hold up to 13 instances in prompt while Naive-ICL can hold all the support set (25) instances in prompt. More details will be added to the revised paper. \n\n   | #instances of prompt (k value) | 5     | 6     | 7     | 8     | 9     | 10    | 11    | 12    | 13    |\n   | ------------------------------ | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | ----- |\n   | CoT-ER                         | 88.40 | 88.80 | 90.80 | 91.60 | 91.20 | 89.20 | 89.20 | 91.80 | 94.20 |\n   | Naive-ICL                      | 73.40 | 78.00 | 78.80 | 82.90 | 81.40 | 82.39 | 81.20 | 82.79 | 84.20 |\n\n   | #instances  of prompt (k value) | 5     | 7     | 9     | 11    | 13    | 15    | 17   | 19    | 21    | 23    | 25    |\n   | ------------------------------- | ----- | ----- | ----- | ----- | ----- | ----- | ---- | ----- | ----- | ----- | ----- |\n   | Naive-ICL                       | 73.40 | 78.80 | 81.40 | 81.20 | 84.20 | 87.60 | 89.0 | 90.40 | 90.80 | 90.60 | 89.80 |\n\n   We can observe that CoT-ER can effectively leverage the information from the provided instances and still perform well with a smaller number of instances. However, the performance of Naive-ICL drops rapidly as the number of instances decreases. Demonstrating CoT-ER has a more stable performance compared to Naive-ICL.\n\n   \n\n2. **Comment**: Given that large language models have the potential to achieve high performance on relation extraction tasks, it would be valuable for the authors to conduct a thorough error analysis to identify the reasons for any mistakes made by the model. This analysis could provide insights into the limitations of the proposed approach and suggest directions for future research.\n\n   **Reply**: Here, we address the error anlaysis issue with a **quantitative** and **qualitative** analysis. We will add more analysis and discussion in the final paper.\n\n   **(1) Quantitative Analysis** \n\n   Taking 5-Way 1-Shot as an example, the table below shows the accuracy of these two methods on different relations of the FewRel. Note that different numbers represent different relation types, and their order is based on the performance ranking of Naive-ICL. These accuracies are the average of multiple test runs conducted with 7 different random seeds.\n\n   The table below shows the experimental results on **FewRel2.0**. We can observe that CoT-ER surpasses Naive-ICL 6 times, with 4 of them showing a significant improvement of over 10%. However, in the 5th and 7th relations, CoT-ER still lags behind Naive-ICL by a few percentage points.\n\n   | Relation Type | 1     | 2        | 3        | 4     | 5     | 6         | 7     | 8         | 9         | 10       |\n   | ------------- | ----- | -------- | -------- | ----- | ----- | --------- | ----- | --------- | --------- | -------- |\n   | Naive-ICL     | 98.71 | 95.71    | 88.82    | 88.19 | 79.33 | 75.54     | 74.07 | 70.71     | 67.69     | 65.46    |\n   | CoT-ER        | 97.84 | 95.95    | 89.29    | 87.3  | 71.32 | 93.01     | 71.32 | 89.84     | 80.41     | 81.16    |\n   | Difference    | -0.87 | **0.24** | **0.47** | -0.89 | -8.01 | **17.47** | -2.75 | **19.13** | **12.72** | **15.7** |\n\n   (Definition of digital relation type: 1 inheritance type of, 2 ingredient of, 3 classified as, 4 gene found in organism, 5 is primary anatomic site of disease, 6 causative agent of, 7 biological process involves gene product, 8 gene plays role in process, 9 is normal tissue origin of disease, 10 occurs in)\n\n   \n\n   The table below shows the experimental results on **FewRel1.0**. We can observe that CoT-ER surpasses Naive-ICL 10 times, with 7 of them showing a relatively high improvement. However, in the 7th and 8th relations, CoT-ER still lags behind Naive-ICL by a few percentage points.\n\n   | Relation Type | 1    | 2    | 3       | 4     | 5       | 6        | 7     | 8     | 9        | 10      | 11    | 12       | 13       | 14        | 15       | 16       |\n   | ------------- | ---- | ---- | ------- | ----- | ------- | -------- | ----- | ----- | -------- | ------- | ----- | -------- | -------- | --------- | -------- | -------- |\n   | Naive-ICL     | 100  | 100  | 99.12   | 98.63 | 98.3    | 98.13    | 97.72 | 96.64 | 96.57    | 94.15   | 92.94 | 91.01    | 85.24    | 84.63     | 83.71    | 82.47    |\n   | CoT-ER        | 100  | 100  | 99.52   | 98.07 | 100     | 98.65    | 95.51 | 89.47 | 100      | 95.25   | 91.72 | 95.66    | 89.31    | 95.98     | 86.95    | 91.39    |\n   | Difference    | 0    | 0    | **0.4** | -0.56 | **1.7** | **0.52** | -2.21 | -7.17 | **3.43** | **1.1** | -1.22 | **4.65** | **4.07** | **11.35** | **3.24** | **8.92** |\n\n   (Definition of digital relation type:1 voice type, 2 position played on team/speciality, 3 original language of film or TV show, 4 constellation, 5 military rank, 6 competition class, 7 member of, 8 spouse, 9 located in or next to body of water, 10 follows, 11 crosses, 12 sport, 13 main subject, 14 child, 15 part of, 16 mother)\n\n**(2) Qualitative Analysis** \n\nIn summary, CoT-ER shows significant improvements in scenarios when Naive-ICL performs poorly. Here, we present a few cases to illustrate the reasoning process of CoT-ER in some representative scenarios. More details will be added to the revised paper. We will showcase two query instances with the digital relation types 8 (the highest difference) and 5 (the lowest difference) in the FewRel2.0 above.\n\n**(i) Positive Example**\n\n- Context:\n\n  In the hippocampus, transcriptional upregulation was observed in nogo-a(one day post-injury), mag and pirb at seven days post-injury.\n\n- Reasoning process of CoT-ER:\n\n  It can be inferred from the context that:  \n\n  - **Entity \"transcriptional\" refers to the process of transcribing DNA into RNA**, which refers to the entity of biological process in the context.\n  - **Entity \"mag\" is the name of a gene**, which refers to the entity of gene in the context.\n  - According to the context, \"transcriptional upregulation was observed in nogo-a (one day post-injury), mag and pirb at seven days post-injury\" indicates that \"mag\" plays role in \"transcriptional\".\n\n  So, the relation between \"transcriptional\" and \"mag\" is **\"gene plays role in process\"**.\n\nIn this case, CoT-ER can not only recognize what \"transcriptional\" and \"mag\" mean in the context but also have concept-level knowledge. And the final prediction is correct. \n\n**(ii) Negative Example**\n\n- Context:\n\n  Loss of hair in the other areas of the skin is present in the majority of cases.\n\n- Reasoning process of CoT-ER:\n\n  It can be inferred from the context that:  \n\n  - **Entity \"Loss of hair\" is the name of a condition**, which refers to the entity of condition in the context.\n\n  - **Entity \"skin\" is the name of a body organ**, which refers to the entity of physical location of body part in the context.\n\n  - According to the context, \"loss of hair in the other areas of the skin\" indicates that \"loss of hair\" occurs in \"skin\".\n\n  So, the relation between \"loss of hair\" and \"skin\" is `\"occurs in\"`.\n\nIn this case, CoT-ER can also recognize the entity well, and the conclusion is semantically right (\"loss of hair\" occurs in \"skin\"). However, **the final prediction is incorrect**, as the predicted relation label is \"occurs in\" while the ground truth label is \"is primary anatomic site of disease\".\n\nThe possible reason is that the label \"occurs in\" in this dataset means \"a condition occurs in a period of the lifetime\". This particular label should be matched with entity pairs like \"condition or disease\" and \"a period of a person's lifetime (such as congenital)\". This issue indicates that the relation description would be a key component in such methods, but it's not included in the FewRel2.0 dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tThe paper proposes a novel CoT-ER method to incorporate task-specific and concept-level knowledge into chain-of-thought prompting for relation extraction.\n\n2.\tThe paper is well-organized and clearly presents the proposed framework.\n\n3.\tThe experimental results are promising and show significant improvement over the compared models.\n", "reasons_to_reject": "1.\tThe paper does not provide a clear explanation of how the prompts used in CoT-ER are selected. The authors should provide more details on how different prompts were evaluated and how they affected the experimental results. It would be better to give a more thorough analysis of the impact of different prompts on the performance of the proposed method.\n2.\tGiven that large language models have the potential to achieve high performance on relation extraction tasks, it would be valuable for the authors to conduct a thorough error analysis to identify the reasons for any mistakes made by the model. This analysis could provide insights into the limitations of the proposed approach and suggest directions for future research.\n3.\tThe paper does not provide a detailed analysis of the impact of the k value on the performance of the proposed method. k-NN search is a key component, a more thorough analysis of the impact of the k value on the experimental results is needed.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a novel approach for few-shot relation extraction called CoT-ER, which utilizes large language models and explicit evidence reasoning to improve performance. The proposed method achieves competitive performance compared to fully-supervised approaches, without the need for training data. Experimental results demonstrate the effectiveness of the proposed approach on the FewRel1.0 and FewRel2.0 datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper tackles the challenge of few-shot relation extraction by employing training-free methods, in contrast to traditional supervised learning. It achieves two key contributions: (1) it investigates the effectiveness of in-context learning for this problem; (2) it introduces a 3-step approach using typed chain-of-thought prompts. This method starts by extracting entity type information and then uses it in conjunction with the context to identify relations. Experimental results show that including entity type information enhances performance on the FewRel 1.0 and FewRel 2.0 validation sets. Reviewers unanimously agreed on the good soundness of the work (one reviewer also increased soundness score after rebuttal), and also shared several areas of improvement. First, there's a lack of error analysis, which could help justify the proposed CoT-ER approach over the baseline. Second, the selection of few-shot instances for reasoning varies widely across setups, and its contribution to performance is unclear. Third, only 100 samples are used from each dataset for evaluation. Thus, the results may not hold up when evaluated on the full test datasets. Finally, the paper does not provide a detailed analysis of the impact of the k value on the performance of the proposed method. k-NN search is a key component, a more thorough analysis of the impact of the k value on the experimental results is needed.
Decision: Accept-Findings