Large Language Model Is Not a Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!
Authors: Yubo Ma
Yixin Cao
Yong Ching Hong
Aixin Sun
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Models
Information Extraction

Abstract: Large Language Models (LLMs) have made remarkable strides in various tasks. Whether LLMs are competitive few-shot solvers for information extraction (IE) tasks, however, remains an open problem. In this work, we aim to provide a thorough answer to this question. Through extensive experiments on nine datasets across four IE tasks, we demonstrate that current advanced LLMs consistently exhibit inferior performance, higher latency, and increased budget requirements compared to fine-tuned SLMs under most settings. Therefore, we conclude that LLMs are not effective few-shot information extractors in general. Nonetheless, we illustrate that with appropriate prompting strategies, LLMs can effectively complement SLMs and tackle challenging samples that SLMs struggle with. And moreover, we propose an adaptive filter-then-rerank paradigm to combine the strengths of LLMs and SLMs. In this paradigm, SLMs serve as filters and LLMs serve as rerankers. By prompting LLMs to rerank a small portion of difficult samples identified by SLMs, our preliminary system consistently achieves promising improvements ($2.4\%$ F1-gain on average) on various IE tasks, with an acceptable time and cost investment.

Large Language Model Is Not Good Few-shot Information Extractor, but a Good Reranker for Hard Samples!
Yubo Ma', Yixin Cao", YongChing Hong', Aixin Sun'
S-Lab, Nanyang Technological University Singapore Management University yubo@01@e. ntu. ntw.:du..ugge....
Abstract
LLMs, prompts a compelling question: Can LLMs be used to boost performance in few-shot information extraction (IE) tasks?
Large Language Models (LLMs) have made remarkable strides in various tasks. Whether LLMs competitive few-shot solvers for in- formation extraction (IE) tasks, however, re- mains an open problem. this work, we aim to provide thorough answer to this ques- tion. Through extensive experiments on nine datasets across four IE tasks, demonstrate that current advanced LLMs consistently ex- hibit inferior performance, higher latency, and increased budget requirements compared to fine-tuned SLMs under most settings. There- fore, we conclude that LLMs are not effec- tive few-shot information extractors in gen- eral Nometheless, we illustrate that with appropriate prompting strategies, LLMs can effectively complement SLMs and tackle chal- lenging samples that SLMs struggle with. And moreover, propose an adaptive filter-then- rerank paradigm to combine the strengths of LLMs and SLMs. In this paradigm, SLMs filters and LLMs serve rerankers. By prompting L.MSs to rerank a small portion difficult samples identified by SLMs, our pre- liminary system consistently achieves promis- ing improvements (2.4% Fl-gain on average) on various IE tasks, with an acceptable time and cost investment. Our code is available at /ttps://githie httt:://ggttuu...ommmmb222333333
To answer this question, we conduct an exten- sive empirical study to compare the performance between LLMs using in-context learning' 2 (ICL) and fine-tuned Small Language Models (SLMs). We fairly evaluate SLMs-based and LLMs-based methods across nine datasets spanning four com- mon IE tasks: (1) Named Entity Recognition, (2)
Relation Extraction, (3) Event Detection and (4)
Event Argument Extraction. For each dataset, we explored four to six settings to encompass typi- cal low-resource extents, from 1-shot to 20-shot or more. Given the potential sensitivity of LLMs' performance the prompt context, we meticu- lously considered variations instruction, demon- stration number and selection strategy, prompt for- mat, etc. Our study reveals that LLMs excel over SLMs only when annotations are extremely lim- ited, i.e., both label types² and the samples* per label are extremely scarce. With more (e.g.. hun- dreds of) samples, SLMs significantly outperform LLMs. Furthermore, LLMs incur greater inference latency and costs than fine-tuned SLMs. Hence, we claim that current LLMs are not good few-shot information extractors in general.
com//mayyoo2333//mLL.. IE.
We further investigate whether LLMs and SLMs exhibit different abilities to handle various types of samples. We categorize samples according to their difficulty measured by SLMs' confidence scores, and compare LLMs' and SLMs' results within each group. We find that LLMs are good at hard sam- ples, though bad easy samples. We posit that the knowledge and reasoning abilities in LLMs en- able them to handle hard samples (which are sim-
1 Introduction
Large Language Models (LLMs, Brown et al. 2020; Chowdhery et al. 2022; Touvron al. 2023) have shown remarkable abilities on various NLP applica- tions such factual question answering (Yu et al., 2023; Sun et al., 2023), arithmetic reasoning (Chen et al., 2022a; Qian et al., 2023) and logical rea- soning (Jung et al., 2022; Pan et al., 2023). Given the reasoning, memorization, instruction-follot and few-shot adaption capabilities emerging from
FAIl LLMs discussed in this paper not fine-tuned, and results for LLMs based in-context learning.
'Label types denote entity/nelaiioneeeee/nu types dif- ferent tasks. We use them interchangeably there-in-after.
AA more precise assertion is that current LLMs, with vanilla prompting setting and without Ie-specific fine-tuning. are not good few-shot information extractors in general.
'Samples refer to (i) demonstrations in ICI. of LLMss, or (ii) training samples for SLMs' fine-tuning.

---

ply beyond SLMs' capabilities) well. Nevertheless, LLMs demonstrate strong predisposition false- positive predictions on negative samples. Since most negative samples are easy samples (which could be solved readily by SLMs), the performance of LLMs on casy samples sometimes collapses and are usually much worse than fine-tuned SLMs.
2.2 Few-shot IE with ICL
Regarding few-shot IE tasks, recent studies inten- sively compare the performance between SLMs and LLMs but yield inconsistent conclusions. Some studies favor LLMs as competent few-shot extractors (Agrawal et al., 2022: Wang et al., 2023b; Li et al., 2023; Zhang et al., 2023a; Wadhwa et al., 2023), while others dispute this claim (Jimenez Gutierrez et al.. 2022: Qin et al.. 2023; Wei et al., 2023; Gao et al., 2023). This discrepancy leaves the question of whether LLMs perform competitively on few-shot IE tasks unre- solved, thus hindering the advances of this domain. We attribute such disagreement to the absence of an comprehensive and unified benchmark. Ex- isting studies usually vary tasks, datasets, and few-shot settings. Furthermore, some studies rely on overly simplistic datasets (Jimenez Gutierrez et al., 2022; Li et al., 2023) and may exaggerate the effectiveness of LLMs. Driven by these find- ings, research undertakes comprchensive ex- periments across four IE tasks, nine datasets with various schema complexities (from coarse-grained to fine-grate aeeeggaiiedd and low-resource settings.
Leveraging these findings, we pursue ap- proach to incorporate LLMs and SLMs within single system and combine their merits. To this end, we propose a novel filter-then-reranke framework. The basic idea is that SLMs serve as a filter and LLMs as a reranker. Specifically, SLMs initially predict and determine the difficulty of each sample. If the sample is hard one, we further pass the top-N most-likely candidate labels from SLMs to LLMs for reranking. Otherwise we view the predic- tion from SLMs as the final decision. By providing easy/hard samples with different solution strategies, our system utilizes each model's ss strengths to com- plement each other. Also, it reranks only small subset of samples and minimizes the extra latency and budgets for calling LLMs. With modest cost increase, our framework yields consistent FI im- provement, averaging 2.4% higher than previous methods on various few-shot IE tasks. To the best of our knowledge, this the first successful attempt to use LLMs to enhance few-shot IE tasks.
In addition to the empirical study, we develop an innovative filler-then-rerant paradigm to combine the strengths of both LLMs and SLMs. It utilizes prompting strategies akin to QA4RE (Zhang et al., 2023a), transforming IE tasks into multi-choice questions. However, our method stands apart by integrating SLMs and LLMs within single frame- work. This incorporation (1) enables our paradigm applicable to various IE tasks by providing candi- date spans in the text and (2) achieves promising performance under low-resource IE scenarios.
2 Related Work
2.1 LLMs for Information Extraction
Recent studies have increasingly explored Informa- tion Extraction (IE) tasks using LLMs. Drawing in- spiration from instruction tuning (Wei et al., 2022a), several methods (Wadhwa et al., 2023; Wang et al., 2023a; Lu et al., 2023) transform annotated sam- ples into instruction-answee tune LLMs, such FlanT5 (Chung et al., 2022), on them. Nonetheless, this method necessitates vast range of samples with diverse schemas and often yields suboptimal results in low-resource sce- narios. In the context of fhw-shot IE tasks, preva- lent strategies bifurcate into two main streams. The first approach perceives LLMs as efficient annota- tors (Ding et al., 2023; Josifoski et al., 2023). In these methods, they produce plethora of pscudo- labeled samples through LLMs and leverage the enhanced annotations train SLMs. Conversely, the latter approach employs L.MMs in inference us- ing the ICL paradigm, which is the focus of our subsequent discussion.
Large LMs v.s. Small LMs
pairs and then fine-
In this section, we compare the performance be- tween LLMs and SLMs to evaluate whether LLMs perform competitively.
3.1 Task, Dataset and Evaluation
We run experiments on nine widely-used datasets across four IE tasks. (1) Named Entity Recognition (NER): CONLLO3 (Tjong Kim Sang and De Meul- der, 2003), OntoNotes (Weischedel et al., 2013)
and FewNERD (Ding al., 2021). (2) Relation Extraction (RE): TACRED (Zhang al., 2017) and TACREV (Alt et al., 2020). (3) Event De- tection (ED): ACE05 (Doddington et al., 2004), MAVEN (Wang et al., 2020) and ERE (Song et al.,

---

Named Entity Recognition
Relation Extraction
Identify the entities then Locate coch enttty to words to sentence, The possible enttty types are butlding-hotel,
Identify the relation between the entities to the sestence. The possible relotion types are prrsticle, org:top_ srropparebers, [tstart]] Eden [sond]
Sentence: 1962 national Football League
hold on chicago, filincis.
Sentence: Internationa: Tatvan,
Maas that
Decceber 42h, 1951 the Sheraten
corpassion
(ryp: organisation- sportstagge, estity: 1982 National football Leogue), (type: suilding hotel, ontsty: Sheraton Hotel)
Founsation,
Agency Tasson, [nstart]
[hend] the Tatkan
Corps.
Sentence:
Critics noted Manual of Detection" combines including eystery fantasy.
Triple: (Subjs IKF, ong: TOAID, Relation: orgrnenber_of)
clements
Sentence: Puller, 17, has beee there before. Triple: (Subj: fuller,
Intities:
Relation:
No entctles Foond.
Persage)
Event Detection
Event Argument Extraction
Given sencence and on evest, you need to ideotify olc orgunents of this evet, and clossify role of each orgusent. Eash-strarpes
Idestify
The possible types Lifee .....
Sentence: Injured Bussian diglorats were arong victins caught in crossfire and friendly fire Sunday.
Trigger production company. Possible
Tramffe--onrrnipp seller, Artifatt,
Events: [type: Life. Infure, trägger word: Insured), (type: Conflict.Actacll triager vard: crossfire)
Arguments: (type: seller, argusens
(type: Aetifect, company) Sentence: Another appeal is penting In the Feterol Court. Event: bustice-dppeal, Triggerr_ appeal, Possible roles: Adudicator, Plaintiff, Place...
argurent:
Sentence: Irwei Kurds fear that Tarkish treaps could move deep enclave in northeen traq and have shirattned clashes.
Events:
Argunents:
(type: Rovenent, Transport, trisper_ word: move), (type: Confiict.Attone trigger_werd word: clashes)
(type:
Adjudicator,
orgument:
Federal Court)
Figure 1: Examples of prompts used. The green, blue and black parts in the top boxes represent the instruction, demonstration (demo) and test sentence the prompt respectively. The red parts represent the outputs from LLMs. We plot only example for convenience of visualization. The actual demo number is usually much larger than 1.
2015). (4) Event Argument Extraction (EAE): ACEOS, ERE and RAMS (Ebner et al., 2020). With label numbers ranging from to 168, we assess LLMs' performance under different schema com- plexities. See their details in Appendix A.1.
(5) UIE (Lu et al., 2022c) for all tasks. See their details in Appendix B.
3.3 Large Language Models
Detailed in Appendix C, we evaluate the ICL abil- ities of LLMs. Given labeled sentences D {(s, y:)} and a test sentence s, our goal to pre-
Few-shot Set We construct few-shot datasets from the original datasets above. For training and vali- dation set, we adopt K-shot sampling strategy, i.e. sampling K samples for each label type. See more details in Appendix A.2. For test set, we down- sample their original test sets to reduce the cost of LLMs. We randomly sample 500 sentences for RE tasks, and 250 sentences for other task. We en- sure that each label has at least one corresponding sample avoid the absence of rare labels.
dict structured information y from & using a frozen LLM L. We feed LLM with prompt (D, 8):
PE,LI(D,s) = =
(I)
We give examples of prompts on four IE tasks in Figure 1. The prompts consist of three parts: in- struction I (color in green in Figure 1), demonstra- tion (EE s)) (demo: color in blue) and the ques- tion f(x (color in black). Here & denotes demo selector and {(D,s) D denotes selected sen- tences as the demo predict s. Prompt format / refers to the template which converts demo E & s) and sample a to input context for LLMs. Then LLM generates f() (color in red) from which we could readily parse the extraction results y. Models L: We explore six LLMs from two sources. (1) OpenAl models 6; we employ Chat-
Evaluation We adopt micro-IF1 score in NER, RE and ED tasks. For EAA task, we follow previous work (Wang et al., 2023b) and adopt head-F1 score, which merely considers matching of the head word rather than the whole content of a text span. We re- port averaged score w.r.t 5 sampled train/validation sets unless otherwise stated.
3.2 Small Language Models
We adopt five supervised methods to evaluate the abilities of SLMs. (1) Vanilla fine-tuning for all
tasks, (2) FSLS (Ma et al., 2022a) for NER and ED
*We slightly abuse the notation to allow s, y and {(s,)) as the input for simplicity.
tasks, (3) KnowPrompt (Chen et al., 2022b) for RE
task, (4) PAIE (Ma et al., 2022b) for EAE task, and
'The versions of model we use are: 3pt-3. 5-5 S-turbo-0301.

---

Fine-tuning
FSLS
UIE
ChatGPT
CODEX
InstrictPPT
LLaMA (138)
Vicuna (138)
100
100
100
score
F1 score
80
60
60
60
40
20
0-
1-shot 5-chot sh- 20-shot
1-shot
5-shot 10-shot Ro-shot OntoNotes
1-shot 10-shot 20-shot
CONLLO3
FewNERD
(a) Named Entity Recognition (NER)
-|- Fine Fine-tuning HOw KnowPrompt
UNE
C PhatGPT
CODEX
InstructGPT
LLaMA (13B)
Vicuna (138)
score
60
60
40
40
20
1-shot 5-shot 10-shot 20-shot 50-shot 100-shot
I-shot
5-shot
10-shot 20-shot 50-shot 100-shat TACRED
TACREV
(b) Relation Extraction (RE)
Fine-tuning
ChatGPT
LLaMA (138)
80
80
80
1cscore
F1 scsce
40
40
40
F1 20
I-shot 5-shot 10-shot 20-shot
1-shot
S-shot 10-shot 20-shot
1-shot 5-shot 10-shot 20-shot
ACE05
ERE
MAVEN
(c) Event Detection (ED)
Fine Tine-tuning
PAIE -
UIE
ChatG PT
Ins testuuctPPT
LLaMA (138)
Vicuna (13B)
80
sccoe
60
F1score
60
60
40
F1 20
1-shot 5-shot 10-shot 20-shot
1-shot
5-shot 10-shot 20-shot
1-shot
5-shot 10-shot 20-shot
ACE05
ERE
RAMS
(d) Event Argument Extraction (EAE)
Overall results of SLM-based methods (dashed lines) and LLM-based methods (solid lines) on nine datasets across four IE tasks. The black, horizontal dashed lines represent the SoTA performance full dataset.
Figure 2:
GPT, CODEX (Chen et al., 2022a) and Instruct- GPT (Ouyang et al., 2022) for main experiments. We also evaluate GPT-4 in Appendix D.3. (2) Open-source models: we use LLaMA-13B (Tou- vron et al., 2023) and its instruction tuned counter- part, Vicuna-13B (Chiang al., 2023).
LLMs usually limits the sentence number de- mos even under few-shot settings. Therefore for each test sentence we demand demo retriever E {(D,s s) which selects small subset from D the sentences in demo. Following previous meth- ods (Liu et al., 2022: Su al., 2022), we retrieve demos according to their sentence embedding simi- larity to the test samples.
Instruction I: The instruction (1) describes the task and (2) enumerates all possible labels for ref- erence. we adopt instructions shown in Figure 1,
Demo selector E: The maximum input length of
Prompt format f: We use simple textual tem- plates format the demos and the test sample in main experiments. For example, the template for NER is "Sentence: [S], Entities: ([type1], [entity1]), ([type2], [entity2])....
code-davinci-802. text-davinci-892 and gpt-4-0314 14, Due to budget constraints, we execute InstructGPT and GPT- only once per setting. We do not conduct EAF task on CODEX since it had been unavailable at that time.

---

60.0
60
60.0
59.0
56
score
score
52
58.0
F1
55.0
57.0
ChatG PT CODEX
40
52.5
36
15
32 64
random Demonstration selection
embed
epr
Instruction format
Demonstration number
Figure 3: LLMs' performance prompt variants 20-shot FewNERD dataset. See full results other datasets in Appendix E.2- E.5. Left: ChatGPT's performance (F1 Score) across six instruction variants. Middle: F1 Score changes over varying numbers of demo. Right: ChatGPT's performance across three demo selection strategies. Random:
Random sampling. Embed: Sentence embedding. EPR: Efficient Prompt Retriever (Rubin et al., 2022).
3.4 Main Results
(2) Among proprietary LLMs, ChatGPT performs better on NER and EAE tasks, but poorer so on RE and ED tasks. InstructGPT and CODEX demon- strate comparable performance across these tasks. LLMs show limited inference speed. We compare the inference speed of different methods and show their results in Table 1. We observe that LLMs much slower than SLMs since they have much more parameters, longer input contexts and extra response decay (if external APIs applied).
We summarize the main experimental outcomes in Figure 2, indicating that L.MMs only outperform SLMs environments with restricted labels and samples. Conversely, SLMs are generally more effective. Given (I) the practicality of fine-grained IE tasks and the manageable effort of obtaining 10- 20 annotations per label and (2) the excessive time and budget demands of LLM inference, we con- clude that LLMs are not effective as supervised SLMs for few-shot IE tasks under real scenarios. We detail our findings as below.
3.5 Analysis Prompt Sensitivity
Previous work (Lu et al., 2022b) indicates that the efficacy of LLMs on specific tasks can be signifi- cantly influenced by the construction of the prompt. To ensure that LLMs' suboptimal outcomes not erroneously ascribed to inappropriate prompt designs, we meticulously examine the impact of diverse prompt variations from four aspects, i.e., in- struction format, demo number, demo selector and prompt format. We leave comprehensive details of the variants and their results to Appendix E.2- E.5, and illustrate salient findings in Figure 3. Our findings include that (1) diverse instruction strate- gies yield comparable results in IE task; (2) in- creasing the number of samples demonstrations does not unequivocally enhance performance; and (3) The selection strategy of demonstration mat- ters, and retrieval based on sentence embedding
Performance w.r.t sample number. The perfor- mance dynamics of SLMs and LLMs are influenced by variations in sample size. Under extremely low- resource (1-shot or 5-shot) settings, L.MMs some- times present superior performance than SLMs. Yet, LLMs tend to reach performance plateau with only modest increases in sample size. Con- versely, SLMs demonstrate marked performance enhancement as sample sizes grow, This trend is evident in Figure 2, where the SLM trajectories (represented by dashed lines) ascend more steeply compared to the LLM ones (solid lines).
Performance w.r. label number. Compared with SLMs, LLMs tend to struggle on fine-graine grained datasets. For instance, LLMs perform relatively worse MAVEN and RAMS datasets (with 168/139 labels) than on CONLL (4 labels only). Detailed quantitative results shown in Ap- pendix 1. illustrating clear negative correlation between the label number and the result disparity between LLMs and SLMs across various IE tasks.
Table 1: The inference seconds over 500 sentences (run on single V100 GPU). Here LLaMA is extremely slow since we set batch size as due to memory limit.
Comparisons among LLMs. We observe perfor- mance variability among LLMss. (1) Open-source models, LLaMA and Vicuna, significantly lag be- hind proprietary LLMs across all few-shot tasks.
Dataset (Task)
Roberta
TS
LLaMA CODEX
FewNERD (NER)
2.8
39.4
1135.4
179.4
TACREV (RE)
1.4
45.6
1144.9
151.6
ACE05 (ED)
6.6
62.5
733.4
171.7

---

(what we used) proves sufficiently effective. Con- sequently, we believe that there unlikely exists a lottery prompt that substantially alters our conclu- sions that LLMs are not good few-shot IE solver.
To mitigate LLMs' drawbacks mentioned above, we propose a filter-then-rerank paradigm to inte- grate both SLMs and LLMs within the same system. This paradigm uses SLMs as filters to select the top-N candidate labels, then LLMs rerank them to make final decisions. By using SLM-generatee candidate answers, the focus of LLMs shifts from sentence-level (i.e., identifying all entities/events in the sentence) to sample-level (i.e. determin- ing single entity/event candidate provided). Each question now corresponds
3.6 Discussion: Why LLMs Fail to Obtain Satisfactory Performance on IE Tasks?
Underutilized Annotations. We notice that LLMs appear to benefit less from additional annotations, Le., more training samples and label types, than SLMs. We speculate that LLMs are constrained by ICL in two ways. (1) More samples: The num- ber of effective samples for LLMs, those in de- mos, is limited by maximum input length. More- over, we also observe LLMs' performance platcaus in some tasks before reaching this limit (see Ap- pendix E.3). Meanwhile, SLMs can continually learn from more samples through supervised learn- ing, widening the performance gap as annotated samples increase. (2) More labels: LLMs struggle with fine-grained datasets. It suggests difficulty in understanding numerous labels and their subtle interactions merely from the given instruction and exemplars for LLMs. Also, the examples per label in demos decrease as label types increase.
single sample, al- lowing us to reframe prompts as multi-choice ques- tions (MCQ; shown in Figure 4) problem. Un- der such format, each candidate label is converted to a choice by pre-defined templates. We claim filter-then-rerank paradigm is more likely to elicit the powers of LLMs and smoothly solve few-shot IE tasks because: (1) LLMs are more familiar with MCQ prompts than IE-format prompts (Zhang et al., 2023a). (2) This paradigm reduces the la- bel scopes significantly, since N is usually much smaller than fine-grained label numbers.
4.2 LLMs are Hard Sample Solver
Our sitee-then-reene paradigm, unfortunately, presents unsatisfactory performance (and even suf- fers longer latency since LLMs rerank candidates per sample). Given LLMS'' abilities in memoriza- tion and reasoning, however, we still believe that LLMs are potential to solve some, if not most, IE samples effectively. We hypothesize that LLMs are more proficient than SLMs on hard samples. These samples are characterized by their requisite for external knowledge acquisition or sophisticated reasoning strategies, areas where LLMs can lever- age their extensive parametric knowledge bases and inherent reasoning mechanisms. In contrast, SLMs often falter with such samples, constrained by their restricted modeling capacities.
Unexplored Task format. As stated in Zhang et al. (2023a), IE-related tasks scarce in the widely-used instruction tuning datasets like Wei al. (2022a) and Wang et al. (2022). Furthermore, the highly-flexibe format of NER and ED tasks impair the ICL abilities? 7, Therefore it is likely that instruction-tuned LLMs are not well-acquainted with such IE-related task formats.
4 LLMs are Good Few-shot Reranker
4.1 Fileer-then-rerank Paradigm
Read following sentences and Edentify what is the entity type of The New Yorker" quated by ct>,
Sentence:
We leverage an unsupervised metric from SLMs to evaluate the difficulty of samples. Given sample in the sentence 8, we define the highest probabil- ity across all labels as the confidence score:
In 2804 Gourevitchh was assigned to cover the 2004 U.S.
prosidential election for "ct> The Now Yorkar cts".
Candidate Choices:
(a) how Yorker does not belong to any known entities. (b) how Yorker is a broadcast progras.
[c]The how Yorker is a kind of written art. (d)the haw Yorker is a redia/nwsspper arganization. Analysis:
The Nes Yorker is a well-inmen aseriean ragazine that has been puhlished since 1925, and is primarily known far its long-fore Journalion, comeentary, and satire. It has reputation for publishing high-quality weiting on a wide variety of topics, inclusing politics, culture, and the arts. So The hew Yarker is a nedia/newspppen organtzation. Correct Answer: (4)
conf(x))
max PSLM (1)2))))
(2)
IEL.
where L denotes the label set and PS (Ix:; s) the probability of a span x (in the sentence s) referring to label computed by SLMs. We classify sam- ples with low confidence scores as hard samples. Otherwise we view them as easy samples.
Figure 4: Multi-choice question (MCQ) prompt.
'These two tasks require unfixed numbers of (label, span) tuple. Furthermore, the length of each span is also unfixed.

---

LLM raraniinng
reranking
Table 2: Comparative ratios of negative to positive sam- ples across various datasets and subsets. We set fixed threshold here for simplicity.
FeWNERD (NER)
100
3
FewNERD TACREV ACEOS
0.25
0.25
0.35
0.05
0.9
0.69
0.75
0.55
0.95
Overall
5.88
3.03
38.2
Confidence Score
TACREV (RE)
Fasy samples (7 0.9)
9.44
3.21
44.0
Hard samples (T < 0.6)
1.28
2.68
1.36
Micro P1
18
0
words, most negative samples are easy samples for SLMs. Here refer negative samples those labeled as None. We speculate that the proficiency of SLMs with negative samples stems from their ability to adeptly discern apparent patterns during the fine-tuning stages. Therefore, SLMs could pre- diet negative samples with (relatively) high confi- dence and accuracy. Due to LLMs' predisposition to false-positive predictions on negative samples, however, the performance of LLMs easy sam- ples collapses. We attribute such false-positive pre- dictions to (1) hallucination and (2) span boundary mismatch. We detail such two kinds of mistakes with cases in Appendix F.2.
0.49
65
0.75
0.55
0.55
Confidence Score
ACE05 (ED)
Micre-Fz
18
8
635
0.25
0.55
0.95
0.65
0.75
0.85
0.25
Confidence Score
and performance with/without LLM reranking. We adopt RoBERTa-large as filter InstructGPT as reranker.
Figure 5: Relationship between confidence
We conduct experiments to confirm our hypoth- esis that LLMs excel on hard samples. We group samples by confidence scores and compare two methods within each group: (a) SLM --sd- meth- ods without LLM reranking, and (b) SLMs as the filter and LLMs the reranker. Method (b) dif- fers from (a) by adding a single LLM to rerank the top-V SLM predictions, using MCQ prompts.
Adaptive Filter-then-reaanh Paradigm
Above findings can be summarized as: (1) SLMs generally outperform LLMs, especially with more training samples and fine-grained labels. (2) SLMs are much more time- and cost-efficient. (3) LLMs serve as powerful rerankers on hard samples that challenge SLMs. Based on them, we propose a simple, efficient, and effective adaptive reranker that combines the strengths of SLMs and LLMs.
The results in Figure 5 substantiate our assump- tion. (1) LLM-based reranking (blue lines) en- hances performance on hard samples (left areas in the figure). We provide a detailed analysis of spe- cific challenging instances where LLM rerankers prove advantageous in Appendix F.I. These in- stances demonstrate the efficacy of LLMs in har- nessing external knowledge and complex reason- ing to rectify erroneous predictions initially made by SLMs (red lines). (2) Conversely, LLM-based reranking impedes performance on easy samples (right arcas), resulting in significant degradation, particularly for very easy samples (rightmost areas). In conclusion, LLMs exhibit greater proficiency in handling hard samples compared to SLMs, yet they underperform relative to SLMs on easy samples.
5.1 Method
Our adaptive filler-then-rerank approach, shown in Figure 6, uses supervised SLMs filter make preliminary decisions. Samples with confi- dence scores exceeding threshold are viewed as easy samples otherwise hard ones, For easy sam- ples, we retain SLM predictions as final results. For hard samples, top-N predictions from SLMs are reranked via LLMs using ICL. Here LLMs employ MCQ prompts (Figure 4), containing demos and sample to be reranked. The LLMs then generate the final answer and optionally provide an explanation.
4.3
Why LLMs Fail on Easy Samples
5.2 Experimental Setup
We investigate why LLMs (relatively) fail on easy samples in this section. As shown Table 2, we observe significant higher negative sample ratios for easy samples across diverse IE tasks. In other
We conduct experiments on FewNERD for NER task, TACREV for RE task and ACE05 for ED task. We employ top- cop-performin SLM-based meth- ods from Section 3 (FSLS or KnowPrompt) as the

---

The lest boskage, Ibalian engineer Eagenio Vagni, released early Sunday,
met the Haptiats" leader, Laurs Silshy of Meridian, Idaho, in our Prince on Jn 26.
Question
Demonstration
Adrien said
Bagtista' leader, Laun Sissy
"The lawper dented Italion reports shi while addnssing the court, hut sald Knnxx was upact as she mocuntod the presoum, the aggressiverees the police who celled (eishe other family member lewyer (biste a lowyer
Meridaan idahoo in Port-an Prince on 26. (stLaur Slisty lives in the state or proviace Merdien (bil, Siikey lives it the city Meeidian
Easy Sample
Hard Somple
(ottaus Slsdy bom
city Merdiin
(60) Hurn Silky his no known islations to Merkilin
Filter Small LM
Azalysis
(clehe has no known relations to lorger
Analyiis The woed She reters to soanecce who was upset while recounting
events wost Sewryer refers someone who deried news repoct about that same person weeping in court. There is no information the seatenoe to inthoate the avo individuals are related any way. Anzwer
Reranker Large LM
implies Slisty associated city Meridlar does not provide information about her bittglace So Laura Slisky lives the Meridarr
percorigin
per: cities of residence
Amswnn (b)
Figure 6: The overall architecture of our adaptive filter-then-reee paradigm. We color easy samples in orange and hard samples in pink. For easy samples, the final predictions are exactly from the SLM- methods. For hard samples, the top-N predictions from SLMs are fed into LLMs as the format of multiple-choice questions (pink box). The question is paired with demos (green box). LLMs rerank these N candidates and generate the final prediction.
filter, and Vicuna-13B, InstructGPT GPT-4 as the reranker. The threshold т to determine sam- ple difficulty is optimized on the valid set. For hard sample, the top-3 SLM predictions and None (if not included) are feed to LLMs for reranking. Each LLM prompt has 4-shot demos. See demo examples in Appendix G.1. We follow templates in Lu et al. (2022a) for TACREV and carefully de- sign others. See these templates in Appendix G.2. We adopt chain-of-thought reasoning (Wei al., 2022b), i.e. prefacing the answer with an explana- tion, to facilitate LLMs' reranking procedure.
erage. This confirms (1) the effectiveness of the LLM reranking and (2) its gains are different and (almost) orthogonal to the SLM ensemble.
5,4 Analysis
Few makes big difference Our method selectively reranks hard samples. Table shows that (1) only a minor fraction (0.5%~10%) of samples are deemed hard and are reranked by LLMs. (2) Despite their limited quantity, reranking results in a substantial performance boost on these samples (10%-25% absolute Fl gains). This uplift on small subset significantly enhances the overall performance. GPT-4 is more aggressive From Tables 3 and 4, GPT-4 generally improves more on hard samples, yet InstructGPT surpasses GPT-4 in NER and RE tasks when evaluated overall. This discrepancy arises from GPT-4's aggressive reranking which introduces more true positives. InstructGPT, how- ever, focuses more on reducing false positives. Few makes small cost Figure 7 demonstrates that our method impressively reduces budget and la- tency by approximately 80%-90% compared to direct ICL. This reduction is due to (1) fewer LLM callings (only for hard samples) and (2) shorter prompts (fewer candidate labels and demos).
Baseline We compare our method with two kinds of baselines to validate its effectiveness.
(I) LLMs with ICL: We follow the prompts in Sec- tion 3.3 and conduct experiments on three LLMs. (2) Supervised SLMs: We follow previous SoTA methods shown in Section 3.4 (FSLS or Know- Prompt). We additionally combine two SLMs with ensemble or reranking approach (i.e., replace the LLM with another SLM as the reranker) verify that improvements from our SLM-LLM integrated system are not solcly due to the ensemble effects.
5.3 Main Results
Table 3 shows that our filter-then-rerntt method consistently improves performance across three datasets and nine settings. For instance, with In- structGPT, reranking provides an average Fl gain of 2.4% without SL.M ensemble (Lines 4 vs. 7). Based on ensemble SLMs as the filter, our method still achieves 2. % (Lines 5 vs. 8) gains on av-
5.5 Ablation Study
We investigate the effectiveness of the modules in adaptive filte-then-rerank system by removing each of them in turn: (1) CoT: We exclude the explantion for each examples in demo. (2) Demo:

---

Table 3: Overall results of LLM-based ICL methods, SLM-based supervised methods, and our proposed filter-then- rerank (SLM+LLM) methods. The best results are in bold face and the second best are underlined. All results except InstructGPT and GPT-4 are averaged over 5 runs, and sample standard deviations are in the round bracket.
FewNERD (NER)
TACREV (RE)
ACE (ED)
5-shot
10-shot
20-shot
20-shot
50-shot 100-shot
5-shot
10-shot
20-shot
CODEX
53.8(0.5) 54.0:1.4)
55.9
(1.44 60.3;2.4)
62.4(2.6)
47.1 47.7(2.8)
47.910.5)
InstructGPT
53.6(-)
54.6(-)
57.2(-)
60.1(-)
58.3(-)
62.7(-)
52.9(-)
52.1(-)
49.3(-)
GPT4
57.8(-)
59.3(-)
52.1(-)
Previous SoTA
59.4(1.5) 61. 6(.4(0.)) 61.9
62.4(3.a)
68.5 6) 72.6(1.5)
|5.1 (4.6))
63.9(u.s) 65. 85.8(2.0)
SLM
Ensemble (S)
59.6(1.7) 61.8(12) 62.6(1.)
64.9(1.5)
71.9(2.2)
74.1
56.9(4.7)
64.2(2.1) 66.5(1.7)
Rerank (S)
59.5(1.5) 61.0a..) 61.5
64.2(2.3)
70.8(2.5) 74.3(2.2)
(6.1 64.0(1.0) 66.7(1.7)
Vicuna- 133
Rerann (l.)
60.0(1.8) 61.9(2.1) 62.2(1.4) (5.2(1.4) 70.8(16)
73.8(1.7)
56.9(4.0)
63.5(2.7) 66.0(2.6
SLM LLM
Ensemble (S) Rerank (L)
59.9(0.7) 62. (o.7) 62.8(1.1)
66.5(o.5) 73.6 75.0(1.5)
57.9(5.2)
64.4(1.2) 66.2
InstructGPT
Rerank (L)
60.6(2.1)
(2.7 63.3(o.0)
66.8(2.8)
72.3:1.4)
75.4(1.))
57.8
(1.3
((23))
Ensemble (S) + Rerank (L)
(1.3 63.2 (3.7(1.8)
(8.9 74.8 (6.8
(9.5
65.3(1.9) 67. 07.8(2.1)
GPT-
Rerank (L)
60.8().3 62.6 (2.7) 63.0(1.a)
65.9(2.7
(2.3 74.5(1.5)
59.6(2.9)
64.9(2.6) 67. (2.6)
Ensemble (S) + Rerank (L)
((22) 6(.8()..) 63.6(1.2)
68.6(1.3 (3.9(1.4) 75.9r2.4) (0.9 65.6(1.5) (1.8(1.7)
Table 4: The Fl-score differences before and after reranking on the reranked samples, as well as their pro- portion of the total samples.
Table 5: Ablation study on three datasets. The filter is ensembled SLMs and the reranker is GPT-4.
FewNERD TACREV ACE05 (20-shot)
GPT-4
InstructGPT before after
CoT
Demo LF
AD
(100-shot)
(20-shot)
before after A
ratio
A
ratio
V
v
/
63.6(1.2)
75.9(2.4)
67.8(L.)
FewNER
31.9 40.7
8.8 3.2%
31.4
28.3
-3.1 3.1.6 3.3%
X
/
d
63.2(1.2)
75.4(2.4)
67.2 67.2(1.7)
TACREV
25.3
43.0
17.7
9.1%
33.8
43.4
9.6
7.1%
X
X
/
63.0(1.4)
74.9(2.2)
66.6(1.5
ACE05
31.1 57.9
26.8
1.6%
35.6
55.7
20.1
0.5%
X
X
V
62. 62.4(2.1)
73.8(2.5)
66.5(1.)
X
X
X
12.5(2.7)
59.9(6.0)
5.4(1.1
Previous SoTA methods
62.6(1.0)
74.1(1.7)
66.5(1.7
We remove all examples, rendering the reranking a zero-shot problem. (3) LF (label filtering): We retain all labels as candidate choices for reranking, instead of only the top-N labels from the SLMs. (4) AD (adaptive): We feed all samples, not just hard ones, to the LLMs.
hence cutting inference costs. (4) The performance collapses without a filter to identify sample diffi culty, reiterating the need for an integrated SLM- LLM system to complement each other.
We show their results in Table 5 and see that (1) Demos with explanations consistently enhance the reranking ability of LLMS across all datasets. (2) Demos without explanations also contribute to performance improvement. (3) Label filtering re- sults in gains and notably reduces the demo length,
Conclusion
Through an extensive empirical study on nine datasets spanning four TE tasks, find that LLMs, despite their superiority in extreme low-resource scenarios, are not effective few-shot information extractors in general. They struggle with IE-related prompts, have limited demonstration capacity, and incur high inference costs. However, LLMs signifi- cantly improve the performance on hard samples when combined with SLM. Building on these in- sights, propose adaptive flter-then-rrean paradigm to leverage the strengths of SLMs and LLMs and mitigate their limitations. This approach consistently achieves promising results, with an av- erage 2.4% Fl gain across multiple few-shot IE tasks, while minimizing latency and budget costs.
DinetICL (IstruckOPT]
Ttar--hhnn-reant
Pine- Fine-tuning (RoBERTa-lare llargl)
150
30
oocondlo)
300
30
FeMNERD
TACREV
ACE05
FOWNERD
TACREV
ACEOS
Financial cost
Time cost
Figure 7: The financial and time cost over 500 sentences. InstructGPT as the reranker.

---

Limitations
Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. 2020. TACRED revisited: A thorough eval- uation of the TACRED relation extraction task. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1558- 1,569, Online. Association for Computational Linguis- tics.
We do work hard to find better prompts to elicit the power of LLMs on few-shot IE tasks in Section 3.5, by exploring various kinds of LLMs, demonstra- tion strategies and prompt formats. We find that dif- ferent prompt variants do not significantly impact in-context learning abilities. As an empirical study, we ucknowledge the potential existence of lottery prompt superior to our explored prompts. However, it seems unlikely that an improved prompt would substantially alter our conclusions.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Another common risk when evaluating LLMs on public benchmark is their potential memoriza- tion of samples tested. To mitigate such poten- tial contamination, we use earlier and stable ver- sions of these models rather than the newer and updated ones (for example, gpt-4-0314 instead of gpt-4). Even if such contamination makes abilities of LLMs overestimated, our primary conclusions remain unchanged because we find that LLMs are NOT good few-shot information extractors.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Pondé de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sas- try, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter. Philippe Tillet, Felipe Petroski Such, Dave Cum- mings, Matthias Plappert, Fotios Chantzis, Eliza- beth Barnes, Ariel Herbert- Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Joshua Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, llya Sutskever, and Wojeiech Zaremba. 2021. Evaluat- ing large language models trained on code. ArXiv preprint, 0bs/2107.0337.
Regarding our adaptive filter-then-eraank paradigm, a key limitation lies in how to assess sample difficulty. In this work, we employ a simple unsupervised metric, i.e,, the maximum probabilities from SLMs. This is predicated on the assumption that SLMs are well-calibrated (Guo et al., 2017). However, it is an obviously imperfect assumption. We envision that calibrating SLMs- based filters developing an advanced difficulty metric could substantially enhance LLM rerankers' performance. We leave them for future work.
Acknowlegement
This study is supported under the RIE2020 In- dustry Alignment Fund - Industry Collaboration Projects (IAF-ICP) Funding Initiative, the Singa- pore Ministry of Education (MOE) Academic Re- search Fund (AcRF) Tier 1 grant, as well as cash and in-kind contribution from the industry part- ner(s).
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. 2020. A simple framework for contrastive learning of visual representations. In Pro- ceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Vir- tual Event. volume 119 of Proceedings of Machine Learning Research, pages 1597-1607. PMLR.
Wenhu Chen, Xueguang Ma. Xinyi Wang, and William W. Cohen. 2022a. Program of thoughts prompting: Disentangling computation from reason- ing for numerical reasoning tasks.
References
Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large language models are few-shot clinical information extractors. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 1998-2022, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng. Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022b. Knowprompt: Knowledge- aware prompt-tuning with synergistic optimization for relation extraction. In WWW '22; The ACM Web

---

Conference 2022, Virtual Event, Lyon, France, April 25 29, 2022, pages 2778-2788. ACM.
George Doddington, Alexis Mitchell, Mark Przybocki. Lance Ramshaw, Stephanie Strassel, and Ralph Weischedel. 2004. The automatic content extrac- tion (ACE) program tasks, data, and evaluation. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LRECC044) Lishon, Portugal. European Language Resources As- sociation (ELRA).
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Seth Ebner. Patrick Xia, Ryan Culkin, Kyle Rawlins. and Benjamin Van Durme. 2020. Multi-sentene gument linking. In Proceedings of the 58th Aunual Meeting of the Association for Computational Lin- guisties, pages 8057-800 15778077, Online. Association for Computational Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez. Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin. Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Frica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways.
Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. 2023. Exploring the feasibility of chatgpt for event extraction.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Procexs- ing, pages 6894 6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Wein- berger. 2017. On calibration of modemn neural net- works. In Proceedings of the 34th International Con- ference on Machine Learning, ICML. 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pages 1331-1330. PMLR.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghami, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le. and Jason Wei. 2022. Scaling instruction instruction-finttueed language models.
Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 in-context learn- ing for biomedical IE? think again. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4497-4512, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Martin Josifoski, Marija Sakota, Maxime Peyrard, and Robert West. 2023. Exploiting asymmetry for syn- thetic training data generation: Synthie and the case of information extraction.
Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pages 11173-11195. Toronto, Canada. Association for Computational Linguistics.
Jachun Jung, Lianhui Qin, Sean Welleck, Faeze Brah- man, Chandra Bhagavatula, Ronan Le Bras, and Yejin Choi. 2022. Maieutic prompting: Logically consistent reasoning with recursive explanations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1266-1279, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguistics.
Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition dataset. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 1 1th International Joint Confer- ence Natural Language Processing (Volume I: Long Papers), pages 3198-3213, Online, Association for Computational Linguistics.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov. and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Compulational Linguistics,

---

pages 7871-7880, Online, Association for Computa- tional Linguistics.
Roth. 2022a. Label semantics for few shot named entity recognition. In Findings of the Association for Computational Linguistics: ACI. 2022, pages 1956- 1971, Dublin, Ireland. Association for Computational Linguistics.
Peng Li. Tianxiang Sun, Qiong Tang, Hang Yan, Yuan- hin Wu, Xuanjing Huang, and Xipeng Qiu. 2023. CodelE: Large code generation models are better few-shot information extractors. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 15339-15353 Toronto, Canada. Association for Computational Linguistics.
Yubo Ma, Zehao Wang, Yixin Cao, Mukai Li. Meiqi Chen, Kun Wang. and Jing Shao. 2022b. Prompt for extraction? PAIE: Prompting argument interac- tion for event argument extraction. I Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 77794,774, Dublin, Ireland. Association for Computational Linguistics.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (Deel1O 2022): The 3rd Workshop on Knowledge Extrac- tion and Integration for Deep Learning Architectures, pages 100-115 14, Dublin, Ireland and Online. Associa- tion for Computational Linguistics.
Yubo Ma, Zehao Wang, Yixin Cao, and Aixin Sun. 2023. Few-shot event detection; An empirical study and a unified view. In Proceedings of the 61st Anmual Meet- ing of the Association for Computational Linguis- rics (Volume I: Long Papers), pages 11211-11 1236, Toronto, Canada. Association for Computational Lin- guistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal. Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net.
I Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. 2023, Logic-Im: Empower- ing large language models with symbolic solvers for faithful logical reasoning.
Keming Lu. I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. 2022a. Summarization as indirect supervision for relation extraction. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 6575-6594, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cheng Qian, Chi Han, Fung, Yujia Qin, Zhiyuan Liu, and Heng Ji. 2023. Creator: Tool creation for disentangling abstract and concrete reasoning of large language models.
Keming Lu, Xiaoman Pan, Kaiqiang Song, Hongming Zhang, Dong Yu, and Jianshu Chen. 2023. Pivoine: Instruction tuning for open-world information extrac- tion. ArXiv preprint, abs/2305. 14898.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general- seneral-purpose natural language process- ing task solver?
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022b. Fantastically ordered prompts and where to find them: Overcoming few- shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Compu- national Linguistics (Volume 1: Long Papers), pages 8086-8098, Dublin, Ireland. Association for Compu- tational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts. Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020, Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:61
Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2022. Learning to retrieve prompts for in-context learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2655-2671, Seattle, United States. Association for Computational Linguistics.
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022c. Uni- fied structure generation for universal information extraction. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), pages 5755-5772, Dublin, Ireland. Association for Computational Linguistics.
Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. 2015. From light to rich ERE: Annotation of entities, relations, and
Jie Ma. Miguel Ballesteros, Srikanth Doss, Rishita Anubhai, Sunil Mallya, Yaser Al-Onaizan, and Dan

---

events. In Proceedings of The 3rd Workshop on EVENTS: Defimition, Detection, Coreference, and Representation, pages 89-98, Denver, Colorado. As- sociation for Computational Linguistics.
chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations (ICLR 2023).
Yizhong Wang. Swaroop Mishra, Pegah Alipoormo- labashi, Yeganch Kordi, Amirreza Mirzaci, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A. Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022. Super-Natural Instrctionn Generaliza- tion via declarative instructions 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empiri- cal Methods in Natural Language Processing, pages 5085-5109, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Hongjin Su, Jungo Kasai, Chen Henry Wu, Weijia Shi, Tianlu Wang, Jiayi Xin, Rui Zhang. Mari Ostendorf, Luke Zeullemoyer, Noah A. Smith, and Tao Yu. 2022. Selective annotation makes language models better few-shot learners.
Zhiqing Sun, Xuezhi Wang. Yi Tay, Yiming Yang. and Denny Zhou. 2023. Recitation-augmenti language models. In International Conference on Learning Representations.
Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142- 147.
I
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric- Aetrie-Aneeeae Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoc V. Le. 2022a. Finetuned language models zero-shot learners. In The Tenth International Conference on Learning Representa- rions, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Somin Wadhwa, Silvio Amir, and Byron Wallace. 2023. Revisiting relation extraction in the era of large lan- guage models. In Proceedings of the 61st Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pages 15566- 15589, Toronto, Canada. Association for Computa- tional Linguisties.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoc Le, and Denny Zhou. 2022b. Chain of thought prompting elicits reasoning in large language models. Proceedings of the 36th International Conference on Neural Information Pro- cessing Systems.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wen- juan Han. 2023. Zero-shot information extraction via chatting with chatgpt.
Xiao Wang, Wei Zhou, Can Zu, Han Xia, Tianze Chen, Yuan Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, J. Yang, Siyuan Li, and Chunsai Du. 2023a, Instructuie: Multi-task instruction tuning for unified information extraction. ArXiv preprint, abs/2304.08085.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Ed- uard Hovy, Sameer Pradhan, Lance Ramshaw, Nian- wen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran- chini, et al. 2013. Ontonotes release 5,0 Idc2013t19. Linguistic Data Consortium, Philadelphia, PA.
Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A Massive General Domain Event Detection Dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language P Processing (EMNLP), pages 1652- 1671, Online. Association for Computational Canatttoonnnnnnuuiii Linguis- tics.
Yi Yang and Arzoo Katiyar. 2020. Simple and effective few-shot named entity recognition with structured nearest neighbor learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6365-6374 Online. Association for Computational Linguistics.
Xingyao Wang. Sha Li, and Heng Ji. 2023b. Code4Struct: Code generation for few-shot event structure prediction. In Proceedings of the 61 An- nual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3640- 3663, Toronto, Canada. Association for Computa- tional Linguistics.
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In International Confer- ence for Learning Representation (ICLR 2023).
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou, 2023e. Self-consistency improves
Kai Zhang, Bernal Jimenez Gutierrez, and Yu Su. 2023a. Aligning instruction tasks unlocks large language

---

models as zero-shot relation extractors. Find- ings of the Association for Computational Linguis- tics: ACI. 2023, pages 794-812, Toronto, Canada. Association for Computational Linguisties.
Algorithm 1 Greedy Sampling
Require: shot number K, original full dataset
D {(X, Y)} tagged with label set E Sort E based on their frequencies in { {Y) as
Yuhao Zhang, Victor Zhong. Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empiri- cal Methods in Natural Language Processing, pages 35-45, Copenhagen, Denmark. Association for Com- putational Linguistics.
an ascending order
2: <- Φ. Counter + dict()
3: for y E E do
4: Counter(y) +- 0
5: end for
6: for y E E do
while Counter(y) y) < K do
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023b. Automatic chain of thought prompt- ing in large language models. In The Eleventh In- ternational Conference on Learning Representations (ICLR 2023).
Sample (X, Y) E D S.t.3j.Wo yj = D Y)
10: Update Counter (not only y but all
event types in Y)
11: end while
A Datasets
12: end for
A.1 Full Datasets
13: for s C do
14:
S + S and update Counter
We construct few-shot IE datasets and conduct the empirical study on nine datasets spanning four tasks, with varying schema complexities ranging from 4 to 168. We show their statistics in Table
if E, s.t. Counter(y) < K then
15:
16:
S+SUs
17: end if
18:
end for
A.2 Details of Few-shot IE Datasets
19: return S
Sampling Algorithm for Train/Valid Datasets. We downsample sentences from original training dataset to construct few-shot training and valid datasets. We adopt K-shot sampling strategy that each label has (at least) samples. We set 6 K- values (1, 5, 10, 20, 50, 100) for RE tasks and K-values (1, 5, 10, 20) for other tasks. For RE task, each sentence has exactly one relation and we simply select K sentences for each label. For NER, ED and EAE tasks, each sentences is possible to contain more than one entities/eventt/arggumett Since our sampling at sentence-level, the algo- rithm of accurate sampling i.e. finding exactly K samples for each label, is NP-complete and unlikely to find a practical solution. Therefore we follow Yang and Katiyar (2020) adopting a greedy sampling algorithm to select sentences for NER and ED tasks, as shown in Algorithm 1. Note that the actual sample number of each label can be larger than K under this sampling strategy. For all three tasks, we additionally sample negative sentences (without any defined labels) and make the ratio of positive sentences (with at least one label) and neg- ative sentences as 1:1. The statistics of the curated datasets are listed in Table 7.
Based on the subsets constructed above, we op- tionally further split them into training and valid sets. For few-shot datasets with more than 300 sen- tences, we additionally split 10% sentences the valid set and the remaining sentences as training set. Otherwise, we do not construct valid set and con- duct 5-fold cross validation to avoid overfitting.
B Details on SLMs
We adopt five representative supervised methods to evaluate the ability of SLMs on few-shot IE tasks.
(1). Fine-tuning (FT): Add classifier head on SLMs to predict the labels of each sentence/word. (2). FSLS (Ma et al., 2022a): The state-of-the-art sirracive-based method for few-shot NER task. Ma et al. (2023) also validate its competitive per- formance on few-shot ED tasks.
(3). KnowPrompt (Chen et al., 2022b): The best extractive-based method for few-shot RE task.
(4). PAIE (Ma et al., 2022b): The best extractive-
based method for few-shot EAE task.
(5). UIE (Lu et al., 2022c): A competitive unified generation-based method for few-shot IE tasks. We introduce their implementation details below: Fine-tuning/FSL.. We implement these two meth-
"The Subset Swn Problem, classical NP-complete prob- lem, can be reduced to this sampling problem.

---

UIE We implement this method based on original Table Statistics of nine datasets used. Note that the #mentions for event detection tasks refers to the number of trigger words, while the #mentiins for event argument extraction tasks refers to the number of arguments.
Named Entity Recognition
Relation Extraction
Event Detection
Event Arg Extraction ACE05 RAMS
Dataset
CONLL OntoNotes FewNERD
TACREV TACRED
ACE05 MAVEN ERE
ERE
#Label Type
18
66
41
41
168
38
Train
14,041
49,706
131,965
68.124
68,124
14.024
32,360
14,736
14,024
7329
14,736
#Sents
Test
3,453
10,348
37,648
15,509
15,509
728
8,035
1,163
728
871
1,163
Train 23,499
128,738
340,247
13,012
13,012
5,349
77,993
6,208
4859
17026
8924
#Mentions
Test
5,648
12,586
96,902
3,123
3,123
424
18,904
551
576
2023
822
ods by ourselves. We use RoBERTa-large (Liu et al., 2019) as the backbones. We adopt Auto- matic Mixed Precision (AMP) training strategy" save memory. We run each experiment on single NVIDIA V100 GPU. We train each model with the A (Loshchilov and Hutter, 2019) opti-
Table The statistics few-shot training sets. set different random seeds and generate 5 training sets for each setting. We report their average statistics.
Dataset Settings
# Labels
Sent # Sample Avg shot
1-shot
4.8
1.4
CONLL'03
5-shot
4
16.2
21.8
5.5
mizer with linear scheduler and 0.1 warm-up steps.
10-shot
29.2
42.6
10.7
20-shot
63.6
82.0
20.5
We set the weight-decay coefficient as le-5 and maximum gradient norms as 1.0. We set the batch
1-shot
20.0
33.4
1.9
OntoNotes
10-shot
5-shot
18
158.6
84.8
281.0
148.0
15.6
8.2
size as 64, the maximum input length 192, the training step as 500 and the learning rate 5e-5.
20-shot
332.8
547.2
30.4
1-shot
89.8
2.2
KnowPrompt We implement this method based on original source code and use RoBERTa-large
FowNERD
5-shot
66
286.2
494.8
7.5
10-shot
538.0
962.0
14.6
20-shot
1027.2
1851.4
28.1
as our backbones. We set 10 maximum epochs for
1-shot
81.6
41.0
1.0
50- and 100-shot datasets, and as 50 epochs for
S-shot
387.6
205.0
5.0
TACREV
10-shot
741.2
406.0
9.9
other datasets. We keep all other hyperparameters
20-shot
1367.2
806.0
19.7
as default, and run each experiment on a single NVIDIA V100 GPU.
50-shot
2872.0
1944.0
47.4
100-shot
4561.0
3520.0
85.9
1-shot
81.6
41.0
1.0
PAIE We implement this method original
5-shot
387.6
205.0
5.0
source codc"". and use BART-large (Lewis et al.,
TACRED
10-shot
41
741.2
406.0
9.9
20-shot
1367.2
806.0
19.7
2020) backbones. We keep all hyperparameters
50-shot
2871.2
1944.0
47.4
100-shot
4575.2
3520.0
85.9
as default for ACE and RAMS dataset. For ERE dataset, we set the training step as 1000, the batch size as 16 and the learning rate as 2e-5. We run each experiment on a single NVIDIA V100 GPU.
I-shot
47.4
41.0
1.2
5-shot
192.8
165.0
5.0
ACEOS
33
10-shot
334.6
319.4
9.7
20-shot
579.4
598.2
18.1
1-shot 5-shot
157.6
298.0
1.8
MAVEN
168
540.4
1262.2
7.5
source !2 and use T5-large (Raffel et al., 2020) the backbones. We each experiment on single NVIDIA Quadro RTX8000 GPU. We
10-shot
891.2
2413.8
14.4
20-shot
1286.4
4611.4
27.4
1-shot
48.4
54.6
1.4
ERE
5-shot
38
175.0
219.2
5.8
set the batch size as with 4000 training steps.
10-shot
304.8
432.4
11.4
20-shot
521.6
806.6
21.2
We set the maximum input length as 800 and the learning rate as le-4.
1-shot
23.4
40.2
1.2
ACEOS
5-shot
33
79.8
178.2
5.4
10-shot
130.8
337.4
C LLMs Implementations
20-shot
213.4
630.2
19.1
1-shot
130.2
332.6
2.4
Regarding our empirical study, we explore the ICL
RAMS
S-shot
133
514.0
1599.6
11.5
abilities of LLMss few-shot IE tasks. We mainly
10-shot
795.2
3193.2
23.0
20-shot
1070.4
6095.4
43.9
use five LLMs from two sources. (1) OpenAl
I-shot
21.6
102.8
2.7
ERE
5-shot
38
74.2
403.4
10.6
10-shot
127.2
775.6
20.4
20-shot
190.2
1397.2
36.8
" https:///gituu..co/mmaaaaa22222

---

models: CODEX (code-davinci-0022 Chen et al.
Table Fi-scores across different t values. Experi-
2021), InstructGPT (text-davinci-o00)) Ouyang
ments 10-shot settings with CODEX.
2022), and ChatGPT - 5ttu1b)-3301),, (2) Open-source models: LLaMA-13B (Touvron et al., 2023) and its instruction-tunn
FewNERD TACREV ACEOS
48.5(1.))
53.7(2.3) (2.9(2.))
counterpart, Vicuna- 13B (Chiang ct al., 2023). We detail their implementation details in the next sections below.
5-ensemble
(3.5
58.6(1.5)
46.3(o.s)
0.7
40.9(2.3)
(9.9(1.)
35.6(1.)
self-consistency 52.1(0.9)
53/(())) 45.6(3.0)
C.1 Open-source Models
We implement multiple ICL approaches on LLaMA-13B and Vicuna-13B without fine-tuning. We set the maximum input length as 2048 and the batch size as 1. We run each experiment on a single NVIDIA V100 GPU. To achieve this, we leverage the Accelerate 13 framework and fp16 inference to save memory. We set maximum output length 96 and sampling temperature 0 (i.e., greedy decoding). We set both frequency_penalty and presence_penalle as O.
original datasets, follow Automatic Chain-of- thought (Auto-CoT; Zhang al. 2023b) method as below. Regarding each sample, we query LLMs
According [sentence], Why [span] a [label].
For example, given the sentence "DSC and Trac- tion Control on all Speed3 models is also stan- dard.", would feed LLM the query that "Could you explain why Speed3 is kind of car". Then we insert the bootstrapped rationales between the sen- tences and ground-truth answers. If a sentence has no positive labels, however, we do not ask LLMs and keep the original format as the vanilla ICL. ap- proach. Here we prompt InstructGPT to generate the rationales with temperature / = 0.7. We com- pare the performance with and without Auto-CoT as shown in Table 9.
C.2 OpenAI Models
We implement multiple ICL approaches on Ope- nAI models by calling their official APIs 14, We set the maximum input length 3600 for all tasks and models. The only exception occurs when we use CODEX on RE tasks, where we the maximum input length as 7000. We unify the maximum out- put length as 32 for RE task, and 96 for other three tasks. We set the sampling temperature coefficient 0, ie., greedy decoding.
Table The Fi-score difference between with and without Auto-CoT. We generate rationales by Instruct- GPT, then adopt ICI. w. Auto-CoT approach and use CODEX as our backbone for inference.
D Pivot Experiments on LLMs
FewNERD TACREV ACE05 (NER)
10-shot train set
D.1 Sampling Temperature
(RE)
(ED)
wo. Auto-CoT
54.0(1.)
57.3(1.8
47.7(2.8)
Existing prompt-engineering discussion suggests setting the sampling temperature = 0 for tasks with structured outputs, including IE tasks. We validate this conclusion in Table 8, from which could see the generated quality when / 0 is much higher than the quality when \ j O. Therefore set 0 in all main experiments, and do not take self self-consistency (Wang et al., 2023c) into account.
w. Auto-CoT
36.6(L.7
22.0(1.2)
43. (3344)
We are frustrated to find Auto-CoT degrades the performance with large margin. We speculate this degration could be attributed three main reasons. (1) The rationale increase the length of each sample and thus decrease the overall example number demos. (2) There exists an obvious discrepancy between sentences with and without positive labels. The rationales are only provided for sentences with positive labels because it is hard to explain why sentence dose not contain any label. (3) Some auto-generated rationales are low-quality, especially for RE tasks. We would explore better strategy to exploit auto-genertuce rationales in the future work.
D.2 Automatic Chain-of-thought
We additionally investigate whether rationales could facilitate LLMs' performance on few-shot IE tasks. Since there exists no golden rationales in
"htts::/hugginggfce http:///wggnnngfaeeoooooocccccce coldocs/accelerate
1 "https://openii.co wwwp://ewww...ooooooo/ooo/aaaaaa
"haps://hclp.ppeeae wttp:/a....//eeeaarro//e/esaaaoa

---

Table 10: Fi-scores difference among GPT-4, CODEX and InstructGPT.
NER (20-shot)
RE (100-shot) TACREV TACRED
ED (20-shot)
EAE (20-shot)
CONLL OntoNotes FewNERD
ACE05 MAVEN ERE
ACE05
RAMS ERE
InstructGPT
77.2
47.7
57.2
62.7
53.8
49.3
25.4
40.8
45.8
42.2
41.9
CODEX
81.1
55.6
55.9
62.4
53.6
47.9
22.8
39.0
GPT4
84.7
65.6
57.8
59.3
50.4
52.1
30.2
40.5
42.9
38.6
38.2
Supervised SoTA
72.3
74.9
61.4
72.6
63.1
65.8
54.7
56.2
55.2
57.7
55.6
D.3 GPT-4 v.s. Others
lation between the label number (row 2) and the performance difference (row 5). In other words, with more label types, LLMs tend to perform rel- atively worse than SLMs. Therefore we conclude that LLMs struggle fine fine-grained datasets.
We tend minimize the GPT-4 calls due to its high price. Thus we utilize 20-/100-shot settings across each dataset to compare GPT-4's performance with other LLMs. Table 10 reveals that GPT-4 does not outperform other LLMs significantly, except on OntoNotes and MAVEN. However, even on these datasets, GPT-4 still falls behind supervised SLMs by significant margin. Consequently, the exclu- sion of GPT-4 does not undermine the conclusions drawn from our main experiments, and we omit from our empirical study.
E.2 Finding Better Instruction
To investigate whether LLMs would benefit from complex instructions, explored six instruction variants from simple to complex. Take NER task as an example, we illustrate them as below. Instruction0: [empty]
Instructionl: Identify the entities expressed by each sentence, and locate each entity to words in the sentence. The possible entity types are: [Type_1], [Type_2],
E
Auxiliary Experiments
E.1 LLMs struggle Fine-grained Datasets
[Type_N]. If you do not find any entity in this sentence, just output 'Answer: No entities found.' Instruction2: Identify the entities expressed by each sentence, and locate each entity to words in the sentence. The possible entity types are:
Based on the results shown in Figure 2, we addi- tionally provide quantitative analysis to show that LLMs struggle with fine graiggraied datasets. Under the 5-shot setting, we compare the performance difference of LLMs (ChatGPT) and SLMs (SoTA few-shot models) among different datasets. For each IE task, we observe a clear negative corre-
[Type_1]: [Definition_]]
Table II: Performance comparison between LLMs (ChatGPT) and SLM--ased methods among datasets with various schema complexities.
[Type_2]: [Definition_2]
Named Entity Recognition CoNLL
[Type_N]: [Definition_N]
OntoNotes FewNERD
Entity Micro-F1 (SLM)
4
18
66
If you do not find any entity in this sentence, just output 'Answer: No entities found.
52.5
59.7
59.4
Miero-FI (LLM)
77.8
59.4
55.5
AFI (LLM. SLM)
25.3
-0.3
-3.9
Event Detection
Instruction3: Assume you are an entity-instance annotator. Given a sentence, you need to (1) identify the word or phrase about the entity in the sentence, and (2) classify its entity type. The possible entity types listed as below: [Type_1], [Type_2], [Type_N]. Please note that your annotation results must follow such
ACE05
ERE
MAVEN
Event
33
38
168
Miero-FI Fl (SLM)
55.1
48.0
49.4
Micro-FI (I.LM)
39.6
33.8
25.3
AFI (LLM, SLM)
-15.5
-14.2
-24.1
Event Argument
Extraction ERE
ACE05
RAMS
Event/ / #ooe
/
38/ 26
139 65
Head-F1 (SLM)
45.9
40.4
54.1
Head-FI (LLM)
52.8
40.7
44.2
AFI (LLMM. SLM)
6.9
0.3
-9.9
format: "'Answer: ([Type_1] <SEP>

---

identified_entie DinniiiieeLentttyy.nniiyyyy., (CType_2) 2] <SEP>
Table 12: Fi-scores across six instruction formats. Ex-
periments 20-shot settings with ChatGPT.
......... "., If you do not find any entity in this sentence, just output :Answer: No entities found.'
FewNERD (NER)
TACREV (RE)
ACE (ED)
ACE
(EAE)
57.6(2.1)
49.1(2.4)
44.0(1.)
(0.9(
Instruction4: Assume you
58.3
49.6(1.2)
42.6(1.0)
51.5(1.1)
entity-instance annotator. objective is perform series of intricate steps for Named Entity Recognition. Firstly, you have to identify a particular word or phrase in the sentence that corresponds to an entity. Following this, classify the entity into one of the potential entity types. The potential entity types are provided below: [Type_1], [Type_2], ..., [Type_N]. Please note that your annotation results must follow such format: 'Answer: ([Type_1] <SEP> ceentified_entity [Entity_1]),
Your
57.7(1.0)
50.0(1.5)
41.8(0.9)
50.3(1.5)
57.6(2.3)
52.3(1.8)
42.9(1.3)
49.2(2.3)
56.8(0.9)
49.6(2.9)
41.6(119)
49.9(1.2)
15
57.8(v.5)
47 47.2(1.8)
43.1(1.8)
50.6(1.5)
and LLMs' performance. Even the prompt with- out instruction (I0) leads to comparable, not bet- ter, results than prompt with complex instructions. Therefore, we use simple instruction (II) in our main experiment.
E.3 Do More Samples in Demos Help?
We wonder whether longer demos bring more pow- erful ICL abilities for LLMs. Thus investigate the impact of increasing the number of demon- strations on LLMs' performance in Figure 8. We observe that: (1) The performance of the RE task consistently improves with more demos, indicating its potential benefiting from additional annotations. (2) The NER and ED tasks reach stable or de- graded performance with increased demo numbers, suggesting that they are limited even before reach- ing the maximum input length. (3) Open-source source LLMs, i.e., LLaMA and Vieuna, have more limited capacities in leveraging demos compared Ope- nAl models, with their performance stagnating or even collapsing with only a few (2-4) demos.
<SEP> (Enneitt ([Type_2) 2] [Entity_21)
........ If you do not find any entity in this sentence, just output "Answer: No entities found.
InstructionS: Assume you are an entity-instance annotator. Given sentence, you need to (1) identify the word or phrase about the entity in the sentence, and (2) classify its entity type. The possible entity types are listed as below:
[Type_1]: [Definition_1]
E.4 Finding Better Demo Selection Strategy
[Type_2]: [Definition_2]
The maximum input length of LLMs usually limits the sentence number in demos even under few- shot settings. For each test sentence we de- mand demo retriever E(D, s) which selects subset from D the sentences in demo. Following previous work, we consider three commonly-used strategies. (1) Random sampling. (2) Sentence- embedding (Liu et al., 2022; Su et al., 2022): re- trieving the top-K nearest sentences measured by sentence embedding. We compute the embeddings by SimCSE-ROBERTa-Lare (Gao et al., 2021).
[Type_N]: [Definition_N]
your annotation results must follow such
Please note that
format:
'Answer: ([Type_1] <SEP>
cenntified_eetttt:tie (Entity_1)), ([Type_2]
<SEP>
If you do not find any entity in this sentence, just output 'Answer: No entities found.
E (D, s) arg- aarg-topK sep (Setteemmeebbt s)] (3)
Regarding these six instructions, we evaluate their performance of ChatGPT on four 20-shot IE tasks. As shown in Table 12, there is no signifi- cant correlation between the instruction complexity
(3) Efficient Prompt Retriever (Rubin et al., 2022):
retrieving by a neural retriever R trained on D.
E(D,s) arg-topK (R((''
(4)

---

ChatGPT CODEX
60
60
56
FLscore
44
48
44
40
40
36
36
32
96
16
32
96
FeWNERD (NER)
TACREV (RE)
ACE05 (ED)
OpenAI LLMs
LLaMA (138)
Vicuna (13B)
28
24
20
12
16
FewNERD (NER)
TACREV (RE)
ACE05 (ED)
(b) Open-source LLMs
Figure 8: Relationship between demo number and Fl-score among three datasets. Note that the x-axis in each subfigure represents the number demos (not shot value K) during ICL. adopt sentence embedding as the demo selection strategy and text prompt in this experiment.
For each test sentence 8, we pre-retrieve M sim- ilar sentences D {( 1 C D. Then we score each sentence in 1 by their likelihoods (c(((())lf(()) where f denotes the prompt for- mat adopted and L the scoring LMM We randomly select positive samples s/(pos) from the top- KD sen- tences and hard negative samples of (hard-nee) from
Table 14; Fl Fl-scores across three prompt formats. Ex- periments run on 20-shot settings with ChatGPT.
FewNERD TACREV
ACE
ACE
(NER)
(RE)
(ED)
(EAE)
Text 57 6(2.3)
49.6.1.) 42.9(1.) 51.5(1.)
Code 53.2(0.9) 50.2(1.8) 44.3(2.0) 17.3(L.)
the bottom-K? Kp ones. Then we train Rp by in- batch contrastive learning (Chen al., 2020). For each sentence sf within the batch, there are posi- tive sentences s/ (pos) and 2B - negative sentences { (hard-neg) Jj=1 U { f Here we adopt M as 40, KD as 5, f as text prompt, the batch size B as 128, and the scoring LM L as FLAN-T5-X1.
E.5 Finding Better Prompt Format
Previous studies LLMs for few-shot IE tasks have explored different prompt formats and high- lighted the importance of selecting an appropri- format for achieving competitive performance. Therefore, we investigate two commonly-used vari- ants in previous work: (1) Text prompt as shown Figure 1. (2) Code prompt: We follow Wang et al. (2023b); Li et al. (2023) and recast the output of IE tasks in the form of code. See more details about this format in their original papers.
Table 13: F1-scores three demo-selection strategies. Experiments run 20-shot settings with ChatGPT.
NEWD TACREV
ACE
(NER)
(RE)
(ED)
Random Sampling
53.2(o.4;
43.0rs.))
38.0(1.))
Table 14 shows comparable performance across all formats. Based on simplicity, we choose the text prompt for our main experiment.
Sentence Embedding
57.6(2.3)
49.6(1.2)
42.9(1.))
Efficient Prompt Retriever
57.2(0.6)
48.0ro.8) 43.5(1.4
Table 13 demonstrates the Fl-score performance different selection strategies. We find that both the sentence embedding and EPR surpass random sampling by large margin. Given the simplicity the sentence embedding, we adopt it, rather than EPR, as our selection strategy in main experiment.
F
Case Study
F.1
Hard Samples
Table 15 showcases some hard examples which benefits from our LLM reranking. In accordance with our intuition, we observe that the LLM

---

rerankers correct two kinds of erroneous predic- tions made by LLMs. (1) The lack of external knowledge, such as first (Tripiolemus is a fig- ure in Greek mythology) and third examples (Mi- nas Gerais is a state instead of city). (2) Limited reasoning abilities, such as the second (His wife's 's children are his children) and the fourth (The word "fought" in this sentence does not involve any phys- ical violence) examples.
The sentence provides information about Flavius Rumoridus, who was a Roman soldier and appointed consul in AD 403. The sentence also mentions that he died in the 5th century AD, which is statement about his death. Therefore, the correct answer (c) because the sentence does not provide any information about Flavius Rumoridus being affiliated with any of the professions listed in option (c).
Analysis:
F.2
Easy Samples
Correct Answer: (c)
As discussed in Section 4.3, attribute the in- ferior performance of LLMs easy samples to their false- predictions for negative samples. Upon examining the incorrect cases, we speculate that two main reasons are responsible: (1) LLM hallucination and (2) span boundary mismatch. We provide details on them with specific cases below.
LLMs predict the word of as entity organization-other (Ground truth: None)
Case
Sentence: During these years he studied at the South Kensington Schools of the Royal College of Art.
LLM Hallucination. The most distinctive mis- takes made by LLMs. Without specific fine-tuning on IE tasks, LLMs (with in-context learning ap- proaches) sometimes hallucinate and provide mean- ingless, even confusing explanations or answers. For instance, LLMs might predict a verb, preposi- tion even special symbols as entity.
Candidate Choices:
(a) of do/does not belong to any known en- tities.
(b) of an educational institute.
(c) of pertains an organization that does not fall under the categories of com- pany, educational institution, govern- ment, media, political party, religion, sports league, sports team, band or mu- sical group.
Case 1: LLMs predict the word die as entity person-other (Ground truth: None)
Sentence: Flavius Rumoridus (died 5th cen- tury AD) Roman soldier who ap- pointed consul in AD 403 in the Western Ro- man Empire the same time as the eastern emperor Theodosius served in the same capacity in the East.
In the given sentence, of is a prepo- sition that indicates the relationship between South Kensington Schools and Royal College of Art. None of the candidate choices directly address the correct relationship between these entities or state that they are not applicable. Therefore, the correct answer (c): of does not pertain to any known entities mentioned in the choices provided.
Analysis:
Candidate Choices:
(a) died do/does not belong to any known entities.
(b) died is kind of geopolitical entity.
Correct Answer: (c)
(c) died is a person, but not affiliated with following professions: actor, artist, ath- lete, author, director, politician, scholar, soldier.
Span boundary mismatch. Different from sentence-level approaches which take one or mul- tiple sentences as an unit and assign each word's label globally, LLMs under our filter-tee--eerakk

---

paradigm adopt sample-level approaches. In other words, LLMs only consider each word/span candi- date independently. The sample-level approaches enable LLMs to rerank candidates with the help of multi-choice question prompt. However, they also aggravate the span boundary mismatching prob- lem in NER and ED tasks. We empirically find easy samples suffering much more from boundary mismatching problem than hard samples.
Sentence: Ralph Fiennes was nominated for the BAFTA Award for Outstanding Debut by a British Writer, Director Producer at the 65th British Academy Film Awards.
Candidate Choices:
(a) 65th do/does not belong to any known entities.
Case 3: LLMs predict Baron, a subspan of Baron Carl Hurleman, as the entity person-other (The ground-trun truth entity type of Baron is None. And the ground-truth entity type of Baron Carl Hurleman is person-arttan artist/author,
(b) 65th kind of award.
(c) 65th refers some event except attack, election, natural disaster, protest, revolu- tion and sports.
Sentence: Additions in the 1740s were by Baron Carl Hurleman (1700- 1753).
The sentence mentions the 65th British Academy Film Awards, which is an event that recognizes outstanding achieve- ments in the film industry. Therefore, option (c) is incorrect as it excludes events like award ceremonies. Option (a) is also incorrect the number 65th is associated with the event. Hence, the correct answer (b) as 65th refers to the specific edition of the BAFTA Awards.
Analysis:
Candidate Choices:
(a) Baron do/does not belong to any known entities.
(b) Baron is person, but not affiliated with following professions: actor, artist, ath- lete, author, director, politician, scholar, soldier.
Correct Answer: (b)
(c) Baron a kind of geopolitical entity
G Details Adaptive Filter-then-rerank Paradigm
Analysis: In the given sentence, Baron is a title given to a person named Carl Hurleman, who is mentioned as the one who made ad- ditions in the 1740s. The sentence does not provide any information about connections to geopolitical entities or any other known entities mentioned in the choices provided. Therefore, the correct answer is (b): Baron person, but not affiliated with following pro- fessions: actor, artist, athlete, author, director, politician, scholar, soldier.
G.1 Demonstration Examples
We convert few-shot IE tasks to multiple-choice questions our filter-then-rerankk paradigm. We show 4 examples used demonstrations for FewN- ERD dataset Table 16, for TACREV dataset Table 17, and for ACE05 datasets in Table 18.
G.2 Template
In our filter-then-reeank paradigm, we utilize tem- plates converting candidate labels to question op- tions. We list the template for FewNERD dataset in Table 19, for TACREV dataset in Table 20, and for ACE05 datasets in Table 21.
Correct Answer: (b)
Case 4: LLMs predict 65th, a subspan of 65th British Academy Film Awards, as the tity other-awards (The ground-truth entity type of 65th is None. And the ground-truth entity type of 65th British Academy Film Awards is other-awards).

---

Table 15: Examples of the samples corrected by LLM reranking. We sample four examples from NER, RE and ED tasks, respectively. Sentences: The sentences in which samples locate. We color the samples (entities trigger words) to be identified. Before: The prediction before LLM reranking. Based on SLM-based methods. After: The reranked prediction using LLMs. Rationales: LLM-generated Explanations.
Sentences (entity/event)
Before
After
Rationales
Named Entity Recognition
According it, King Eleusis and Cothonea (Cyntinia). parents of Trip- tolemus, are visited by Demeter, who rears their son, feeding him divine milk by day and placing him into the fire night, which makes Triptolemus grow faster than mortal children normally do.
person-other
other-god
According the context. Triptolemus is figure in Greek mythology. the son of King Eleusis and Cothonea (Cyntinia). He was demigod who mankind the of agriculture, So Triptole- mus is god in some legend/religous story.
X
sent by Demeter to teach
Relation Extraction
Survivors include his wife, Sandra; four sons, Jeff, James, Douglas and Harris; daughter, Leslie; his mother, Sally: and two hrothers, Guy and Paul.
None
per:children
The sentence 'Survivors include his wife, Sandra; four sons, Jeff, James, Douglas and Harris; daughter, Leslie; his mother, Sally; and two brothers, Guy and Paul.' This indicates "his" the father of Jeff, James, Douglas and Harris, and the husband of San- dra. So his parent Jeff.
Eliza Samudio, Bruno 's ex-girlfriend his 4-month-old baby boy's mother, disappeared from her hometown of Belo Horizonte, Minas Gerais.
The sentence states that "Eliza Samudio, Bruno's 's ex-girlfriend and his 4-month-old baby boy's mother. disappeared from hometown Belo Horizonte, Minas Gerais." This indicates that Samudio from city of Belo Horizonte, which located in the state Minas Gerais. Eliza Samudio lives the state or province Minas Gerais.
perciitiss of residence
perssttes of residence
Event Detection
We have fought every institution, every law firm it 's cost us a fortune but now we 'r going on a massive expansion.
ConflictAAtacck
None
In the given sentence, the word fought is used to describe the action of the speaker fighting against various institutions and law firms. This does not involve any physical violence or court proceedings, so the word fought does not trigger any known event.
X

---

Table 16: Demo examples used in FewNERD dataset. We color the entity blue.
Instruct: Read following sentences and identify what is the entity type of 392 quoted by <D>
Sentence: Powell v. Texas, <D> 392 <>> U.S. 514 1968 ). was United States Supreme Court case that ruled that Texas statute criminalizing public intoxication did not violate the Eighth Amendment protection against cruel and unusual punishment.
(a) 392 is legal document, term or convention in legal sense.
(b) 392 does not belong to any known entities.
(c) 392 refers to protest, uprising or revolution event
(d) 392 refers to govermment govemmental ageney
Analysis: In the context you provided, 392 refers to the volume number in the United States Reports where the Supreme Court's decision in Powell Texas can be found. However, 392 itself does not refer to legal document, So 392 doldoes not belong any known entities.
Answer:
Instruct: Read following sentences and identify what is the entity lype of The New Yorker quoted by <t> Sentence: In 2004 Gourevitch was assigned to cover the 2004 U.S. presidential election for <<>> The New Yorker <t>>> ". (a) The New Yorker does not helong to any known entities.
(b) The New Yorker is broadcast program.
(c) The New Yorker is kind of written art.
(d) The New Yorker is media/newspaper organization.
Analysis: The New Yorker a well-knows American magazine that has been published since 1925, and primarily known for its long- long-form journalism, commentary, and satire. has reputation for publishing high-quality writing
wide variety of topics, including politics, culture, and the arts. So The New Yorker is media/newssaper organization.
Answer: (d)
Instruct: Read following sentence and identify what is the entity type of St. quoted by <t>
Sentence: The May 1980 cruption of Mount <t> St. <t> Helens in the state of Washington seriously affected both 47th Air Division and 92d Bombardment Wing operations at Fairchild AFB, resulting in dispersal of Fairchild 's B-52 and KC-133 aircraft to various hases while mound-the-elock shifts removed the volcanic ash from facilities within the hase perimeter. (a) St. does not belong to any known entities.
(b) St. is natural disaster event.
(c) St. is geographic position about mountain.
Analysis: According the context, St. is abbreviation of Saint, used in the name of Mount St. Helens, which is active volcano in the state of Washington. However, St. itself does not refer anything. So St. doídoes not belong to any known entities.
Answer:
Instruct: Read following sentence and identify what is the entity type of Ridzuan quoted by <t>
Sentence: <t> Ridzuan <t> was promoted to Harimau Muda A for 2014 season
(a) Ridzuan does not belong to known entities.
(b) Ridzuan is person, hut not affiliated with following professions: actor, artist, author, director, politician, scholar, soldier.
(c) Ridzuan is athlete.
Analysis: The mention of 'Harimau Muda A' indicates that it is sports-related context, and 'promoted' implies progression advancement within the sports team. So Ridzuan athlete.
Answer:

---

Table 17: Demo examples used in TACREV dataset. We color the subject and object entities in blue.
Instruct: Read sentence and determine relation between she and lawyer quoted by <t>
Sentence: The lawyer <t> denied Italian news reports she wept while addressing court, but said Knox was upset as <D> she <D> recounted the pressure, the aggressiveness of police who called liar
(a) she the other family member of lawyer
(b) she a lawyer
(c) she no known relations lawyer
Analysis: the sentence, the word 'she' refers to someone who was upset while recounting certain events court. The word 'lawyer' refers to someone who denied a news report about that same person weeping court. There is no information in the sentence to indicate that the two individuals are related in any way. So she has no known relations to lawyer.
Answer: (c)
Instruct: Read sentence determine the relation between and Myanmar Equestrian Federation quoted by <D> Sentence: YANGON,
-LRB- Xinhua RRB- Myanmar will hold horse race Yangon to commemorate the country 63rd Anniversary Independence Day. the <D> Myanmar Equestrian Federation <D> -LRB- <D> MEF <D> -RRB- confirmed to Xinhua on Monday.
(a) MEF also known as Myanmar Equestrian Federation
(b) MEF has political affiliation with Myanmar Equestrian Federation
(c) MEF has no known relations to Myanmar Equestrian Federation
Analysis: The symbols -LRB- and -RBB in the sentence stand for left and right round brackets and are used to enclose the abbreviation MEF' to indicate that it is replacement for the longer name 'Myanmar Equestrian Federation. So MEF also known as Myanmar Equestrian Federation.
Answer: (a)
Instruct: Read sentence and determine relation between Douglas Flint chairman quoted by <t> Sentence: At same time. Chief Financial Officer <t> Douglas Flint <t> will become <t> chairman <D>, succeeding Stephen Green who leaving to take government job.
(a) Douglas Flint has no known relations to chairman
(b) Douglas Flint is a chairman
(c) Douglas Flint is the employee of chairman
Analysis: The sentence states that Chief Financial Officer Douglas Flint Douglas Flint will succeed Stephen Green as chairman, So Douglas Flint is chairman.
Answer: (b)
U.S. quoted by <D> Sentence: On Web site <t> U.S. <t> <t> FAA <t> says the Category rating means the country lacks laws or regulations that are needed for the certification oversight of air carriers, according to minimum international standards. (a) FAA is also known as U.S.
Instruct: Read sentence and determine relation between
(b) FAA has no known relations to U.S.
(c) FAA has a headquarter in the country U.S.
Analysis: The sentence states that the FAA says the Category rating means the country lacks the laws or regulations needed for the certification and oversight of air carriers, indicating that the FAA is responsible for overseeing aviation regulations in the country. Actually the FAA (Federal Aviation Administration) is U.S. government agency responsible for regulating and overseeing civil aviation in the United States, and it has its headquarters in Washington, D.C.. So FAA has headquarter in the country U.S.. Answer: (c)

---

Table 18: Demo examples used in ACEOS dataset. We color the trigger word in blue.
Instruct: Read following sentences and identify what event triggered by word loan quoted by <t>, Sentence: Separately former WorldCom CRO Bernard Ebbers failed on April make first repayment million dollars, plus interest, on a 400-million dollar <D> loan <t> from MCI, the Journal said, citing SEC documents. (a) The word Joan does not trigger any known event.
(b) The word loan triggers a COONSSERR-ONNESS event: giving. receiving, borrowing, or lending money when is NOT
the context of purchasing something.
(c) The word loan triggers
BELLLRRR-AAKKRPTCY event: an Entity officially requests legal protection from deht
collection due to an extremely negative balance sheet.
Analysis: In the given sentence, the word loan is used to describe the 400-million-dolla. loan from MCI to former WorldCom CEO Bernard Ebbers, which he failed to repay on time. This situation clearly involves transfer of money without the context of purchasing something, which falls under the TTRANSFERMMORN triggers a TRANSFER-MONE MONEY event: giving, receiving, borrowing, or lending money when it is NOT in the context of purchasing something. Answer: (b)
event. So the word loan
Instruct: Read following sentences and identify what event is triggered by words treated quoted by <D> Sentence: When 's Germany Lynch will be <t> treated <t> for bullet wounds and broken bones (a) The word treated triggers an INJURE event: a PERSON gets/got injured whether it occurs accidentally, intentionally or even self-inflicted.
(b) The word treated does not trigger any known event.
(c) The word treated triggers TRANSPORT event: an ARTIFACT (WEAPON or VEHICLE) or a PERSON moved from one PLACE ITCCLL ENTITY, FACILITY, LOCATION) to another.
Analysis: The sentence suggests that Lynch has already been injured and will receive medical treatment in Germany for her injuries. The word 'treated' simply describes the medical care she will receive and does not indicate a new event or action taking place. So the word treated does not trigger any known event. Answer: (b)
Instruct: Read following sentences and identify what event is triggered by words buy quoted by <t> Sentence: And won't 't dwell on the irony of an Oracle employee being driven out Oracle, starting his own company. and forcing Ellison to spend 10.3 billion to get his company- - but not him back ( though it does rather delightfully remind me of Coca - Cola basically giving away the bottling franchise and then spending billions to <t> buy <t> it back ) (a) The word buy triggers a DECI ARR. BANKRUPTCYY event: an Entity officially requests legal protection from deht collection due to an extremely negative balance sheet.
(b) The word buy triggers a TR TRANNFER- OWNERSHIP event: The buying, selling, loaning, borrowing, giving, or receiving of artifacts or organizations by an individual or organization.
(c) The word buy does not trigger any known event.
Analysis: In the given sentence, the word buy is used to describe the action of Oracle spending $10.3 billion to get a company back. This clearly involves the transfer of ownership of the company from one entity to another. So the word buy triggers a TRANSFER BRANSFER-OWNERSS -W-ERSSIIPPPPPPP event: The buying. selling, loaning, borrowing, giving, or receiving of artifacts or organizations by an individual or organization.
Answer: (b)
Instruct: Read following sentences and identify what event is triggered by the words set quoted by <t>
Sentence: British forces also began establishing the country's 's first postwar administration Tuesday, granting a local sheik power to <D> set <D> up an administrative committee representing the groups in the region.
(a) The word set triggers a START- TERTT-OOOIIONNN event: a PERSON elected or appointed begins working for (or changes
offices within) an ORGANIZATION or GOVERNMENT.
(b) The word set triggers a START-ORG event: new ORGANIZATION is created.
(c) The word set does not trigger any known event.
Analysis: The phrase set up" specifically implies the creation or establishment of new organization or entity, rather than simply the word 'set'. So the word set does not trigger any known event.
Answer: (c)

---

Table 19: Templates for FewNERD dataset, where (ent) the placeholder for entity type.
Entity
Template
no-entity
{ent) do/does not helong to any known entities.
(ent) an artist or author.
person-actor
{ent)
art art-writtenart
{ent} kind of writtenart.
person-director
{ent) director.
person-other
(ent) person, but not affiliated with following professions: actor, artist, athlete, author, director, politician, scholar, soldier.
organization-othe
(ent) pertains to an organization that does not under the categories company, educational institution, government, media, political party, religion, sports league, sports team, band or musical group.
organization-company
(ent) company
organization-spootsteann
(ent) is sports team
orgamization-sportstsqquu
(ent) sports league
product-car
(ent) kind car
event-protest
(ent) refers protest, uprising revolution event
organization-
{ent] refers government governmental ageney
swverneenigoveemmenttgnnn
other-bioloyything
(ent) is special term about biology / life science.
location-GPE:
(ent) kind geopolitical entity
location-other
(ent) geographic locaton that does not fall under the categories geopolitical
entity, body of water, island, mountain, park, road, railway and transit.
person athleee
{ent} is athlete coach.
art- broadcastprogram
(ent) broadcast program.
product-other
(ent) a kind of product that does not fall under the categories of airplane, train, ship, car, weapon, food, electronic game and software.
building-other
(ent) kind building that does not fall under the categories of airport, hospital, hotel, library, restaurant, sports facility and theater
product-weapon
(ent) is kind of weapon.
building-airport
(ent) is airport.
building-sportsfacclltyy
(ent) sports facility building.
person-scholar
(ent) scholar.
art-music
(ent) music.
event-other
[ent] refers to some event except attack, election, natural disaster, protest, revolution and sports
other-language
(ent) kind of human language.
other-chemicalthing
(ent) some special term about chemical science.
art-film
[ent) film.
hosspitl
(ent) hospital.
other-law
(ent) legal document, term or a convention in legal sense.
product-airplane
(ent) is kind of airplane product.
location-
[ent) geographic position about roadways, railways, highways public transit systems.
rod//ailvayhhgghwaytassss
person-soldier
(ent) soldier
location location-mountain mountain
(ent) is geographic position about mountain.
organization- edduattioonn
(ent) an educational institutelorganization.
oggaizatioonmmediineeee
(ent) media/newwsppper organization.

---

product-software
(ent) is software product.
location-island
(ent) geographic position about island.
watee
[ent) geographic position situated near body water.
Hallding-librrrg
(ent) library.
other-astromomythhngg
[ent] special term about astronomy.
person-politician
(ent) is politician or lawyer judge.
building-hotel
(ent) is a hotel building.
product-game
(ent) a electronic game product.
other-award
[ent] kind of award.
event-sportsevent (ent) refers some event related sports.
orgamization-shooorraaizaaiionn
[ent] is a band or musical organization.
other-educattionaldegre
(ent) a educational degree.
building-theater
[ent) theater.
other-disease
[ent] kind of disease.
event-clection
[ent] is an event about election.
ugganizatioonpoliiicaaaar
(ent) a political partylorganization.
other-currency
[ent) kind of currency.
event-
[ent] is event about attack, hattle, war or military conffict.
product-ship
(ent) is ship.
building-restauran restaurant
(ent) a restaurant.
other- liingthivng
(ent) a living aninal/creature//gaaniss
art-other
[ent] is work of art, but belong to the categories of music, film, written broadcast painting.
event -ent-disaster
(ent) is natural disaster event.
organizationn-reliion
(ent) a religious organization.
other-medical
(ent) refers some kind callicine.entig
location-park
[ent] park.
other-god
(ent) is god in some legend/religious story.
product-food
(ent) is a kind of food.
product-train
(ent) a kind of train(vehicle).
art-painting
[ent] painting.

---

Table 20: Templates for TACREV dataset, where (subj} and (obj) are the placeholders for subject and object entities. Copied from (Lu et al., 2022a)
Relation
Template
no. relation
{subj} has no known relations [obj)
per:stateorrrovince.. of death
(subj) died in the state province (obj)
per:title
[subj] is [obj]
org:member
(subj) the member of [obj)
per:other_famme
[subj} the other family member of [ohj]
org:country. of headquarters
[subj} has headquarter in the country [obj)
org:parents
[subj] has the parent company {obj}
persstatcorrroovice. of birth
(subj) was born in the state or province [obj]
per:spouse
[subj} the spouse {obj)
per:origin
[subj} has the nationality [obj]
per:date_od of_ birth
[subj] has birthday on {obj}
per:schools, atteedd
(subj] studied in [obj]
org:members
(subj} has the member [obj)
org:founded
[subj] founded in [obj]
(subj} lives in the state province (obj)
per:date_ sfreisoo____ee death
(subj] died in the date (obj)
org:shareholders
(subj) has shares hold in [obj]
org:wehsite
{subj} has website (obj)
org:subsidiaries
(subj) owns (obj)
per:charges
(subj) is convicted of [obj]
org:dissolved
(subj) dissolved [obj]
[subj} headquarter the state or province (obj)
per:country of birth
(subj) born in the country (obj)
per:siblings
(subj) is the siblings of (obj)
org: membens/employees
(subj) has the high level member (obj)
per:cause_off death
(subj) died because (obj)
per:altemate names
(subj) has the alternate name (obj)
org:number_.om _fraa_oo______ empooyee///meebeerr
(subj) has the number of employees (obj)
per:cities of residence
(subj) lives in the city [obj)
org:city_ood headquarres
(subj) headquarter the city (obj)
per:children
(subj} the parent of (obj)
per:employee of
(subj) is the employee of (obj)
org:politicallllliiooss affiliation
(subj) has political affiliation with [obj]
per:parents
(subj} parent (obj)
per:eity_of birth per:age
(subj} born in the city [ohi]
(subj) has the age (obj)
per:countries_ of residence
[subj] lives in the country [obj]
org:alternte names
(subj} also known as (obj)
per:religion
(subj} has the religion {obj]
per:eity_or of death
(subj] died in the city (obj)
per:country_of__ _dchhhhtthhhhhhhho
(subj) died in the country {obj]
org:founded_ by
{subj} was founded by [ohi)

---

Table 21: Templates for ACE05 dataset, where (evt] is the placeholder for event type.
Event
Template
no-event
The word [ovt] does not trigger any known event.
Mrnsooot.raanport
The word {evt) triggers a TRANSPORT event: an ARTIFACT (WEAPON or VEHICLE) or a PERSON is moved from one PLACE (GEOPOLITICAL ENTITY, FACILITY, LOCATION) to another.
EesoonelEEeec
The word (evt) triggers an ELECT event which implies an election.
Personnel Position
The word [evt) triggers begins working for (or changes offices within) ORGANIZATION or GOVERN- MENT.
event: PERSON elected or appointed
Personnel.ommnntee
The word [evt] triggers a NOMINATE event: PERSON is proposed for position through official channels.
Conflict.Attack
The word (evt) triggers an ATTACK event: a violent physical act causing harm or damage.
Personnel. Ennd-Postioon
The word [evt] triggers an END-POSITION event: a PERSON stops working for (or changes offices within) an ORGANIZATION or GOVERNMENT.
Mentact.Meet
The word [evt] triggers a MEET event: two or more entities come together at a single location and interact with one another face-to-face.
Life.Marry
The word [evt] triggers MARRY event: people are married under the legal definition.
Contact.Phone-
The word (evt) triggers a PHONE-WRITE event: two or more people directly engage in discussion which does not take place 'face-to-face..
Transaction. TransferMMoney
The word (evt) triggers TRANSFER-MONNE
event: giving, receiving. borrowing, or lending money when it is NOT in the context of purchasing something.
Justice_Sue
The word [evt] triggers SUE event: a court proceeding has been initiated for the purposes of determining the liability of a PERSON, ORGANIZATION or GEOPO- LITICAL ENTITY accused of committing a crime or neglecting commitment
Conflict.emmostter
The word [evt] triggers DEMONSTRATE event: large number of people come together public area to protest demand some sort of official action. For cample: protests, sil ins, strikes and riots.
Business.End-OG End-Org
The word [evt] triggers an END-ORG event: ORGANIZATION ceases exist (in other words, goes out of business).
Injure
The word [evt] triggers an INJURE event: PERSON gets/got injured whether it occurs accidentally. intentionally or even self-infficted.
Dife.Die
The word [evt] triggers a DIE event: a PERSON dies/died whether it occurs acci- dentally, intentionally or even self-inflicted.
Justice.Anest-UU
The word (evt) triggers a ARREST-JAIL event: a PERSON is sent to prison.
Transaction. Transfer-- Ownership
The word [evt] triggers loaning, borrowing, giving, or receiving of artifacts organizutions by individual or organization.
event: The buying, selling,
Justice.Fxecute
The word [evt] triggers an EXECUTE event: a PERSON is/was executed
Justice. Trial- Trial-Hanni
The word [evt] triggers a TRIAL-HEARING event: a court proceeding has been initiated for the purposes of determining the guilt or innocence of a PERSON, ORGANIZATION or GEOPOLITICAL ENTITY accused of committing a crime.
Sentece
The word (evt) triggers a SENTENCE event: the punishment for the DEFENDANT is issued
Life. Be-rrBe
The word (evt) triggers a BE-BORN event: a PERSON given birth to.
Justice.Charge-Bee
The word (evt) triggers CHARGE- INDICTTTIITTTTTEEEEE event: PERSON, ORGANIZATION or GEOPOLITICAL ENTITY accused of crime
Business. Start-Org
The word (evt) triggers a START-ORG event: new ORGANIZATION is created.
Justice.Convia
The word (evt) trigges a CONVICT event: a PERSON, ORGANIZATION or GEOPOLITICAL ENTITY is convicted whenever it has been found guilty of a CRIME.
Business. BankerpBBnkkrtyyy
The word [ev]] triggers a DECLARE..RR PORF-BANKKUPCCY event: an Entity officially requests legal protection from debt collection due to an extremely negative balance sheet.
Justice.Release-PP
The word [evt] triggers HELEASE-PPROLL event.

---

Justice.Finne
The word (evt) triggers FINE event: a GEOPOLITICAL ENTITY, PERSON ORGANIZATION get financial punishment lypically as result of court proceedings.
Mustice.Pardon
The word [evt) triggers PARDON event: head-of-state or their appointed repre- sentative lifts a sentence imposed by the judiciary.
Justice.Appeal
The word (evt) triggers a APPEAL event: the decision a court taken to higher court for review
Business Mrrge-Orr
The word [evt] triggers MERGE-ORG Entities come together form ORGANIZATION Entity.
more ORGANIZATION
Justice.Extraid
The word [evt] triggers EXTRADITE event.
Life.Divorce
The word [evt] triggers a DIVORCE event: two people are officially divorced under the legal definition of divorce.
Acquit
The word (evt) triggers ACQUIT event: trial ends but fails to produce a convic- tion.


Reviews:
[{"review_id": "knMkreixLA", "reviewer": "Reviewer_A8RC", "summary": "", "questions": "Please refer to the Reasons To Reject.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your reviews! Your professional reviews offer us great advice towards writing a more comprehensive and competitive paper! And, we appreciate you finding our empirical study thorough and our proposed filter-then-rerank paradigm insightful.\n\n-----------------\n\n> Q1: The instructions used in this paper are simple. Would this directly affect the performance of LLMs on information extraction tasks? Did the authors try other complex instructions?\n\n1. We fully understand your concern that the prompt, especially the instructions, might significantly affect the performance of LLMs on IE tasks. To investigate whether LLMs would benefit from complex instructions, we explored six instruction variants from simple to complex as below. We briefly introduce them as below and show their formats in detail at the bottom of our response.\n\n\n\n    **I0**: No instruction.\n\n    **I1**: Simple instruction, used in our main experiment.\n\n    **I2**: Instruction with label explanation: We concatenate the explanation/definition of each label after their names in the instruction.\n\n    **I3**, **I4**: Instruction with format explanation: We provide two more detailed instructions about the task and the input/output format. These two instructions are written by two authors respectively. \n\n    **I5**: Instruction with both label and format explanation: We combine I2 and I3 to a more complex instruction.\n\n\n    Regarding these six instructions, we evaluate the performance of ChatGPT (gpt-3.5-turbo-0301) on four 20-shot IE tasks. We conduct 3-run experiments and report their mean values and standard deviations (in bracket).\n\n    | | FewNERD (NER)  | TACREV (RE) | ACE (ED) | ACE (EAE) |\n    | --- |  ----  | ----  |  ----  | ----  |\n    | I0 | 57.6 (2.1) | 49.1 (2.4) |  44.0 (1.4) | 50.9 (0.1) |\n    | I1 | 58.3 (0.5)  | 49.6 (1.2) |  42.6 (1.0)  | 51.5 (1.1)  |\n    | I2 | 57.7 (1.0) | 50.0 (1.5) | 41.8 (0.9) | 50.3 (1.5) |\n    | I3 | 57.6 (2.3)  | 52.3 (1.8) |  42.9 (1.3)  | 49.2 (2.3)  |\n    | I4 | 56.8 (0.9)  | 49.6 (2.9) |  41.6 (1.9)  | 49.9 (1.2)  |\n    | I5 | 57.8 (0.5)  | 47.2 (1.7) |  43.1 (1.8)  | 50.6 (1.8)  |\n\n    We observe that there is **no significant correlation between the instruction complexity and the LLMs’ performance**. Even the prompt without instruction (I0) leads to comparable even better performance than prompt with complex instructions.\n\n2. As we state in lines 196-210, the prompt consists of instruction, demonstrations and the input query. **From our previous experience, the demonstration is the more determined factor than the instruction for the LLMs' performance under few-shot IE task settings.** Therefore, we additionally conduct a series of pivot experiments to find the (nearly) optimal demonstration number, selection strategy and format. We detail them in Appendix D.3, D.4 and D.5 respectively. We adopt the best combination we found in the main experiments. \n3. In summary, we work hard to find the best prompts, in terms of both their instructions, demonstrations and formats, to evaluate LLMs’ IE abilities. As an empirical study, we acknowledge in the Limitations (lines 509-518) that we do not, and have no possibility, to enumerate all prompt (either instruction or demonstration) combinations. But we believe there unlikely exists a lottery prompt which largely improves the LLMs’ performance and alters our conclusion that LLM performs poorly on few-shot IE tasks.\n\nInstruction 0 (Empty instruction)\n```\n```\n\nInstruction 1 (Simple instruction. Take NER task as an example, the same below.)\n```\nIdentify the entities expressed by each sentence, and locate each entity to words in sentence. The possible entity types are:\n[Type_1], [Type_2],.., [Type_N]. \nIf you do not find any entity in this sentence, just output '''Answer: No entities found.'''\n```\n\nInstruction 2 (+ label explanation)\n```\nIdentify the entities expressed by each sentence, and locate each entity to words in sentence. The possible entity types are:\n[Type_1]: [Definition_1]\n[Type_2]: [Definition_2]\n..\n[Type_N]: [Definition_N]\nIf you do not find any entity in this sentence, just output '''Answer: No entities found.'''\n```\n\nInstruction 3 (+ format explanation v1)\n```\nAssume you are an entity-instance annotator. Given a sentence, you need to (1) identify the word or phrase about the entity in the sentence, and (2) classify its entity type. The possible entity types are listed as below:\n[Type_1], [Type_2], …, [Type_N].\nPlease note that your annotation results must follow such format: '''Answer: ([Type_1] <SEP> identified_entity:[Entity_1]), ([Type_2] <SEP> identified_entity:[Entity_2]), ......'''. \nIf you do not find any entity in this sentence, just output '''Answer: No entities found.'''\n```\n\nInstruction 4 (+ format explanation v2)\n```\nAssume you are an entity-instance annotator. Your objective is to peform a series of intricate steps for Named Entity Recognition. Firstly, you have to identify a particular word or phrase in the sentence that corresponds to an entity. Following this, classify the entity into one of the potential entity types. The potential entity types are provided as below:\n[Type_1], [Type_2], …, [Type_N].\nPlease note that your annotation results must follow such format: '''Answer: ([Type_1] <SEP> identified_entity:[Entity_1]), ([Type_2] <SEP> identified_entity:[Entity_2]), ......'''. \nIf you do not find any entity in this sentence, just output '''Answer: No entities found.'''\n```\n\nInstruction 5 (+ label&format explanation)\n```\nAssume you are an entity-instance annotator. Given a sentence, you need to (1) identify the word or phrase about the entity in the sentence, and (2) classify its entity type. The possible entity types are listed as below:\n[Type_1]: [Definition_1]\n[Type_2]: [Definition_2]\n..\n[Type_N]: [Definition_N]\n'''\nPlease note that your annotation results must follow such format: '''Answer: ([Type_1] <SEP> identified_entity:[Entity_1]), ([Type_2] <SEP> identified_entity:[Entity_2]), ......'''. \nIf you do not find any entity in this sentence, just output '''Answer: No entities found.'''\n```\n\n-----------------\n\n-----------------\n\n> Q2: Are there any quantitative analyses to support that LLMs struggle with fine-grained dataset?\n\nThanks for your constructive suggestion. Based on the results shown in Figure 2, we additionally provide a quantitative analysis to show that LLMs struggle with fine-grained datasets. Under the 5-shot setting, we compare the performance difference of LLMs (ChatGPT) and SLMs (SoTA few-shot models for each IE task) among different datasets. For each IE task, we observe **a clear negative correlation between the label number (row 2) and the performance difference (row 5)**. In other words, with more label types, **LLMs tend to perform relatively worse than SLMs**. Therefore we conclude that LLMs struggle on fine-grained datasets.\n\n**NER (5-shot, SLM: FSLS, LLM: ChatGPT)**\n|---| CoNLL | OntoNotes | FewNERD |\n|---|---|---|---|\n|#Entity| 4 | 18 | 66 |\n|SLM Performance | 52.5| 59.7 | 59.4 |\n|LLM Performance | 77.8 | 59.4 | 55.5 |\n|$\\Delta$ (LLM, SLM)  | 25.3 | -0.3 | -3.9 |\n\n\n**ED (5-shot, SLM: FSLS, LLM: ChatGPT)**\n|---| ACE05 | ERE | MAVEN |\n|---|---|---|---|\n|#Event| 33 | 38 | 168 |\n|SLM Performance | 55.1 | 48.0 | 49.4 |\n|LLM Performance | 39.6 | 33.8 | 25.3 |\n|$\\Delta$ (LLM, SLM)  | -15.5 | -14.2 | -24.1 |\n\n**EAE (5-shot, SLM: PAIE, LLM: ChatGPT)**\n|---| ACE05 | ERE | RAMS |\n|---|---|---|---|\n|#Event/#Role| 33/22 | 38/26 | 139/65 |\n|SLM Performance | 45.9 | 40.4 | 54.1 |\n|LLM Performance | 52.8 | 40.7 | 44.2 |\n|$\\Delta$ (LLM, SLM)  | 6.9 | 0.3 | -9.9 |\n\n---\n\nWe hope that our explanations and newly-added experiments have addressed your concern about (1) the effect of instructions on LLMs' performance and (2) the quantitative analysis on LLMs' struggling on fine-grained datasets. If not, would you mind telling us your remaining concerns? We would try our best efforts to further address your any concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper is well written and easy to understand.\n2. The authors thoroughly discussed why LLMs have unsatisfactory performance on information extraction tasks.\n3. The proposed filter-then-rerank paradigm is simple but insightful.\n4. The experiments are thorough and effectively validate the author's hypotheses. Moreover, experiment results demonstrate hat the proposed method is effective.", "reasons_to_reject": "1. The instructions used in this paper are simple. Would this directly affect the performance of LLMs on information extraction tasks? Did the authors try other complex instructions? \n2. The authors argue that LLMs struggle with fine-grained datasets. Are there any quantitative analyses?", "questions_to_authors": "Please refer to the Reasons To Reject.", "paper_topic_and_main_contributions": "The authors show that LLMs are not effective few-show information extractors in general, but can well complement SLMs and effectively solve hard samples that SLMs struggle with. They propose an adaptive filter-then-rerank paradigm, which utilizes LLMs to rerank a small portion of difficult samples identified by SLMs.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EuTFzZHSqy", "reviewer": "Reviewer_ifw3", "summary": "", "questions": "- Could you add detailed statistics around your test subsets? I know you downsampled randomly but I would like to know the relation percentage as well as the negative vs positive ratio in the different datasets (previous work has found that this is very important for IE performance). ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your reviews! Your professional reviews offer us great advice towards writing a more comprehensive and competitive paper! And, we appreciate you finding our empirical study covering broadly and our proposed filter-then-rerank paradigm practical.\n\n-----------------\n\n> Q1: The authors did not do a more thorough analysis on why LLMs are failing on easy samples. A case study on this or at least some deeper discussion on this topic could greatly improve the paper.\n\nThanks for your constructive suggestion. We summarize the reason for LLMs' failure on easy samples from two aspects: (1) SLMs’ remarkable and better performance on easy samples. (2) LLMs’ predisposition to false-positive predictions on negative samples (most of which are easy samples). The tendency towards false-positive predictions, specifically, are due to (a) LLM hallucination and (b) Span-boundary mismatch. We provide more detailed analysis and case studies as below:\n\n1. The performance of LLMs (red lines in Figure 4, the same below) are more stable than that of SLMs (blue lines) among samples with various difficulties (LLMs’ curves are more flatten). Specifically, SLMs trail LLMs on hard samples (left parts of the figures) but gradually surpass LLMs on easier samples (right parts of the figures). Therefore, the LLMs’ failure on easy samples are largely **relative**. In other words, we could find a better alternative, i.e. SLM-based methods, to solve the hard samples.\n2. On NER and ED tasks, however, we indeed observe **absolute** performance drops when the samples are extremely easy (see the blue curves’ drops in the first and third subfigures at the very right parts in Figure 4). Since NER and ED are sequence-labeling tasks, i.e. determining the label type of every word/span in the sentence, almost all samples (words/spans) are actually negative samples. Also, **most of these negative samples are extremely easy for SLMs** (see the last table in Q2, in which the sample-level negative-positive ratios in easy samples are much higher than hard samples). Due to LLMs’ **predisposition to false-positive predictions on negative samples**, the performance of LLMs on easy samples collapses. Specifically, we attribute such false-positive predictions to two main reasons: (1) LLM hallucination and (2) span boundary mismatch. We take NER task as an example and detail such two kinds of mistakes with cases as below:\n\n    2(a). **LLM Hallucination**: the most distinctive mistakes made by LLMs. Without specific fine-tuning on IE tasks, LLMs (with in-context learning approaches) sometimes hallucinate and provide meaningless, even confusing explanations or answers. For example, LLMs might predict a verb, preposition or even special symbols as an entity. Here are two cases:\n\n    Case (1): LLMs predict the word ```die``` as entity ```person-other``` (False-positive prediction, ground truth should be ```None```)\n    ```\n    Sentence: Flavius Rumoridus ( died 5th century AD ) was a Roman soldier who was appointed consul in AD 403 in the Western Roman Empire at the same time as the eastern emperor Theodosius II served in the same capacity in the East .\n    Candidate Choices: \n    (a): died do/does not belong to any known entities.\n    (b): died is a kind of geopolitical entity\n    (c): died is a person, but not affiliated with following professions: actor, artist, athlete, author, director, politician, scholar, soldier.\n\n    Analysis: The sentence provides information about Flavius Rumoridus, who was a Roman soldier and appointed consul in AD 403. The sentence also mentions that he died in the 5th century AD, which is a statement about his death. Therefore, the correct answer is (c) because the sentence does not provide any information about Flavius Rumoridus being affiliated with any of the professions listed in option (c).\n    Correct Answer: (c)\n    ```\n\n    Case (2): LLMs predict the word ```of``` as entity ```organization-other``` (False-positive prediction, ground truth should be ```None```)\n    ```\n    Sentence: During these years he studied at the South Kensington Schools of the Royal College of Art .\n    Candidate Choices: \n    (a): of do/does not belong to any known entities.\n    (b): of is an educational institute/organization.\n    (c): of pertains to an organization that does not fall under the categories of company, educational institution, government, media, political party, religion, sports league, sports team, band or musical group.\n\n    Analysis: In the given sentence, \\\"of\\\" is a preposition that indicates the relationship between \\\"South Kensington Schools\\\" and \\\"Royal College of Art.\\\" None of the candidate choices directly address the correct relationship between these entities or state that they are not applicable. Therefore, the correct answer is (c): \\\"of\\\" does not pertain to any known entities mentioned in the choices provided.\n    Correct Answer: (c)\n    ```\n\n    2(b). **Span boundary mismatch**: Different from sentence-level approaches which take one or multiple sentences as an unit and assign each word’s label globally, LLMs under our filter-then-rerank paradigm adopt sample-level approaches (lines 326-330). In other words, LLMs only consider each word/span candidate independently. The sample-level approaches enable LLMs to rerank candidates with the help of multi-choice question prompt. However, they also aggravate the span boundary mismatching problem in NER and ED tasks. We empirically find easy samples suffering much more from boundary mismatching problem than hard samples. Here are two cases:\n\n    Case (3): LLMs predict  ```Baron```, a subspan of ```Baron Carl Hurleman```, as the entity ```person-other```. It is a false-positive prediction, the ground-truth entity type of ```Baron``` is ```None```. And the ground-truth entity type of ```Baron Carl Hurleman``` is ```person-artist/author```.\n    ```\n    Sentence: Additions in the 1740s were by Baron Carl Hurleman ( 1700 - 1753 ) .\n    Candidate Choices: \n    (a): Baron do/does not belong to any known entities.\n    (b): Baron is a person, but not affiliated with following professions: actor, artist, athlete, author, director, politician, scholar, soldier.\n    (c): Baron is a kind of geopolitical entity\n\n    Analysis: In the given sentence, \\\"Baron\\\" is a title given to a person named Carl Hurleman, who is mentioned as the one who made additions in the 1740s. The sentence does not provide any information about connections to geopolitical entities or any other known entities mentioned in the choices provided. Therefore, the correct answer is (b): Baron is a person, but not affiliated with following professions: actor, artist, athlete, author, director, politician, scholar, soldier.\n    Correct Answer: (b)\n    ```\n    \n    Case (4): LLMs predict ```65th```, a subspan of ```65th British Academy Film Awards```, as the entity ```other-awards```. It is a false-positive prediction, the ground-truth entity type of ```65th``` is ```None```. And the ground-truth entity type of ```65th British Academy Film Awards``` is ```other-awards```.\n\n    ```\n    Sentence: Ralph Fiennes was nominated for the BAFTA Award for Outstanding Debut by a British Writer , Director or Producer at the 65th British Academy Film Awards .\n    Candidate Choices: \n    (a): 65th do/does not belong to any known entities.\n    (b): 65th is a kind of award.\n    (c): 65th refers to some event except attack, election, natural disaster, protest, revolution and sports\n\n    Analysis: The sentence mentions the \\\"65th British Academy Film Awards,\\\" which is an event that recognizes outstanding achievements in the film industry. Therefore, option (c) is incorrect as it excludes events like award ceremonies. Option (a) is also incorrect as the number \\\"65th\\\" is associated with the event. Hence, the correct answer is (b) as \\\"65th\\\" refers to the specific edition of the BAFTA Awards.\n    Correct Answer: (b)\n    ```\n\nBased on the analysis above, we would add an extra subsection in Section 4 to discuss why LLMs work on hard samples but fail on easy samples. Specifically, we would combine the current Section 5.6 (LLMs on hard samples, with cases in Appendix E.1) and our response for Q1 (LLMs on easy samples). We hope that this new subsection (1) illustrates clearer about the advantages and disadvantages of LLMs and SLMs, and (2) provides a more meticulous and comprehensive motivation explanation of our proposed filter-then-rerank paradigm.\n\n--------------------------\n-----------------\n\n> Q2:  Could you add detailed statistics around your test subsets? I know you downsampled randomly but I would like to know the relation percentage as well as the negative vs positive ratio in the different datasets (previous work has found that this is very important for IE performance).\n\nWe appreciate your insightful suggestion. We provide more detailed statistics about the test subsets as below, including their (1) sentence numbers and (2) sample (entity/relation/trigger word/argument for different IE tasks) numbers as below. \n\n| | NER-CoNLL | NER-OntoNotes | NER-FewNERD | RE-TACREV | RE-TACRED | ED-ACE | ED-ERE | ED-MAVEN | EAE-ACE | EAE-ERE | EAE-RAMS |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| # Sentence| 250 | 250 | 250 | 500 | 500 | 250 | 250 | 250 | 250 | 250 | 250 |\n| # Sample | 388 | 342 | 745 | 376 | 361 | 151 | 176 | 689 | 395 | 731 | 589 |\n\nRegarding negative-positive ratios (NP ratios; we also view the relation percentage in RE task as a special negative-positive ratio), we consider two variants, i.e. (1) sentence-level and (2) sample-level NP ratios. They correspond to the scenarios of our main empirical studies (Section 3) and filter-then-rerank paradigm (Section 4 and 5), respectively. We investigate their effects on LLMs' performance. Briefly speaking, we do observe the remarkable relations of **sample-level** NP ratios with regard to sample difficulty. As mentioned in Q1, the high NP ratios in easy samples and the LLMs' tendency towards false-negative predictions largely cause LLMs failing on easy samples. We detail these two kinds of NP ratios as below:\n\n1. **Sentence-level**: If a sentence does not include any positive labels, we view this sentence a sentence-level negative sample. Otherwise it is a sentence-level positive sample. We show the sentence-level NP ratios of all datasets we used in our empirical studies in Section 3 as below.\n\n    *When downsampling test subsets, we ensure that each label has at least one corresponding sample to avoid the absence of rare labels in our evaluation.*\n    | | NER-CoNLL | NER-OntoNotes | NER-FewNERD | RE-TACREV | RE-TACRED | ED-ACE | ED-ERE | ED-MAVEN | EAE-ACE | EAE-ERE | EAE-RAMS |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    | Sentence-level NP ratio | 0.30 | 1.03 | 0.13 | 3.03 | 2.59 | 1.35 |1.25 | 0.03 | 0.23 | 0.02 | 0.02 |\n\n    We found that there are **no significant correlations between the negative-positive ratio of the datasets and their results on LLM evaluation**. For example, both the CoNLL and MAVEN have few negative sentence-level samples. But LLMs perform much better on CoNLL than on MAVEN (see Figure 2). Compared with other factors such as task difficulty and/or label number, the sentence-level NP ratios are not a determining factor to affect the IE performance on LLMs.\n    However, we admit the importance of the sentence-level NP ratios. To decouple the other confounders among different datasets, we plan to curate multiple test subsets, having different sentence-level NP ratios, for each dataset. We hope that the results on these different test subsets from single original dataset could better reveal the relation of LLMs' IE performance with regard to different sentence-level NP ratios. We promise to have a deeper analysis about the effect of sentence-level NP ratios in our updated version.\n\n2. **Sample-level**: For NER and ED tasks, each sentence could have zero, one or more entities / trigger words. Therefore LLMs in our filter–then-rerank paradigm adopt another prompt strategy: they predict a single entity/event candidate once a time (lines 326-330).  We call such word/span a sample-level negative sample if it were not any entities or trigger words. Otherwise it were a sample-level positive sample. Given two NER and ED datasets used in Section 4 and 5, we show sample-level NP ratios on them and their easy/hard subsets as below. We set fixed threshold $\\tau$ here for simplicity.\n   \n    | | NER-FewNERD | ED-ACE |\n    | --- | --- | --- |\n    | Overall | 5.88 | 38.2\n    | (very) Easy samples ($\\tau$ > 0.9) | 9.44 | 44.0 |\n    | Hard samples ($\\tau$ < 0.6) | 1.28 | 1.36 |\n\n    We observe that **the sample-level NP ratios of easy samples are much higher than that of hard samples**. As we mentioned in Q1, LLMs tend to provide false positive predictions (more likely failing on negative samples). Thus the LLMs under filter-then-reranker paradigm have relatively more competitive performance on samples with low sample-level NP ratio. From this perspective, our classification on hard and easy samples is a simple but effective strategy to lower the NP ratios of samples feeding to LLMs. This strategy greatly avoids the limitations and leverages the powers of LLM rerankers.\n\n-----------------\n\nWe additionally appreciate your kind and valuable comments on the typos, grammer, and writing styles of our paper. We would revise our paper to correct these problems. We hope that our explanations and newly-added experiments have addressed your concern about (1) the reasons for LLMs failing on easy samples and (2) the potential effect of negative-positive ratios on LLMs' performance. If not, would you mind telling us your remaining concerns? We would try our best efforts to further address your any concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Systematic, comprehensive and timely empirical study\n- Provides some insight into what IE examples and tasks are challenging for LLMs\n- Proposes a way to utilize each model type optimally by keeping performance up and costs down (strong practical contribution)", "reasons_to_reject": "- The empirical and practical contributions of this paper are robust and I would like it to be accepted to the conference.\n- Nevertheless, this paper’s breadth does somewhat hurt its depth. It is unfortunate that the authors were unable to do a more thorough analysis on why LLMs are failing on easy samples. A case study on this or at least some deeper discussion on this topic could greatly improve the paper.", "questions_to_authors": "- Could you add detailed statistics around your test subsets? I know you downsampled randomly but I would like to know the relation percentage as well as the negative vs positive ratio in the different datasets (previous work has found that this is very important for IE performance). ", "paper_topic_and_main_contributions": "In this thorough study, the authors set out to determine whether LLMs can solve information extraction tasks effectively or whether these tasks are still dominated by smaller LMs. In a broad and systematic evaluation over 9 datasets and 4 IE tasks, they provide evidence that training small models through different techniques is still the most effective way to perform well on IE tasks. Especially since strong performance is achieved with fewer than 100 examples and small models (RoBERTa-large and T5-large). \n\nApart from a solid empirical study, the paper explores the idea that SLMs are better than LLMs at easier examples while LLMs are better at more complex examples (if the conditions of the example are simplified first by the smaller supervised model). The authors then propose a new method which leverages this insight by using the SLM for easy samples and LLMs for hard samples. This approach yields large improvements in examples which are challenging for SLMs while still leveraging the strengths of trained models in high-confidence examples.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- “Strength (and deficiency)” L076\n- “Instruction I: The instruction locates at the very” L219\n- “(Identity the entities...in the sentence)” L221\n- For in-text section titles like “LLMs show limited inference speed” in L275, place a period after it unless the title is above. Looks a bit messy otherwise.\n- L282: “Discussion: Why do LLMs not Present Satisfactory Performance on IE Tasks?” -> “Discussion: Why LLMs Fail to Obtain Satisfactory Performance on IE Tasks?” (Title sounds a bit awkward)\n- “labels, then LLMs subsequently rerank.” L325\n- “LLMs are and only are Hard Sample Solver” L343 (sounds awkward)\n- “with low confidence scores as hard otherwise easy.” L364"}}, {"review_id": "ysCl7e3dq7", "reviewer": "Reviewer_CUTz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks a lot for your reviews! Your professional reviews offer us great advice towards writing a more comprehensive and competitive paper! And, we appreciate you finding our empirical study extensive and our proposed filter-then-rerank paradigm combining the strength of both SLMs and LLMs.\n\n---\n\n> Q1: It is not certain that the training set was used in the pre-training process of the two LLMs, and this matter is not discussed in the paper.\n\nReducing the effect of data contamination when evaluating LLMs is an open, challenging and not well-solved problem. It is beyond the main scope of this paper. However, **we still discuss this potential problem in the Limitations part (lines 519 to 528)**. We argue that this potential problem does **NOT** affect the validity of the filter-then-rerank paradigm because: \n1. Data contamination makes the results overestimated. And the inferior overall performance of LLMs on our experiments (see Figure 2) demonstrates that the extent of data contamination on these datasets is limited.\n2. A reasonable assumption is that the effect of data contamination is equal on hard samples and easy samples. Since we observe a distinct performance difference on both easy and hard samples, we believe LLMs’ proficiencies on hard samples are due to their parametric knowledge and reasoning abilities (Line 85-88) instead of sample  leakage (otherwise LLMs should perform well on easy samples as well).\n\n---\n---\n\n> Q2: The validity of the filter-then-rerank paradigm is not discussed in the context of small-scale generative language models such as LLaMA and Vicuna in Figure 2.\n\nWe appreciate your valuable suggestion to discuss the validity of the filter-then-rerank paradigm on smaller, open-source LLMs. We add experiments with Vicuna-13B, which is instruction-tuned on LLaMA-13B, as the reranker and extend the Table 3 as below (We do not run experiments on LLaMA because we require the rerankers being LLMs with instruction tuning, on which the multi-choice question (MCQ) prompts could leverage their powers). We conduct 3-run experiments and report their mean values and standard deviations (in bracket).\n\n| | NER-FewNERD (5-shot) | NER-FewNERD (10-shot) |  NER-FewNERD (20-shot) | RE-TACREV (20-shot) | RE-TACREV (50-shot) | RE-TACREV (100-shot) | ED - ACE (5-shot) | ED - ACE (10-shot) | ED - ACE (20-shot) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| SLM | 59.4 (1.5) | 61.4 (0.8) | 61.9 (1.2) | 62.4 (3.8) | 68.5 (1.6) | 72.6 (1.5) | 55.1 (4.6) | 63.9 (0.8) | 65.8 (2.0) |\n| SLM + Vicuna| 60.0 (1.8) | 61.9 (2.1) | 62.2 (1.4) | 65.2 (1.4) | 70.8 (1.6) | 73.8 (1.7) | 56.9 (4.0) | 63.5 (2.7) | 66.0 (2.6) |\n| SLM w. ensemble | 59.6 (1.7) | 61.8 (1.2) | 62.6 (1.0) | 64.9 (1.5) | 71.9 (2.2) | 74.1 (1.7) | 56.9 (4.7) | 64.2 (2.1) | 66.5 (1.7) |\n| SLM w. ensemble + Vicuna | 59.9 (0.7) | 62.1 (0.7) |62.8 (1.1) |66.5 (0.5) | 73.6 (1.4) |75.0(1.5) | 57.9 (5.2) | 64.4 (1.2) |66.2 (2.4) |\n\nWe observe that:\n1. **Our filter-then-rerank paradigm still works and outperforms SLM-only approaches under most cases.** \n2. The reranking gains from Vicuna-13B are significantly less than that of GPT-4 (0.8% v.s. 2.4% absolute F1-gain, on average). \n3. Vicuna-13B presents inconsistent reranking capabilities among different tasks. It shows strongest performance on RE task, while shows minimal even negative effects on ED task.\n\nWe argue such observation does **NOT** undermine the validity of our filter-then-rerank paradigm because \n1. Being a LLaMA-based model distilled from GPT-4 auto-generated data, Vicuna has its limitations on both the pre-training and instruction-tuning scale. **It is reasonable that Vicuna presents inferior and less balanced performance than GPT-4.** We believe more powerful open-source LLMs would be likely to achieve stronger performance than Vicuna.\n2. As stated in Section 5.4, we only call LLMs for a small portion of samples. So the overall time and budget cost is affordable even with GPT-4 as the reranker.\n\n---\n---\n\n> Q3: Some of the conclusions in the paper (such as those corresponding to lines 64 to 70) are not exciting, or even almost taken for granted. \n\n1. The conclusions shown in lines 64 to 70 are somewhat unsurprising from the posterior perspective (after reading our paper). Even if some guesses seem reasonable and intuitive, however, they still require empirical studies to support or falsify them to be a conclusion. To our best knowledge, this paper is the first work to conduct extensive and systematic experiments to evaluate the performance of LLMs on diverse IE tasks. **It provides supportive references for the imprecise intuitions of the community and is valuable from this point**.\n\n2. Additionally, conclusions in lines 64 to 70 are only some of conclusions in our work and by no means encompass the full breadth of conclusions drawn from our empirical analyses. For example, (1) we explore the effect of various demonstration selection strategies, demonstration number and prompt formats in Section 3.3, (2) we observe that LLMs are relatively more proficient on hard samples than easy samples in Section 4.2, and (3) a detailed analysis and case study on our filter-then-rerank paradigm in Section 5.4 and 5.6.\n\n---\n\nWe hope that our explanations and newly-added experiments have addressed your concern about (1) the potential data contamintation problem, (2) the validity of filter-then-rerank paradigm on smaller, open-source LLMs, and (3) the novelty and excitement of our work. If not, would you mind telling us your remaining concerns? We would try our best efforts to further address your any concerns.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper provides a thorough evaluation of LLMs as few-shot solvers for IE tasks, comparing them to fine-tuned SLMs on multiple datasets and across various IE tasks.  This extensive evaluation helps the NLP community gain a deeper understanding of the capabilities and limitations of LLMs in this context.\n\n2. The proposed adaptive filter-then-rerank paradigm, leveraging the complementary strengths of LLMs and SLMs, offers a novel and practical solution to improve IE performance while managing time and budget constraints.", "reasons_to_reject": "The paper revolves around InstructGPT and GPT-4 to discuss the filter-then-rerank paradigm, and there are two potential problems:\n1. It is not certain that the training set was used in the pre-training process of the two LLMs, and this matter is not discussed in the paper.\n\n2. The validity of the filter-then-rerank paradigm is not discussed in the context of small-scale generative language models such as LLaMA and Vicuna in Figure 2.\n\nIn addition, some of the conclusions in the paper (such as those corresponding to lines 64 to 70) are not exciting, or even almost taken for granted.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper investigates the use of Large Language Models (LLMs) as few-shot solvers for information extraction (IE) tasks and their comparison with fine-tuned small Pre-trained Language Models (SLMs).  The main contributions include extensive experiments on multiple datasets, highlighting that LLMs are not effective few-shot information extractors but can complement SLMs by handling difficult samples.  The paper proposes an adaptive filter-then-rerank paradigm where SLMs filter challenging samples, and LLMs rerank them to achieve promising improvements (average 2.4% F1-gain) on various IE tasks while maintaining reasonable time and cost efficiency.  The research contributes insights into building efficient and effective IE systems by leveraging both LLMs and SLMs.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "none", "Typos_Grammar_Style_and_Presentation_Improvements": "none"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a comprehensive empirical study that leads to the observation that LLMs are generally not as competitive for IE tasks than fine-tuned small LMs. The paper then proposes a hybrid strategy that uses LLMs to rerank for "hard" examples that a small LM finds challenging. Empirical results demonstrate the effectiveness of this strategy.

Strengths:
- Solid and comprehensive empirical study on an important problem
- Clear and professional writing
- An intuitively plausible hybrid strategy with demonstrated effectiveness

Weaknesses:
- As the reviewers pointed out, the paper falls short in investigating and explaining why LLMs are underperforming, especially given the known sensitivity to prompts. The authors have responded to this criticism with new analysis and experiments. The authors are strongly encouraged to add these to the revised version.
- As Zhang et al. (2023), which is already cited in this paper, have shown, LLMs' RE performance can be substantially improved by reformulating RE as a QA problem. One possibility demonstrated through that work is, perhaps it's not that LLMs are inherently bad at IE, but that we haven't found the best way to use them for IE yet. LLMs are also becoming better over time. A blanket statement, like "LLMs are not good few-shot information extractors in general", may need more qualification, like "the **current** LLMs **under vanilla prompting settings** are not good few-shot information extractors in general". This doesn't significantly undermine the contribution of this paper, and I think it's still a worthy paper to be presented at the conference. But it's important to send the most accurate message to the community to inform future research.

Kai Zhang, Bernal Jimenez Gutierrez, Yu Su. Aligning Instruction Tasks Unlocks Large Language Models as Zero-Shot Relation Extractors, Findings of ACL 2023.
Decision: Accept-Findings