Guideline Learning for In-Context Information Extraction
Authors: Chaoxu Pang
Yixuan Cao
Qiang Ding
Ping Luo
Venue: EMNLP 2023 Main (2023)
Keywords: information extraction
in-context learning
large language models
TL;DR: We propose a framework that enables LLMs to automatically generate and utilize guidelines for alleviating the underspecified task description problem.

Abstract: Large language models (LLMs) can perform a new task by merely conditioning on task instructions and a few input-output examples, without optimizing any parameters. This is called In-Context Learning (ICL).  In-context Information Extraction (IE) has recently garnered attention in the research community. However, the performance of In-context IE generally lags behind the state-of-the-art supervised expert models. We highlight a key reason for this shortfall: underspecified task description. The limited-length context struggles to thoroughly express the intricate IE task instructions and various edge cases, leading to misalignment in task comprehension with humans. In this paper, we propose a Guideline Learning (GL) framework for In-context IE which reflectively learns and follows guidelines. During the learning phrase, GL automatically synthesizes a set of guidelines based on a few error cases, and during inference, GL retrieves helpful guidelines for better ICL. Moreover, we propose a self-consistency-based active learning method to enhance the efficiency of GL. Experiments on event extraction and relation extraction show that GL can significantly improve the performance of in-context IE.

Guideline Learning for In-Context Information Extraction
Chaoxu Pangl? Yixuan Cao!.2+
1.2+ Qiang D Ping Luo 1.2.3*
Key Lab of Intelligent Information Processing of Chinese Academy of Sciences (CAS)
Institute of Computing Technology, CAS, Beijing 100190, China
Unniversity of Chinese Academy of Sciences, Beijing 100049, China
*Peng Cheng Laboratory, Shenzhen 518066, China
(pangchaoxu21b, caoy dingqiang22z, luti)
Abstract
In-C ontext Learning Example:
the tell me the most precise relation between two entities. CONTENT AND CONTAINER X and have CONTENT AND CONTAINER relation X abjectphyskall stored delineated area of space Y. AND DESTINATION:
context,
Large language models (LLMs) perform new task by merely conditioning task in- structions and a few input-output examples, without optimizing any parameters. This is called In-Context Learning (ICL). In-context Information Extraction (IE) has recently gar- nered attention in the research community. However, the performance of In-context IE generally lags behind the state-of-the-art pervised expert models. We highlight key reason this shortfall: underspecifed task de- scription. The -eitted-enntth context struggles to thoroughly express the intricate instructions and various edge cases IE tasks, leading to misalignment in task comprehension with hu- mans. In this paper, we propose Guideline Learning (GL) framework for In-context IE which reflectively learns and follows guidelines. During the learning phrase, GL automatically synthesizes set of guidelines based few етrоr cases, and during inference, GL retrieves helpful guidelines for better ICI. Moreover, we propose self-consistency- based active learn- ing method to enhance the efficiency of GL. Experiments on event extraction and relation extraction show that GL. can significantly im- prove the performance of in-context IE.
DESTINATION re relation if is an entiy moving towards destinaton Y.
...
Context shipments arrived Question: Whatts the re relation between shipments and stock?
CONTENT AND CONTAINER
user answer
DESTINATION
Behind the scene:
User's Target Concept
LLM's Comprehanded
Concept
C uhai shipments the suggests that ship- ments are being stored in the stock. Therefore..
motoon verbs prevailing over
"stative relations
Errors caused Underspeciffed Task Description
Figure An example of conceptual bias the relation classification task (SemEval 2010 Task 8).
extend over 33 pages (Consortium, 2008). the past, the supervised learning paradigm has been ap- plied to fine-tune numerous parameters on massive data to accurately learn the concept (Li et al., 2020; Zheng al., 2019). This approach, while effec- tive, is data-intensive, hard to train, and difficult update.
1 Introduction
Information extraction (IE), whose primary goal is to extract structured information from unstruc- tured plain text, serves as a critical foundation for numerous downstream tasks such as question an- swering and knowledge base construction (Wang et al., 2022a; Fei et al., 2022). IE tasks typically have complex task settings due to their requirement of translating diverse real-world facts into few predefined classes. This often necessitates a large number of rules and examples thoroughly and accurately define the target concept of the task. For example, the guidelines for ACE relation extraction "Corresponding author: Yixuan Cao and Ping Luo.
Recently, however, the NLP community wit- nesses the rapid rise of large language models (LLMs), such as PaLM (Chowdhery et al., 2022), ChatGPT (OpenAI, 2023a) and LLaMA (Touvron et al., 2023). These LLMs have achieved great per- formance
wide range of NLP tasks with their superior language understanding power, but fine- tuning them faces closed-source and high-training- cost issues. In-Context Learning (ICL. (Brown et al., 2020). a characteristic feature of LLMs, of- fers a solution to hamness the power of LLMs while

---

sidestepping these issues. ICL enables LLMs to perform new tasks without tuning any parameters. Instead, they are given only the task instruction and few input-outpur examples as the prompt. achieves promising performance on many tasks like natural language inference and sentiment classifi- cation (Brown et al., 2020), demonstrating new paradigm in the NLP community.
Instance
Query
Guidelines
①
Retrieve
Rules
g
Query
Task instruction
Instance
Generatel
Selected Rules
& Update
Several recent studies have explored the ICL paradigm for IE (Han et al., 2023; Wei et al., 2023). Impressively, by merely providing task instructions and handful of in-context examples, LLMs can achieve significant performance on many IE tasks. However, they still lag behind supervised SOTA models (Han et al., 2023).
Reason
Reflect
Response
Label
Answer
Reference
Inference
Leaming
We underline one primary reason for the sub- optimal performance: underspecified task descrip- tion. As discussed earlier, the target concepi of IE is inherently complex. But the input context uti- lized for elucidating the target concept to LLMs is constrained by its limited length. Consequently, the comprehended concept by LLMs might de- viate from the target concept. An example of this illustrated Figure 1. In the sentence "The shipments have arrived into the stock", the pre-defined relation types Content-Container and Entity- Detiiy--Dettnttionn presents grey area concerning the relation between the entities "shipments" and "stock". The target concept is embodied in rule in the annotation guidelines' "motion verbs pre- vailing over stative relations" which is misaligned with the LLM's 's comprehended concept.
Figure 2: The Guideline Learning Framework, includ- ing inference and training phrases. denotes LLM agents are applied in this phrase.
ing mechanism, similar to Madaan et al. (2022), is inspired from the theory of recursive reminding in psychology (Jacoby and Wahlheim, 2013). This theory suggests that human learn from the error cases and recall the most helpful experiences when encountering new case.
Furthermore, we incorporate seff-consistency. based active learning method to enhance the effi- ciency of label utilization. eralizer" to assist in the generation and retrieval of guidelines. Finally, we conduct in-depth exper- iments on two representative IE tasks: (1) event extraction on financial documents, and (2) relation extraction on general domain resources, which both feature relatively complex target concepts. Exper- imental results indicate that the use of 50 labeled samples per class can greatly boost the performance of ICL in both tasks.
also propose "gen-
This paper attempts to mitigate this problem by introducing Guideline Learning (GL) framework. This framework replicates the human annotation process, which first gathers annotation guidelines, and then annotates accordingly. Specifically, it has two phrases. In the learning phase, set of guide- lines are iteratively learned from scratch based on a few labeled instances. A guideline here is a natural language rule derived by integrating the appropriate extrapolation of an eтror instance and its true label. This is different from previous supervised learning methods, which learn set of model parameters. In the inference phase, given new instance, it retrieves relevant rules from the guideline to com- pose prompt, which includes the task instruction, the retrieved rules, a few examples, and the input instance. then asks an LLM agent to finish the task given the prompt. This failure-driven remind-
2
Guideline Learning Framework
2.1
Overview
Figure presents an overview of the Guideline Learning (GL) framework. For the inference phase, assuming we have collected set of guide- lines for task. Given input instance a, the GL framework first retrieves set of relevant rules from the guidelines. A query is constructed by assem- bling the task instruction, few in-context examples, the instance, and the retrieved rules. The query is then forwarded to an LLM agent, which generates
Data Creation Guidelines for the SemEval 2010 Task 8

---

both the answer and the references (rules that the agent deems beneficial for this particular instance), During the training phrase, the framework iterates over few training instances to generate and learn guidelines from scratch. For each instance, if the predicted answer from the LLM agent is different from the annotation, another LLM agent generates new rule and update the existing guidelines. In the following sections, will detail the infer- ence phrase (See 2.2), the learning algorithm (Sec 2.3), and an active instance selection method for effective guideline learning (Sec 2.4).
update guidelines immediately ensure stable rea- soning inside one epoch. After the iteration, we update rules in the cache to the guidelines. Besides, we keep a score for each rule based on whether it leads to correct answers. At the end of an epoch, rules with score below a threshold are regarded as harmful and are removed from the guidelines. Specifically, the rules are generated as follows. If the predicted answer ý is wrong, the instance 2, the predicted ÿ, and the true label y are given to an LLM agent to write rule:
Reflect(2, 9.
2.2 Inference
The score of rule is computed follows. For rule r 9, we compute its prior score based on its statisties:
this section, we introduce how to predict the answer of an instance x in the GL framework. Suppose we have collected the Guidelines g (reH1 which is a of rules that supports read, write, and retrieve operations. Each rule, expressed as a natural language description, explicates an as- pect of the task, while the guidelines implicitly reflect the target concept of the task. The inference process unfolds as follows.
Nua Nerong
score(r)
Nyetrieve
where Nretrs Nbat and Nurrng are the number of instances in which the model retrieves r (r R), refers r (r Rt) and predicts correctly, and refers to and predicts wrongly. The prior score indicates the helpfulness of a rule based on the historical responses.
Retrieve. We retrieve the top-k rules R. from G that are most relevant x:
R Retrieveex,, g)
Algorithm 1: Guideline Learning
where R C C. We can also retrieve some input- output examples N from the training dataset D. Reason. The task instruction T, the instance x, the few-shot examples N, and the retrieved rules R are integrated to create a query which is used to ask the reasoner about which class the instance belongs to:
Input :number of epoch Ne, task description T, training set
D {Im vmm) ANa
Output : guidelines G
Initialize G cache Ø;
2 for e = 1...N, do
3
for [ y) in do
4
R = retrieve 9);
g [T, Ú, R = (gason(g)
N = retrieve_entt patoove__emmlle((::___ D);
q = f(T, x., R, N);
where reasoning is performed by an LLM agent with ICL capability, ý is the predicted answer, and R" C R is a returned subset of retrieved rules that the agent deems helpful during reasoning. R' is used to evaluate the quality of the rules in Sec 2.3.
R reason(q);
update_score (R' 9 v.
ÿ / then reflect 2. 0, y): cache cache U (r):
10
2.3 Learning Algorithm
foreach cache do
In this section, we introduce the learning algorithm which reflectively learns guidelines from collec- tion of instance-label pairs. The pseudo code of this algorithm is presented in Algorithm 1. In each epoch, first predict on all instances to get the re- sponse comprising the answer ý and references R* If the answer is wrong, an LLM agent will generate a new guideline and append it in a cache. We don't
update_ Sillate_guidelinerr,,, 9):
13
forget_ haget_harmful_geee (aamfullguidellnes(())
2.4 Active Instance Selection
In this section, we investigate how to seleet in- stances for annotation, to construct the training

---

dataset for effective guideline learning (Sec 2.3). Random sampling could result in a low efficiency as the model may already be capable of accurately predicting large portion of instances. To allevi- ate this problem, we propose an active learning upproach that prioritizes instances where the model is most uncertain.
2.
event trigger classification (ETC) that classi- fies candidate event triggers event classes;
3. event argument extraction (EAE) that iden- tifies the event arguments of a given trigger and recognize the specific roles they play.
Assume we have a collection of instances I {xm] { m=1- Following self-consistency chain-of- thoughts (Wang et al., 2022b), for each instance n, we first sample 7 reasoning paths and answers {(re, }o)) with relatively high temperature. Then we obtain the model's probability on each class e by marginalizing out the sampled reasoning paths:
For this task, we apply guideline learning to ETC. Specifically, given event schema and set of candidate triggers in text, one instance here is the text and one candidate trigger. Note that it's also feasible to apply guideline learning to EAE. We leave it as future work.
3.2 Relation Extraction
1 T
T
Relation extraction (RE) aims to predict semantic relations between pair of entities in texts. Figure 1 presents example of RE. According to recent report (Han al., 2023), even when equipped with chain-of- oftthhthht prompting, ChatGPT can only achieve a maximum performance of 43% compared to state-of-the-a RE methods.
x(c/x)=
Σ
H
= cc
The consistency of the sampled answers indicates the model's confidence. A sharp probability distri- bution indicates high confidence on this instance, whereas flatten distribution indicates low con- fidence. We compute the negative entropy of the probability distribution to measure the model's con- fidence on this instance:
For RE, we directly apply guideline learning to assist in distinguishing relation concepts. Specif- ically, given set of relation types and one entity pair from text, instance here is the text and one entity pair.
confid(x)) Ep(clz) log p(c|x)
We select the top-k instances with the lowest con- fidence score. The underlying assumption here is that the model is more prone to committing errors for instances with lower confidence.
3.3 Implementation of Base Components
LLM Agent db For all LLM agents, we use the official API2 of ChatGPT (OpenAI, 2023a) to gen- crate outputs. To prevent the influence of dialogue history. we generate the response separately for each testing sample.
3 Task and Implementation
Initially, we implement the guideline learning framework for two information extraction tasks: event extraction (Sec 3.1) and relation extraction (Sec 3.2). We choose these tasks because the target concepts of these tasks are relatively complex.
Generalizer We introduce important LLM agent generalizer narrow the shallow semantic gap between instances and rules. The generalizer is an LLM agent which extrapolates the instance x properly to a more general form by abstract- ing common properties, such as company names, dates. We use the ă instead of x to retrieve and generate rules. Figure 3 presents example of the generalizer in EE. We provide some intuition of the generalizer in Appendix A.3.
3.1 Event Extraction
Event extraction (EE) aims to extract structured events from unstructured texts. Figure 3 gives an example of EE. The event structure is predefined by an event schema, consisting of event classes and corresponding event roles. For example, the equity repurchase event has roles like company name, repurchased shares, closing date, etc. In this paper, we decompose EE into three sequential sub-tasks:
Retrieval For an input instance, we use its gen- eral form ã to sort rules in guidelines by the seman- tic similarity between o and the rules. Specifically, we use the embedding API (text-embedding ada- 002) from OpenAl (2023b) to obtain the embed- dings of ã and and use cosine similarity the
1. event trigger identification (ETI) that identi- fies all candidate event triggers from the text;
3gpt-3.5-turbo-0300

---

Document
Group Co., repurchosed 26,858,570 highest repurchase price yuan/share aporoximately 282,270,000.0 (inclucing commission)...
yuan/share, After the repurchase transaction, the company helds 23,000,000 shares...
lowest
Event E Extraction Example
① E vent Trigger Iddetifiiation
23,000, 0,0,000000000.0000000 shares, 26,858,570 shares
2 Event Trigger classiffcaaton
126,858,570 shares, EE Equiity Repurchase', (23,000,000 shares, 'None')
Guideline Leaminng
Event argument extraction
Highest Trading Price
Lowest Trading Price
Closing
Reparchased Shares
Repurchase Amount
Group LTD.
yuon
(E)
26,858,570 shares
262,270,000. yuan
(summarized): 23,000,000 shares total
Group Co., DTD. has repurchased une 30, 2011. General form: The total shares that companyy has repurchased as of specific date.
Implamentation of Trigger Classification
General form
Label: Equity Repurchase
Rule: hh tatal shares that company has repurchased of specific date triggers equity repurchase event.
Figure 3; An example (translated) of event extraction from ChFinAnn dataset (Zheng et al., 2019). We decompose EE into three sub-tasks: event trigger identification, event trigger classification, event argument extraction. We present the output of each sub-tasks.
semantic similarity score. The few-shot demonstra- tions are randomly chosen from the training data, and are fixed for all instances and methods in each task.
set due to the token length limit of OpenAl's API. More details are presented in Appendix A.1.1.
Metries We use role-level micro precision, re- call, and Fl for evaluation, following previous work (Zheng al., 2019).
Reflect In this paper, simply concatenate the general form & of the instance and the golden la- bel to generate rule. Figure presents an example this process in EE.
Method Though only working on ETC, we also provide simple solutions for the other two subtasks for comparison with other methods. Specifically, for ETI, as all event types are related to equity transaction, we identify text spans with the for- mat "(number) shares" as candidate triggers via string matching. For ETC, apply guideline learning framework and conduct binary classifi- cations for each event type. As the documents in this dataset are long, we apply an extra L.MM agent to generate a description for each trigger about its meaning according to the document. We use the generated description as the input instance to con- duct the classification. For EAE, we apply an LLM agent to generate an event table in the markdown format given predicted event triggers.
Note that our implementation only requires the official APIs without any parameter updating.
4 Experiments
We conduct experiments to demonstrate the ef- fectiveness of the GL framework on event extrac- tion (See 4.1) and relation extraction (Sec 4.2). In the last section, we analyze the quality of learned guidelines and conduct case studies (Sec 4.3).
4.1 Event Extraction
4.1.1 Setup
Dataset We use the ChFinAnn dataset (Zheng al., 2019), distant-supervised document-level event extraction dataset on Chinese financial doc- uments, to conduct our experiments. Zheng et al. (2019) highlighted one challenge is to detect multi- plc event instances in one document. We focus on four event types: Equity Freeze (EF), Equity Repur- chase (ER), Equity Underweight (EU), and Equity Overweight (EO). For the test set, We randomly sample at most 200 documents with proper token length for each event type from the original test
Compared Models (1) ReDEE (Liang al.. 2022) and DE-PPN (Yang et al., 2021): Two su- pervised methods. We reproduce DE-PPN on the entire dataset strictly following the official code. ReDEE runs out of memory on 12G GPU so we do not reproduce it. (2) EE-ICL: Prompt the LLM to directly output the event table without predict- ing event triggers.
(3) EE-GL-b: Bascline ver- sion of our guideline learning method with empty guidelines. (4) EE-GL-r: Our guideline learning method. We randomly sample 50 documents from the training set and annotate event triggers. (5) EE- GL-a: We actively select 50 documents out of 400
All prompts and hyper-paramter settings are detailed in the Appendix. All datasets and our annotations are publicly available for rosearch purposes here.

---

EU
ER
EO
EF
Method
R.
R.
P.
F1.
R.
F1.
DE-PPNI
69.7
79.9
74.4
91.1
89.3
85.6
87.4
81.0
71,3
78.2
69.4
73.5
ReDEE
82.5
69.2
91.1
90.3
90.7
83.7
73.1
78.1
78.0
70.6
74.1
DE-PPN
71.2
66.1
68.6
84.3
88.2
86.2
70.9
71.9
71.4
72.6
56.0
63.2
EE-ICL
51.8
74.0
60.9
85.2
88.4
86.8
60.4
75.9
67.3
43.2
65.6
52.1
EE-GL-b
54.3
71.0
61.5
85.0
89.3
87.1
62.0
74.6
67.7
44.7
63.5
52.5
EE-GL-r
56.3
72.6
86.5
89.4
87.9
66.5
70.1
66.7
53.9
BE-GL-a
55.0
76.0
63.8
86.7
89.2
88.0
65.8
76.2
70.6
48.6
66.6
56.2
Table I: Overall event-level precision (P.). recall (R.) and scores evaluated on test set (distant-superrise) label). t: results from Liang et al. (2022). Note that these performances not comparable as they evaluate on the entire test set. : SOTA supervised model. We reproduce this work following Yang et al. (2021).
Single R.
Multi
Method
P.
F1.
P.
R.
F1.
P.
R.
F1.
DE-PPN
78.7
79.8
79.3
72.9
42.2
53.4
73.9
57.1
64.4
EE-ICL
64.6
88.9
74,8
70.8
79.4
75.0
68.1
83.3
74.9
EE-GL-b
71.5
87.8
78.8
73.0
72.2
72.6
72.8
77.9
75.3
EE-GL-r
72,4
88.7
79.7
74.4
74.5
74.4
73.5
80.
76.6
EE-GL-a
71.0
89.3
79.1
71.7
81.3
76.2
71.4
84.5
77.4
Table 2: Results for the Equity Underweight type on the single-event and multi-event sets (buman-annotated label).
randomly sampled documents from the training set and annotate event triggers.
ing method. Note that DE-PPN trained on 25631 fully annotated examples, while our methods are trained on 200 examples total with only trigger annotation.
We use the same human-annotated demonstra- tions for all EE methods.
Results Human-Annotated Test Set As the label constructed by distant supervision is noisy, manually annotate the test set of Equity Under- weight. The results on this test are shown in Table 2. shows that: (1) GL-r and GL-a im- prove 1.7, 2.5 Fl scores over ICL, respectively. (2) ICL and GL-r/a outperform DE-PPN by over 10% micro-F1. This implies that though only pro- vided few manual labels, LLMs are more capable of aligning with human annotation than supervised methods trained large-scale weakly-superviser dataset. (3) Supervised method DE-PPN performs much poorer on multi-event documents than single- event document (53.4 vs. 79.3), while ICL-based methods are more robust (more discussion on Ap- pendix A.1.4).
4.1.2 Results and Analysis
Main Results We show our main experimen- tal results in Table We can observe that: (1) ICL achieves promising results (-7.7, +0.6, -4.1, -11.1 micro-F1 compared with DE-PPN) on four event types. Note that previous studies (Han et al.. 2023; Wei et al., 2023) have shown that in-context learning performs poorly other event extraction datasets. We suppose that the performance is better on this dataset because the financial disclosure doc- uments are required to organize in highly homo- geneous format. This result indicates the power of in-context learning. (2) Both GL-r and GL-a out- perform ICL on four event types by at most +2.9, +1.2, +3.3, +4.1 micro-F1. Note that we only use extra trigger labels of 50 documents per class. (3) Though out three-step methods and the summary agent can slightly improve the performance (GL-b vs. ICL), the main performance gain comes from the learned guidelines (GL-r vs. GL.-b). (4) GL.-a consistently outperforms GL-r by a small margin, which verifies the effectiveness of our active learn-
4.2 Relation Extraction
4.2.1 Setups
Dataset We use SemEval 2010 task 8 (Hen- drickx al., 2010) relation extraction dataset conduct our experiments. This task focuses on se- mantic relations (e.g., "component and container".

---

Method
P.
R.
Fl.
91.3), which is consistent to previous studies (Han ct al., 2023).
RIFRE
91.3
RE-ICL
58.3
62.7
4.3 Analysis
RE-GL-b
59.3
67.1
63.0
4.3.1 Quality Evaluation of Guidelines
RE-GL-r 62.3
69.7
65.88
RE-GL-a 63.5
66.9
We manually evaluate the quality of learned guide- lines. Specifically, for each task, we randomly sam- ple guidelines from the best epoch and compute the accuracy where count hit if the guideline is precise and unambiguous. The results are shown in Figure 4. For both GL-r and GL-a, which are provided manual labels, the accuracy above 90%. This indicates that LLMs can well perform the gen- eralizing task when appropriately prompted. To investigate how the label quality effects the quality of generated guideline, we conduct experiments (GL-r-ds) with the same setting GL-r but pro- viding the distant supervised labels. The accuracy drops dramatically by 17.2 points. The forgetting mechanism (w/ discard, detailed in Sec 2.3) helps to discard harmful guidelines boosting the accuracy by 3.3 points, but it is still significantly lower than GL-r. This indicating the necessity of label quality for generating high-quality guidelines.
Table 3: Results on the SemEval dataset. supervised model (Zhao et al., 2021).
SOTA
100
90
Accuracy (%)
80
GL-f
70
GL-a
GL-r-ds
60
w/ discard
50
ChiFinAnn-EU
Semeval
Figure 4: The manual evaluation results of the learned guidelines on ChFinAnn-BU (EE) and SemEval (RE) dataset (randomly select 50 for each evaluation).
"entity and destination") between pairs nominals and contains 10.717 annotated examples covering nine relations collected from general domain re- sources. We randomly sample 1000 test samples from the original test set for evaluation.
4.3.2 Case Study of Guidelines
Note that we generate guidelines by first general- izing the input instance to its general form, then combining it with its golden label. This implemen- tation can successfully generate helpful guidelines, while inevitably makes some mistakes. We show some cases in Figure 5. We find some helpful guidelines imply annotation rules in the annotation guidelines (e.g., He-4). The cause of the harmful guidelines mainly due to the inadequate gener- alization (c.g. Ha-1, Ha-3) and annotation error (e.g. Ha-2). Besides, in extreme cases, the relation between two entities is only based on the literal meaning of the entity (e.g. Ha-4), which is hard to generate general guideline.
Method We directly apply guideline learning to conduct the relation extraction lask as detailed in Sec 3.2.
Compared Models (1) RIFRE (Zhao et al., 2021): SOTA supervised model. (1) RE-ICL: For a pair of entities in a text, we prompt the LLM to directly output the relation type. (3) RE-GL-b: Baseline version of our guideline learning method with empty guidelines. (4) RE-GL-r: Our guide- line learning method. We randomly sample 500 instances (50 instances per relation class on aver- age) from the training set to learn guidelines. (5) RE-GL-a: We actively sample 500 instances out of 1000 randomly sampled instances from the training set to learn guidelines.
4.3.3 Comparison with DE-PPN in Data Scarcity Settings
We conduct experiments to investigate how ICL- based approaches compare to alternative super- vised approaches in settings where annotation is scarce. Specifically, train DE-PPN on (1) the 192 annotated documents available to ICL ap- proaches (50 documents per event type); (2) 5k an- notated documents (random sampled); (3) all 29k annotated documents. We compare DE-PPN with vanilla fhw-shot ICL (EE-ICL) and our guideline
4.2.2 Results and Analysis
The results shown in Table 3. We can observe that (1) GL-r and GL-a outperform ICL by 3.1, 4.2 Fl-scores, respectively. This verifies the effective- ness of applying our guideline learning framework for relation extraction.
(2) The performance of ICL- based RE is still far behind SOTA methods (66.9 vs.

---

Helpful
Harmful
He-1 The shares sold through the trading system trigger
Ha-1. The shares bought and sold mistakenly triggers an equity underweight event.
equity underweight event.
EE
He-2. freely tradable shares held before the reduction don't trigger equity underweight event.
Ha-2. The outstanding shares sold through the centralized bidding
trading system don't trigger an equity underweight event.
Ho-3. "ss the Y (Technology) to inform about (Topic or Subject)" MESSAGE TOPIC.
Ha-3, "early in Yhe (Product"" indicates that the relation between MESSAGE TOPIC.
indicates
relation between He-4, "Y (Agriculture)
inside the upper part of the rock X (Building)" indirates that the relation between and Y ENTITY AND DESTINATION.
Ha-4, "X (Food) (Food)" indicates that relation between X and Y ENTITY AND ORIGIN,
RE
(IOT CONTAIN AND CONTAINER because of the motion verbs prevalling over "stafive"" relations cftaria.)
(The original sentence: Homemade tomato better the shop bought vers
Figure 5: Case study of guidelines learned in EE and RE task. We use colors for better illustration.
annotated datasets and may fail in new scenario. On the other hand, Brown et al. (2020) shows that in-context learning (ICL) of large language models (LLMs) can perform numerous tasks when provided few examples in a natural language prompt. ICL is a highly promising new learning paradigm because it is tuming- freen user- seer-friendly, and data-efficient. There are many studies applying in-context learning to perform IE tasks. Wan et al. (2023) proposes GPT-RE to bridge the gap between ICL and finetuning baselines for RE via two strate- gies: entity-aware demonstration retrieval and gold- label induced reasoning. Chen et al. (2023) propose an in-contex. learming-based NER approach and model PLMs as a meta-function, which can inject in-context NER ability into PLMs and recognize entities of new types on-the-fly using only few demonstrative instances. However, though focus- ing on ICI., these methods still requires training over large-scale datasets.
80
F1 5c00e
40-
Brent Type
BU
ER
20
Model
200
Number of labeled documents used for training
Figure 6: F1 scores of different methods trained on different training dataset sizes. We use different colors, markers to distinguish different event types and models, respectively.
learning approach (EE-GL) on the same test set. The F1 score of each event type is shown in Figure 6. We find that DE-PPN fails when only providing 192 labeled documents, with very low Fl scores on all event types. The problem is alleviated when providing 5k labeled documents. DE-PPN relies on a large amount of annotated data to work well. This indicates the superiority of ICL approaches over data-hungry supervised approaches. Our guideline learning approach further improves the few-shot ICI. approach (EF-ICL) on all event types.
Recently, ChatGPT (OpenAl, 2023a) has stimu- lated the research boom in the field of LLMs. Chat- GPT has been the most well-known and powerful LLM far, with amazing ability of ICL and in- struction following. There are many studies ex- ploring ChatGPT's capability on IE tasks. Many studies (Han et al., 2023; Wei et al., 2023; Gao et al., 2023) evaluate ChatGPT's capability on IE tasks by directly prompting and find a huge perfor- mance gap between ChatGPT and SOTA results. They mainly focus on performance evaluation with- out in-depth investigations to boost ICL ability for IE tasks.
5 Related Work
5.1 In-Context Information Extraction
Information extraction (IE) extracts structured knowledge of interest from unstructured text, in- cludes entities, relations between entities, event arguments, etc. Previous studies mainly focus fine-tuning task-specific model under the super- vision from large-scale datasets (Zhao al., 2021; Zheng et al., 2019; Yang et al., 2021; Liang et al., 2022). Though achieving remarkable performance, these models heavily rely on high-quality manually-
5.2 Retrieval-augmentee ICL
Many studies propose retrieve relevant evidence from extra knowledge sources to enhance the per- formance ICI. Demonstration retrieval aims at designing more effective strategics for judiciously selecting in-context examples from a large training

---

set. For example, Liu et al. (2022) applies KNN- retrieval based on sentence-level representations. GPT-RE (Wan et al., 2023) further finetunes an entity-aware representation on the training set for better retrieval. However, similar to the supervised paradigm, these methods still rely on large-scale annotated dataset. Some studies retrieve relevant information from an extra memory assist in ICL. Madaan et al. (2022) proposes memory-assisted framework that correct errors via user interactions. They pair the GPT-3 (Brown et al., 2020) with a growing memory of recorded cases and user feed- back, which allows the system to produce enhanced prompts for any new query. However, their method heavily replies on the quality of user interaction. As they use simulated user feedback in experiments, the effectiveness and stability have not been veri- fied in real-world cases.
learning phrase, and retrieving helpful guidelines to assist in reasoning during inference. Our ex- periments on event and relation extraction show that a straightforward implementation guideline learning can enhance vanilla in-context learning by approximately 4%.
Limitations
The guideline learning (GL) framework establishes powerful and reproducible starting point for in- context learning research. However, our work still lacks depth in certain aspects and many potential research directions within this framework warrant further investigation.
Broader applications In this paper, we only ap- ply GL to IE tasks to alleviate the underspecified task description problem. It's encouraging to trans- fer GL. to other tasks with complicated task specifi- cations.
Our approach utilizes similar memory and re- trieval mechanism. With a focus on IE, our frame- work can automatically learn high-quality guide- lines from few error cases, obviating the need for user feedback, which is more efficient and stable.
More specialized retriever We implement an elementary retriever by utilizing OpenAl's embed- ding API. Though sufficient verify the effective- ness of our framework, the performance is sub- optimal. It's promising to establish a more power- ful retriever that specializes in retrieving relevant guidelines based on input cases.
5.3 Instruction Learning
Guideline Learning differs from two main branches of previous work on instruction learning: Instruction induction via ICL. Honovich et al. (2023) predict the task instruction by prompting instruction-tuned LLMs. They conduct explorative experiments, focusing on tasks that have "clear and simple instructions". In contrast, our GL frame- work focuses more complex instructions with highlight on IE tasks: extraction of complex con- cepts. We propose the "guideline" as bridge to learn and utilize more specific instructions from error cases automatically, which can be viewed as an de-tepth extension of previous work.
More sophisticated generalizer We generate guidelines by prompting an LLM agent to prop- erly extrapolate each error case. The guidelines are mostly precise but still lack generality. It's possi- ble to design a more sophisticated generalizer to summarize a guideline based on multiple similar eттor cases.
Enhance the rule-following capability of LLMs One key necessary capability of the reasoner is generate responses while faithfully following input rules. We observe that gpt-3.5-turbo the backbone LLM agent our experiments, still struggles to truly refer to relevant rules. We present a prelimi- nary discussion in Appendix A.4. would be in- triguing to evaluate and enhance the rule-following ability of LLMs.
Instruction learning for meta- meta-training. Ye et al. (2023) propose utilize instruction learning better finetune LLMs and boost the zero-shot performance. Our GL framework aims at boost- ing the model performance under the tuning-free setting, which orthogonal to their work.
Acknowledgements
This work has been supported by the National Nat- ural Science Foundation of China (No. 62076231, 62206265), and the China Postdoctoral Science Foundation (No. 2021M703271), We thank all the anonymous reviewers for their valuable and constructive comments.
6 Conclusion
This paper explores the underspecified task descrip- tion problem in in-context information extraction. We propose guideline learning framework to al- leviate the problem, which automatically learns guidelines from few labeled instances during the

---

References
Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang. Lu Liu, and Xiang Wan. 2023. Is information extrac- tion solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. CoRR, abs/2305.14450.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan. Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurIPS 2020, December 6-12. 2020, virtual.
Iris Hendrickx, Su Nam Kim, Zornisa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti. Lorenza Romano, and Stan Szpakowicz. 2010. Semeval- 2010 task 8: Multi- way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Workshop on Semantic Evaluation, SemEval ACL 2010, Uppsala University, Uppsala, Sweden, July 15-16, 2010, pages 33-38. The Association for Com- puter Linguistics.
Or Honovich, Uri Shaham, Samuel R. Bowman, and Omer Levy. 2023. Instruction induction: From few examples to natural language task descriptions. In Proceedings of the 61st Annual Meeting of the As- sociation for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9- 14, 2023, pages 1935-1955 1952. Association for Computa- tional Linguistics.
Jiawei Chen, Yaojie Lu, Hongyu Lin, Jie Lou, Wei Jia, Dai Dai, Hua Wu, Boxi Cao, Xianpei Han, and Le Sun. 2023. Learning in-context learning for named entity recognition. CoRR, abs/2305. 1038.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko. Joshua Maynez, Abhishek Rao, Parker Barnes. Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin. Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph. Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lec, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways. CoRR, abs/2204.0233
Larry L. Jacoby and Christopher N. Wahlheim. 2013. On the importance of looking back: The role of re- cursive remindings in recency judgments and cued recall.
Xiaoya Li, Jingrong Feng, Yuxian Meng, Qinghong Han, Fei Wu, and Jiwei Li. 2020. A unified MRC framework for named entity recognition. In Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics. ACL 2020, Online, July 5-10, 2020, pages 5849-5859. Association for Computational Linguistics.
Yuan Liang, Zhuoxuan Jiang, Di Yin, and Bo Ren. 2022. RAAT: relation- auggmenedd attention transformer for relation modeling in document- level event extraction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 4985 Association for Computa- tional Linguisties.
Linguistic Data Consortium. 2008 ACE (automatic content extraction) english annotation guidelines for relations v6.2.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in in-context examples for gpt-3? In Pro- ceedings of Deep Learning Inside Out: The 3rd Work- shop on Knowledge Extraction and Integration for Deep Learning Architectures, DeeLIO (ALL 2022, Dublin, Ireland and Online, May 27, 2022, pages 100-114. Association for Computational Linguistics.
Zichu Fei, Qi Zhang, Tao Gui, Di Liang, Sirui Wang, Wei Wu, and Xuanjing Huang. 2022. CQG: A sim- ple and effective controlled generation framework for multi-hop hop question generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volue 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 6896669066 Association for Computational Linguis- tics.
Aman Madaan, Niket Tandon, Peter Clark, and Yim- ing Yang. 2022. Mcmory-assisted prompt editing to improve GPT-3 after deployment. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2833-2861. Association for Computational Linguis- tics.
Jun Gao, Huan Zhao, Changlong Yu, and Ruifeng Xu. 2023. Exploring the feasibility of chatgpt for event extraction. CoRR, hbb//2303.0886...

---

OpenAI. 2023a. Introducing chatgpt.
Kang Zhao. Hua Xu, Yue Cheng, Xiaoteng Li, and Kai Gao. 2021. Representation iterative fusion based heterogeneous graph neural network for joint entity and relation extraction. Knowl. Based Syst., 106888.
OpenAl. 2023b. New and improved embedding model.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Fric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, 139771
Shun Zheng, Wei Cao, Wei Xu, and Jiang Bian. 2019. DocZedag: An end-to-end document-level frame- work for chinese financial event extraction. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing, EMNLP-IJCNIPP 2019, Hong Kong, China, November 3-7, 2019, pages 337-346. Association for Computational Linguisties.
Zhen Wan, Fei Cheng, Zhuoyuan Mao, Qianying Liu, Haiyue Song, Jiwei Li, and Sadao Kurohashi. 2023. GPT-RE: in-context learning for relation extraction using large language models. CoRR, abs/2305.02105.
Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. 2022a. Simkge: Simple contrastive knowledge graph completion with pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4281 0281-4294. Association for Com- putational Linguistics.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed Chi, and Denny Zhou. 2022b. Self- consistency improves chain of thought reasoning in language models. CoRR, abs/2203.11171.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang. Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wen- juan Han. 2023. Zero-shot information extraction via chatting with chatgpt. CoRR, abs/2302. 10205.
Runxin Xu, Tianyu Liu, Lei Li, and Baobao Chang. 2021. Document-level event extraction via heteroge- neous graph-base interaction model with a tracker. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the I Itth International Joint Conference on Natural Lan- guage Processing, ACI/ICCIIP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 3533-33 3533-3546. Association for Computational Linguis- tics.
Hang Yang. Dianbo Sui, Yubo Chen, Kang Liu, Jun Zhao, and Taifeng Wang. 2021. Document-level event extraction via parallel prediction networks. In Proceedings of the 59th Annual Meeting of the Asxo- ciation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021. (Volume 1: Long Pupers), Virtual Event, August 1-6, 2021, pages 6298- 6308, Association for Computational Linguistics.
Seonghyeon Ye, Doyoung Kim, Joel Jang, Joongbo Shin, and Minjoon Seo. 2023. Guess the instruction! Nipped learning makes language models stronger zero-shot learners. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.

---

A Appendix
Event
# Candidate
Negative
Ratio
A.1 Event Extraction Experiment Details
EU
790
468
59.2%
ER
260
60
23.1% %
A.1.1 ChFinAnn Dataset
EO
534
341
63.9%
The ChFinAnn dataset (Zheng al., 2019) is con- structed from real-world Chinese financial docu- ments via event-level distant supervision. It con- tains 32040 documents in total, focusing on five event types: Equity Freeze (EF), Equity Repur- chase (ER), Equity Underweight (EU), Equity Overweight (EO) and Equity Pledge (EP). We don't conduct experiments EP events we suppose there exists event confusion that both equity pledge and release of pledge are labeled as EP events. As the official API 1app-3.5-ttrbo--00
EF
477
262
54.9%
Table 5: Dataset statistics about the number candidate triggers and negative triggers in test
A.1.2
Prompts
For guideline learning, we conduct binary classifi- cation for each event type. We present the prompt of the equity underweight events. Only the demon- strations in the prompt different across different event types. We use 6-8 demonstrations for each LLM agent. We introduce our method in section 4,1.1 Here we briefly recap the input and output of each LLM agent:
has a max token length of 4096 tokens, we only keep the doc- uments with a length less than 1000 Chinese char- acters. We sample at most 200 documents for each event type from these documents. Table 4 presents the data statistics.
We calculate the ratio of negative triggers (i.e. candidate shares that refer to non-events) in each event type on our test set. The results are shown in Table 5. The ratio of negative triggers varies across different event types, ranging from min- imum of 23.1% % to a maximum of 63.9%. The simple trigger expression "number shares" we for this dataset ensures high recall (every event record on this dataset involves such expression). however, it also introduces unnecessary negative triggers, resulting in additional cost of event trigger classification. This indicates that identifying and classifying triggers on this dataset is non-trivial. Note that our experiments are designed to validate the GL framework, with focus trigger clas- do place much emphasis on trigger identification. In practice, it's more efficient design powerful event trigger identifier beyond the simple pattern. For example, it's promising to prompt the LLM to identify can- didate triggers with few in-context demonstrations. We leave it as future work.
1. The summarizer takes document and one share in it the input and output a summary of this share, which we call share description. The prompt presented in Figure 7.
2.
The generalizer takes the instance (share de- scription) input and output its general form by abstracting common properties. The prompt is presented in Figure 8.
3.
The reasoner takes the instance (share descrip- tion) and the retrieved guidelines input and output the reasoning process (CoT), the pre- dicted answer and the index of the used guide- line. The prompt presented in Figure 9.
sification. Consequently,
For EAE, we prompt the LLM to output the event table in the markdown format. As the documents in this dataset are long, only use demonstra- tions in each prompt. EE-ICL and EE-GL use the same task instruction and demonstrations. The only difference that EE-GL provides the candidate trigger shares identified by the ETC method. The prompts are presented in Figure 10 and Figure 11.
Event
Test
Our Test
Ratio
EF
204
174
85.3%
ER
282
200
70.9%
A.1.3 Hyper-parameters
EU
346
193
55.8%
EO
1138
165
14.5%
Note that for the reasoner, we apply Self- CoffCoonsittet Chain-of- Chain-of-Thougee -h-ughtss (SC-CoT) prompting (Wang et al., 2022b). We show the hyper-paramete set- tings in Table For EAE, we use a very low temperature 0 to generate stable outputs.
Table 4: Dataset statistics about the number of docu- ments for the test set (# Test) and the test set in our experiments (# Our Test).

---

EE
G Generate hare Descriptions
Please describe the shares according the document content. Please ensure that the reply concise and clear, refrain from providing other explanations outputting additional content.
Demonstration:
Document After
reduction. Mr. Xunde holds 100 shares company's stock, Previously, the company had disclosed that the actual controllers the company, Mr. Huang S henggiu, had reduced his stock holdings 168,700 shares November 2018.
Please describe: 168,700 shares
Answer: 158,700 shares the number of shares that Mr. Huang Shengoiu, the actual controller of the company, reduced November 2018.
Document: On December 25, 2018, the pledgee securites company disposed of 464,900 shares of "Qianshan harmaccutical Machinery' stock pledged by P eng Xunde due to defauli, with transaction amount of 1,904,900, 0 yuan and average transaction price of 4.097 yuanishare. Prior to this passive reduction, Mr. Peng Xunde held 505,000 shares of the company's stock.
Please describe: 464, 464,900 shares
Answer: 464,90 shares are the number of shares disposed of by Mr. Peng Xunde due to default on stock pledge on December 25, 2018.
more demonstrations...
Document: (document)
Please describe: (hare)
Figure 7: The prompt (translated) of the summarizer for the equity underweight event (ChFinAnn dataset). The [document] and (share) denotes the input document and share, respectively.
Module
Hyper-parameter
Value
the number of epochs
the maximum number of retrieved guidelines retrieval threshold
3
Recall
0.95
SC-CoT trials
8
Reason
SC-CoT sampling temperature
Reflect
the score threshold to discard harmful guideline
Table 6: dyyer-paramese settings for event extraction.
A.1.4 Discussion on Single-F1 vs. Multi-F1
A.2 Relation Extraction Experiment Details
A.2.1 Prompts
More precisely, "Multi" denotes "multi-record" rather than "multi-event", which means there are multiple records for one event type in document. The fact that multi-record performance is lower than single-record performance is widely observed among all supervised approaches on this dataset. For example, Doc2EDAG (Zheng et al., 2019): sin- gle 82.3 vs. multi 67.3, GIT (Xu al., 2021): sin- gle 87.6 vs. multi 72.3 (averaged Fl scores), One possible reason data imbalance (only 28.5% of all documents contain multi-record events). Another possible reason is the difficulty of multi-record doc- uments. It's interesting that ICL approaches seem to be more robust against the number of records in documents than supervised approaches. Specif- ically, the gap between single-Fi and multi-FF is relatively low for Guideline Learning, as shown in Table This is out of the scope of this paper and we leave it as future work.
For guideline learning, we directly conduct multi- class relation classification. There are two main components:
The generalizer takes the instance sentence and one entity pair) as input and output the general form. This is decomposed into two steps: extracting relevant text pieces and ab- stract entity types. The prompt is presented in Figure 12. The generalizer combines the two responses (the text span and entity types) get the final general form.
2.
The reasoner takes the instance (a sentence and entity pair) and the retrieved guide- lines input and output the reasoning process, the predicted answer and the index of the used guideline. The prompt is presented in Figure 13.

---

EE Gneralizeer
Please generate general description of the given share description. This description focuses the trading behavior involved in shares. Please careful retain descriptions related trading behavior concise general possible. To ensure generality, de not contain specific company names, personal names, and dates.
Demonstration:
Share description: 48574700 shares are number of company shores held by Mr. Wang Yun before this reduction.
Answer: The shares held before the reduction.
Share description: 3670000
total number of shares that Daguan Investment sold through S Shanghai Stock Exchange's bulk trading system during implementation period reduction plan.
Answer: The total shares reduced through the bulk trading system.
demonstrations..
Share description: (description)
Answer:
Figure 8: The prompt (translated) of the generalizer for the equity underweight event (ChFinAnn datasct). The (description) denotes the imput share description.
For RE-ICL, we apply thain-of-thoure prompt- ing. The prompt is presented in Figure 14. We use 10 demonstrations for the reasoner and RE-ICL.
G2: Jinying Technology actively re- duced its shareholdings of this company by 200,000 shares today. This triggers EU event.
A.2.2 Hyper-parameters
General form of input case: One com- pany actively reduced its shareholdings of another company.
Note that for the reasoner and RE-ICL, ap- ply Self-Consistent Chain-of-Thoughes (SC-CoT) prompting (Wang et al., 2022b). We show the hyper-parameter settings in Table
General form of G1: One company pas- sively reduced its shareholdings of other company. This does trigger an EU event.
A.3 Discussion on Generalizer
The intuition of the generalizer in two folds. First, the guideline should have some generalizability to cover/handle similar cases. Secondly, in practice, the generalizer is helpful to the guideline retrieval task based on the input case. If the guidelines are composed of corrected error cases, the retrieval would be case-to-case, which is very sensitive. For example in the following quote block, the input case is more similar G1 literally. However, G2 is more relevant as they both describe an active underweight event. we generate their general form by abstracting common properties (company name, number of shares, date), it will be more similar to G2.
General form of G2: One company ac- tively reduced its shareholdings of an- other company. This triggers EU event.
In our experiments, implement the general- izer by few-shot prompting an LLM agent. Though the generalizer critical for the GL framework, we don't 't put the implementation details into section 3 as there may other underlying implementations, for example, finetuning a more effective general- izer, and we want to highlight our contribution on the guideline learning framework itself.
A.4 Discussion on Rule-following Capabilities
Input case: Xinguang Investment ac- tively reduced its shareholdings of this company by 300,000 shares.
We manually evaluate the following aspects of re- sponses from Reasoner: 1. relevant: whether the rules referred by the model are truly relevant the instance; well-referred: whether the model genuinely follows the rules, i.e. the response is consistent with the rules refers to. We analyze 50 responses generated by the gpt-3.5-turf and gpt-4
G1: Xinguang Investment passively re- duced its shareholding in the company by 300,000 shares. This does not trigger an EU event.

---

EE Reasoner
Please detarmine whether the number of shares meets the given statement based on the share description and guidelines. If you need to refer to guideline, please refer the one is most relevant to the share description. there is no guideline to refer to, please make direct judgment. The output format is ("reasoning": "reasoning process', "used guideline": "Index of the used guideline", "answer": "yes or no").
Share description: 168,700 shares are the number of shares that Mr. Huang Shengqiu, the actual controller of the company, reduced November 1, 2018.
Statement The shares reduced by an equity holder equity underveight
Guideline None.
Answer: ("easoning": 'According the share description, number shares that Mr. Huang Shenggiu reduced. Without any guideline, only make direct judgment. The shares reduced the shareholder (Mr. Huang Shengqiu) meets the statement, therefore the answer yes.", "used_ guideline": "1", "answer": "yes")
Share description: 464,900
the mumber of shares disposed by Mr. Xunde due default stock pledge December
2018.
Statement The shares reduced by an holder equity underveightevent
Guideline:
shares held before passive reduction The disposed due default stock pledge the the statements.
number shares held reduction shares disposed due Peng Xunde's default stock Answer: "reasoning": "According the description,
pledge.
guideline description guideline shares disposed default pledge by therefore the yes.", "used_ "used_guidellne" "answer": "yes")
shareholder Peng Xunde)
more demonstrations...
Share description: (descritiin)
Statement The shares are reduced by an equity holder in equity underweight event. (jetrieve_ guideelies))
Guideline:
Answer:
Figure 9: The prompt (translated) of the reasoner for the equity underweight event (ChFinAnn dataset). The (description) denotes input share description. (retrieved_guidelinee denotes guidelines re- trieved from the knowledge base.
agents". The results (accuracy) are shown in Table 8. We find that gpt-3.5-tune is capable of figur- ing out and following relevant rules, while gpt-4, known as the most powerful LLM, makes fewer mistakes, We utilize gpt-3.5-turbo as the backbone LLM in our experiments, which is sufficient to verify framework. Moreover, framework may potentially gain additional advantages from the increasing fule-following capabilities of these backbone LLMs.
"gpt-3.5-turbo-003 and gpt-4-0613

---

EE L
Please perform the event extraction task. Please output table in markdown format, where the first row is the event roles, and the and order the event roles should be consistent with those given in the question. Starting from second row, represents the event arguments an event record, each argument must text span document. The shareholder can only include one company name or person's name. for company names, you should extract the full name of a company Instead of its abbreviation. If the document does not provide the event role, output "none". Demonstration:
Document: Announcement of Shanghal Furen Industrial (Group) Co., Ltd. on the Reduction of hhareholding by Company Shareho iders. The Board of D irectors of our company and its directors guarantee that the Information contained In this report does not contain any false records, misleading statements, major omissions, and
individual and joint responsibiltt completeness iss contents, Our company received a notice from) Development Co., Ltd. (hereinafter referred to as inli Company) on May 24, 2012, stating that from March 21, 2012. to May 23, 2012. inii Company had cumulatively reduced Its holdings of our company's outstanding shares 2.321,997 shares the secondary market of the Shanghal Stock Exchange, with average price 4.35 capital.
the truthfuiness, accuracy, and
yuanishare, accounting for 1.31% of company's total shares of our company's outstanding shares, accounting for 3.57% of our company's total share capital.
the of May 2012, inli Company held 6,348,746
Question: Please extract all equity underweight events and output markdown table. The following elements of each event should be included: shareholder, number shares traded, start date, end date, number of shares held after the transaction, and average
traded date end date -1-1-1-1-11 Development Co., Ltd. 2,32219977 shares March 21, 2012 May 23, 2012 6,348,746 shares (4.35 yuan
shareholder
transaction average price | |-|
more demonstrations...
Document dlccument)
Question: Please extract all equity underweight events and output markdown table. The following elements of each event should be
included: shareholder, number shares traded, start date, end date, number of shares held after the acaiin and average Answer:
Figure 10: The prompt (translated) of the EE-ICL the equity underweight event (ChFinAnn dataset). [document] denotes the input document.
EE G
Please perform event extraction task. Please output table markdown format, order of the event roles should be consistent with those given in the question. Starting from the second row, each row represents the event arguments of event record, each argument must text span document. shareholder can only include one company name person's name. For company names, you should extract full name of a company instead of its abbreviation. the document does not provide event role, output "none".
row event roles, the names
Damonstration:
Document Announcement Shanghal F uren Industrial (Group) Co., on the Reduction Shareholding by Company Shareholders. Beard Directors of our company and its directors guarantee that the information contained this report does not contain any false records, misleading statements, or major omissions, and assume Individual and joint responsibilty the tuthfulness, accuracy, and completeness of contents. our company received a from jini Development Co., (hereinafter
as Company) May 24, 2012, stating that from March 2012, May 23, 2012. Company cumulatively reduced Its holdings of our company's outstandiing shares by 2.321.997 shares on the secondary ma nket the Shanghai Stock Exchange, with an average price 4.35 yuanishare, accounting for 1.31% our company's total share capital. As of close May 23, 2012. Inll C Company held 6,348,746 shares of our company's outstanding shares, accounting 3.57% of companys total share capital. Question: known that 2,321,997 shares, 6,348.746 shares may be involved equity undenveight events. Please output following elements for share number
event: shareholder, number shares start datr, date, number shares hald
transaction,
average price.
In inll Development Co., Ltd. 2,,221 shares March 2012 May 23, 2012 66,48,7746 shares 4.35 yuan transaction average price I'n (/n--I-1 1-1-1-111 1-1-1-1
Answer: sharaholder number shares traded start end date number of shares
demonstrations...
(focument)
Question:
eventts it known that (hares) may be involved in equity undenweight events. Please output the following elements each share number involved event: shareholder, number shares traded, date, end date, number of shares transaction, average price.
Answer:
Figure II: The prompt (translated) of the EE-GL for the equity underweight event (ChFinAnn dataset). The [document] and (shares) denotes the input document and candidate trigger shares identified by previous ETC methods, respectively.

---

RE Generalizer
Find the text span in the quoted sentence that may indicate the relation between two entities. Remove irrelevant words in the text span and make sure your answer only text span.
Context: These city dwellers have sunk into aboominatons, after raiin.
Entitles: city dwellers and abomiiations
city
sunk abominations
more demonstrations....
Sentence)
Entities:
RE G eneralizerr
expert. You knowledgeable in taxonomy. category entity. that category should and procise. example, the following catagory Location, Organization, Product, Action, Time. Your should only
Sentance: (eenence)) The catagory (ancties)
Figure 12: The prompt of the generalizer in RE, The (sentence) and (entities) denotes the input sentence and the entity pair, respectively.
RE reasoner
You are knowledgeable
You will solve the relation extraction task. Given the context, will first consider whether the most precise relation between two entities belongs the following nine possible relations. yes, you will the most precise relation, otherwise you will output NULL:
CAUSE AND EFFECT: and have CAUSE AND EFFECT relation If x an event object leads to effect that PRODUCT AND PRODUCER: X and have PRODUCT AND PRODUCER relation is producer that causes product Y to exist.
INSTRUMENT AND AGENCY: x and have INSTRUMENT AND AGENCY relation if
Instrument
relations
The output format should be (C"reasoning": "my reasoning process", "used, guideline". "the index of the guideline that used the question", "answer": "the most precise relation"))
Damonstration:
Context: The apple blossom season usually runs from mid-apri earty
Entities: apple and blossom
Guideline: None.
Answer: ("reasoning': 'According to the meaning of the Context, the 'blossom' is comgonent of an apple tree. Therefore, the most precise relation between apple'' and blossom's is COMPONENT AND WHOLE." "used_ guideline: +1", 'answer": "PRODUCT AND PRODUCER")
Context: public brand products were donated to charities.
Entitie: brand and charities
Guideline: 1. * donotad Y* indicates that the relation between X and Y ENTITY AND DESTINATION. 2. * caused Y indicates that the
relation between X and Y is CAUSE AND EFFECT.
Answer: ("rassning": "The context indicates the brand (X) is the entty that is being donated and the charities (Y) are the destination towards which the brand products being danated. Therefore, the most precise relation between brand and 'charities' is ENTITY AND DEST DESTINATION::
lused_ Boceed__ideiiiet.. '1", 'answerf: "ENTITY AND DESTINATION")
more demonstrations...
Context Sentence)
Entiies: (entlies)
Guideline: puidelde_eeeleess
Answer:
Figure 13: The prompt of the reasoner in RE. The (sentence), (entities), and {retrieved_guidelings denotes the input sentence, entity pair, and the retrieved guidelines, respectively.

---

0.88 RE ICL
You will solve the relation extraction task. Given the context, will first consider whether the most precise relation between two entities belongs the following nine possible relations. will output NULL:
You are knowledgeable
will output the most precise relation, otherwise you
CAUSE AND EFFECT: and Y have CAUSE AND EFFECT relation If event or object that leads to effect
INSTRUMENT AND AGENCY: x and have an INSTRUMENT AND AGENCY relation X an agent that uses Instrument
PRODUCT AND PRODUCER: X and have PRODUCT AND PRODUCER relation X is producer that causes product Y to exist.
relations
The output should be (( reassonig:": "my reasoning process", "answer": "the most procise relation"))
Demonstration:
Context Public brand products were donated charttis.
Entibis:: brand and charises
Answer: t"rasoning": "The context indicates that the brand is the enthy that being danated and the charttes are the destination towards which the brand products are being denated. Therefore, the most precise relation between fbrand' and 'charlies' is ENTITY AND DESTINATION.: "answer":
'ENTITY AND DESTINATION'S
more demonstrations...
Context Sentence)
Entities: (entlles)
Answer:
Figure The prompt of the RE-ICL RE. The (sentence) (entitien tintiiies) denotes the input sentence the entity pair, respectively.
Module
Hyper- pperm-prreetter
Value
the number of epochs
the maximum number of retrieved guidelines retrieval threshold
Recall
0.92
SC-CoT trials
Reason
SC-CoT sampling temperature
Reflect
the score threshold to discard harmful guideline 0
Table 7: Hyper-parameter settings for event extraction.
Model
Relevant
Well-Referred
gpt-3.5-turbe gpt-4
0.84
0.94
0.98
Table 8: Manual evaluation of rule following capabilities.


Reviews:
[{"review_id": "osx5Ri6afz", "reviewer": "Reviewer_6ZCo", "summary": "", "questions": "A. Have the authors thought about the faithfulness of the rules returned by the Reasoner? In Lines 232-233, it is mentioned that R* is the set of rules that \"the LLM refers to as being helpful during reasoning\". Are these rules truly relevant to the instance and are they actually used by the LLM? \n\nB. ~~Because the rule scoring and rule update rely on the true label y, instance selection can only be done over the instances that have labels.  This might be ok when compared with fully-supervised models, but the authors should note that this is not true few-shot learning (where the assumption is that few-shot examples are provided as input and the model does not have access to any other labels)~~\n\nC. The generalizer is only mentioned in Section 4.3 and not in the methodology part (Section 3).  Can you provide some intuition on why the generalizer is needed or show some results that ablate this component? If the generalizer is critical for the model to work, I'd suggest moving it into Section 3.\n\nD. In Table 2, it seems that guideline learning is not helpful for multi-event documents (75.0 F1 for ICL vs 74.4 F1 for GL), can the authors explain this behavior? \n\nE. Can the authors think of a way to apply this framework to datasets that do not have readily available guidelines?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments, and they are exceedingly helpful for us to improve our paper. We feel sorry to not include the mandatory Limitations section, but we do address the limitations of our approach in the conclusion (as reviewer B35m summarized). We have appended Limitations section at the end of this comment and will add it to our next revision. It appears there are some misunderstandings regarding our framework, and one important thing we want to highlight is that our model **does NOT** need annotated guidelines. \n\nIn the following, we address your concerns one by one. \n\n**Q: The idea is related to instruction learning/induction, which aims to automatically generate or rewrite the instruction of the task. The guideline in this paper can also be viewed as part of the task instruction.**\n\nA: We tend to think of Guideline Learning (GL) as an innovative approach (as reviewer NFNq said) in the realm of instructional learning/induction. Our GL framework is designed to tackle complex tasks, learning guidelines iteratively from many error cases. In contrast, previous work, particularly the reference you provided, focuses on learning simple and clear instructions, prompting the LLM to predict the instruction contextually in a single pass. Specifically, GL differs from two main branches of previous work on instruction learning:\n 1. **Instruction induction via ICL.** Honovich et al. [2] predict the task instruction by prompting the instruction-tuned LLMs. They conduct explorative experiments, focusing on tasks that have \"clear and simple instructions\". Our GL framework focuses on more complex instructions with a highlight on IE tasks: extraction of complex concepts (event/relation). We propose the \"guideline\" as a bridge to learn and utilize more specific instructions from error cases automatically, which can be viewed as an in-depth extension of previous work. \n\n2. **Instruction learning for meta-training.** Ye et al. [3] propose to utilize instruction learning to better finetune LLMs and boost the zero-shot performance. Our GL framework aims at boosting the model performance under the tuning-free setting, which is orthogonal to their work. \n\nTo the best of our knowledge, we are the first to apply instruction learning (specifically guideline learning) to alleviate the \"underspecified task description\" problem for information extraction. (Please note that we have revised the phrase \"conceptual bias\" to \"underspecified task description\", following the suggestion from reviewer B35m.)\n\n**Q: Few datasets have rule-based guidelines, which limits the application of the framework. & Question E: Can the authors think of a way to apply this framework to datasets that do not have readily available guidelines?**\n\nA: It seems there may be a misunderstanding about our framework. It's important to note that our framework **does NOT** require the readily available guidelines of datasets. Our framework learns guidelines automatically from error cases. GL can be easily transferred to other datasets just with simple modifications on prompts. We've provided our prompts in the appendix for replication.\n\n**Q: Because the rule scoring and rule update rely on the true label y, instance selection can only be done over the instances that have labels. The authors should note that this is not true few-shot learning.**\n\nA: There is probably a misunderstanding. Our active instance selection **does NOT** require instance labels. We select instances according to the model's confidence in predicting the instance. The model's confidence is computed by measuring the consistency of multiple sampled responses of the model, where the true label is not necessary (detailed in Sec 3.4).\n\n**Q: Have the authors thought about the faithfulness of the rules returned by the Reasoner? Are these rules truly relevant to the instance and are they actually used by the LLM?**\n\nA: We conduct an additional experiment. Specifically, we manually evaluate the following aspects of responses from Reasoner: 1. **relevant**: whether the rules referred to by the model are truly relevant to the instance; 2. **well-referred**: whether the model genuinely follows the rules, i.e. the response is consistent with the rules it refers to. We analyze 50 responses generated by the gpt-3.5-turbo and gpt-4 agents. The results (acc) are shown below:\n\n| model | relevant | well-referred |\n| --- | --- | --- |\n| gpt-3.5-turbo | 0.84 | 0.88 |\n| gpt-4 | 0.94 | 0.98 |\n\nWe find that gpt-3.5-turbo is capable of figuring out and following relevant rules, while gpt-4, known as the most powerful LLM, makes fewer mistakes. We utilize gpt-3.5-turbo as the backbone LLM in our experiments, which is sufficient to verify our framework. Moreover, our framework may potentially gain additional advantages from the increasing rule-following capabilities of these backbone LLMs. \n\n**Q: Can you provide some intuition on why the generalizer is needed or show some results that ablate this component?**\n\nA: The intuition of the generalizer is in two folds. First, the guideline should have some generalizability to cover/handle similar cases. Secondly, in practice, the generalizer is helpful to the guideline retrieval task based on the input case. If the guidelines are composed of corrected error cases, the retrieval would be case-to-case, which is very sensitive. For example in the following quote block, the input case is more similar to G1 literally. However, G2 is more relevant as they both describe an active underweight event. If we generate their general form by abstracting common properties (company name, number of shares, date), it will be more similar to G2. \n\n> **Input case**: Xinguang Investment actively reduced its shareholdings of this company by 300,000 shares.\n> \n> **G1**: Xinguang Investment passively reduced its shareholding in the company by 300,000 shares. This does not trigger an EU event.\n> \n> **G2**: Jinying Technology actively reduced its shareholdings of this company by 200,000 shares today. This triggers an EU event.\n> \n> **The General form of input case**: One company actively reduced its shareholdings of another company.\n> \n> **The General form of G1**: One company passively reduced its shareholdings of another company. This does not trigger an EU event.\n> \n> **The General form of G2**: One company actively reduced its shareholdings of another company. This triggers an EU event. \n> \nIn our experiments, we implement the generalizer by few-shot prompting an LLM agent. Though the generalizer is critical for the GL framework, we don't put the implementation details into section 3 as there may be other underlying implementations, for example, finetuning a more effective generalizer, and we want to highlight our contribution on the guideline learning framework.\n\n**Q: In Table 2, it seems that guideline learning is not helpful for multi-event documents (75.0 F1 for ICL vs 74.4 F1 for GL), can the authors explain this behavior?**\n\nA: Though EE-GL-r is slightly below EE-ICL (74.4 vs 75.0), EE-GL-a surpasses EE-GL-r by a relatively large margin (76.2 vs 74.4). We suppose the reason is that some learned harmful guidelines in EE-GL-r slightly hurt the performance. \n\n**Q: Since some of the experiment settings (EE-GL-r, RE-GL-r) rely on random sampling in-context examples, the variance should be reported for results.**\n\nA: Thanks for your suggestion. Since manually annotating more examples requires some time, we will report the variance after we finish it. \n\n**If you have any further questions or concerns, please do not hesitate to inform us.**\n\n**Best regards.**\n\n[1] Memory-assisted prompt editing to improve GPT-3 after deployment. EMNLP 2022. \n\n[2] Instruction Induction: From Few Examples to Natural Language Task Descriptions. ACL 2023.\n\n[3] Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners. ICLR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Overall, this guideline learning framework is interesting and meaningful as it does not require fine-tuning the entire model. Although the paper only applies the idea to EE and RE, the idea is transferrable to a wide range of tasks.", "reasons_to_reject": "- **This paper does not include the mandatory Limitation section, and thus should be desk-rejected.** \n- The idea is related to instruction learning/induction, which aims to automatically generate or rewrite the instruction of the task. The guideline in this paper can also be viewed as part of the task instruction, the difference being only a subset of the rules are used for each instance. \n- Few datasets have rule-based guidelines (this might be the reason why not the paper does not include results for more popular RE datasets such as TACRED / DocRED), which limits the application of the framework. \n- Since some of the experiment settings (EE-GL-r, RE-GL-r) rely on randomly sampling in-context examples, the variance should be reported for results.", "questions_to_authors": "A. Have the authors thought about the faithfulness of the rules returned by the Reasoner? In Lines 232-233, it is mentioned that R* is the set of rules that \"the LLM refers to as being helpful during reasoning\". Are these rules truly relevant to the instance and are they actually used by the LLM? \n\nB. ~~Because the rule scoring and rule update rely on the true label y, instance selection can only be done over the instances that have labels.  This might be ok when compared with fully-supervised models, but the authors should note that this is not true few-shot learning (where the assumption is that few-shot examples are provided as input and the model does not have access to any other labels)~~\n\nC. The generalizer is only mentioned in Section 4.3 and not in the methodology part (Section 3).  Can you provide some intuition on why the generalizer is needed or show some results that ablate this component? If the generalizer is critical for the model to work, I'd suggest moving it into Section 3.\n\nD. In Table 2, it seems that guideline learning is not helpful for multi-event documents (75.0 F1 for ICL vs 74.4 F1 for GL), can the authors explain this behavior? \n\nE. Can the authors think of a way to apply this framework to datasets that do not have readily available guidelines?", "paper_topic_and_main_contributions": "This paper introduces a new paradigm of guideline learning (or more specifically guideline refinement) that can improve in-context learning. Given a few training examples and a guideline consisting of a set of rules, the model will update the rules for wrongly-predicted examples. Then the updated rules will be added into the context for model inference. \nThe paper evaluates this framework on the classification component of event extraction and relation extraction tasks and shows that it improves performance.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Missing related work on instruction tuning, especially using the model to generate instructions, which is similar to generating guidelines.\nTo list a few: \n- Honovich, Or et al. “Instruction Induction: From Few Examples to Natural Language Task Descriptions.” Annual Meeting of the Association for Computational Linguistics (2022).\n- Ye, Seonghyeon et al. “Guess the Instruction! Flipped Learning Makes Language Models Stronger Zero-Shot Learners.” ICLR (2023)", "Typos_Grammar_Style_and_Presentation_Improvements": "Presentation improvements:\n- - It would be beneficial to accompany Section 3 with some examples of the prompt and input/output. The current writing is very abstract. For example, can you give an example of a guideline before and after the update? \n\n\nTypos: \n- Line 48: fine-tuning LLM faces with the .... issues  -> fine-tuning LLM faces ... issues\n- Line 57: promising performances -> promising performance \n\n- Line 95: different with -> different from \n- Line 374: as all the event is related to -> as all event types are related to \n- Line 377: extra \"we apply guide\""}}, {"review_id": "3YXuNCMYHZ", "reviewer": "Reviewer_B35m", "summary": "", "questions": "A. I didn't really understand the observation that DE-PPN perform poorly\non multi-event documents (\"due to data imbalance\").  Please explain.\n\nB. Table 4 in the appendix is useful, but doesn't answer a question I had\nin assessing the event classification results.  The trigger expression\nseems very inclusive.  How often does it nominate an expression that\nmatches none of the classes you targeted?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed comments. We are glad that you like our presentation of the approach and its motivation. We agree that the phrase \"conceptual bias\" is imprecise and some claims are too strong. We would like to use the phrase \"underspecified task description\" instead. Besides, it seems that you are mostly concerned about some aspects of our experiments. We conduct some additional experiments for them. In the following, your comments are first stated and then followed by our point-by-point responses.\n\n**Q: It's not clear how this approach compares to alternative supervised approaches in a setting where annotation is scarce.**\n\nA: We conduct an experiment to evaluate the supervised model DE-PPN in the data-scarcity setting. Specifically, we train DE-PPN on (1) the **192 annotated documents** available to ICL approaches (50 documents per event type); (2) **5k annotated documents** (random sampled); (3) **all 29k annotated documents**. We use 20% of the data as the dev set. We compare several models on the same test set in our paper: 1. **DE-PPN**, 2. vanilla few-shot ICL (**EE-ICL**), 3. Our guideline learning approach (**EE-GL**). The F1 score of each event type is shown below: \n \n| model | EU | ER | EO | EF | \n| --- | --- | --- | --- | --- |\n| EE-ICL_192 | 60.9 | 86.8 | 67.3 | 52.1 |\n| EE-GL_192 | 63.8 | **88.0** | 70.6 | 56.2 |\n| DE-PPN_192 | 6.7 | 0.0 | 6.4 | 3.2 |\n| DE-PPN_5k | 56.6 | 68.0 | 58.0 | 56.8 |\n| DE-PPN_29k | **68.6** | 86.2 | **71.4** | **63.2** |\n\nWe find that DE-PPN fails when only providing 192 labeled documents, with very low F1 scores on all event types. The problem is alleviated when providing 5k labeled documents. DE-PPN relies on a large amount of annotated data to work well. This indicates the superiority of ICL approaches over data-hungry supervised approaches. Our guideline learning approach further improves the few-shot ICL approach (EE-ICL) on all event types.\n\n**Q: The approach includes a focused summarization step in response to limited GPT input buffers. While understandable, this expedient has an unclear impact on performance.**\n\nA: For event extraction (EE), we introduce two expedient changes: (1) we decompose EE into three sub-tasks: trigger identification, trigger classification, and argument extraction; (2) we introduce a summarization step to fit the limited context length. In order to exclude the influence of these factors on the performance, we use the exact same experimental setup as the GL method but do not provide any guideline to the reasoner, denoted as **EE-GL-w/o. gd**. The F1 score of each event type is shown below:\n\n| setting | model | EU | ER | EO | EF | \n| --- | --- | --- | --- | --- | --- |\n|1-step | EE-ICL | 60.9 | 86.8 | 67.3 | 52.1 |\n|3-step + summarization | EE-GL-w/o. gd. | 61.5 | 87.1 | 67.7 | 52.5 |\n|3-step + summarization | EE-GL-r | 63.4 | 87.9 | 70.1 | 53.9 |\n|3-step + summarization | EE-GL-a | **63.8** | **88.0** | **70.6** | **56.2**|\n\nWe find that though the two expedient changes can slightly improve the performance (0.6, 0.3, 0.4, 0.4) compared to EE-ICL, **the main performance gain comes from the learned guidelines (2.3, 0.9, 2.9, 3.7).**\n\n**Q:  The financial EE task seems relatively simple. However, the typical event extraction problem features: (1)  a large number of possible event types; (2) triggers that may refer to non-events.**\n\nA: In this paper, we decompose EE into three subtasks to apply our GL framework, with a highlight on the event trigger classification. When there are a large number of possible event types, the reasoner can't classify all event types in a single session, since it's hard to concretely describe all event types due to the limited input length. In our implementation, we decompose the multi-class event trigger classification into multiple binary classifications for each class. Specifically, we prompt the reasoner (LLM agent) to binarily classify whether the input instance triggers a specific event type. This alleviates the challenge at the cost of more rounds of sessions. After the trigger classification, we extract event records of each event type, given the triggers classified to this event type. The triggers that refer to non-events will not be provided in the next step. \n**To sum up, our implementation is capable of addressing the two typical challenges of event extraction without any modifications.** However, as our paper focuses on addressing the \"underspecified task description\" problem via guideline learning, these features are not highlighted. The cost of trigger classification is still relatively high, which calls for further research on efficient classification for a large number of event types. \n\n**Q: I didn't really understand the observation that DE-PPN performs poorly on multi-event documents (\"due to data imbalance\"). Please explain.**\n\nA: More precisely, the term \"multi-event\" should be \"multi-record\", which means there are multiple records for one event type in a document. The fact that multi-record performance is lower than single-record performance is widely observed among all supervised approaches on this dataset. For example, Doc2EDAG [1]: single 82.3 vs. multi 67.3, GIT [2]: single 87.6 vs. multi 72.3 (averaged F1 scores). One possible reason is data imbalance (only 28.5% of all documents contain multi-record events).  Another possible reason is the difficulty of multi-record documents. It's interesting that ICL approaches seem to be more robust against the number of records in documents than supervised approaches. Specifically, the gap between the \"single-F1\" and \"multi-F1\" is relatively low for Guideline Learning, as shown in Table 2. This is out of the scope of this paper and we leave it as future work.\n\n**Q: The RE experiments randomly sample test examples from the SemEval test set, complicating comparison with reported SemEval results and making it hard for follow-up research to replicate the paper's results.**\n\nA: We don't test the model on the whole SemEval test set in the current version due to limited funds to access the OpenAI API. We will make our test set publicly available for comparison and replication.\n\nAdditionally, we are having difficulty following your question B. Specifically, what is \"trigger expression\"?\n\n**We've appended our Limitations section below our rebuttal to Reviewer 6ZCo. If you have any further questions or concerns, please do not hesitate to inform us.**\n\n**Best regards.**\n\n[1] Doc2EDAG: An End-to-End Document-level Framework for Chinese Financial Event Extraction. EMNLP 2019.\n\n[2] Document-level Event Extraction via Heterogeneous Graph-based Interaction Model with a Tracker. ACL 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "In an area of active research and considerable interest to the NLP\ncommunity, the paper proposes an interesting approach and demonstrates\nits value convincingly.  The work will be of interest to practitioners\nattempting to address IE problems where annotated data is in short\nsupply.  Because IE admits so many variants, such problems are not\nhard to come by.  Thus, the practical contribution of the paper is\nreal.\n\nThe paper is well written and mostly clear.  The related work section\nis better written than in many papers, with a nice focus that helps\norient the reader to this relatively active research area.\n\nThe idea pursued in the paper is intuitive and described with\nsufficient clarity that the work should be eminently replicable.  The\npaper's appendices include ample information useful for other\nresearchers seeking to build on this work.", "reasons_to_reject": "It's not clear how significant this work ultimately is.  Yes, it\nappears to improve the SOTA in the area of in-context learning for IE,\nbut performance improvements over generic ICL, although consistent and\napparently significant, do not really approach those of the fully\nsupervised approaches.  In addition, it's not clear how this approach\ncompares to alternative supervised approaches in a setting where\nannotation is scarce.  The evaluated supervised baselines were all\ntrained on all available training data. It would be useful to see what\nperformance levels they achieve when limited to the 500 examples\navailable to the ICL approaches.  \n\nThe phrase \"conceptual bias\" is decidedly vague and doesn't really\nseem to correspond to the definition the paper gives it.  The problem\nthe paper targets is that the tasks we're asking the model to perform\nare chronically underspecified, because it's difficult to describe\nthem succinctly.  It's clear that this is a problem that impacts\ninter-annotator agreement in human annotation, so the idea that it\nnegatively impacts ICL is intuitive, and the paper's idea appealing.\nHowever, the paper makes the rather strong claim that \"the suboptimal\nperformance of in-context IE is attributable to...conceptual bias.\"\nThe paper does not substantiate this claim.  It shows that this kind\nof \"guideline learning\" provides lift, but the resulting models are\npresumably still suboptimal (e.g., they presumably lag behind human\nperformance).  Bottom line: I think a little more modesty in the\nframing of the work would be good.\n\nAfter admirable presentation of the approach and its motivation, the\nexperiments underwhelm in certain respects.  First, the approach\nincludes a focused summarization step in response to limited GPT3\ninput buffers that is not part of the retrieve-reason-reflect\nframework articulated in the paper.  While understandable, this\nexpedient has an unclear impact on performance.  Second, the RE\nexperiments randomly sample test examples from the SemEval test set,\ncomplicating comparison with reported SemEval results and making it\nhard for follow-on research to replicate the paper's results.\nFinally, despite the paper's framing of IE as complex, the financial\nIE task seems relatively simple--perhaps one reason it is shown to be\ncompetitive with DE-PPN in the experiments reported in Table 2.  In\ncontrast to the typical event extraction problem, where there are a\nlarge number of possible event types, this problem has four types.  In\nconventional event classification, triggers may refer to non-events.\nIt's not clear from the paper's explication whether and how much this\nis a feature of the problem the paper addresses.", "questions_to_authors": "A. I didn't really understand the observation that DE-PPN perform poorly\non multi-event documents (\"due to data imbalance\").  Please explain.\n\nB. Table 4 in the appendix is useful, but doesn't answer a question I had\nin assessing the event classification results.  The trigger expression\nseems very inclusive.  How often does it nominate an expression that\nmatches none of the classes you targeted?", "paper_topic_and_main_contributions": "The paper addresses the problem of information extraction (IE),\nspecifically event and relation extraction, through in-context learning\nusing instruction-tuned large language models, specifically GPT3.\nNoting mounting evidence that the strength these models exhibit for\nother NLP tasks does not necessarily extend to IE, the paper argues\nthat this is due to insufficient context for correct performance of\nthese relatively complex tasks.  It therefore proposes an approach\ncalled \"guideline learning,\" a pre-training phase in which the models\ngenerate elaborations of the guidelines in an error-driven fashion\nusing a modest number of labeled examples.  At test time, the\nguidelines, including the generated ones, as well as illustrative\ntraining examples, are retrieved to form part of the context used to\nprime the model for a particular example.  In experiments involving\nevent classification in a Chinese financial domain and relation\nclassification on a SemEval problem set, the paper demonstrates\nconsistent improvement over a generic in-context learning framing\nusing the same model, though performance lags behind fully supervised\nbaselines trained on many more examples.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The English in the paper is very good, but it had difficulty with\nnumber agreement between subject and verb in a number of places.  A\nnative English speaker could find and correct these things quickly.\nHowever, these grammatical discrepancies never damaged the clarity of\nyour explication.\n\nYou address limitations of the approach in the conclusion, and the\nlimitations you cite seem thoughtful.  However, it's more typical to\nhave a separate Limitations section.  I believe such a section does\nnot count toward the page budget, so moving this discussion to such a\nsection would both help the reader and buy you more space."}}, {"review_id": "aEJZt9yoFu", "reviewer": "Reviewer_NFNq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your positive feedback! We are glad that you think our GL approach is innovative and effective, and the error analysis and case study can provide some insights. In the following, we address your concerns one by one.\n\n**Q: The evaluation probably needs multiple runs, or at least addresses why it is not needed.**\n\nA: Thanks for your suggestion. We have set a very low temperature in our experiments, which makes the responses more focused and deterministic. We only do a single run for each experiment to lower our cost. We will also report the variance in our next revision.\n\n**Q: Only one dataset is used, for the two tasks.**\n\nA: Though we only conduct experiments on one dataset for EE and RE task, our framework can be easily transferred to other datasets, with simple modifications on prompts. We would like to apply the GL framework to other tasks in the future.\n\n**If you have any further questions or concerns, please do not hesitate to inform us.**\n\n**Best regards.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. GL is an interesting and promising direction in the era of LLMs. This paper proposes an innovative and interesting approach. \n2. Results suggest this framework is effective, compared to the vanilla ICL method.\n3. The error analysis and case study can provide some insights.", "reasons_to_reject": "1. GPT-3 is known to give unstable outputs, although that is more of an issue in text generation. The evaluation probably needs multiple runs, or at least addresses why it is not needed.\n2. Only one dataset is used, for the two tasks.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a Guideline Learning (GL) framework for improving the in-context learning (ICL) in information extraction (IE) tasks. \nThe authors identify a challenge in ICL, namely the conceptual bias, where the \"comprehended concept by LLMs might diverge from the target concept\". To address this, they introduce a GL framework that learns to generate guidelines from a few annotations during the learning phase, and retrieves helpful guidelines for better ICL during inference. \n\nThe authors demonstrate the effectiveness of their approach through experiments on event extraction and relation extraction tasks. The results show the GL framework can enhance ICL, although it still falls behind SOTA supervised models in general. The authors also manually labeled a test set (Equity Underweight), and found their approach outperforms supervised models in this particular case.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "L075: examples --> example"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new model for guideline learning/refinement to enhance in-context learning. The idea is to provide a few training examples and a set of rules (guidelines). If the model makes an incorrect prediction, it updates the rules. The updated rules are then used for model inference.

The paper makes significant contributions to an active area of research and would be of considerable interest to the NLP community. The idea pursued in the paper is intuitive and clear, and it is convincingly demonstrated.

The paper lacks a mandatory "Limitations" section, which violates the EMNLP policy and could result in desk rejection. However, the authors have included text for Limitations section as part of the discussion and will include in the next revision. Moreover, as part of the discussion, the authors conducted additional experiments.The paper is well-written and easy to follow, with only a few typos and missing references.
Decision: Accept-Main