GoLLIE: Annotation Guidelines improve Zero-Shot Information-Extraction
Authors: Oscar Sainz
Iker García-Ferrero
Rodrigo Agerri
Oier Lopez de Lacalle
German Rigau
Eneko Agirre
Venue: ICLR 2024 poster (2024)
Keywords: Information Extraction
Zero-Shot
Annotation Guidelines
Large Language Models
LLM
prompt
Primary area: generative models
TL;DR: We propose GoLLIE (Guideline-following Large Language Model for IE), a model able to improve zero-shot results on unseen IE tasks by virtue of being fine-tuned to comply with annotation guidelines.

Abstract: Large Language Models (LLMs) combined with instruction tuning have made significant progress when generalizing to unseen tasks. However, they have been less successful in Information Extraction (IE), lagging behind task-specific models. Typically, IE tasks are characterized by complex annotation guidelines which describe the task and give examples to humans. Previous attempts to leverage such information have failed, even with the largest models, as they are not able to follow the guidelines out-of-the-box. In this paper we propose GoLLIE (Guideline-following Large Language Model for IE), a model able to improve zero-shot results on unseen IE tasks by virtue of being fine-tuned to comply with annotation guidelines. Comprehensive evaluation empirically demonstrates that GoLLIE is able to generalize to and follow unseen guidelines, outperforming previous attempts at zero-shot information extraction. The ablation study shows that detailed guidelines is key for good results. Code, data and models will be made publicly available.

Published as a conference paper at ICLR 2024
GOLLIE: ANNOTATION GUIDELINES IMPROVE
ZERO-SHOT INFORMATION-EXTRACTIOON
Oscar Sainz', Iker Garcia-Ferrero"
Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, Eneko Agirre HiTZ Basque Center for Language Technology Ixa NLP Group University of the Basque Country (UPV/EHU) (oscar.saing sainz, iker. iker.garee garciaf)eehu.eus
ABSTRACT
Large Language Models (LLMs) combined with instruction tuning have made sig- nificant progress when generalizing to unseen tasks. However, they have been less successful Information Extraction (IE), lagging behind task-specific models. Typically, IE tasks are characterized by complex annotation guidelines that de- scribe the task and give examples to humans. Previous attempts to leverage such information have failed, even with the largest models, as they not able to fol- low the guidelines out of the box. In this paper, we propose GoLLIE (Guideline- following Large Language Model for IE), model able improve zero-shot re- sults on unseen IE tasks by virtue of being fine-tuned to comply with annotation guidelines. Comprehensive evaluation empirically demonstrates that GoLLIE is able to generalize to and follow unseen guidelines, outperforming previous at- tempts zero-shot information extraction. The ablation study shows that detailed guidelines are key for good results. Code, data, and models are publicly available: /tttps://gite thub. com tzzitzz-ai zen.
INTRODUCTION
The task of Information Extraction (IE) is highly challenging. This challenge is evident in the detailed guidelines, which feature granular definitions and numerous exceptions, that human anno- tators must follow to perform the task. The performance of current SoTA models heavily depends on the quantity human-annotated data, as the model learns the guidelines from these examples. However, this performance significantly decreases when tested in new annotation schema (Liu al.) 2021a). The common practice in IE to achieve good results is to manually annotate each new domain and schema from scratch, almost transfer exists across application domains. Unfortunately, this is unfeasible, both, in terms of financial cost and human effort.
Recent advancements in Large Language Models (LLM) (Min et al. 2023 have enabled the de- velopment of models capable of generalizing to unseen tasks. Thus, current zero-shot IE systems
* Equual contribution
GAGe GPT-3.5
SOTA
GOLLIE
63
63
57
47
Restaurant
Music
Figure Out of domain zero-shot NER results. GPT results are not available for domains.

---

Published as a conference paper at ICLR 2024
leverage the knowledge encoded in LLMs to annotate new examples (Sainz al. 2022a: Wang et al. 2023a). As a by-product of the pre-training process, models possess now a strong represen- tation of what person
organization is. Therefore, they can be prompted to extract mentions of those categories from a text. However, this has a clear limitation: not every annotation schema") defines "person" (or any other label) in the same way. For example, ACE05 (Walker et al., (006) annotates pronouns as persons, while, CoNLL03 (Tjong Kim Sang & De Meulder] (2003) does not. IE tasks require more information than just label names, they require annotation guidelines.
Current LLMs have been trained to follow instructions, but they fail to follow annotation guidelines out of the box. For instance, Figure | shows results on comain-specific zero-shot Named Entity Recognition. The results of gpt-3.5-turbo when prompted with guidelines (Zhang et al. 2023a) are low, around 20 F1 score on Music or Politics domains. Building a system that enables high- performance zero-shot information extraction, reducing the dependence on costly human annota- tions, remains an open challenge.
In this work, we present ( GoLLIE (Guidelinefollowinng Large Language Model for IE), an LLM fine-tuned to learn how to attend to the guidelines on a small set of well known IE tasks. Compre- hensive zero-shot evaluation empirically demonstrates that GoLLIE outperforms the SoTA (Wang ct al. 2023a) in zero-shot information extraction (see Figure T
2 RELATED WORK
Large Language Models (LLMs) have made significant advancements toward the development of systems that can generalize to unseen tasks (Min et al. (003). Radford et al. (2019) trained LLMs using vast amount of internet data, finding that pre-trained models given natural language task descriptions can perform tasks such as question answering, machine translation, or summarizing without explicit supervision. Building on this discovery, instruction tuning, often referred to as multitask fine-tuning, has emerged as the leading method to achieve generalization to unseen tasks. This process involves pre-training model on massive amount of unlabeled data and subsequently fine-tuning it on a diverse collection of tasks (Wang et al.) 2022 Chung et al. 2022) phrased as text-to-text problems (Raffel et al. 2020). A natural language instruction or prompt is given to the model to identifyy the task it should solve (Schick & Schütze 2021 Scao & Rush 2021). Research
has demonstrated that increasing the parameter count of the language model (Brown et al. 2020),
coupled with improvements in the size and quality of the instruction tuning dataset, results in
hanced generalization capabilities (Chen et al., 2023 Zhang et al. 2022; [Chowdhery et al. 2022]
Muennighoff et al. 2023 Touvron et al. 2022a b). LLMs have displayed impressive zero-shot gen- eralization capabilities in various challenging tasks, including coding Wang & Komatsuzaki (202 I);
Black et al. (2022): Rozière et al. (2023), common sense reasoning Touvron et al. (2023a). and
medical applications Singhal et al. (2023). among others.
In the field of Information Extraction (IE), recent shared tasks (Fetahu et al. 2023) have shown that
encoder-only language models such as MoEMRaaERTTa (Conneau et al., 2020) and mDEBERTA
(He et al.) 2023) remain the most effective models. Attempts utilize LLMs and natural language instructions for IE have been less successful (Tan et al., 2023 Zhou et al.] 2023 Zhang et al.) 2023a), as their performance lags behind that of encoder-only models. Before the billion parameters
T.LMs, indirectly supervised methods improve zero-shot IE by utilizing the knowledge learned from tasks like Textual Entailment (Sainz et al. 2021 2022a b) and Question Answering (Levy et al.) 2017). Obeidat al. (2019) propose an entity typing method that encodes label descriptions from Wikipedia as embeddings using an LSTM, which is then used to score the inputs. Methods that leveraged external knowledge were also successful on fine-grained zero-shot NER (Chen et al., 2021]. Lu et al. (2022a) introduced a unified text-to-structure generation that can model different IE tasks universally. Lou al. (2023) proposed converting IE tasks to a semantic matching problem, allowing their method to generalize to new domains and label ontologies not seen during training. Wang et al. (2023a) framed IE tasks as natural language descriptive instructions and trained an LLM across a diverse range of IE tasks. In evaluations on tasks with unseen label ontologies, their model outperformed other instruction-tuning -utinng methods.
'ee define schema as the set of labels and their definitions.
2

---

Published as a conference paper at ICLR 2024
Schema definition
The folloxing lines describe the task definition @totaclass
class Progrowinggooggggeeeeettt
Guideliine are introduced as docstrings
""Rffers to programning language used in the development of At applications research. langunge, such as Java and Python.
programing
Representative condidates are introduced as commants
spon: str
Such es: "Dava", 'R", "CLIPS", "Python",
Blataclass
class Metric(Entity);
Labals are defined as python classes
"tafees to evalustion setrics used to assess the performance of Al models and algorithes. Annotate spectfic metrics 1ske Fa-score,
span: str
Such as: "rean squared enror", "DCG",
Input text
This the text to anolyze
text "Hore accuracy is neasured by error rate which is defined as..."
Output annotations
annotation instances take place text above are listed result
Annotations are represented as instances
peppeee(pp""cccursse Metric arror rate
Figure 2: Example of the input and output of the model.
Most instruction tuning attempts for IE share a limitation: they only consider label names in the prompts (e.g.. "List all the Persons"). This poses two major challenges. Firstly, not all datasets share the same definition for labels like Person (some exclude fictional characters or pronouns). Secondly, a label name alone doesn't i sufficiently describe complex or less common labels. While there have been attempts to prompt LLMs using guidelines (Zhang et al.) 2023a), strong prior knowledge of LLMS regarding task labels (Blevins et al. 2023) deter the model from adhering to those guidelines.
3 APPROACH
Different from previous approaches, GoLLIE forces the model to attend to the details in the guidelines, performing robustly on schemas not seen during training. On this section we deep dive into the details of our approach, describing how the input and output was represented and the regu- larization techniques used to force the model to attend the guidelines.
3.1 INPUT-OUTPUT REPRESENTATION
We have adopted Python code- representation (Wang et al. 202.3b Li et al. 2023) for both the input and output of the model. This approach not only offers a clear and human-readable structure but also addresses several challenges typically associated with natural language instructions. It enables the representation of any information extraction task under a unified format. The inputs can be automatically standardized using Python code formatters such Black. The output is well- structured and parsing it is trivial. Furthermore, most current LLMs incorporate code in their pre- training datasets, indicating that these models are already familiar with this representation.
Figure shows the three main parts of the format: schema definition, input text, and output annota- tions. Schema definition forms the initial segment of the input. This section contains information about the labels that are represented as Python classes; guidelines, articulated as docstrings; and representative annotation candidates presented in the form of code comments. The number of class definitions corresponds to the number of labels in the dataset. Classes are flexible and vary for each task. For example, classes for a NER dataset merely require an attribute to specify the text span that corresponds to the class. On the other side, more complex tasks such as Event Argument Ex- traction (EAE) or Slot Filling (SF) demand more class attributes to categorize the task, such as a list of participants in an event (refer to examples in Appendix (A). Input text is the second part of the input. The input text is represented as a string variable Python. Output annotations the part generated by the model. The model starts generating after result The annotations are represented as a list of instances of the classes defined on the schema definition part. Parsing the output is straightforward; executing the generated code in Python yields list containing the result. This ease of parsing the output stands as a significant advantage of our model. A further detailed analysis of the efficiency of this approach is available in Appendix H
3

---

Published as a conference paper at ICLR 2024
Alataclass
@dataclass
class Wciersbbiityaaaccceeeett
class wybnenab11eeaaaccccceee
"** Valnerabil ityPacch happens
corpeny by releasing describinng appropriate update, tan
mention:
List(sor)
tssues addressed: List[ate]
rention:
supported_
[stes]
triggers
winerrbiitty: Listfate)
patch, fixed, sodresses, ieplomorted, released
muinerbblc_oysten List(str]
releaser: List[ste]
cve: Lisifste] a vulierability idettifler
patche List[ste]
issues addressed: List[ste] suppotte__lattane Hist[str] the platfores that support the patch oulnerahi Tity: [ist[ste] a the vilrenabliity vuinersble, syster List[ste] i The affected systers releaser: List[str] The entityy relensing the patch patch: List[str] what was the patch about patch, number; Lististe] Numben on none of the patch List[stn] i The version of the vuinerable syston time: List[str] à khen was the patch impleenneed the date
the patch
nuterr List(ste]
system version: [SSt[ste)
time: [ist[stn]]
Figure 3: Example of the input representation. (left) example of an event definition w/o guide- lines information. (right) The same example but with guideline information as Python comments.
3.2 GUIDELINES ENHANCED REPRESENTATION
The main contribution of this work is the use of the guidelines as part of the inference process to improve the zero-shot generalization. An example of class definition with and without guidelines shown in Figure 3 Different datasets usually define guidelines in many different ways: some provide a complex definition of a label with several exceptions and special treatments and others just give few representative candidates of the fillers of the label. To normalize the input format, we included the label definitions as class docstrings and the candidates as a comment for the principal argument (which is usually mention or span). Complex tasks such as EAE or SF require additional definitions for the arguments or slots, to that end, we included small definitions comments on each class argument. In this paper, we will refer to the model without guidelines as Baseline and the model with guidelines as @ GoLLIE.
3.3 TRAINING REGULARIZATION
We want to ensure that the model follows the guidelines and does not just learn to identify specific datasets and perform correctly on them. To do this, we introduce various kinds of noise during training. This stops the model from recognizing particular datasets, recalling specific labels, or attending only to the label names rather than learning to follow the actual description for each label in the guidelines.
We applied the following regularizations. Class order shuffling, for each example, the order of the input classes is randomly shuffled. This makes it more difficult for the model to memorize entire task definitions. Class dropout, we delete some of the input classes randomly. By eliminating few classes from both the input and output, we force the model to learn to only output instances of classes defined in the input. This not only encourages the model focus on the schema definition but also minimizes the occurrence of hallucinations during inference. Guideline paraphrasing, we generate variations of the label definitions to prevent the model from easily memorizing them. We also think this will make the method more robust to different variations on the definition. Represen- tative candidate sampling, similar to what we do with the paraphrases, for each input we sample different candidates from a fixed pool of 10 per class. Class name masking involves substituting the label class names (e.g.. PERSON) with placeholders, such as LABEL_1. This prevents the model from exploiting the label names during training and forces it to attend and understand the guidelines.
EXPERIMENTAL SETUP
4.1 DATA
Evaluating zero-shot capabilities requires dividing the data into training and evaluation datasets. However, many benchmarks for Information Extraction based the same domain or share part of their schema. To ensure that the zero-shot evaluation is not affected by similar data, we have divided our set of benchmarks based on the domain of the data (a related topic is data contamination,
4

---

Published as a conference paper at ICLR 2024
Table I: Datasets used the experiments. The table shows the domain, tasks and whether for training, evaluation or both.
Dataset
Domain
NER
RE
EE
EAE
SP
Training
Evaluation
ACEOS Walker
2006
News
BCSCDK at. 2016
Biomedical
CoNLL 2001 Txny Kim Sang De Meukder 2003
News
DIANN (Palweggi
Biomedical
NCBIDiscaee (ssimay Logan
2012
Biomedical
Ontonotes 5 (Pramaa
News
[Ebuc
News
TACRED ZZhan di 20/
News
WNUT 2017 Dereeyys
2017
News
Brotewwitter Dereryas
CASIE Satyaaanne sinie mul 4020
Cybercrime
CrosaNEX L mi 2022
Mony
Mapolii 64M1
Riomedical
FabNER (Klimir FabNeR((Klinir < Staryy 2022
Science
HarveyNER (Chen
Twitter
(Lli 2r
Queries
MIT Restauraitss (Li al
2013
Queries
MuliNERD Tedesen
2022
Wikipedia
Wikilivents
Wikipedia
which we discuss in Appendix G For training we kept mostly datasets from News and Biomedical domains, for evaluation instead, used datasets from diverse domains. This approach helps to avoid introducing any noise into the evaluation process. Among the evaluation datasets we included CrossNER (Liu al.. 2021b), a dataset that is split into many domains, for simplicity, we will call each domain as a separate dataset: AI, Literature, Music, Politics, and Science. Also, we will refer MIT Movie and MIT Restaurant as Movie and Restaurant. Table contains the information about the data used in the experiments.
We have trained the model perform different tasks: Named Entity Recognition (NER), Relation Extraction (RE), Event Extraction (EE), Event Argument Extraction (EAE), and Slot Filling (SF). However, we only evaluated the model on the three main tasks of interest: NER, EE, and EAE. The other two tasks are added to the training data to add diversity and improve the flexibility of the model.
A few modifications have been made to two datasets to improve the quality of the model. First, the training data of Ontonotes was reduced drastically as it was automatically annotated. Second, the TACRED dataset converted from RE to SF to increase the complexity of the task. These modifications make our system not comparable with the state the art on those tasks. However, focus of interest is the zero-shot evaluation and, therefore, the benefits (see Appendix ( are more interesting than adding 2 more comparable points on the supervised setup. In the CASIE dataset, we detected that the annotated event spans are inconsistent. The models typically annotate a sub-string rather than the entire span. Therefore, we evaluate all the models based on the predicted event categories, without considering the exact text span. For arguments, we partial matching.
We use the guidelines released by the authors of each dataset (More details in Appendix FD. When such guidelines are not publicly available, we ask human experts create them, based on the anno- tations from the development split. The representative candidates are extracted from the guidelines when available, otherwise, the candidates are sampled from the the train split based on word fre- quency or manually curated based on the guidelines. Paraphrases are automatically generated using Vicuna 33B v1.3 (Zheng et al. 2023).
4.2 LANGUAGE MODELS AND BASELINES
Backbone LLMs: GoLLIE is a fine-tuned version of Code-LLaMM Rozière et al. 20233 Other backbone LLMs, such as LLaMA (Touvron et al. 2023a), LLaMA-2 Touvron et al. 2023b) or Falcon Penedo et al. (2023) were considered during the development, however, as approach uses code to represent the input and output, CodA model worked better on the preliminary experiments. In order to perform fair comparisons the baseline developed in this paper is based on Code-LLaMA as well. All the development of this paper was done with the 7B parameter version of CCode-LLamee but, for scaling analysis, we also trained the 13B and 34B parameter models.
5

---

Published as a conference paper at ICLR 2024
Table 2: Supervised evaluation results. "** indicates that results not directly comparable.
Dataset
SoTA
Baseline
13B
e 34B
ACE05NER
WWang ct al.
2023a
86.6
89.1
88.1 a8
89.4
89.6
ACE05RE
Lu ct at.
2022b
66.1
63.8
63.6g
67.5
70.1
ACE05EE
Lu al.
2022b
73.4
71.7
72.2gas
20.
21.9.1
ACE0SEAD
Lu cl al
2022b
54.8
65.9-ur
66.0+as
67.8 -0.
68.6-12
BC5CDR
Zhang ct al.
20236
91.9
87.5
87.5+a2
87.9-au
88.4-o2
CoNLL 2003
Lu al.
2022b
93.0
=2.9
92.8
93.0gua
93.1
DIANN
Zabala et a
2018
74.8
80.3
79,4=11
82.6a13
84.1=11
NCBIDisease
Wang ct al.
202.5a
90.2
g6.2
85.4ga
86.5g08
85.8g03
Ontonotes
g3.4gma
83.4gax
84.0g03
84.6gm4
RAMS
et al.
2021
48.6
48.9
48.7
49.62a1
51.2201
TACRED
56.6.n2
57.1
26.7
58.7
WNUT 2017
Wang et al.
2021
60.2
53.7
52.0gas
20.5gnv
54.3g04
Average
73.3
73.0ga0
23.92n
75.0
Training setup:
To train the models we use QLoRA eHu
2022
Dettmers et al. 2023 LoRA
freezes the pre-trained model weights and injects trainable rank decomposition matrices into linear layers of the Transformer architecture. In preliminary experiment, this setup outperformed fine- tuning the entire model the zero-shot tasks, while training much faster (more details in Appendix D.4). We applied the LoRA to all linear transformer block layers as recommended by Dettmers et al. (2023). The models were trained for 3 epochs with an effective batch size of 32 and learning rate of 3e-4 with a cosine scheduler. Our training infrastructure was 2 NVIDIA's A100 with 80gb each. More details about the training are given in the Appendix [
Comparable systems: Our main point of comparison is Instruct-UIE (Wang et al., 2023a) as it the approach closest to our system, but does not use guidelines. Another system considered for comparison is PromptNER (Zhang et al. 2023a), which proposes to prompt GPT-3.5 and T5 with definitions using Chain-of-Thoughe in order to perform few-shot NER. Different from us, they did not fine-tune the model to attend to the guidelines. For fair comparison, only considered the zero-shot results reported in the paper. In addition, other SoTA systems are added for comparison when results from Instruct-UID and PromptNER are available. Given that system is designed for the zero-shot scenario, the supervised experiments are intended to verify that our system does not degrade its performance. Thus selected, for the supervised scenario, those systems among SoTA that share the most comparable setting with us.
5 RESULTS
5.1 SUPERVISED EVALUATION
The results on the supervised datasets are shown in Table Comparing GoLLIE with the baseline, they both obtain very similar results, with an absolute difference of 0.3 F1 points on average. This expected, the baseline model implicitly learns the guidelines for annotating the datasets based on the data distribution during fine-tuning. In addition, despite the noise introduced to GoLLIE fine-tuning in order generalize from guidelines, the performance close to that of the baseline.
Compared to other systems our model achieves similar results in general. Focusing on the two datasets where our model under-performs significantly. WNUT and NCBIDisease, we find that task- specific techniques are still needed. For instance, [Wang et al. (2021) uses external knowledge to detect emergent and rare entities. In the NCBIDisisease dataset, models pre-trained on Biomedical domain corpora achieve the best results (Kocaman & Talby) (02T). (Wang et al. (023a) leverages
Flan-T5, which has great proficiency on Biomedical domain tasks (Singhal et al. 2022). These improvements, however, complementary our proposal.
5.2 ZERO-SHOT EVALUATION
The results on the zero-shot are shown in Table Overall, compared to the baseline, the results are improved significantly when using guidelines almost every dataset, with absolute difference

---

Published as a conference paper at ICLR 2024
Table 3: Zero-shot evaluation results. "*" indicates results obtained using the original code.
Dataset
SoTA
Baseline
13B
34B
BroadTwitter
39.0aus
49.5
11.4-1x
50.3
CASIEEE
33.9 +3.9.65
59.3
62.2=00
65.5:18
CASIERAR
47.9.14
50.0
52.6-a2
55.2
AI
(Wang ct al. 202
49.0
32.3
56.7
61.6:19
Literature
Wang et al 202.5g
47.2
39.4:07
62.7 e32
59.7=03
59, 1:26
Music
(Wang ct al.
20238
53.2
56.2+13
67.8
65.5 3.
68.4
Politics
Wang el
202.50
48.2
38.3
$7.2
$4.4=41
60.2
Science
(aag
202.50
49.3
37.1
55.5
56.2:10
56.3
E3C
59.8
59.0
59.0
60.0
FabNER
06.1
24.8
25.4gas
26.3
HarveyNER
23.2:04
37.3
:1.3
38 9:95
Movie
Wang et al.
2023a
63.0
43.4gm
63.0
62.5
62.4gM
Restaurants
Waan el 20458
21.0
31.3an
43.4gaa
49.8
52.7
MultiNERD
55.0at
76.0
77.5
77.2
WikiEventsNEeR
Sainz al.
2022b
+49.1
76.9
80.7
80.2
81.3
WikiFventsge
Sanz el al.
41226
#10.4
47.5gos
43.0ga6
45.7
47.0g19
WikiEventSEAR
Sainz ct al.
20226
35.9
51.6.us
51.9
52.5 122
50.7 soa
Average SoTA
42.6
45.4
58.4gas
58.3
60.0g10
Average
42.3ana
55.3
56.02na
57.2
of 13 F1 points on average. Despite dividing the evaluation benchmarks based on the domain, there always some overlap between labels of train and evaluation benchmarks. For instance, the datasets E3C and WikiEvents share large part of their schema with datasets like BC5CDR, ACE05, and RAMS. This phenomenon is reflected in the results.
GoLLIE surpasses by large margin the current zero-shot SoTA methods Instruct-UIE (Wang et al., 2023a) and Entailment based IE (Sainz al.] 2022b). Compared to Instruct-UIE, the main differ- ences are the backbone model, the amount of training data, and, the use or not of the guidelines. Instruct- UIE leverages the FlanT5 which is a T5 fine-tuned on 473 NLP datasets. With respect to the data, Instruct-UII leverages total of 34 IE datasets (counting different tasks as datasets) from diverse domains, we only leverage datasets. Contrary to our method they do not use guideline information. Still, our method performs significantly better suggesting that the guidelines have an important effect on the results.
PromptNER Zhang al. 2023a) also adds some definition information into the prompt in order to perform zero-shot NER. We compare our approach with them (represented as GPT-3.5) in Figure Although their approach leverages guidelines too, approach performs significantly better all datasets, showing that LLMs (even with 175B parameters) struggle to follow guidelines. They solve this by adding examples in the context but still far behind comparable setting (T5-XXL).
Seen unseen labels: Not all labels in the zero- shot datasets are unseen; there an overlap between the labels in the training and zero-shot datasets. Al- though these labels may have very different annotation guidelines, also report results the of labels to which it has not been exposed during training, to bet- ter understand the generalization capabilities of GoL- LIE. The list of seen and unseen labels, as well as an extended analysis available in Appendix B Figure aggregates the F1 scores across datasets for seen and unseen labels in the zero-shot scenario. All models ex- hibit slightly lower performance on unseen labels. For the baseline model, the performance drop is more pro- nounced. In contrast, GoLLIE demonstrates better gen- eralization ability, showing a smaller gap in Fl scores between the seen and unseen labels. Also, the gap is smaller the parameter count of our model increases.
beswite
7B
13B
34B
Urewwn
Figure 4: Seen vs unseen label zero-shot performance, results aggregated from all datasets.
7

---

Published as a conference paper at ICLR 2024
Model scaling: Recent research has shown that increasing the parameter count of language models leads to improved generalization capabilities Brown et al. (2020). Higher parameter count yields superior average zero-shot performance. However, some datasets and tasks greatly benefit from larger L.LM, while others do not. We believe that some datasets do not see benefits from increasing the L.M size because their performance is hindered by the issues with the guidelines that discuss in Section 5.3 While, in general, larger models achieve better results in both supervised and zero- shot settings, GoLLIE with a 7B parameter backbone already exhibits strong zero-shot capabilities.
5.3 ABLATION STUDY
We have performed an ablation to see the contribution of
Table 4: Ablation results.
several components in the zero-shot evaluation. We ana-
lyzed the different regularization techniques proposed in
Model
p-value
Section 3.3 Additionally, we represent the baseline, i.c. when removing all components including guidelines, "w/o all". Along with the mean zero-shot FI we also pro- vide the one-sided p-value with respect to GoLLIE.
GoLLIE
55.3202
w/o Shuffling
55.9ta2
7.2c
w/o Paraphrases
54.8gna
1.le_
w/o Masking
54.6.as
1.0e-1
w/o Dropout
54.0amz
4.0e
The class order shuffling, guideline paraphrasing, and class name masking seem have significant contribu- tion to the final result, while class dropout although sig- nificant improvements are small. As further explained Appendix D the loss is only computed over the result tokens, inherently limiting the model's po- tential to overfit the guidelines. In contrast, the representative annotation items give a stronger signal to the model. We see how definitions and representative candidates from the guidelines are complementary and help to improve each other.
w/o Candidates
49.9202
2.2e
w/o all (baseline)
42.3201
5.1e-13
ERROR ANALYSIS
In this section, we aim to better understand the effect of prompting LLMs with guidelines. We focus on specific labels across various datasets, with the results displayed in Table Our analysis covers both successful and unsuccessful cases of entity labeling by GoLLIE. For the latter, we also aim to identify the reasons why the model fails to correctly label these entities. Further analyses malformed outputs or hallucinations are discussed in Appendix
Table 5: This table shows the F1 scores for specific labels from different datasets. The guideline
column is small summary of the actual guideline used to prompt the model.
Dataset
Label
Guideline
Baseline
MultiNERD
Media
Titles of films. books, songs, albums, fictional characters and languages.
13.6
69.1
CASTE
Vul. Patch
When software company addresses a vulnerability by re- leasing an update.
27.7
70.5
Movic
Trailer
Refers to short promotional video or preview movic.
00.0
76.4
AI
Task
Particular rescarch task or problem within specific Al re- search field.
02.7
63.9
MuliNERD
Time
Specific and well-defined time intervals, such as eras, his- torical periods, centurics, years und important days.
01.4
03.5
Movic
Plot
Recurring concept, event, motif that plays significant role in the development of movie,
00.4
05.1
AI
Misc
Named entities that not included in any other category.
01.1
05.2
Literature
Misc
Named entities that Individual actively engaged in the creation literary works.
included in any other category.
03.7
30.8
Litcrature
Writer
04.2
65.1
Literature
Person
Person name that is not writer.
33.5
49.4
Science
Scientist
A person who is studying or has expert knowledge of nat-
02.1
05.8
ural science field.
Science
Person
Person that scientist.
46.1
45.9
Politics
Polit. Party
Organization compete particular country's elec-
11.2
34.9
tions.
8

---

Published as a conference paper at ICLR 2024
The details are in the guidelines: Labels such as MEDIA, VULNERABILITYPATON TRAILER, and TASK are inherently polysemous, making it challenging to determine the appropriate catego- rization based solely on the label name. As result, the baseline struggles effeectivly classify items under these labels due to having insufficient information. Conversely, GoLLIE successfully follows the guidelines, underscoring their utility.
When the annotations do not comply with the guidelines: In the case of the TIME label of the MultiNERD dataset, we found that our model labels years as TIME entities. This is correct according the annotation guidelines. Surprisingly, years are not labeled as entities in the dataset. In this case, GoLLIE successfully follows the guidelines; unfortunately, the dataset annotations do not.
Ambiguous labels: The MISCELLANEOUS category, used by CoNLL03 and CrossNER datasets, refers to any named entity that is not included the predefined categories set by the dataset. This definition is highly ambiguous and serves catch-all for various elements that do not fit into any of the predefined categories. Similarly, the PLOT category the Movie dataset is used to label a wide range of elements. For example, events in movie (e.g., murder, horse racing), characters (e.g., vampires, zombies), and the country of origin (e.g., British), among others. This lack of specificity hinders the development of consistent rules guidelines for tagging such elements (Ratinov & Roth 2009). which is problem for humans and machines alike. As consequence, GoLLIE also fails to label them accurately.
Conflicts Between Fine-Grained and Coarse Entities: The CrossNER dataset introduces two labels for person names within each domain. For example, in the Science domain, the labels SCIEN- TIST and PERSON are used. The former used to label any person that is not a Scientist. Similarly, the Literature domain includes the labels WRITER and PERSON. The guidelines assist GoLLIE in correctly labeling entities as WRITER. However, GoLLIE still categorizes individuals as Per- even when they are Scientist, despite the guidelines. This is not technically incorrect, as every scientist is, by definition, also a person.
Strong Label Preconceptions: In its Political domain set, CrossNER includes the label POLITI- CAL PARTY. GoLLIE outperforms the baseline, once again demonstrating the utility of providing the model with guidelines. However, we often find that the model categorizes political parties as or- ganizations. As listed in Table [ most of the pre-traiming datasets originate from the news domain, where political parties are common entity. However, none of the fine-tuning datasets include the POLITICAL PARTY entity; they are instead categorized as ORGANIZATION. Consequently, during inference, the model consistently labels political parties organizations. We believe this issue can be resolved by expanding the number and diversity of the fine-tuning datasets.
In summary, we anticipate that GoLLIE will perform well on labels with well-defined and clearly bounded guidelines. On the other hand, ambiguous labels or very coarse labels pose challenges. In this regard, we believe that GoLLIE would benefit from learning to follow instructions such as "Label always the most specific class" or "Annotate this class in the absence of other specific class". We also expect that GoL.LIE would benefit from expanding the number and diversity of the pre-training datasets.
7 CONCLUSIONS
In this paper, we introduce J GOLLIE, an LLM specifically fine-tuned comply with annotation guidelines that were devised to help humans annotate the dataset. A comprehensive zero-shot eval- uation empirically demonstrates that annotation guidelines are of great value for LLMss, as GoLLIE successfully leverages them. GoLLIE achieves better zero-shot results than previous attempts at zero-shot IE which do not leverage the guidelines, or use models not finetuned for following guide- lines.
GoLLIE is a significant progress towards the development of models that can generalize to unseen IE tasks. In the future, we plan to enhance GoLLIE by using larger and more diverse set of pre- training datasets. We will also improve the model's performance with ambiguous and coarse labels by expanding the set of instructions that the model can follow.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work has been partially supported by the Basque Government (Research group funding IT- 1805-22 and ICLALANG project. grant no. KK-2023/00094). We are also thankful to sev- eral 1103///11...009/511111111001 10001 1033 projects: (i) DeepKnowledge (PID2021-127770B-CZ2
and by FEDER, EU; (ii) Disargue (TED2021-1308) 10B OB-C21) and European Union NextGen- erationEU/PRTR: (iii) AWARE (THD2021-131617B 100- and European Union NextGenera- tionEU/PRTR. This work has also been partially funded by the LUMINOUS project (HORIZON- CL4-2023-HUMANNN0-22--2 1-37222))337244 Oscar Sainz is supported by a doctoral grant from the Basque Government PPRE_2033220000 Rodrigo Agerri currently holds the RYC-2017-23647 fel- lowship IMCINAAE//O..503555110000
and by ESF Investing in your future).
REFERENCES
Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Ho- race He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang. and Samuel Weinbach. GPT- NeoX-20B: An open-source autoregressive language model. In Angela Fan. Suzana Ilie, Thomas Wolf, and Matthias Gallé (eds.). Proceedings of BigScience Episode #5- Workshop on Chal- lenges & Perspectives in Creating Large Language Models, pp. 95- 136, Dirbual+Dublin, May 2022. Association for Computational Linguistics. doi: 10. wtgp/0///2222.. bi www..ciiiiggei.......o.
URL
Terra Blevins, Hila Gonen, and Luke Zettlemoyer. Prompting language models for linguistic struc- ture. In Anna Rogers, Jordan I.. hoyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL. 2023, Toronto, Canada, July 9-14, 2023, pp. 6649-6663, Association for Computational Linguis- tics, 2023, doi: 18.18653/v aaw.0023..llllo..661.11 URL
2023 1023.ac1-1e acl- long. 367.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Horbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tice Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL https: Kttps::/prooeedding neurisseeuppppp wttppppppee/2///aaa..... 1457c0d6bfcb4996 2tT.://88..44444444.8...88821222
8bfb8acc 42f64a Abst -
Pei Chen, Haotian Xu, Cheng Zhang, and Ruihong Huang. Crossroads, buildings and neighbor- hoods: A dataset for fine-grained location recognition. In Marine Carpuat, Marie-Cathering de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. Linguisties. doi: 2022. www.aaaalllcmmmm main. 243
Seattle, United States, July 2022. Association for Computational org/
URL https::/aalanttholl//
Yi Chen, Haiyunn Jiang, Lemao Liu, Shuming Shi, Chuang Fan, Min Yang, and Ruifeng Xu. An empirical study on multiple information sources for zero-shot fine-grained entity typing. In Marie- Francine Moens, Xuanjing Huang. Lucia Specia. and Scott Wen-tau Yih (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2668-2678 Online and Punta Cana, Dominican Republic, November 2021. Association for Computational org/
Linguistics. doi: 10. 00665///1/221.emmmmpppp main.210. URL www.hhtpps.. htt.p:://aclathhhaate 2021. www.p-nanlpp...e in..niii... .221
Zekai Chen, Mariann Micsinai Balan, and Kevin Brown. Language models are few-shot learners for prognostic prediction. CoRR, abs/2302. 12692, 2023, doi: 10.488550/a2iw..2322..cm 12692. URL
https://doi http:///oo..rrg/1///
www./org.........o. wttp://....0///00e111..o www.w.wwww.......m 12692
10

---

Published as a conference paper at ICLR 2024
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsyyashchenko, Joshua Maynez, Abhishek Rao. Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Lev- skaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowyez, Er- ica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee. Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Palm: Scaling language model- ing with pathways. CoRR, abs/2204.02311, 2022. doi: 00448550/r2iw.2244..222 https: //d::://ooo
URL
wwt.S//arxiv... 2204 02311
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery. Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang. Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction- finetuned language models. CoRR, abs/2210. 11416, 2022. doi: 10.485500arXi..221000 11466. URL https://den http:://dooooogg///// www.orgg.................. 4.8.0..4 / 2210 11416
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Na- talie Schluter, and Joel R. Tetreault (eds.), Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pp. 8440-8451, Association for Computational Linguistics, 2020. doi: 10.18653/v12020..e
747. URL
acl main. 747
Leon Derczynski, Kalina Bontcheva, and Ian Roberts. Broad Twitter corpus: A diverse named entity recognition resource. In Yuji Matsumoto and Rashmi Prasad (eds.), Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pp- 1179-1179, Osaka, Japan, December 2016. The COLING 2016 Organizing Committee. URL bttps://aclantholocyy -111:011111///011111111111111111 1111
Leon Derezynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. Results of the WNUT2017 shared task novel and emerging entity recognition. In Leon Derczynski, Wei Xu, Alan Ritter, and Tim Baldwin (eds.), Proceedings of the 3rd Workshop on Noisy User-generated Text, pp. 140-147, Copenhagen, Demmark, September 2017. Association for Computational Linguistics. doi: 10.18653/11/W117444 URL/EtTp:://aLLatttoloog:
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. CoRR, abs/2305. 14314, 2023. doi: 10.485550/rxxi.22355..33 URL https:
//doi. oew.loorg110. 10. 00..0088550550arr.. 2305. 14314
Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. Documenting large webtext corpora: A case study on the colossal clean crawled corpus. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 1286-1305, Online and Punta Cana, Dominican Republic, November 1005:3//12221..mmpp-mo
2021. Association for Computational Linguistics. doi: hhtpps://an http:::s/aaaaaatholloooo www.aaatttooolooooo/aoo.. wtt.:20mmmmmmmmmmmmmmmmmmm in..iii.. 98)
URL
Seth Ebner, Patrick Xia, Ryan Culkin, Kyle Rawlins, and Benjamin Van Durme. Multi-sentence ar- gument linking. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceed- ings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8057-8077, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v 1/2020 main. 718. h https: / anthologyae org. /2000/22220mmmm mm...mmannnn....

---

Published as a conference paper at ICLR 2024
Henmenegildo Fabregat, Juan Martínez-Romo and Lourdes Araujo. Overview of the DIANN task: Disability annotation task. In Paolo Rosso, Julio Gonzalo, Raquel Martínez, Soto Montalvo, and Jorge Carrillo de Albornoz (eds.), Proceedings of the Third Workshop on Evaluation of Human Language Technologies for Iherian Languages (IberEval 2018) co-located with 34th Conference of the Spanish Society for Natural Language Processing (SEPLN 2018), Sevilla, Spain, September 18th, 2018, volume 2150 of CEUR Workshop Proceedings, pp. 1-14. CEUR-WS.Org 2018. URL https: / ceur Ws. org/ hoog/Vol---- 2150/ overview diann task. pdf
Besnik Fetahu, Sudipta Kar, Zhiyu Chen, Oleg Rokhlenko, and Shervin Malmasi. Semeval- 2023 task 2: Fine-grained multilingual named entity recognition (multiconer 2). In Atul Kr. Ojha, A. Seza Dogruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Ku- mar, and Elisa Sartori (eds.), Proceedings of the The 17th International Workshop on Seman- tic Evaluation, SemEval @ ACL 2023, Toronto. Canada, 13-14 July 2023, pp. 2247-2265. As- sociation for Computational Linguistics, 2023. doi: 10. 10.18653/v11 10.18653/v1/2023..0m s/002/ 1.310. URL
https:
semeval-1. 310
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with disetat-dissettanlledd embedding sharing. In The Eleventh International Confer- ence on Learning Representatiinss, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL
net /aae/ppdff.. wwppfiid=-a. aE
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL http:///ppeeeeeiieeeee net//fortrroe wtw.rrmmiidneeeeeeeee.
Rezarta Islamaj Dogan and Zhiyong Lu. An improved corpus of disease mentions in PubMed cita- tions. In Kevin B. Cohen, Dina Demner-Fushman, Sophia Ananiadou, Bonnie Webber, Jun't ichi Tsujii, and John Pestian (eds.), BioNLP: Proceedings of the 2012 Workshop on Biomedical Nat- ural Language Processing, pp. 91-99, Montréal, Canada, June 2012. Association for Computa- tional Linguistics.
org/ M12-2411
Veysel Kocaman and David Talby. Biomedical named entity recognition at scale. In Alberto Del Bimbo, Rita Cucchiara, Stan Sclaroff, Giovanni Maria Farinella, Tao Mei, Marco Bertini, Hugo Jair Escalante, and Roberto Vezzani (eds.), Pattern Recognition. ICPR International Work- shops and Challenges, pp. 635-646, Cham, 2021. Springer International Publishing. ISBN 978- 3-030-68763-2.
Aman Kumar and Binil Starly. "fabner": information extraction from manufacturing process science domain literature using named entity recognition.
J. Intell. Manuf, 33(8):2393- URL https: http://ti..rrgg...../o org or../0..0/117
doi: 10.1007/110845-0210018866 2407, 2022.
s
Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. Zero-shot relation extraction via reading comprehension. In Roger Levy and Lucia Specia (eds.), Proceedings of the 21st Con- ference Computational Natural Language Learning (CoNLL 2017), pp. 333-342, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. (0.11853/11/KK17-1033
URL httpps://alaatthlodsttooo
1034
Peng Li, Tianxiang Sun, Qiong Tang, Hang Yan, Yuanbin WU, Xuanjing Huang, and Xipeng Qiu. Large code generation models are better few-shot information extractors. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61 Annual Meet- ing the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15339-15353, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 00.18653/11/0233, http:::://aaaaatiee
CodelE:
acl-long URL https: / aa//lllallla wwwanthology.....ooo....oooo /tg/ acl-iong. 855
Sha Li, Heng Ji, and Jiawei Han. Document-level event argument extraction by conditional gener- ation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Belt- agy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Compu- tational Linguistics: Human Language Technologies, pp. 894-908, Online, June 2021. Asso- ciation for Computational Linguistics. doi: 10.18653/v1/20 1/2021 mancii URL https:
7/acl antholl ogy. org/2221 naacl -main. 69
12

---

Published as a conference paper at ICLR 2024
Ying Lin, Heng Ji, Fei Huang, and Lingfei Wu. A joint neural model for information extraction with global features. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Anmual Meeting of the Association for Computational Linguisties, pp. 7999-8009, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.acl- mall--miin.1.... 713. wtt.httss:/ antho ogy wrg/ wttrggg2222cco2 acl00.aallmminn. 713
Xiao Ling and Daniel S. Weld. Fine-grained entity recognition. In Jörg Hoffmann and Bart Selman (eds.), Proceedings of the Twenty-Sixth AAAI Conference on Artificial Intelligence, July 22-26, 2012, Toronto, Ontario, Canada, pp. 94- AAAI Press, 2012. doi: 10.1609/AAAI 8122.
htgl 609/ aaai. w2w.1. 8122
Jingjing Liu, Panupong Pasupat, Scott Cyphers, and James R. Glass. Asgard: A portable architecture for multilingual dialogue systems. In IEEE International Conference on Acoustics, Speech and Signal Processing. ICASSP 2013. Vancouver: BC, Canada, May 26-31, 2013, pp. 8386 8390. 10. 10.
IEEE, 2013. doi: 10. 11111//CSSS.22203666990000 TCASSP 2013 663930T
URL https: /ecii
Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. Crossner: Evaluating cross-domain named entity recognition. In Thirty- Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Thire Conference on In- novative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Edu- cational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pp. 13452 13460. AAAI Press, 2021a. URL https: aaai. org. wwgg/nndxx...o php/AAAI/ AAAI/
article wttlvview//// ew757
Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. Crossner: Evaluating cross-domain named entity recognition. In Thirly-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 202 pp. 13462-13460. (t0.0/9/aai.3515......
AAAI Press, 2021b. doi:
URL https: / do org / 10. 1609/
aaai. v35115 17587
Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. Universal in- formation extraction as unified semantic matching. In Brian Williams, Yiling Chen, and Jennifer Neville (eds.), Thirty-Seventh AAAI Conference on Artificial Intelligence, AAAI 2023, Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence, IAAI 2023, Thirteenth Sympo- sium on Educational Advances in Artificial Intelligence, EAAI 2023, Washington, DC, USA, February 7-14, 2023, pp. 13318-13326 AAAI Press, 2023. doi: 10.1609/aaai.va v3771 117711.200 1.26563. URL https: //ol:////o..../ www.ddo..orr/o/oooooooo 10. 1 609 /aae// www.w11.111.... 26563
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. Unified structure generation for universal information extraction. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, Liriiiisniiiiiiiiiiiiiiiiiieie Linguisties, 2022a. doi: 10.18653/v 1/2022. aci 1- long. 395
2022, pp. 5755-5772 Association for Computational acl-long.395. URL https: /tw..//o../////// orq 10.1000100 18 21086653vvv122
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. Unified structure generation for universal information extraction. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 5755-5772, Dublin, Ireland, May 2022b. Association for Computational Linguistics. doi: 10. 1t1.18653/1122222.ad/ae acl-long....l- URL attps://aclantinn
org/ 2022. acl acl-long.................o 395
Inbal Magar and Roy Schwartz. Data contamination: From memorization to exploitation. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 157-165, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/ v v1/2022.acl ach-ortt. ht //acl anthol ogy ogg..ogg2...222.ooom/ acl waoellsshh.tt1. 18
13

---

Published as a conference paper at ICLR 2024
Bernardo Magnini, Begoña Altuna, Alberto Lavelli, Manuela Speranza, and Roberto Zanoli. The E3C project: European clinical case corpus. In Jon Alkorta, Itziar Gonzalez-Dios Aitziber Atutxa, Koldo Gojenola, Eugenio Martínez-Cámann
Álvaro Rodrigo. and Paloma Martínez (eds.), Proceedings of the Anmual Conference of the Spanish Association for Natural Language Processing: Projects and Demonstrations (SEPLN-PD 2021) co-located with the Conference of the Spanish Society for Natural Language Processing (SEPLN 2021), ), Málaga, Spain, September, 2021, volume 2968 of CEUR Workshop Proceedings, Pp. 17-20. CEUR-WS.org. 2021. URL https: cww.r//eerrreeeee ws worg/Vol--ao 2968 2tw.229668pppp.pp.e. pdf
Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth. Recent advances in natural language processing via large pre-trained language models: A survey. ACM Comput. Surv., 56(2), sep 2023. ISSN 0360- 0300. doi: 10. 145/3605943, URL hww.:tttp://ee http://oooorrg//// 10. 45/3605943
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong. Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak. Samuel Albanic, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning. In Anna Rogers, Jordan I.. Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 15991-16111 Association for Computational Lin- guisties, 2023, doi: 10. 10.18653/v1/20333 lon.o.n........o........ 891. URL https: /tte::/ddooo. wwt./o.g/1110.... v1/2023. acl ong. 891
Rasha Obeidat, Xiaoli Fern, Hamed Shahbazi, and Prasad Tadepalli. Description-based zero-shot fine-grained entity typing. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 807 814, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/ v1/N19-1087. URL /tLLttts//accaatteeee
org/N19-1081
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The refinedweb dataset for falcon LLM: outperforming curated corpora with web data, and web data only. CoRR, doi: 10.488500axxx..2366.111. URL https::///ao /tt:://ooi..... org/10.
abs/2306.01116, 2023.
48550/ 2306. 001.1.0166666. 01116
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. Towards robust linguistic analysis using OntoNotes. In Julia Hockenmaier and Sebastian Riedel (eds.), Proceedings of the Seventeenth Conference on Computational Natural Language Learning, pp. 143 152, Sofia, Bulgaria, August 2013. Associ- ation for Computational Linguistics. URL httppps://aaaatthhlogoooo org/ 0o0gg//w33351191
Ofir Press, Noah A. Smith, and Mike Lewis. Train short, test long: Attention with linear bi- ases enables input length extrapolation. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL net https:://ppenrew/ wwwp:eeeeeeetteeeeeeeeeeee. /tte://ooommo---000GS..
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8);9, 2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to- text transformer. J. Mach. Learn. Res., 21:140:1-140:61 2020. URL stttp://jml. org/
papers/v21/20-00 htmll
Lev Ratinov and Dan Roth. Design challenges and misconceptions in named entity recognition. In Suzanne Stevenson and Xavier Carreras (eds.), Proceedings of the Thirteenth Conference on Com- putational Natural Language Learning (CoNLL-2009), pp. 147-155. Boulder, Colorado, June 2009. Association for Computational Linguistics. URL https:://acaattaaaet: httpaaaaaaaaanooooooaaoaoa .xgg W09 1119
14

---

Published as a conference paper at ICLR 2024
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu. Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Ilama: Open foundation models for code. CORR, 222222288..29900 2023. doi: 10448550/rxiv.2238..2222 2308 12950
URL httpp:://ci..ooo http:///0...rr11.885555550
Oscar Sainz, Oier Lopez de Lacalle, Gorka Labaka, Ander Barrena, and Eneko Agirre. Label ver- balization and entailment for effective zero and few-shot relation extraction. In Marie-Francits Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 1199-1212, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguis- tics. doi: 10. 1t2.221.2mn2.....
URL
em. p- ma iimaaini.. 92
Oscar Sainz, Itziar Gonzalez-Dios, Oier Lopez de Lacalle, Bonan Min, and Eneko Agirre Textual entailment for event argument extraction: Zero- and few-shot with multi-source learning. In Ma- rine Carpuat, Marie-Cathering de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACI. 2022, pp. 2439-2455, Seattle, United States, July 2022a. Association for Computational Linguisties. doi: 10. 18653/ www.://2.....nnnaaaiaae 187. URL https: : /atpaaaaaanntoo.y....om.. org/ 2022. findings naacl. 187
Oscar Sainz, Haoling Qiu, Oier Lopez de Lacalle, Eneko Agirre, and Bonan Min. ZS4IE: A toolkit for zero-shot information extraction with simple verbalizations. In Hannaneh Hajishirzi, Qiang Ning, and Avi Sil (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies: System Demonstrations, pp. 27-38, Hybrid: Seattle, Washington Online, July 2022b. Asso- ciation for Computational Linguistics. doi: 10.18653/v 1/1155///12222222maldddeeeee //aclanthology.oo wwwp:hhoowoyy...//22..oooo wtwp://///2...aaaceeeeeeeea.eeee www.:aanaacllaaaaaeeoeeee
URL https:
Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. NLP evaluation trouble: On the need to measure LLM data contamination for each benchmark. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 10776-10787, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v 1/2023. findings-emnlp.222 URL BEtps://acLantholog. www.//ggg.rrr///ooo /ttpo///2002222 wtt.2222......no..eeeeeme www.iiiinnnniiie. emn.pp 722
Oscar Sainz, Jon Ander Campos, Iker Ferrea- Julen Etxaniz, and Eneko Agirre. Did
chatgpt cheat your test?, Jun 2023b. URL https::/hitz www..o.................. io/
1m-contami htt.a/imianntn//oooooo/oooo
Taneeya Satyapanich, Francis Ferraro, and Tim Finin. Casie: Extracting cybersecurity event infor- mation from text. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8749- 8757, Apr. 2020. doi: www./////.aa.........o.o 6600. URL https://ojs aaai wrg.orgggooo... index. php/ www.///Atattt/////.e.eaaaA 6441
Teven Le Scao and Alexander M. Rush. How many data points is prompt worth? In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tüür Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Con- ference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pp. 2627-2636 As- sociation for Computational Linguistics, 2021. doi: 10. 10.18653/v1/2011 http:///2221....1mnnnea .naacl-main.200 URL 1tt.oooorrr...1133///0/000002020
https:://oi.org http::////oi.rr/11.111 wtt.ll11633ww/2/
nww.l -aa--mainn.. 208
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty (eds.), Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computa- tional Linguistics: Main Volume, EACL 2021, Online, April 19 23. 2021, pp. 255-269. As- sociation for Computational Linguistics, 2021. doi: 10. 100118653v11 10t/S3//v1222... 1a..022eeaclmmaatt 20. URL
https://dos http:///oo.ogg//.... 1653/v1 htt.:/oorrr.00..1833111//3/011/0
eaall meac-mmain .0
15

---

Published as a conference paper at ICLR 2024
Karan Singhal, Shekoofeh Azizi, Tao Tu, S. Sara Mahdavi, Jason Wei, Hyung Won Chung. Nathan Scales, Ajay Kumar Tanwani, Heather Cole-Lewis, Stephen Pfohl, Perry Payne, Martin Senevi- ratne, Paul Gamble, Chris Kelly, Nathaneal Schärli, Aakanksha Chowdhery. Philip Andrew Mans- field, Blaise Agüera Arcas, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Katherine Chou, Juraj Gottweis, Nenad Tomasev, Yun Liu, Alvin Rajkomar, Joelle K. Barral, Christo- pher Semturs, Alan Karthikesalingam, and Vivek Natarajan. Large language models encode clinical knowledge. CoRR, abs/2212.13138, 2022. doi: 10.45550/RXXV.2222111333 URL
attpss://oi.... org/10..g..... 112111..88555 48550/ ar www..iiw.......................e 2212.133381
Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. Nature, pp. 1-9, 2023.
Zeqi Tan, Shen Huang, Zixia Jia, Jiong Cai, Yinghui Li, Weiming Lu, Yucting Zhuang, Kewei Tu, Pengjun Xie, and Fei Huang. DAMO-NLP semeval-2023 task 2: A unified retrieval- augmented system for multilingual named entity recognition. In Atul Kr. Ojha, A. Seza Dogruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, and Elisa Sartori (eds.), Pro- ceedings of the The 17th International Workshop on Semantic Evaluation, SemEval @ ACL 2023, Toronto, Canada, 13-14 July 2023, pp. 2014-2028. Association for Computational Linguistics, 2023. doi: 10. 2211866/1122223semmats 2023. semeval 1.277
URL https:: wtt.d/oooo///..1100////11 wrg.ooggg//... 0080///..1166///110
Simone Tedeschi and Roberto Navigli. MultiNERD: multilingual, multi-genre and fine- tne-grained dataset for named entity recognition (and disambiguation). In Marine Carpuat, Maric-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz. (eds.), Findings of the Association for Compu-
tational Linguistics: NAACL 2022, pp. 801-812, Seattle, United States, July 2022. Associa-
tion for Computational Linguistics. doi: 10. 10.18633/11/0000 (tt.://///22...nnnnnn--iaee finding-nnacll.66 URL https:
wollanthoooyy.rrg///e 2022. findings naacl. .60
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the CoNLL-2003 shared task: Language-independen named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pp. 142-147, 2003. URL https:
//aclanthology.o 2M0////33--00 0419
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302.13971, 2023a. doi: 10.488550/axXw.2300211999 httpps:/ddi. www.oogg........... 48550 arxiv. 200.2.192..111111
URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang. Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov. and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, 2tt.222777.9222.. 2023b. doi: 10.48550//rXxw.2307 .09288. URI. https: //oii www.orr................o 48550 arriiiiiiii 2307 09288
Christopher Walker, Stephanie Strassel, Julie Medero, and Kazuaki Maeda. Ace 2005 multilin- gual training corpus. Linguistic Data Consortium, Philadelphia, 57:45, 2006. URL https: //catalog. Idc. upenn. ecu LDC.C///DDC0006066
Ben Wang and Aran Komatsuzaki. Gpt-j-6b: 6 billion parameter autoregressive language model, 2021.
16

---

Published as a conference paper at ICLR 2024
Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, Jingsheng Yang. Siyuan Li, and Chunsai Du. Instructuie: Multi- task instruction tuning for unified information extraction. CoRR, abs/2304.08088, 2023a. doi: URL bhttts:://ooii..
2304. 08085.
Xingyao Wang. Sha Li, and Heng Ji. Code4Struct: Code generation for few-shot event struc- ture prediction. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3640-3663, Toronto, Canada, July 2023b. Association for Computational Linguis- URL https: /ttps//acllaan hatp:sss/aaaaaathcloooso/ org 2023.
tics. doi: 10. 1/200112222223./og acl- ong. 202
Xinyu Wang. Yong Jiang, Nguyen Bach, Tao Wang, Zhonggiang Huang, Fei Huang, and Kewei Tu. Improving named entity recognition by external context retrieving and cooperative learning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1800-1812, 1812, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653//1/2221..all lal--long 142.
URL https: htt.pps//aclannhooooy...oo wtw..r//..0.....1oooo........
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganch Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Gary Lai, Ishan Purohit, Ishani Mondal, Jacob Ander- son, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A. Sumanta Patro, Tanay Dixit, and Xudong Shen. huperrrnaturllinsttutiionss Generalization via declarative instructions on 1600+ NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arah Emirates, December 7-11, 2022, pp- 5085- 5109. Association for Computational Linguistics, 2022. doi: 10. 12211//6/1//2222.como wtt.:222.mmmmmmmii.....2 URL /ttt:ps///ol..og/000...666
emnlp ma in..in.... 340
Chih-Hsuan Wei, Yifan Peng, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Jiao Li, Thomas C Wiegers, and Zhiyong Lu. Assessing the state of the art in biomedical relation extrac- tion: overview of the biocreative v pnemical-diseass relation (cdr) task. Database, 2016, 2016.
Renzo M. Rivera Zabala, Paloma Martinez, and Isabel Regura-Bedmar. A hybrid bi- crf model to recognition of disabilities from biomedical texts. In Proceedings of the Third Workshop Evaluation of Human Language Technologies for Iberian Languages, 2018. URL. httph:tt///
ceur-ws. wwg.w/..rrgg.... Vol -1 BtT//////DDDaSSSSSSSSSS wtwppppaprr.......
Mozhi Zhang, Hang Yan, Yaqian Zhou, and Xipeng Qiu. Promptner: A prompting method for few- shot named entity recognition via k nearest neighbor search. CoRR, abs/2305. 2023a. doi: 10.485500/r2Xw.2205.. 12217. URt/tttss: doi /10 205504/222....305.555.5 12217
Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. Optimizing bi-encoder for named entity recognition via contrastive learning. In The Eleventh International Conference on Learning Representations, 2023b. URL URRp:ttts:///peeeee///
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. OPT: open pre-trained transformer language models. CoRR, abs abs/2205.01066 2022. doi: 405.0/4450///.. 2205 01068
10.48850 01068.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position- aware attention and supervised data improve slot filling. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel (eds.). Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pp. 35-45, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17--
URL https: /ettts::/accle //apaaaaaaatot......
brq/DD7-1004]
17

---

Published as a conference paper at ICLR 2024
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang, Joseph Gouzalez. and lon Stoica. Judging lim-as-a-judge with mt-bench and chatbot arena. CoRR, fbbs/2306.05685, 2023, doi: ttt.htts:://ddo
10.48550/a2X122296.556333
wtw.1..85/////...22000000 05685
Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. Universalner: Targeted dis- tillation from large language models for open named entity recognition. CoRR, abs/2308.03279, 2023. doi: 10.48500/2Xi..2200003777 URL htttp:://c...oggg//.. /ttpp//...../..../////a...//
2308 03279
18

---

Published as a conference paper at ICLR 2024
@dataclass
class Launcher(Templatt))
"RRfeers to a vehicle designed primarily to transport payloads from the Earth surface to space, Launchers can carry various payloads, including satellites, spacecraft, cargo, various arbits even beyond orbit. They are usually stat vehicles that use rocket engines for propulston.
sention:
of launcher vehicle.
such as: "Sturn v", "Atlas v", "Seyuz", "Ariane
space_ company: str corpany operates the launcher. Such crew: List[str]
"Blue origin", "ESA", "Booing"
o* Naats of the
bearding the Launcher.
Such as: "Nell Arnstrong", "Michael Collins", "Buzz Aldrin"
@dataclass
class Mission(Terplate):
"*" planned accomplished journey beyond Earth's atrosphere with specific objectives, either crewed uncreved. It includes nissions to satellites, the International bodies, deep space."" ...
Space Station (ISS),
rention: nae
The nane of the mission.
Such "Apollo "Artenis", "Hercury"
date: str The start date of the nission
vehicle launched. "Florida", "Houston" destination: str The place or planet to which the launcher will be sent. Such as "Moon", "low-orbit"
departure:
This is the text analyze
"The Ares nission to Hars is schedulad for 2832. The Starship rocket huild by Spacex will take off" "fron Boca chica, carrying the astronauts Rutherford, Elena and Jake Martinez."
annotation instances that result
listed hore
Mission(rention= date='2832' departure: Chica', destinations "Hrr
(mentionn "Starshipp" ceapce_ companyy""Spacee" 'SpaceX", crew- 'Max Rutherford', "Elena Soto", "Jake Martinez'))
Figure 5: Example of generalization custom tasks defined by the user.
A EXAMPLES
In addition NER, EE and EAE for which examples are shown Figures and 3 respectively, we also feed the model with data from RE and SE. The formulation of RE is similar to the NER but with two argument attributes. However, the SF task is more complex as shown Figure With this task, added several layers of complexity to the input: extended definitions for each possible attribute (slot), optional arguments, and, fine-grained definitions of types such as Names, Values, or Strings. We also added constraints into the prompt condition the model to just output the information of our desired query instead of every single template on the text. In the future, we would like to add more complex tasks the training and evaluation improve the capabilities and flexibility of the model. For more examples refer to the GitHub repository.
A.1 EXAMPLE OF GENERALIZATION TO NEW CUSTOM TASKS
Our model allows the user to define custom annotation schemas using Python code. We provide an example where we define two new types of entities: Launcher and Mission. As shown in Figure Launcher and Mission are not simple entities, they correspond to what we call Template, class similar to Entity but with additional arguments, like the SF task. For example, the space pompanyy or the crew of the launcher in the example, the model's output (everything after result = () satisfies the type constraints defined in the guidelines, attributes defined strings are filled with strings and, the arguments defined as lists (like crew) are filled with lists. The model can correctly analyze the given sentence with our newly created annotation schema.
5]
some of the additional arguments we added to the schema. As shown
19

---

Published as a conference paper at ICLR 2024
# The following lines describe the task definition Édataclass
analyze
toxt "Hongolion Prine Ministar N. net Hongcal vice ninister of International Department the Chinese Comunist
class personTerplateeTorpp Torplate):
Person tesplates encodes the Information about the Person entity. **.
Party Contral Comsittee on Horday here
query: The Person entity query
The 1ist called result contains the terplates instances for the foll following entity querless
alternate_nawes nawes: (ptional[Lsst(Nne]]] Hone
""uuees used refer to query person that are distinct from the 'officiol" nane, Including: allases, stage names, abbreviations ... date of birth: optional[value] None
Enkhbold: PersonTempiate
""h date on which the query person was born."
PersonTemplate:
age: [Valo) None
query""N. Enkthold",
*"* reported age of the query person."
countries_ contrres__f_reee (haaeeecceeaaeeeHooeeeeee
bibrth: [ptsonal[have]
tiele-[string(" "Prine Nisi
sunicipality level (eity, town, village) which auery person was born"
"he geopol entiity
date of death: optional[valus]
"he date the query person's 's death
(Collapsed 36 more slots)
Figure 6: Example of the TACRED dataset converted to Slot Filling task represented as code.
B PERFORMANCE IN SEEN VS UNSEEN LABELS: FURTHER ANALYSIS
Table 6: List of labels in the zero-shot datasets that overlap with the ones in the training datasets (seen) and the labels that do not overlap with the ones in the training datasets (unseen)
Dataset
Seen Labels
Unseen Labels
BroudTwitter
Location, Organization, Person
DetabreschAmack PhisingAttack, RansomAttack, Vulnerabili- tyDiscover, Valnerabilityatte Field, Tisk, Algoritim, Researcher, Metrie, University, Pro- grammingl anguage, Conference Book, Writer, Award, Poem, Magazine, LitearyGeare MusieGenre, Song, Band, Alhum, MusicalArtist, MusicalIn- strument, Award
CASIE
Product, Country, Person, Organization, Location, Miscella-
neous
Literature
Event, Person, Location, Organization, Country, Misccilancous
Music
Event, Country, Location, Organization, Person, Miscellancous
Person, Organization, Location, Election, Event, Country, Mis- cellaneous
Polities
Politician, PoliticalParty
Person, Organization, Country, Location, ChemicalElement, CoempcalCoomouudd,, Event, Miscellanenus ClinicalEatity
Seientist, University, Discipline, Enzyme, Protein, Astronem- icalObject, Acnlemiclouraal, Theary, Award
Science
ESC
Material, ManufacturinPrroess Equipneett Apolica-
tion, Engineeringfeattrrs MechamicalProppetiic PracesCharr acterization, ProcessParameterr cepiPrinciples, Mumufacturingiannlrrk
FaRNER
Biomedical
Technoloyy......... Con-
HarveyNER
Point, Area, Road, River
Movie
Year
Actor, Character, Director, Genre, Plot, Rating, RatingsAver- age. Review, Song. Title, Trailer
Restaurants
Location, Price, Hours
Rating, Amenity, RestaurantName, Dish. Cuisine Animal, Colestial, Food, Instrumentt, Media, Plant, Mythologi- cal
Location, Organization, Biological, Discase, Event,
MuliNERR
Time, Vehicle
CommeccciaPrrduust Facility, GPE, Location. Modical- Healthlssue Money, Organization, Person, JobTitle, Numeric, Vchicke, Weapon
WikiEventSees
BodyPart, Information, SideOIConflict
ConflictEvent. ContactEvent, GenericCrimeivent, Jus- ficeEveat, MedicalEvent, Movemen[ThanporrEvee Person- nellEvent, Tramsactionlivent
ArifaentrissameeIieen, Cognitivelivemt ControlEvent, Disas- terEvent, LifeEvent
WikiBventter
Table 6 categorizes the labels for each zero-shot dataset into those that overlap with the training dataset and those that are completely unseen. We adhere to a strict approach in this classification. For instance, although the label COUNTRY does not appear in the training datasets, similar labels such as GEOPOLITICAL entity do. Therefore, we consider that the model has been exposed to this label during training.
While some labels in the zero-shot datasets overlap with those in the training dataset, the annotation guidelines for each label may vary significantly between datasets. Table 7 presents the miero-F1
20

---

Published as a conference paper at ICLR 2024
Table 7: Micro F1 score for the seen and unseen labels in the zero-shot datasets.
Baseline
13B
34B
Dataset
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
BroadTwitter
39.0-06
49.5
51,4+L8
50.3+2.1
CASIEEE
33.9
29.3
62.2209
65.5218
AI
43.5_14 14
21.1
57.8
60.0a12
57.8
55.8147
57.7
64.2213
Literature
34.6-02
43.6
54.6236
67.4
52,4
64.6105
52.742.
63.7a28
Music
46.8-10
62.2+16
53.7=002
74.9.03
$2.8+39
72.7
54.0#38
76.3+12
Politics
45.9-1.1
4.6+26
64.0=2.2
31.9447
62.0+22
12.4
64.4
45.8+93
Science
38.7=0x
34.7=33
52.7=17
18.8
52.7
60.4
52.5
60.5+a7
E3C
59,8=03
59.0
59,0+09
60.0+04
FabNER
0.0=0.0
6.2+0.4
22.6=2.3
24.9
23.9
25.5
20.7+29
26.5+06
HarveyNER
23.2+0.4
37.3+18
41.3+09
38.9
Movie
31.5
46.1+1.5
58.7=2.
63.8+05
47.3+3.1
65.3+10
42.7+2.
66.1+14
Restaurants
18.0
38.7128
33.2
49.9
38.0236
57.1
46.0142
57.2109
MultiNERD
58.0
39.5
81.2200
44.6209
82,4
47.7
82.3105
49.1
WikiEventsweR
77.2
0.0.00
81,5207
0.02n0
80.9
0.0gao
82.1
3.5226
WikiEventser
43.3-03
57.2+15
41.7+0.1
45.0+1.
43,9+0a
48.8+177
45,0+1.1
50.4+09
Average 41.2_04
31.6106
54.6203
47.5a06
54.2104
48.0a13
54.7409
51.4210
scores for both seen and unseen labels across each zero-shot dataset. Generally, the models perform better on seen labels compared to unseen ones. However, there are instances where the reverse is true. This occurs when dataset contains labels that, although overlapping with those in the train- ing dataset, have vastly different annotation guidelines. As discussed in Section 5.3] the model has strong preconceptions for some labels, adversely affecting zero-shot performance. GoLLIE, trained to adhere to specific annotation guidelines, demonstrates greater robustness against these label pre- conceptions than the baseline model. Consequently, achieves better results for both seen and seen labels. GoLLIE can successfully handle both, seen and unseen labels from datasets that were used during training. This ability underscores GoLLIE's superior generalization capabilities, largely attributable to its capability of leveraging annotation guidelines.
C MODEL HALLUCINATIONS
Table 8: Number impossible to parse outputs and number predicted labels that are hallucinations. Fl scores on the dataset are shown for reference.
Dataset
Impossible to Parse
Hallucinations
Fl Score
BroadTwitter
O2o 2002
O2o 1664
49.5.08
CASIFEE
1 / 199
6д1 / 1548
59.3
CASIEFAE
3+2 2804
50.0+11
A1
0+o / 431
1+1 / 1292
59.1+11
Literature
0+o / 416
0+o / 2059
62.7+32
Music
020 / 465
642 / 3080
67.8102
Politics
040 / 651
342 / 4142
57.2-10
Science
040 543
7+1 2700
16.5
E3C
0:o / 851
1+0 / 688
59.0
FabNER
1 / 2064
13t3 / 4474
24.8
HERveyNER
0:0 1303
111 708
37.3-18
Movie
040 2443
1 3919
63.0=
Restaurants
0+o / 1521
3+0 / 1451
43.4=008
MultiNERD
49+11 / 32908
51+8 / 67142
76,0
WikiFventsSNER
O2o / 573
111 2666
80.7
WikiFventsge
040 /53
311 630
43.0m24
WikiFvenntsAAe
241 /311
O4o /
51.9204
21

---

Published as a conference paper at ICLR 2024
In this section, evaluate the hallucinations generated by the model. We examine two different phenomena. First, we consider instances where the output is so corrupted that it is impossible to parse. In such cases, treat the output model outputs a label Hallucination, that is, a label not defined among the input classes, In these instances, we remove the label from the output. As demonstrated in Table [8 for all the zero-shot datasets, both phenomena occur in less than 1% of the predictions. This demonstrates that GoLLIE highly resistant hallucinations and closely adheres the classes defined in the input.
empty list. Second, we look at instances where the
D EXTENDED TRAINING DETAILS
D.1 LOSS CALCULATION
We have used the standard Next Token Prediction (NTP) loss to train our models. However, sev- eral regularizations that we applied to the models made the loss computed over the guideline tokens much higher than the actual output tokens. This is because we randomly shuffle the guidelines order, mask names, or, drop classes, which makes it impos- sible to predict what goes next. To avoid the loss of the guideline tokens overshadowing the actual out- put tokens loss, we decided to only compute the loss over the output tokens. This way, we can also avoid some overfitting of the guidelines. This resulted in faster training and better results overall.
Table 9: Number of examples for each train-
ing and zero-shot dataset
Dataset
Train
Dev
Test
ACE05NER
19217
676
901
ACE05RE
19217
901
676
ACE05EF
19217
676
901
ACE0SEAB
3843
397
368
ACE05RC
5691
ACE05VER
19217
BCSCDR
4561
4582
4798
CoNLL 2003
14041
3250
3453
DIANN
3976
793
1309
NCBIDisease
5433
924
941
D.2 DATASET DETAILTS
Ontonotes 5
30000
15680
12217
RAMS
7329
924
871
Table shows the number of examples for each training and zero-shot dataset. OntoNotes gen- erated semi-automatically and is orders of magni- tude larger than the other ones. Therefore, for each training epoch,
TACRED
10027
3896
2311
WNUT 2017
3394
1009
1287
Total
165163
33708
30033
sample 30.000 random examples from the training set. The models were trained for 3 epochs with an effective batch size of 32 and learn- ing rate of 3e-4 with a cosine scheduler. Therefore, we perform 15,485 training steps.
BroadTwitter
2002
CASIEEE
199
CASIERAE
199
AI
431
Literature
Music
465
Regarding the splits, we use the standard train, dev test splits for every dataset. the case of ACE, we follow the split provided by Lin et al. (2020). In the case of CASIE, we took the first 200 instances as validation and the last 2000 as test.
Politics
651
Science
543
E3C
851
FabNER
2064
HarveyNER
1303
Movie
2443
Restaurants
1521
MultiNRR
32908
WikiEventsNEr
573
WikiEventsar
573
WikiFveentgAe
321
Total
47463
22

---

Published as a conference paper at ICLR 2024
Table 10: Details about the training resources required for each model.
Model
Hardware
FL.Pss
Time (h)
CO2eq (kg)
Baseline
IxA100
4.5c
17.3
0.61
GoLLIE
1xA100
11.9e
44.5
1.57
13B
1xA100
22.7e
79.5
2.80
34B
2xA100
55.8e
94.6
6.67
D.3 CARBON FOOTPRINT
tine-tuning LLMs not as expensive as pre-training these models. Still, we believe that it is im- portant to measure and document the costs that our experiments have our planet. We provide the resources required for a single run of our experiments Table[T0 All the experiments were done on private infrastructure. For the carbon footprint estimation, estimated the values considering a 400W consumption per GPU with a 0.141 kg/kWh carbon intensity
D.4 LORA VS FULL MODEL. TUNN-TUNING
We conducted preliminary experiments to compare the performance of QLoRA (Hu et 2022
Dettmers et al. 2023) with that of training all the parameters in the model. These preliminary ex- periments were conducted using the LLaMA2 7B model Touvron et al. (2023b) and an early version of the code. However, the experimental setup for both models was identical. Both approaches were prompted with guidelines. First, we compared the training loss of both approaches. F shows that when fine-tuning all the parameters, the loss decreases much more rapidly than when training only the LoRA layers. It also achieves a lower loss at the end training. However, when eval- uating the model at the end of the first and third epochs, observed that training the full model performs very poorly, as shown in Table 11 We hypothesize that when training all the parameters, the model overfits quickly (indicated by the lower training loss) and memorizes the training data. On the other hand, training only the LoRA layers, which represent around 0.5% of the total model weights, introduces bottleneck that prevents the model from memorizing the training dataset. is also noteworthy that the QLoRA approach was trained using just one Nvidia A100 80GB GPU thanks to the 4-bit quantization of the frozen model Dettmers et al. (2023). Training the full model required minimum of four Nvidia A100 80GB GPUs to fit the model into memory. We used Deep- Speed [ to distribute the model across the four GPUs for training. Due to the high cost training, we did not perform extensive hyperparameter search for the full model.
Figure 7: Training loss of fine-tuning the full model training LoRA layers only.
train/loss
Full Model
QLORA
0.1
M M pau Mand n n. Ma Mn "Wn
0.05
0.05
w Mombh bha mymo www.wnm wwmmmmmmn www.mmaaaaoo..........
train/spoch
500m
15
2.5
Statistic taken from ftomhttss://ppp.lleeee rict
riniriic tymaps ricttyapps com/map
g tituub. www.mommmcmcccccc wttp://mmmcooooooo//eepppppeeee
23

---

Published as a conference paper at ICLR 2024
Table 11: Fl scores achieved when training the full model vs only training the LoRA Layers the end of the first and third epoch.
Training
Epoch
Precision
LR
HarveyNer
FabNER
Restaurant
Movic
CASIE
CoNLL03
Full
BF16
le-4
0.00
0.00
0.25
4.74
0.00
85,57
Full
3
BF16
lc 4
3,45
0.21
46.7
16.72
0.42
84.83
QLORA
4Bi + BF16
2e-3
34.98
20.78
45.01
51.14
55.83
91.41
QLoRA
3.
4Bit + BF16
2c 3
35.34
16.21
39.07
44.18
$7.93
93.14
E HANDLING DATASETS WITH HUNDREDS OF LABELS AND CODE-STYLE PROMPT OVERHEAD
In our research, we focus on datasets with fewer than 20 labels. However, some datasets, such as FIGER Ling & Weld (2012), include hundreds of fine- ome-grained labels. Including guidelines in datasets with hundreds of labels can make inputs excessively long, exceeding the context size of current Large Language Models (LLMs). This is a known constraint LLMs, and recently, signifi- cant research effort has been directed towards algorithms that efficiently increase the context window size Press et al. (2022). We anticipate that future LLMs will have a context window large enough to accommodate not only more labels but also more detailed guidelines. For the time being, this problem can be mitigated by batching the labels into multiple input examples. Instead of prompting the model with, for example, 100 labels in single input, it is possible to prompt the model with 10 inputs, each incorporating 10 labels, and then combine all the outputs into a single response. In any case, handling datasets with large number of labels remains limitation of GoLLIE.
Figure 8: Percentage of characters from the input required to represent the code-style prompt for different labels. For detailed guidelines, the code is a small fraction of the input.
Ontonotes Code ratio: 0.43
Actor Coce ratio: 0.13
HarveyNER Point
Code ratio: 0.08
Edataclass
@dataclass
Adataclass
Actor(Entity):
class (oint(Entiyy):
Peoplo. inclusing fictional
"* "ofees to individual
**
lecation that
movic including
buiiding Landrart, Intersection
Bush", "Clinbon", "Barak"
surcocti
Intorsoctionn speciee address
with
"Neeegs" "Putie"
Tr Individial bnoun for their In valevisio, thrater, other adia but alsn ancears In envles, they be ieered Actor in this contest.
Inte/reservnt
rensrit conpany/tranchit unless It accompanted with meecis
Inratimn, for exeeple, IIB Kirkaend Franchised smill unfgue
"dotnny "brad pitel, "ton "tos crutie "elint eastiend"
bustnestes
considered point. Tgnore locatiom the Twitter usemane, unless does refer Teitter accourt High School.
For
exsemle.
span: str P "600", "GEONGE E. HIDM" "Likovood Church" "Bayou Oaks", "Nortigate Sundivision S. of toll road!.
Our approach uses Python-based code-style prompts, this requires including tokens in the input to represent the code structures. Figure illustrates various labels formatted in our code-style input alongside their respective guidelines. For very generic guidelines, such as those for the PERSON entity OntoNotes the code structure accounts for almost half of the input's characters. However, for detailed guidelines, like those for the POINT entity in HarveyNER, the code structure constitutes only a small portion of the input. While there is an overhead tokens to represent the code structure, when dealing with datasets with a very large number of labels, the primary limitation is fitting the guideline definitions into the model's s input, rather than accommodating the Python code structure.
24

---

Published as a conference paper at ICLR 2024
F HUMAN EFFORT TO BUILD THE PROMPTS
GoLLIE requires formatting the input in a Python-based code representation. We achieve this by filling pre-defined templates for each task (NER, EE, EAE, RE, SF). We will make this predefined set of templates publicly available along with our code. Implementing new datasets only requires defining list of labels and the guidelines for each label. We the annotation guidelines provided by the dataset authors. Therefore, for most datasets, this process is straightforward and requires very little human effort. For datasets with very large and complex guidelines. such as TACRED, manual summarization of the guidelines was necessary. In any case, the involvement of a domain expert not required. Additionally, since the inputs are automatically generated using templates, implementing new datasets does not require knowledge of Python coding.
Some datasets did not have publicly available guidelines, either due to semi-automatic generation or because the authors chose not to disclose them. For these specific datasets, human experts were needed to construct the guidelines from examples in the development split. We plan to release generated guidelines to support future research. Human domain experts necessary to adapt GoLLIE to new tasks where guidelines or annotations are unavailable. However, this requirement is common any other Information Extraction (IE) model.
G DATA-CONTAMINATION STATEMENT
We believe data contamination is relevant problem that affects the NLP evaluations nowadays, becoming more prevalent with LLMs (Dodge et al.] 2021 Magar Schwartz 2022 Sainz et al.] 2023b a). Detecting whether dataset was inside LLM pre-trained corpora challenging even with the pre-training data itself. In this paper, unfortunately, we do not have access to the pre-training data used train Code-LLaMA the backbone LLM of our model. This issue particularly worrying for us because one big source of contamination is probably GitHub and other code repositories which are also used to upload evaluation benchmarks Dodge et al. (202T). As Code-LLaMA trained on code, there is chance for this particular data leakage. However, all of our comparisons were made against our baseline, which has the same backbone LLM as GOLLIE. Even the results were impacted, the improvements of our model over the baseline would not be affected by data contamination as both share the same pre-training. We hope that the future more transparency will allow to perform safer evaluations.
25


Reviews:
[{"review_id": "xYwjxxfwLU", "reviewer": "Reviewer_y8gw", "summary": "This paper focuses on zero-shot information extraction. Their goal is to train a large model that can generalize well to unseen domains and datasets. They main idea is using code as the intermediate layer for all the information extraction tasks and considering annotation guidelines or additional information during pre-training or testing. This additional information makes the model to understand tasks better and therefore be able to generalize. Experiments demonstrate the potential of the proposed method.", "questions": "- See above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The performance is impressive.\n- The pre-trained model is valuable.", "weaknesses": "- My main concern is as follows. Although the authors try to split the datasets for training and testing based on domains. I believe that there is still a large overlap among their output spaces. For example, RAMS, which is considered as one of the training tasks, contains role label *transporter*, *vehicle*, and *place*, while WikiEvents, which is consider as one of the testing tasks, also contains those role labels. In NER tasks, I believe the same situation happens as well. This makes the setting not *truly* zero-shot because the model has already learned those concepts. The model can get improvements just because including more *in-domain* supervised training signals.\n- I think the authors should be careful when referring the state-of-the-art models. In fact, in Table 2, the reported SOTA numbers and models are not SOTA anymore.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4RvXjBIyqs", "reviewer": "Reviewer_bJoP", "summary": "This paper proposes an LLM fintuned to comply with annotation guidelines for information extraction tasks. Specifically, with a focus on NER, EE, and EAE (19 datasets in total), the task format is unified into a code style: the label class definitions as class docstrings and the candidates as a comment for the arguments.  Some regularization techniques are proposed to ensure that the model follows the guidelines and does not just learn to identify specific datasets. Experiments show that a finetuned version of Code-LLaMA achieves comparable performance compared with a naive baseline finetuned without guidelines, and beats the baseline in zero-shot settings by a large margin.", "questions": "1. How many samples are used to supervise finetuning LLaMA?\n2. The guidelines used in this paper are specifically the basic definitions of entity/relation/event types, which should have been provided to LLMs. However, the full annotation guidelines usually cover many edge cases and may contain tens of pages. Have the authors considered incorporating more fine-grained guidelines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper proposes to utilize the annotation guidelines to boost the performance of zero-shot information extraction. The idea is intuitive and compelling.\n2. The experimental results properly demonstrate the effectiveness of the proposed method and the bonus of the annotation guidelines.\n3. The presentation of this paper is clear, with few typos.", "weaknesses": "1. The motivation behind the code style and its effects are unclear. Note that incorporating the annotation guidelines in a natural language style rather than the code style is also feasible. The LLMs pretrained on codes may lack natural language understanding capabilities that are crucial to conducting IE tasks. This may result in the low performance of the baseline. The advantages may lie in the capability of generating outputs following code grammar, which mitigates the need for parsing outputs. The authors should justify the necessity of the code style and its underlying effects compared to the natural language style.\n2. The ablation studies of class order shuffling and guideline paraphrasing are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yEtUdh7S1w", "reviewer": "Reviewer_xrh5", "summary": "The paper introduces GoLLIE (Guideline-following Large Language Model for IE), a model fine-tuned to effectively utilize annotation guidelines for enhanced zero-shot results in Information Extraction tasks. A primary distinction of GoLLIE is its use of a Python code-based representation for both input and output. This method provides a clear, human-readable format, unifying the representation of various IE tasks and addressing challenges associated with traditional natural language instructions. Empirical evaluations validate GoLLIE's superior capabilities in leveraging guidelines, indicating promising directions for future research and model enhancements, including the exploration of more extensive pre-training datasets and refining its response to ambiguous labels.", "questions": "1. **Sampling Strategy**: Datasets differ in scale, and a naive mixture could introduce imbalance. How did the authors handle the sampling process for the datasets used in this study? Were experiments conducted to assess the effects of different sampling techniques on the results? This aspect wasn't addressed in the paper and could provide clarity on the method's robustness across varied data distributions.\n2. **Scalability with Numerous Labels**: For datasets with a large number of labels, say in the order of hundreds, the input length for your method could become substantially lengthy, leading to efficiency concerns. In such complex scenarios with multiple labels, how does the inclusion of guidelines impact the performance? This wasn't touched upon in the paper but would provide deeper insights into the method's scalability and effectiveness in real-world applications.\n3. **QLoRA vs. Full Model Fine-tuning**: The authors mentioned employing QLoRA for training, citing its superior performance over fine-tuning the entire model on zero-shot tasks and faster training speed. However, the paper lacks empirical evidence to support this. What are the relative impacts of full-parameter SFT and techniques like LoRA and QLoRA on zero-shot IE performance?\n4. **Benchmark Selection**: Many of the state-of-the-art methods used for comparison in the paper, such as UIE, appear outdated. Why weren't more recent and potentially stronger methods like USM, InstructUIE, and GPT4 considered for benchmarking?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper presents a novel solution to the long-standing challenge where large models struggled to leverage intricate annotation guidelines inherently. Through targeted fine-tuning and a unified Python code-based representation for both input and output, it significantly improves zero-shot outcomes in IE tasks.\n\n2. The detailed error analysis provides invaluable insights into the challenges of zero-shot IE, especially when leveraging guidelines. The study methodically identifies specific issues like the ambiguity of labels, conflicts between fine-grained and coarse entities, and the repercussions of strong label preconceptions. These findings offer a valuable roadmap for subsequent research in the domain, emphasizing the need for clearer guidelines, broader fine-tuning datasets, and more specific instruction-following capabilities.\n\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The paper's novelty appears somewhat limited as similar approaches have been explored in past research. [1] proposes a zero-shot entity typing approach that utilizes the type description available from Wikipedia to build a distributed semantic representation of the types.\n\n2. The choice of code-based prompting can potentially hinder usability for general users. Those unfamiliar with coding might find it challenging to interact with or fully leverage the model, thereby narrowing its applicability.\n\n3. The method's efficacy remains unclear due to incomplete experimental validation. Specific concerns include:\n\n   a. **Prompt Style Impact**: The rationale for using a Python code style prompt is not experimentally validated. It's untested whether natural language prompts with guidelines might yield similar outcomes. Although CodeIE validated code style prompts for OpenAI models, the necessity of this design for open-source LLMs during instruction tuning remains unproven.\n\n   b.**Prompt Sensitivity**: The paper omits experiments examining the sensitivity of prompts, leading to concerns about the method's stability. The model's performance under varying definitions, code structures, and code comment styles remains unexplored, making its robustness questionable.\n\n4. The paper lacks experiments analyzing the impact of training dataset diversity on the LLM's ability to follow unseen guidelines. It remains unclear how much data is required to achieve satisfactory performance, leaving questions about the scalability and efficiency of the approach.\n\n\n\n[1] Description-Based Zero-shot Fine-Grained Entity Typing", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uDRvnLLGEG", "reviewer": "Reviewer_zWZU", "summary": "This paper proposes GoLLIE, a large language model fine-tuned to follow label descriptions and few-shot examples to improve zero-shot information extraction. Key ideas include formulating tasks as Python code, regularizing training, and comprehensive evaluation on diverse IE tasks.", "questions": "* How does the prompt length vary in the proposed method? Translating IE tasks into Python classes might make them more accessible for LLMs, but could also significantly inflate the token count. For instance, a fine-grained entity type classification task with over 1000 entity types would result in a considerable token overhead.\n* The paper doesn't detail the human (or domain expert) effort required to craft the prompts. Given the variety of labels and tasks covered in the study, it would be beneficial to track this for subsequent research.\n\nThese are important questions that need to be answered in the updated paper.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* LLMs have excelled at numerous complex tasks, but underperformed in IE tasks. This paper addresses this gap, diving deep into the annotation guidelines and error analysis using an LLM approach: zero-shot learning, natural language instructions, programming language output format, and leveraging one model to tackle multiple tasks.\n* The study's thorough experiments span various IE tasks, coupled with an extensive error analysis when applying the unified model across these diverse tasks. Such detailed error analysis offers valuable insights into the model's strengths and limitations.\n* Another significant contribution is crafting prompts for intricate IE tasks. Demonstrating the model's ability to adhere to these instructions, especially given the recognized challenges of current LLMs in following complex instructions, signifies substantial effort and expertise from the IE domain.", "weaknesses": "* The claim of making use of “annotation guideline” may be an overstatement - this paper only considered label name, label description and few-shot examples, however, annotation guideline in IE domain are very complicated and was curated by linguists. E.g., For TACRED slot filling (https://tac.nist.gov/2015/KBP/ColdStart/guidelines/TAC_KBP_2015_Slot_Descriptions_V1.0.pdf), section 3.6 per:city_of_birth, they use “GPEs below the city level (e.g. 5 boroughs of New York City) are not valid fillers.“ as an example rule to guide annotators. The prompts proposed by this paper might not fully capture the depth of true guideline understanding.\n* The paper omits key references from the era before LLMs that discuss label descriptions and verbalization. Examples include:\n    * Zero-Shot Relation Extraction via Reading Comprehension\n    * An Empirical Study on Multiple Information Sources for Zero-Shot Fine-Grained Entity Typing\n    * Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction\n\nSmall formatting issues:\n* Use ``’’ instead of ‘’’’.\n* Table 6 could be more user-friendly. Presenting exact numbers (like 10/10,000) would be clearer than percentages.\n\nPlease see more in the question section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)