PIVOINE: Instruction Tuning for Open-world Entity Profiling
Authors: Keming Lu
Xiaoman Pan
Kaiqiang Song
Hongming Zhang
Dong Yu
Jianshu Chen
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Model
Information Extraction
Instruction Tuning

Abstract: This work considers the problem of Open-world Entity Profiling, a sub-domain of Open-world Information Extraction (Open-world IE). Unlike the conventional closed-world IE, Open-world IE is considered a more general situation where entities and relations could be beyond a predefined ontology. We seek to develop a large language model (LLM) that can perform Open-world Entity Profiling with instruction tuning to extract desirable entity profiles characterized by (possibly fine-grained) natural language instructions. In particular, we construct INSTRUCTOPENWIKI, a substantial instruction-tuning dataset for Open-world Entity Profiling enriched with a comprehensive corpus, extensive annotations, and diverse instructions. We finetune pretrained BLOOM models on INSTRUCTOPENWIKI and obtain PIVOINE, an LLM for Open-world Entity Profiling with strong instruction-following capabilities. Our experiments demonstrate that PIVOINE significantly outperforms traditional methods and ChatGPT-based baselines, displaying impressive generalization capabilities on both unseen instructions and out-of-ontology cases. Consequently, PIVOINE emerges as a promising solution to tackle the open-world challenge of entity profiling.

PIVOINE: Instruction Tuning for Open-world Entity Profiling
Keming Lu Xiaoman Pant, Kaiqiang Song,, Hongming Zhang Dong Yu', Jianshu Chen
University of Southern California, Los Angeles, CA
Tencent Al Lab, Bellevue, WA TkemingluBusc.co edu
{ (xiaomanpan, riversong, honggzzhag @global tencent...... com Idyu, jianshuchen howpee/nnnneooaaal...oaa tencent com
Abstract
Open-world IE Generalization
Unseen Instruction
This work considers the problem of Open- world Entity Profiling, which is a sub-domain of Open-world Information Extraction (Open- world IB). Unlike the conventional closed- considers a more general situation where entities and relations could be beyond a predefined ontology. We seek develop large language model (LLM) that can perform Open-world Entity Profil- ing with instruction tuning to extract desir- able entity profiles characterized by (possi- bly fine-grained) natural language instructions. particular, we construct INSTRUCTOPEN- WIKI, substantial instruction-tune -uning dataset for Open-world Entity Profiling enriched with a comprehensive corpus, extensive annotations, and diverse instructions. We finetune pre- trained BLOOM models on INSTRUCTOPEN- WIKI and obtain PIVOINE, an LLM for Open- world Entity Profiling with strong instruction- following capabilities. Our experiments demon- strate that PIVOINE significantly outperforms traditional methods and ChatGPT- based base- lines, displaying impressive generalization ca- pabilities on both unseen instructions and out- of-ontology cases. Consequently, PIVOINE emerges promising solution to tackle the open-world challenge in entity profiling.
Out-of-Ontology Entity
Open hard
wes odttion the Open part the ATP Tour 250 swr the 2025 ATP look placa Paleia das Sports Marsellle Marsellle, F france, from 20 through 2e rebruay 2022
toumament played indcar
world IE, Open-world
Instruction: Please identify tennis circuits in Marseile.
Mention
2023 ATP Tour
Types
Tennis tour edition
Description
men's tennis circuit
Relations
(sport, tennie)
(sports season of laague or compattion, ATP Tour)
(kllows, 2022 ATP Tour)
Figure I: Illustration of Open-world Entity Profiling and its two main challenges: generalization to unseen instructions and out-of-ontology entities.
IE plays a vital role in knowledge graph construc- tion (Schneider et al., 2022), search engine (Wang et al., 2022), and document analysis (Chiticariu et al., 2010; Wang al., 2018; Zhong et al., 2020) Most existing IE methods center around closed- world setting with predefined ontologies. For instance, NER generally extracts named entities within several categories (Lample et al., 2016); EL focuses on associating mentions with predefined ontology (Kolitsas al., 2018). To better break this limitation, we introduce Open-world Entity Profil- ing, a series of entity-centric tasks within Open- world Information Extraction (Open-world IE) to accommodate broad and diverse requests related to entity profiles surpassing predefined ontologies' limits. Past research open-setting IE has pre- dominantly focused individual tasks of IE, such as EL (lurshina al., 2022; Ruas and Couto. 2022) and OpenIE (Niklaus et al., 2018; Bhardwaj et al., 2019a). Consequently, a noticeable gap exists comprehensive end-to-end studies aiming to create more extensive entity profiles within an open-world setting.
1 Introduction
Information extraction (IE) aims to discern meaningful information from unstructured data sources (Grishman, 2015). A traditional IE pipeline contains an array of tasks, which in- clude, but are not limited to, Named Entity Recognition (NER) (Lample et al., 2016), En- tity Linking (EL) (Kolitsas et al., 2018), Entity Typing (ET) (Ren et al., 2016), Relation Extrac- tion (RE) (Huguet Cabot and Navigli, 2021), etc.
Work done during Keming Lu's internship at Tencent Lab.
'Checkpoints and datasets are available at ht tptps: ////ttttttt//t//////////////tttt wtcchu..ooooeeee www//okkkmingeeeooooooo... inghua// Instruction inpructiioniii on- on-Tuning- for world-r-
Furthermore, closed-world IE usually extracts all information without focusing on desired targets.

---

Therefore, we design Open-world Entity Profiling accepting an unstructured corpus and an instruc- tion that characterizes target entities, identifies all entities within the context, and generates entity pro- files, as shown in Fig. 1. With the emergence of large language models (LLMs) (Zhao et al., 2023), generative IE based on LLMs holds substantial promise in addressing the open-world challenge, given their exceptional generalization capabilities on various instructions. Open- world Entity Profil- ing can also be a pivotal capability for integrating plugins into the ChatGPT system since it provides a flexible communication interface between LLMs and their plugins. Nevertheless, existing research on LLMs reveals that they typically perform poorly in IE under zero-shot scenarios, necessitating ap- propriate instruction tuning to enhance their IE capabilities (Ma et al., 2023; Wadhwa et al., 2023). Therefore, instruction tuning (Wei al., 2022) be- comes critical in endowing LLMs with Open-world Entity Profiling abilities.
STRUCTOPENWIKS to assess Open-world capabil- ities thoroughly, focusing on the generalization of unseen instructions and out-of-ontology enti- ties. Our contributions are verified with experi- ments and multifaceted analysis. Most notably, PIVOINE exhibits impressive generalization capa- bilities on unseen instructions and out-of-ontology cases, demonstrating its robust potential to address the open-world challenge effectively.
2 Related Works
Large Language Models. Large language mod- els (LLMs) is an emerging topic summarized in a recent survey Zhao et al. (2023). Therefore, we only provide highly-selective review. Brown et al. (2020) train an auto-regressive language model GPT-3 with 175 billion parameters, showing ex- traordinary task-agnostic few-shot performance. Chowdhery et al. (2022) develop a Pathways Lan- guage Model PALM and scale it up to 540 billion parameters. Scao et al. (2022) propose BLOOM, open-access LLMs from 560 million to 175 bil- lion parameters. Touvron et al. (2023a) develop LLAMA, more efficient public-accessible LLM. We use BLOOM as the backbone since it the latest public LLM pretrained on diverse corpus, including codes, when we conducted this study. However, other latest LLMs, such as LLAMA, can also be easily tuned on our dataset to acquire open- world IE abilities.
To generalize well on unseen instructions and ontologics, we develop PIVOINE (Instruction- following Open-world Information Extraction). PIVOINE is an LLM designed for Open-world En- tity Profiling. We formulate Open-world Entity Pro- regresseeee
cnntructioo-oollowiiin generation task to generate comprehensive entity profiles in JSON. We explore eight popular cate- gories of instructions various granularities. Each category imposes specific constraints on candidate entities. In pursuit of generalization over unseen cases, we develop an instruction tuning dataset INSTRUCTOPENWIK for Open-worth world Entity Pro- filing, including diverse instructions that endow PIVOINE with solid instruction -orruuttiooooolowinng capabil- ity. INSTRUCTÖPEN WIKI incorporates rich anno- tations, various instructions, and a delicate out-of- ontology evaluation set, significantly contributing to the generalization of unseen cases.
filing
Instruction Tuning, Instruction tuning is emer- gent paradigm that finetunes LLMs on datasets described by instructions. Wei et al. (2022) fine- tune an LLM with 175 billion parameters on var- ious NLP datasets with instruction templates and proof instruction tuning can significantly improve zero-shot performance. Ouyang et al. (2022) show supervised instruction tuning and finctuning with human feedback helps L.MSs align with human in- tent. This work is further extended by OPENAI and becomes the product CHATGPT2 used as baseline in our work. In this work, we create an instruction- following dataset INSTRUCTOPEN WIKI for open- world IE and employ instruction tuning to empower LLMs with Open-world Entity Profiling abilities.
The contributions of this work are fhree-fold. First, we define Oorn-world Entity Profiling and develop PIVOINE, which performs entity-centric IE without the limitations of predefined ontology. This flexibility allows for its generalization abilities and application across diverse downstream scenarios. Second, we construct substantial Open-world En- tity Profiling dataset INSTRUCTOPENWIIK Third, we explore comprehensive evaluation for Open- world Entity Profiling. We meticulously design an open- open-world world evaluation set incorporated in IN-
Information Extraction. Instraction-followinn IE reformulates IE into generation task with instruc- tions describing target information. We mainly present two concurrent works as this is an emerg-
Phttps://openal wtteeeaaa.cccmeeeem www.c//m/oo.oo/. bbtg//loo////oooo//ooo chatgpt.

---

ing topic. Wei et al. (2023) solve IE a multi- turn question -answernnn format by providing pre- defined instructions to CHATGPT. Wang et al. (2023) proposes an instruction-tunit tuningg IE bench- mark and develops a unified TE method. How- ever, all these works are based on the closed-world setting and have not adapted to Open-world IE, precisely our focus in this work. To our best knowledge, PIVOINE is the first work exploring cntructiion-ollowiiii open-world entity profiling. Previous explorations are limited to sub-fields of IE, such as the NIL problem EL (Lin et al., 2012) and open information extraction (Zhou et al., 2022).
Then, we augment the dataset with diverse instruc- tions and rephrase them to enrich semantic diver- sity, We also comprehensively evaluate PIVOINE on the open- world evaluation set (84).
3.2 Instruction Tuning for Open-world Entity Profiling
Leveraging the strong generalization abilities of LLMs to pursue generalization on unseen instruc- tions and out-of-ontolog cases, we reformulate Open-world Entity Profiling following generation task. We create diverse in- structions and linearize structured IE outputs into JSON sequences. And then, we finetune LLMs in a supervised instruction tuning (SFT) setup, training LLMs to generate the targeted ISON sequence of entity profiles as the output.
instruction-
3 Methods
We describe PIVOINE, an Open-world Entity Profiling method with instruction-followinn abil- ities. We will introduce preliminaries ($3.1), instructionnfoolooing open-world entity profil- ing ($3.2), and construction of the dataset IN- STRUCTOPENWIKI ($3.3).
The diversity of instruction categories is essential for the generalization of un- seen instructions. We manually designed eight in- struction categories with varying granularities:
Instruction Design.
Extract all entities and relations in the input without additional requirements.
Default:
3.1 Preliminaries
Base Type: Extract entities of given base types.
Problem Definition. Open-world Entity Profiling aims to extract entity profiles from unstructured texts without predefined ontologies by following specific instructions. Entity profiles include men- tions, canonical titles, types, descriptions, aliases, and relations, shown in Fig. I. Specifically, men- tions are text spans linked to entities; types are a list of phrases an entity is an instance of; aliases are a list of synonyms; relations are a list of relation titles between extracted entities within the input. Given a document and an instruction describing a specific constraint about target entities, world Entity Profiling methods are expected to generate entity profiles that fix the instruction.
We define base types as fine-grained types Wikidata, building from the "P31 (instance of)"
properties of entities in Wikidata.
A Abstract Type: Extract entities of given abstract types. We define abstract types
coarse- grained types obtained by finding parent "P279 (subelass of)" properties of base types in Wiki- data. This instruction category only designed for extending type semantics during the training and helps LLMs learn type hierarchy. Both in- structions with types share the same prompt, so we do not distinguish them in inference.
Description: Extract entities that fit given de- scriptions, providing ultra fine-grained instruc- tions that require Open-world Entity Profiling methods directly understand diverse descriptions. As shown in Fig. 2, descriptions can be phrases or sentences describing entitics' propertics. Importance: Extract the top-K most important entities. The importance of entities is defined as entity priorities in Wikidata. This instruction requires Open-world Entity Profiling methods to
Method Overview. We solve Open-world Entity Profiling by instruction tuning LLMs. As shown in Fig. 2, we first reformulate Open-world Entity Profiling into auto-regressive generation by lin- earizing the structure knowledge into the ISON format ($3.2). We apply instruction tuning to em- power PIvOINE to extract different entities follow- ing instructions in different granularities. To do so, we build INSTRUCTOPENWO W IKI, large-scale instruction-following Open-world Entity Profiling dataset. As presented in Fig. 3, INSTRUCTOPEN- WIkI is created by weak supervision between large- scale corpus and existing knowledge base ($3.3).
rank entities with inherent priorities properly.
Number: Extract specific number of entities in the input document. Unlike other categories, instructions with number aim to extract partial information from the input, and the answer is not

---

Instruction Categories
JSON
Auto rogrosiet Gencration
JSON
Asto-iieessive Gottration
Octalt
Boso Type
Atahas Doe
Nocovtonce
down fordty Extraction
"cntities": U& 2 rumas entities
"matitias". [t estities
"title"; "T.G.dhepaard"
Descsplot
"esstee: Heaustey susict, E1
hyo"" ("aaa" country singe--ioppeitet
|ooorer 1. 210i0esc.1 "descristion"
"Farty Tåre" is a song wratton by Bruce Charnel,
"Allases": [-]
country
"aliasse"s [-] sie sares)
auste artist T. 6. Sheppent. It was releaned Jure 1981 the secord single fros aibun Love 'tis AI.", The Sheppard's is eighth number one the country
"waation" "Bruce Ouareet".
"nentioe":
"cista": love "Ts AET
sypp": ("aadd") "dscriptiin"" 22,0)
"type: l'Alsur I, "dsecriptin":
Hi doos Rolacioo extractieo
JSON
Aute-rogrerate Generation
chart. The single ore week and spent total of thirteen veeks
"troplate", It
"tall" "ourtry
"antitias"; eatity fit the descriptioe the query "cantise:: "Eruce thams
country shart.
ratatioraa I'gerral
"type" ["honee"]
"description" --" y1,
PLease Loentify entities in the foloningg categoracs: Geocs, Albus,
J8ON
Auto-rognoscb
Plaase extract bol basins 2n the context,
Please Adbicity ac, enEccy w2th the followseg Seseriptrnn: the Jancican coustry sutic
ontities"s estity the description the query "ranttoat: Sheppare", Sxeppard",
LLM
soep "Party
Please voeatity on sariry with the rollnving Sescriphrens the Zeectcon comstry miviC
"allasas" (Vitii Browten", "Brian Stacy") )1,
"dscripttoon" To --"
Figure 2: Overview of the open-world entity profiling method PIVOINE. This figure shows four generation cases of PIVOINE under three instruction categories colored in orange, blue, and green. PIvoINE takes corpus and instructions as inputs and regressesyyyvlly generates JSON sequences. PIVOINE can extract different entity profiles based on different instructions from the same corpus. The auto-regressive generation of ISON targets aligns well with various IE tasks marked as green comments in the JSON.
unique. Therefore, we separately analyze these categories in evaluation.
key-value pairs, capturing additional semantics in natural language by keys and providing flexible structures. This linearization aggregates multiple IE subtasks, revealing the chain of thoughts in IE employed in traditional pipeline methods.
Number+Base Type: Cross instructions be- tween categories Number and Base Type. Number+Abstract Type: Cross instructions be- tween categories Number and Abstract Type. We manually design an instruction template for each category. Then we ask CHATGPT to rephrase the manually designed templates, enhancing the se- mantic diversity of these templates. Details of orig- inal and rephrased templates are shown in Tab. 8. CHATGPT prompts for rephrasing are shown in Tab. 22. We train PIVOINE to follow single instruc- tions. And we add the last two cross-instructions to evaluate generalization on complex instructions.
3.3 Instruction Dataset Construction
Learning from large-scale instruction tuning dataset with rich corpus and large ontology con- tributes to the generalization of out-of-ontology cases. However, building a large-scale Open-world Entity Profling dataset by manual annotations is infeasible since identifying entities in text and link- ing them with entity proñies require tremendous human effort. Therefore, develop weakly supervised method that automatically creates the dataset INSTRUCTOPEN for instruction tun- ing.
Linearization. Previously, various techniques have been explored to linearize the structured informa- tion in the generative IE (Ye et al., 2022) but either lack semanties or require additional training for spe- cial tokens (Lou et al., 2023; Wang et al., 2021; Lu et al., 2022). To better leverage pretrained knowl- edge, we present a novel idea that uses the ISON (JavaScript Object Notation) format to linearize heterogeneous structured entity profiles. JSON is primarily used to transmit data in web applica- tions, so it frequently appears in codes. Therefore, LLMs pretrained on codes are familiar with the JSON schema, avoiding additional training for spe- cial tokens or manually-defin defined structure templates. Furthermore, JSON uses a text-based syntax with
Weak Supervision. INSTRUCTOPEN is cre- ated by aligning anchor links in Wikipedia to en- tity profiles in its corresponding knowledge base Wikidata* by the wiki identifiers, shown in the left part of Fig. 3. Wikipedia is large corpus cov- ering various domains, while Wikidata contains rich world knowledge. Wikipedia and Wikidata are frequently revised by contributors worldwide, en- suring precision and being up-to-date, All anchor links in Wikipedia are manually annotated, so link-
Phttps:/ ///www wi wikipedia... org/ "https: //www//www/////////////// wi kidata.orgg

---

ing between mentions and entities is reliable. We only use the leading paragraph in each Wikipedia article since it contains the richest anchor links. Be- sides, anchor links referring to the same entity may only be marked once the first time within an article, so using the rest of the paragraphs will face higher risks of missing mention annotations. We retrieve four fields from Wikidata as its profile for each entity, where canonical titles are English labels of entities, and types are derived from "instance of (P31)" properties. After identifying all entities in a paragraph, we employ distant supervision to iden- tify relations between these entities from Wikidata as described at the top of Fig. 3. Specifically, we link relation triplet in the KB to this paragraph if both head and tail entities are mentioned. A rela- tion triplet is represented by mentions of head and tail entities and list of relation names. Detailed statisties are presented in Appx. SB.
dumps, 06/20/2022 and 03/01/2023, And we select new entities appearing in the Wikidata dump on 03/01/2023 but not presented in the Wikidata on 05/30/2022 as out-of-ontology entities S. Using time difference to build an out-of-ontology eval- uation set minimizes potential data leakage and maximizes completeness of training annotations.
Metrics. Although defining Open-wort Entity Profiling as an end-to-end task, we still split it into six tasks in evaluation to provide more compre- hensive analyses: (1) Mention Detection (MD) corresponds to the correctness of mention extrac- tion in "mention" keys. (2) Entity Linking (EL) evaluating whether models generate proper canon- ical titles for mentions in "title" keys. We use hard and soft matching based on ROUGE-L Fl threshold as the criterion.
(3) Entity Typing (ET) requires models generate entity types. (4) Open Relation Extraction (RE) evaluates extracted re- lations under "triplets". Inspired by OpenlE (Zhou et al., 2022), we use CaRB with the ROUGE-L matcher (Bhardwaj al., 2019b) to evaluate triplet generation performance. (5) Description Genera- tion (EIG-Desc.) requires models to generate de- scription. (6) Aliases Generation (EIG-Aliases) expects models to generate aliases "aliases" keys. We report precision, recall, and Fl scores on each task except description generation, where we re- port ROUGE-L instead. We randomly select three rephrased templates for each sample during evalua- tion and report the average with standard deviation.
Instruction Augmentation. We further augment the dataset with predefined instructions as shown in the middle of Fig. 3. We generate an instruction- tuning sample with the default instruction and ran- domly augment one another from six training cat- egories. All instructions focus on entities, and also filter triplets to ensure alignment.
4 Evaluation
A well-designed open-world evaluation set is es- sential for evaluating Open-world Entity Profiling methods without bias. So we create an open-world evaluation set with rich out-of-ontology cases and design metrics for evaluating such performance.
Unseen Ontologies. We explore the out-of- ontology generalization by separately analyzing the recall of training (Before 05/30/2022) and out-of- training (After 05/30/2022) entities. For instance, 2023 ATP Tour
Open-world Evaluation Set. Previous work con- structs open-world test sets by simply holding out a portion of entities from the training ontology. How- ever, such methods may introduce potential risks and lead to insufficient evaluation. First, holding out entities from the training ontology also removes corresponding links in the training corpus, hinder- ing the completeness of annotations. Moreover, mentions of held-out entities still frequently appear in the training corpus even if they are not explic- itly annotated, resulting in potential data leakage. Therefore, we propose delicate method that uses the time difference between Wiki dumps con- struct strictly open-world test set. As shown in the bottom of Fig. 3. we use the Wikidata dump on 05/30/2022 and the Wikipedia dump on 06/20/2022 to build the training set. As the evaluation corpus, we filter all new articles between two Wikipedia
/27) shown in Fig. 1 is unseen entity introduced to Wikidata after 05/20/2022. Open-world Entity Profiling methods are proven have a great generalization of unseen ontologies if they can extract this entity from the latest corpus.
Unseen Instructions. We also split the test set into samples with unseen and seen instructions un- der the most fine-grained category "Description". Unseen instructions query target entities that are not in the training instructions. The proportions of queries with unseen entities are shown in Tab. 4.
'The ROOTS corpus (Laurençon et al., 2022), pretrained corpus of BLOOM, only includes Wikipedia dump before 06/20/2022, so this evaluation set also remains unseen for the BLOOM pretraining.

---

Instruction Dataset Construction
Supervision
JSON
"Party Tsme!! song vrstten by Bruce Chancel, and recorded by American country music 823563 artist 6. Sheapard, ne released in June 1911 as the second single from albux
Nettin
"entities"s It
Extraction
"santion": "country rusiet, NO
Tye
Dhalsgans brieg
WHI0A7A
"eitie": "Conntry muste",
Decrppiee
Serve of Anwiee copuler mute
"type"; ET "description"
Alana
weulity meando
"aliases": L.]
Iarwnn
"santion":
Shespard's eighth runber one on the country chart. The
Sesreke
Tue
Xin
150) Sheperd
"type": ('Atbus"), "deacription"
"titte": Love
kingle stayed
Instruction Categories
11.
Relation Extraction
for one vesk and spent total of thirteen weeks on the country chart.
Delsa)
triplets":
head": "I Love 'On ALT,
Murderr
"tail: "conntry wustc",
ralations": ["geare"]
Please 2doncify Iat oncitacs da the catagertes:
Filber wi Instructions
Linearization
Detology
1543625222
Cut-of-ontsingy
030112923
Open-world Evaluation
Linking
Training
06.20 2022
Eveluation
0310 12223
Figure 3: Overview of instruction -ostruttoo-followingg spen-world Entity Profiling dataset construction. The top of this figure shows INSTRUCTOPEN WIKI is created by aligning anchor links Wikipedia entities and relations in the knowledge base Wikidata, then augmented with instructions within eight categories. Entity profiles are identified from Wikidata and lincarized into sequence in the JSON format. The bottom of this figure shows how we create open-world evaluation set with the time difference between Wiki dumps.
Furthermore, we create three more instruction tem- plates by mamually writing and GPT-4 generation, respectively, and make sure they have no overlap with rephrased training instructions. So we can analyze the performance difference between the two new sources and training instructions. We also separately evaluate number-related instructions as these partial extraction instructions have no unique correct answers.
model LLaMa-2 (Touvron et al., 2023b). We also introduce two traditional closed-worf IE baselines: (3) GENRE (Kumar and Bojar, 2022) is the first system retrieving entities by generating canonical names, which can address MD and EL but con- strained by the KILT entities. (4) OPENIE6 (Kol- luru et al., 2020) is a well-developed neural OpenIE system. We employ it baseline of RE.
Configurations. We start from BLOOM (Scao et al., 2022) checkpoints with 1 billion and 7 billion parameters and run instruction tuning on INSTRUCTOPEN WIKI. We use suggested hyper- parameters for finetuning each model in Scao al. (2022). PIVOINE-Ib is trained on 64 NVIDIA V100 GPU for 92 hours. PIVOINE-7b is trained on 256 NVIDIA V100 GPU for 54 hours. Detailed configurations are shown in Appx. $E.
5 Experiments
We carry out comprehensive evaluation of PIVOINE, including experimental setup ($5.1),
main results ($5.2), and further analysis ($5.3).
5.1 Experimental Setup
Baselines. We employ ChatGPT as our main basc- line since no previous Open-wond world Entity Profiling methods exist to our best knowledge. (1) CHAT- GPT is an instruction-follooiin LLM that can handle various tasks. Detailed configurations of CHATGPT are described in Appx. SC. (2) CHAT- GPT w/DEMO is stronger baseline with one- shot demo based on CHAIGPT. We provide CHAT- GPT an instruction example in the conversation history. Furthermore, we also involve the latest Vi- CUNA-7B-vi.. (Zheng et al., 2023) as a baseline, an open-source instruction -uutinn model trained on over 150k dialogues based on the foundation
5.2 Main Results
We present main results in three aspects: over- all performance, generalization study on unseen ontologies, and unseen instructions.
Overall Performance. Tab. shows overall per- formance on six tasks. We report macro average F1 scores with all instruction categories. And we only report performance on default instruction for GENRE and OPENIE6 for reference since they have no instruction following abilitics. They fail to achieve outstanding performance on correspond-

---

MD
EL
ET
OpenRE FI(CaRB)
EIG (Desc.) FIRROUGE-L)
EIG (Aliases)
Method
FI(T=I)
FI(T=088)
F1
Cloldd-world Baselines
43.77
17.2
20.17
15.2
CHATGPT
50.5u.2
25.0u.2
26.1a.2
7.90.0
22.5u.
39.0o.2
15.80.0
CHATGPT w/DEMO
51.10.1
38.6u.0
39.9u.e
14.80.1
21.40.9
52.00.2
13.0v.1
VICUNA-7B
30.6a.a
16.5u.o
17.2a.0
3.40.0
20.0o.9
28.40.0
10.8o.1
PIVOINE-Ib 61.4a.a
49.6n.o
50.6o.e
40.4o.o
56.
68, 68.6p.0
72.To.1
PIVOINB-7b
79.60.o
69.8o.1
70.70.1
56.40.o
67.80.2
80. 40.2p.1
80.5o.
Table I: Main results of overall performance end-to-end evaluation. We report the macro average of Fl scores with all instruction categories mention detection (MD), entity linking (BL), entity typing (ET), open relation extraction (OpenRE), and entity information generation (EIG) for descriptions and aliases. Subscript scores are the deviation under three different rephrased instructions for each sample. Comprehensive results are demonstrated in Appx. SF. We only report the performance of closed-wort baselines on default instructions since they focus on specific tasks without instruction following abilities, such as GENRE MD, EL, ET and OPENIE in OpenRE.
MD R
EL
ET
OpenRE R(CaRB)
EIG (Desc.)
EIG (Aliases)
Partition
Method
R(T=1)
R(T=0.8)
R(ROUGE-L)
R
Closed-world Baselines
55.31
23.8t
24.7*
21.9*
CHATGPT
59.30.3
31.80.2
32.3p.2
9.70.1
34.1o.2
10.20.4
10.1p.1
Before
CHATGPT w/DEMO
56.5p.0
45.0n.1
45.70.1
15.90.2
38.30.0
53.6n.2
8.40.0
05/30/2022
VICUNA-7B
26.50.0
14.90.0
15.1p.
3.30.0
36.40.0
29.40.0
8.80.0
PIVOINE-Ib
59.2o.1
51.8u.0
52.20.0
44.4p.1
61.8u.a
71.4u.1
69.7p.1
PIVOINE-7h
83.7o.o
78.80
79.10.1
66.80.1
79.40.1
82.50.0
78.0p.o
Clased- world Baselines
33.6*
gt
3.6*
18.2
CHATGPT
58.90.2
23.80.2
26.50.3
6.9b.1
27.1a.n
36.0a.3
14.3p.a
After 05/30/2022
CHATGPT w/DEMO
58.2o.o
39.
42.
14.2p.1
29.3u.s
48. 18.
13.3p.1
VICUNA-7B
26.7o.o
13.0a.a
14.3p.0
3.10.0
25.5a.0
25.8a.a
10.5p.o
PIVOINE-1b
55.1n.0
36.3n.1
38.3p.1
22.80.0
34.5n.1
55.7o.3
20.5p.a
PIVOINE-7b
71.70.1
51.70.2
53.8p.2
24.10.2
36.10.1
64.1o.2
22.7o.1
Table Main results of generalization analysis in end-to-end evaluation. Headers the same Tab. except report recalls in each task. The partition "Before 05/30/2022" denotes mentions linked to the training ontology (seen entities), while "After 05/30/2022" denotes mentions linked to entities that are out of training ontology (unseen entities). Comprehensive results demonstrated in Appx. $F. 1 We only report the performance of closed world baselines on default instructions since they focus on specific tasks without instruction following abilities, such as GENRE in MD, EL, ET and OPENIE in OpenRE.
ing tasks, showing Open-world Entity Profiling is challenging for closed-world IE methods as we ex- pect. CHATGPT can partially address Open-world Entity Profiling but only has 7.9% Fl score in ET and 15.8% Fl score in EIG(Aliases). CHATGPT w/DEMO significantly outperforms CHATGPT in EL and EIG(Desc.) and has comparable perfor- mance on other tasks, showing the demo in history benefits CHATGPT Open-world Entity Profiling. PIVOINE-Ib lb outperforms CHATGPT w/DEMO by about 10% absolute improvement in Fl over all tasks. PIVOINE-7b achieves the best performance among all methods, significantly outperforming PIVOINE-Ib by nearly 20% Fl in most tasks. This result suggests larger models will boost the perfor- mance and even can potentially address some of the tasks.
rately analyze recalls of entities before and after 05/30/2022 in the open- world test set and present results in Tab. 2 to evaluate generalization on un- ontologies. The partition "Before 05/30/2022" denotes mentions linked to the training ontology while "After 05/30/2022" denotes those that are out training ontology. We first witness con- sistent performance drop on unseen entities for all methods, especially closed-world baselines. Even for two ChatGPT-based baselines, performance on EL, ET, OpenRE, and EIG(Desc.) also drop dra- matically. PIVOINE-1b lb outperforms CHATGPT w/DEMO four tasks in the unseen partition, and surpasses CHATGPT w/DEMO on all tasks in the seen partition. PIVOINE- 7b still achieves the best performance on tasks in both partitions, showing can successfully generalize to out-of-ontology entities in Open-world Entity Profiling.
open-world
Generalization to unseen ontologies. We sepa-

---

Mention Datection
Entity Linking
Entity Typing
After 05 232222
Abstract
Nestract
srost
Defore DSANN 2022
Basn 1ype
Description
Bane Type
Description
Base Type
Dewcription
Delautt
Importacc
Defaut
Importance
Dettut
Imporiance
Importance
Descripton
Dare Type
Descrition
Dase Type
Description
Dooe Type
Abstract hype
Nostradd Type
Pedrost Type
ChatGPT DhsiSPT mDDemo PIVOrNE-77
Open Relation Extraction
Description Generation
Alias Generation
Base Type
Descritioon
Base Type
Description
Base Type
Descrition
Defa.
Importence
Dafauti
Importence
Deluzk
Importance
Importance
Importance
Descrittion
Base Typo
Desoritton"
Bose Type
Desorition
Boss Type
Abstrcl
Abstrad
Aestract
Figure 4: Main results of the ontology generalization for each instruction category in each task. The partition "Before 05/30/2022" denotes mentions linked to the training ontology (seen entities), while "After 05/30/2022" denotes mentions linked to entities that are out of training ontology (unseen entities). Descriptions as instructions are not considered in description generation tasks to avoid data leakage. Original scores are shown in Appx. $F.
MD
EL
ET
OpenRE FI(CaRB)
EIG (Aliases)
Instruction Source
Partition
FI(T=1)
FI(T=0.8)
Fl
Training
83.60.2
55.10.1
57.80.1
28.2a.1
35.0p.a
12.80.a
83.3u.2 -0.3
54.4u.2 -0.7
57.1u.1
28.40.1
34.5o.1 -0.5
13.00.2
Manual Written
-0.7
-0.2
+0.2
83.40.2
54.8u.1 0.3
56.3u.1
27.8g.1
34.6u,2
12.7p.2
GPT-4 Generation
-.22
-0.4
-0.4
-0.1
Table 3: Main results of the instruction generalization on "Description" instruction category with PIVOINE-7b the unseen entities described in $4. A refers absolute differences between instructions from new sources and training instructions. Headers are the same as Tab. 1. To avoid data leakage, descriptions as instructions are not considered in description generation tasks. Subscript scores are the deviation under three different rephrased instructions for each sample. The best scores in each partition marked in bold.
Fig. 4 shows more detailed analysis each in- struction category. Comparing six radar charts, all methods are easy to generalize to out-of-ontology entities on MD, EL, and EIG(Desc.) as their up- per and lower parts of plots are nearly symmet- ric. However, such generalization is harder in RE since it an end-to-end task requiring precise out- of-ontology MD first. ET and ElG(Alisses) are the most challenging for all methods because they require a comprehensive list of types and aliases. We also find PIVOINE-76 7b consistently outperforms other basclines on all instruction categories.
uate the performance difference PIVOINE-7b with unseen entities queried by instructions from various sources of instructions. The performance difference on most tasks is within I absolute per- cent of Fl scores. Therefore, PIVOINE-7b shows extraordinary generalization unseen instructions in Open-world Entity Profiling.
5.3 Analysis
We provide further analyses to reveal model behav- iors on Open-worle world Entity Profiling.
Instruction Following, We analyze instruction fol- lowing qualities in three aspects shown in Fig. 5. First, we analyze the JSON decoding error rates, generating the correct JSON schema is essential for parsing extracted information. As the left figure in Fig. 5, ChatGPT based baselines fail to ensure
Generalization unseen instructions. We eval- uate the generalization on unseen instructions and present the results Tab. As introduced in the last paragraph of $4, unseen instructions are con- structed by manual writing and GPT-4. We eval-

---

JSON Dacoding Error Rate (%)
Eniity Number Instruction (%)
Enilty Type Instruction
(%)
Macss.g.
Nuroen Gone
100
Macro
Usilad
Nanner
Nember
Neroonddaatoo Syo
See
Nustoundooae Sye
Bean lase
Nacmendeal lype Necoenttattoo
epulaxe
80
Awoad Type
50-
30-
40
20
10
DnaiGPT
ChaiGPT wDonn
PIVDINE
PIVOINE T6
ChaiGht
ChairPT vDDon
PIVONN
PIVOIN T0
ChaiGPt
ChstcPP w/fano
PNVOE to
PNONNE To
Figure 5: Analysis of the instruction- -irruction-following capabilities, including JSON decoding error rate, entity number instruction failure rate, and entity type instruction failure rate. Original scores are shown in Appx. $F.
the valid JSON schema, especially on unseen cross instructions. PIVOINE-lb is better than baselines on average but still faces decoding errors. However, PIVOINE-7b is a trustworthy model with no JSON decoding errors all instruction categories, even for unseen cross instructions.
4 to provide judgment and explanation based on evidence supplied by PIVOINE. We also manually evaluate the correctness of relation triplets based on the same evidence without additional world knowl- edge. The accuracy of relation triplets provided by GPT-4 is 87%, and manual checking accuracy is 83%. The agreement between GPT-4 and the annotator is 87%, suggesting GPT-4 is capable of evaluating relation correctness. The latest Wiki- data also verifies 8% cases. This evaluation shows PIVOINE-76 can precisely excavate new entities and relations from the corpus.
We also compare entity number instruction fail- ure rates on four number-related instruction cate- gories as shown in the middle of Fig. PIVOINE- 7b still has the lowest failure rates extracting the correct number of entities, which is close to zero on two trained instruction categories. All methods fail to follow number instructions when receiving untrained cross instructions, suggesting cross in- structions still challenging for current methods. Furthermore, results in Tab. 21 shows that number- related instructions do not hinder the precision of MD and EL. Instructions with number constraints provide partial extraction abilities to PIvOINE so that we can control the precision-recal trade-off by specifying the number in instructions.
Conclusion
We propose Open-world Entity Profiling, chal- lenging task that aims to extract out-of-ontology entity profiles with instructions. Towards this grand mission, we create a large-scale Open-world En- tity Profiling dataset INSTRUCTOPENWII and develop PIVOINE by instruction tuning. We con- duct extensive experiments on diverse instruction categories and different model scales, showing PIVOINE is trustworthy LLM capable of fol- lowing (possibly unseen) instructions in various granularities and extracting out-of-ontology entity profiles. Valuable future works include extending PIVOINE to a larger scale and exploring more com- prehensive instruction categories.
Similar patterns are observed in entity type in- struction failure rate analysis in the right part of Fig. 5. PIVOINE-7b only has half failure rates cross instruction categories compared with the vanilla CHATGPT, showing PIVOINE-7b has much better type instruction following abilities. We also notice that following instructions with abstract types is significantly easier for PIVOINE than fine- grained base types.
Limitations
Human Evaluation. PIVOINE may extract correct relation triplets that are out of the scope of exist- ing Wikidata. Therefore, we randomly select 100 relation triplets with unscen entities predicted by PIVOINE-7b analyze its precision further. We consider the evidence of relation triplet as the context from which PIVOINE-76 7b extracts the triplet. To minimize the manual effort, we first reformulate the output of PIVOINE-77 7b to a prompt and ask GPT.
The contents in PIVOINE outputs are limited by the output length constraint of BLOOM. However, techniques for extending such length limitation are rapidly developed (Yu al., 2023). Furthermore, instructions investigated in this work mainly focus on entities. Future works can extend the scope of instruction categories to cover demands about relations or events.

---

Ethics statement
targeting nil-linking cases. In Proceedings of the st ACM International Conference on Information & Knowledge Management, pages 4069-4073.
All data and results are obtained and processed ac- cording to the respective data and API usage policy. PIvOINE is developed based on BLOOM (Scao et al., 2022). Although PIVOINE has great perfor- mance in Open-world Entity Profiling, PIVOINE may still generate potential counterfactual outputs. Outputs from PIVOINE need careful verification before being applied real-world applications.
Nikolaos Kolitsas, Aetavian-Eugen Ganea, and Thomas Hofmann. 2018. End-to-end neural entity linking, In Proceedings of the 22nd Conference on Computa- rional Natural Language Learning, pages 519-529, Brussels, Belgium. Association for Computational Linguistics.
Keshav Kolluru, Vaibhav Adlakha, Samarth Aggarwal,
Mausam, and Soumen Chakrabarti. 2020. Open1E6: Iterative Grid Labeling and Coordination Analysis for Open Information Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1748-3761. Online. Association for Computational Linguistics.
References
Sangnie Bhardwaj, Samarth Aggarwal, and Mausam Mausam. 2019a. CaRB: A crowdsourced benchmark for open IE. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pages 6262-6267, Hong Kong, China. Association for Computational Linguistics.
Nalin Kumar and Ondrej Bojar. 2022. Genre transfer in NMT:ereating synthetic spoken parallel sentences us- ing written parallel data. In Proceedings of the 19th International Conference on Natural Language Pro- cessing (ICON), pages 224-233, New Delhi, India Association for Computational Linguistics.
Sangnie Bhardwaj, Samarth Aggarwal, and Mausam Mausam. 2019b. Carb: crowdsourced benchmark for open ie. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCNLP) pages 6262-6267.
Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association forr Computational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sustry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901.
Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villamova del Moral, 'Tvven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, Jörg Frohberg, Mario Šaško, Quentin Lhoest, Angelina McMillan-Major, Gérard Dupont, Stella Biderman, Anna Rogers, Loubna Ben allal, Francesco De Toni, Giada Pistilli, Olivier Nguyen, Somaieh Nikpoor. Maraim Masoud, Pierre Colombo, Javier de la Rosa, Paulo Villegas, Tristan Thrush, Shayne Longpre, Se- bastian Nagel. Leon Weber, Manuel Romero Muñoz. Jian Zhu, Daniel Van Strien, 7aid Alyafeai, Khalid Almubarak, Vu Minh Chien, Itziar Gonzalez-Dios, Aitor Soroa, Kyle Lo, Manan Dey. Pedro Ortiz Suarez, Aaron Gokaslan, Shamik Bose, David Ife- oluwa Adelani, Long Phan, Hieu Tran, Ian Yu, Suhas Pai, Jenny Chim, Violette Lepercq, Suzana Ilic, Mar- garet Mitchell, Sasha Luccioni, and Yacine Jernite. 2022. The bigscience ROOTS corpus: A 1.6TB composite multilingual dataset. In T'hirtysixth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track.
Laura Chiticariu, Yunyao Li, Sriram Raghavan, and Frederick R Reiss. 2010. Enterprise information ex- traction: recent developments and open challenges. Proceedings of the 2010 ACM SIGMOD Interna- tional Conference on Management of data, pages 1257-1258.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
Ralph Grishman. 2015. Information extraction. JEEE Intelligent Systems, 30(5):8-11 15.
Pere-Lluis Huguet Cabot and Roberto Navigli. 2021. REBEL: Relation extraction by end-to-end language generation. Findings of the Association for Com- putational Linguistics: EMNLP 2021, pages 2370- 2381, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Thomas Lin, Oren Etzioni, et al. 2012. No noun phrase left behind: detecting and typing unlinkable entities. In Proceedings of the 2012 joint conference on em- pirical methods in natural language processing and computational natural language learning, pages 893- 903.
Anastasiaa Jurshina, Jiaxin Pan, Rafika Boutalbi, and Steffen Staab. 2022. Nilk; Entity linking dataset

---

Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xi- anpei Han, Le Sun, and Hua Wu. 2023. Universal information extraction unified semantic matching. arXiv preprint arXi::2301.0328.
Somin Wadhwa, Silvio Amir, and Byron C Wallace. 2023. Revisiting relation extraction in the era of large language models. arXiv preprint arXiv:2305.0500
Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. 2021. Zero-shot informa- tion extraction as unified text-to-triple translation. arXiv preprint arXiv:2109.0 11171.
Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu, 2022. Uni- fied structure generation for universal information extraction. arXiv preprint arXiv:2203. 12277.
Qifan Wang, Yi Fang, Anirudh Ravula, Fuli Feng. Xiao- jun Quan, and Dongfang Liu. 2022. Wehformer: The web-page transformer for structure information ex- traction. In Proceedings of the ACM Web Conference 2022, pages 3124-3133.
Yubo Ma, Yixin Cao, YongChing Hong, and Aixin Sun. 2023, Large language model is not good few-shot information extractor, but a good reranker for hard samples! arXiv preprint arXiv:2303.08.. 085599
Christina Niklaus, Matthias Cetto, André Freitas, and Siegfried Handschuh. 2018. A survey on open in- formation extraction. In Proceedings of the 27th International Conference on Computational Linguis- tics, pages 3866-3878, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Xiao Wang, Weikang Zhou, Can 7u, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al. 2023. Instructuic: Multi- task instruction tuning for unified information extrac- tion. arXiv preprint arXiv:2304.08085
Yanshan Wang, Liwei Wang. Majid Mustegar-Mojarall Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, et al. 2018. Clinical information extraction appli- cations: a literature review. Journal of biomedical informatics, 77:34 49.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroli Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. 2022. Finetuned language mod- els are zero-shot learners. In International Confer- ence on Learning Representations.
Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. 2016. AFET: Automatic fine- grained entity lyping by hierarchical partial-label em- bedding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Process- ing, pages 1 1369-1378, Austin, Texas. Association for Computational Linguistics.
Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang. Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Mcishan Zhang, et al. 2023. Zero- shot information extraction via chatting with chatgpt. arXiv preprint 2302.120000000
Pedro Ruas and Francisco M Couto. 2022. Nilinker: Attention- based approach to nil entity linking. Jour- nal of Biomedical Informatics, 132:104137.
Hongbin Ye. Ningyu Zhang, Hui Chen, and Huajun Chen. 2022. Generative knowledge graph construc- tion: A review. arXiv preprint arXiv:2210.12714.
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ili,, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arx.v:2211.05/00
Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, and Mike Lewis. 2023. Megabyte: Predicting million-byte se- quences with multiscale transformers. arXiv preprint 2rX.v.3305.07185.
Phillip Schneider, Tim Schopf. Juraj Vladika, Mikhail Galkin, Elena Simperl, and Florian Matthes. 2022. A decade of knowledge graphs in natural language pro- cessing: a survey, Xiviv preprint ar X1v:2210.000000
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang. Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.1822.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023a. llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.1 13977
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, 7i Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging aud-eee-judge with mt-bench and chatbot arena. arXiv preprint arXiv:2306.00 0566.
Hugo Touvron, Louis Martin, Kevin Stone, Peter AI- bert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023b. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. How does nip benefit legal system: A summary of legal artificial intelligence. arXiv preprint arXiv:2004.12155

---

Shaowen Zhou, Bowen Yu, Aixin Sun, Cheng Long. Jingyang Li, and Jian Sun. 2022. A survey on neu- ral open information extraction: Current status and future directions. arXiv preprint 172.

---

Appendix
E Hyper-parameters
Detailed Instructions
We develop our models with from BigScience. We trained PIVOINE-lb for 10,294 steps with global batch size of 1,024 and PIVOINE-7b for 5,000 steps with a global batch size of 2,048. The training steps and learning rates selected by performance on the development set with grid search. We infer on 256 NVIDIA V100 GPU within 30 minutes with the official BLOOM inference project provided by HuggingFace We generate with beam search without sampling and the number of beams is 4. The maximum number of generated tokens is 2,048. The other generation parameters are set to default.
Tab. 8 shows original manually-designed seed in- struction, rephrased instructions, and number of rephrased instructions for each category. We use ChatGPT rephrased our original instructions. The prompt we used is shown in Tab. 22.
B Dataset Statisties
We display the statistics of INSTRUCTOPENWIK in Tab. 9. INSTRUCTOPEN contains rich corpus, which includes all head paragraphs of all Wikipedia articles with 39 million mentions and 19 million triplets. And INSTRUCTÖPEN WIKI is also annotated with large ontology containing over million entities with 21 thousand entity types and 962 relation types, ensuring covers a wide range of domains. Besides, the entity information den- sity of INSTRUCTOPEN IKI is also abundant, so models can be trained for extracting entity profiles efficiently.
F
Comprehensive Results
ChatGPT w/Demo
Instruction
ChatGPT
PIVOINE-16 lb PIVOINE-7b
Default
2.50.1
1.90.1
1.7a.0
0.0g.0
Base Type
3.40.3
3.60.4
2.3a.2
0.0n.0
Abstract Type
3.80.4
2.50.a
2.3a.2
0.0g.0
Description
3.10.3
2.20.2
1.3u.3
0.0g.a
Importance
3.90.s
3.90.9
3.9u.3
0.0g.0
Partition
Base Type
Abstract Type
Description
Number
2.70.s
3.70.1
2.30.1
0.0o.a
Number+
Unseen
18.9%
17.4%
82.3%
3.50.2
5.50.s
3.8a.1
0.0p.0
Base Type
Seen
81.1%
82.6%
17.7%
Number+
3.60.4
2.81.0
3.30.4
0.0o.1
Abstract Type
Table 4: Proportions of unseen and seen instructions in "Base Type", "Abstract Type", and "Description" in the test set.
Macro Avg.
3.30.2
3.30.4
2.10.a
0.0o.a
Table Analysis of JSON format correctness all instruction categories. We report the JSON decoding error rates (%) in this table. Subscript scores are the deviation under three different rephrased instructions for each sample.
We also create small development set to se- lect the best checkpoint balancing performance on seen and unseen cases. As shown in Tab. 9, the open- world evaluation test set contains rich unseen mentions (27.1%, 29,612/109,411 annotated by unseen entities (24.2%, 9,464/39,086).
CharGPT ChatGPT w/Demo
Instruction
PIVOINE-Ib PIVOINE-7h
CHATGPT Configurations
Number
26.8o.4
13.4g.s
6.5g.0
0.60.a
Importance Number+
25.61.1
18.7a.2
7.90.3
0.8a.a
We provide ChatGPT with the input context, in- struction, and output JSON schema to prompt Chat- GPT to solve dnstruction-followiig open-we open-world IE, We use ChatGPT in May 2022 and query it with the official API provided by OpenAI The detailed prompt used is shown in Tab. 23.
13.4o.4
37.3o.0
38.0p.9
29.00.2
Base Type
Number+ Abstract Type
48.30.9
40.8u.4
42.3p.s
37.30.1
Macro Avg.
36.0o.1
27.5o.0
23.7o.1
16.9o.o
Table 6: Analysis of following entity number constraints. We report error rates (5) of predictions that do not have same number entities as the instruction. Subscript scores the deviation under three different rephrased instructions for each sample.
D Details of Human Evaluation
We use the May 1st, 2023 version of GPT-4. We query it via the chat platform of OpenAl?.
hhttps://openat com/blog/ hattps://chat.ore com/
/ wttp:t.cccchheoeeeeeeeeeeeeeooee wwtp.///wgggggttac...oooooo transformers Seanafformerr--bloo-----o wlloo-iinference

---

ChatGPT w/Demo
Instruction
CharGT
PIVOINE-Ib PIVOINE-7b
Base Type
72.4p.8
48.8p.1
38. lu.
22.4u.s
Abstract Type
65.12.1
50.7o.7
16.9p.
4.10.2
Number+
65.8p.s
17.3a.4
42.5g.s
21.77.33
Base Type
Number+
57.11.6
43.5p.8
31.1g.7
25.5
Abstract Type
Macro Avg.
65.1o.s
47.6g.1
32.2g.n
19.2o.s
Table 7: Analysis of following entity type constraints. We report rates (%) of predictions that only have entities in specific types from instructions. Subscript scores are the deviation under three different rephrased instructions for each sample.

---

#Rephrased Templates
Categories
Manually Designed Templates
Rephrased Templates
Default
Extract entities.
Identify the entitics present in the text. Please identify the entities falling under the categories [types]. ).
219
Base Type
Extract entities in types {types).
48
Abstract Type
Extract entities in types (types).
Please identify the entities falling under the categories (types).
48
Description
Extract entities with following
Can you identify the entities described as followed: [descriptions]:
104
descriptions: [descriptions).
Importance
Extract the most important [num] entities.
Retrieve the (number) most essential entities.
62
Number Number+Base Type
Extract [num] entities.
Fetch [umber]] entities. Could you identify (number) entities
49
Extract [num] entities in types (types).
117
helonging to [types]?
Abstrct Type
Extract [num] entities in types (types).
Retrieve [number) entities belonging to [types).
117
Table 8: Details of Instructions. (num), [types], ddsccriptions) are placeholders for entity numbers, types, descriptions. We only display plural forms of templates in this table, but the templates will be different for singular or plural. We only show one example of rephrased templates for each category.
Corpus #Mention
Ontology
Entity Info Density
Split
#Article
#Triplets
#Ent.
#Aliases
#Re1.
#Types
DDese.
SAliases % TTypes
Train
11,447,454
39,930,663
19,184,948
2,234,052
840,401
962
21,350
93.5
64.2
71.5
13,601
6,868
Dev
2.710
5.915
8,812
234
1.163
94.6
55.6
70.2
unseen:3038
unseen:1417
109,411
39,086
Test
24,393
45,758
37,809
398
3,306
92.7
52.6
70.7
unseen:29.612
unseen:9,474
Table Statisties of the cntrruction-oolowinng open-world dataset INSTRUCTOPENWIK: We report the statisties of the corpus, including the number of articles, mentions, and triplets in the left section. The ontology statistics, including the number of unique entities, aliases, relations, and types, are reported in the middle section. The right section shows the proportions of mentions with descriptions, aliases, and types, respectively. Ent., Rel., and Desc. are short for entities, relations, and descriptions.

---

MD
EL
ET
OpenRE EIG (Desc.) FICCARE)
EIG (Aliases)
Method
Partition Instruction
F1
FI(T=I)F
F1
Fl
Base Type 56.31.a
31.41.9
34.32.3
16.5u.s
23.30.9
41.40.s
17.91.2
Abstract Type 47. 23.7
24.60.9
10.30.4
23.01.6
39.52.1
19.70.4
unseen
Description 51.80.1 24.50.4
25.30.3
3.3p.1
22.0o.o
56.30.2
14.70.c
Macro Avg. 51.80.5 26.50.5
28.1o.s
10.0n.3
22.80.6
45.80.6
17.40.4
ChatGPT
Base Type 50.1o.s 26.
27.4a.9
15.30.6
21.0o.s
37.81.1
13.6p.6
Abstract Type 47. 18, 1o.s
19.31.0
7.60.s
21.0p.o
36.6p.s
16.71.4
Description 16.60.s 21.61.0
22.91.2
1.7o.3
22.0p.s
45.11.3
25.0g.8
Macro Avg. 48. lo.2 22. 1o.7
23.2o.8
8.20.4
21.30.s
39.81.0
18.41.6
Base Type 58.40.7 42.00.4
45.50.3
30.0a.2
24.0o.o
45.80.6
17.414
Abstract 17.30n. 37.6n. 6o.s
39.20.9
16.80.4
21.51.5
51.91.4
25.02.7
unseen
Description 5 39.20.2
40.0o.3
6.7p.1
17.50.5
83.5p.3
24.40.5
Macro Avg. 52.6n.2 39.60.3
41.60.3
17.80.2
21.0o.3
60.4o.8
22.30.3
CharGPT w/Demo
Base Type 55. lu.4 40.5u.s
41.50.5
32.90.3
21.0o.u
47.70.1
12.60.6
Abstract Type 49. lo.3 37.00.2
38.4a.2
17.20.2
21.01.0
52.6o.6
18.70.a
Description 47.6u.s 34.11.4
35.41.5
4.60.1
17.0o.u
78, 68.6
38.71.2
Macro Avg. 50.6o.2 37.20.3
38.4a.4
18.2..2
19.70.3
59.7o.2
23.4p.7
Type 62.7u.s 51.3u.4
52.8u.a
38.3u.1
62.7u.o
64.7u.s
74.1e.1
Abstract 54, 0o.2 6o.5
45.2a.c
32.2p.c
53.01.4
68.71.a
70.0..
unseen
Description 62. 62.4 50.6
51.4p.3
44.6g.2
56.7o.s
75.1u.1
70.3p.1
Macro Avg. 59.8 48.8
49.80.3
38.3a.2
37.40.c
69.5o.s
71.50.3
PIVOINE-1b
Base Type 61.70.2
50.1o.1
51.00.2
11.0p.1
56.3o.s
64.2p..
70.9p.2
Abstract 59.c59.00. 48.5o.2
49.30.2
40.8o.1
56.0o.s
62.0o.2
70.10.2
seen
Description 67.8n.s 55.3p.s
57.10.s
42.30.2
62.71.7
90.6p.o
75.0p.7
Macro Avg. 62.8u.3 51.3u.2
52.4g.3
42.4g.1
58.3u.s
72.30.u
72.0p.3
Base Type 84.1o.4 74.40.5
75.60.6
56.0g.3
69.0p.s
74.10.2
76.7o.1
Abstract Type 65.0n.2
65.5o.1
52.0o.3
66.3o.s
78.8o.1
80.6p.1
unseen
Description 88.3o.1 75.90.1
77.0p.1
64.0p.1
72.30.5
87.00.0
79.40.1
Macro Avg. 82.3u.2 71.70.2
72.Ta.2
57.3p.2
69.20.2
80.00.0
78.9..1
PIVOINE-76 7b
Type 76.30.2 66.70.1
67.30.1
59.2g.2
65.76.9
76.90.2
78.60.0
Abstract Type 78.5 69.8u.2
70.4g.2
59.6g.1
69.7o.s
76.8u.1
80.0p.1
seen
Description 90.5o.1 80.9o.1
81.40.1
59.60.3
80.0p.o
97.0p.0
81.3p.0
Macro Avg. 81.8o.1 72.5o.1 73.1a.1
59.5a.1
71.80.4
83.6o.1
80.0p.o
Table 10: Main results of the instruction generalization evaluation on three instruction categories "Base Type". "Abstract Type", and "Description". Headers are the same as Tab. 1. Partition denotes the unseen and seen instructions. Subscript scores are the deviation under three different rephrased instructions for each sample.
GENRE R
ChaiGPT R
ChatGPT w/ Demo
PIVOINE-IP lb
PIVOINE-7b 7b
Instruction
P
FI
P
FI
P
R
FI
P
R
FI
P
R
F1
Default
38.8
50.0 43.7
53.4o, 10.1 57.50.2 55.40.1 51.2p.1 51.90.1 51.6..1
71.1g.o o 57-3o 63.5g.o 82.0.. 76.40. 79. 1o.0
Base Type
41. 90.4 66.31.0 51.4 47.90.: 66. 66.8o.s 55.8 65.8g.1 1 58.5u.e 01.9u. 76.0u.z 80, 20.1 78. lo
Abstract Type
38. 20.4 62.50.2 47, 40.3 42.50.3 57 1o.1 48.7o.2 61.2u.1 54.9u.2 57.9u.1 8..1 78.30.2 77.50.
Description
43. 2 61. 51.1o,0 44.7o.o 60.9p.2 51.6a.1 65.9g.1 60.8g.s 63.2a.a 78.1 88, 40.1 88.50.1
Importance
86.8 47.90.2 40.4 47 57.4 48 2o.1 47 8p.1 64.1n.1 57 60.5 60.7 74.8 74. 74.7o.1
Macro Avg.
44.90.2 59.1o.1 30.50.2 86.8o.1 37.0p.0 51.1p.1 65.6g.o 57.8 61.4n.a 79.7a.1 79.6o.a 79.6o.a
Table 11: Main results of mention detection (MD). We report precision, recall, and Fl score of each model in
this table. The subseript scores are the deviation under three different rephrased instructions for each instruction category. Subscript scores are the deviation under three different rephrased instructions for each sample. GENRE is an closed-workd generative IE baseline without the instruction- follooiio ability, so we only report GENRE scores with the default instruction.

---

GENRE
ChatGPT
ChatGPT w/Demo
PIVOINE-Ib
PIVOINE-7b
Instruction
FI(T=I) FI(T=0.8) FI(T=I) FI(T=0.8)
FI(==1) FI(R=0.8)
FI(R=1) FI(R=088)
FI(R=1) FI(R=0.8)
Default
17.2
20.1
27.4a.1
28.3a.
39.60.o
41.0a.0
50.3o.0
51.3n.o
69.4o.1
70.1o.0
Base Type
27.4a.4
28.80.4
40.8p.4
42.3a.4
50,4o.0
51.4n.1
68.5o.1
69.3p.1
Abstract Type
19.3o.5
20.46.6
37.10.3
38.60.3
17.60.2
48.4p.2
68.70.2
69.2p.2
Description
24.2a.3
24.96.1
38.50.0
39.40.0
51.20.2
52.1n.2
76.40.1
77.50.1
Importance
26.70.4
28.00.4
36.80.1
38.2p.0
48.70.2
49.80.3
66.2o.1
67.2p.1
Macro Avg.
25.00.2
26.1o.2
38.60.0
39.9g.o
49.6p.0
50.66.0
69.8n.
70.7o.1
Table 12: Main results of end-to-end entity linking (EL.). We report FI score calculated by title matching with ROUGE-L Fl threshold 1 and 0.8 in this table. Subscript scores are the deviation under three different rephrased instructions for each sample. GENRE an closed- world generative IE baseline without the instraction-folloiing ability, only report GENRE scores with the default instruction.
Partition
Instruction
GENRE
ChatGPT
ChatGPT w/Demo
PIVOINE-I
PIVOINE-7b
Default
55.3
56.5p.2
50.1o.2
56.80.o
77.8p.0
Basc Type
66.11.2
65.
59.4
86.5p.1
Before
Abstract Type
63.6p.7
56.7n.
56.4a.
82.9p.1
05/30/2022
Description
60.9p.3
61.1o.2
62.8o.s
90.4g.1
Importance
4
49.0n.1
60.7o.s
80.7o.1
Macro Avg.
59.3p.3
56.5u.o
59.2o.1
83.7g.v
Default Base Type
33.6
60.5p.1
57.40.2
58.70.0
71.8p.0
66.71. 1
69.0g.0
57.10.2
70.4p.4
After 05/30/2022
Abstract Type Description
60.6p.s
57.8p.0
52.50.4
70.9p.5
61. 11.5..
60.5u.2
56.5o.2
84.1p.2
Importance
45.3p.s
46.4o.1
50.9o.5
61 61.6p.a
Macro Avg.
58.9b.2
58.20.0
55.1o.a
71.70.1
Table 13: Results of generalization study mention detection (MD). We report recalls of mention spans in this table. The partition "Before 05/30/2022" denotes mentions linked to the training ontology (seen entities), while "After 05/30/2022" denotes mentions linked entities that are out training ontology (unseen entities). Subscript scores the deviation under three different rephrased instructions for each sample. GENRE is closed-woo world generative IE baseline without the instruction-following ability, so we only report GENRE scores with the default instruction.
GENRE
ChatGT
ChatGPT w/Demo R(T=0.8)
PIVOINE-1 R(T=1)
PIVOINE-7b
Partition
Instruction
R(==)) R(T=0.8) R(T=1) R(T=0.8)
R(T=1)
R(T=0.8)
R(T=1) R(T=0.8)
Default
23.8
24.7
29.6o.1
30.2a.1
40.2p.1
41.0o.1
48.5p.o
49.0o.o
73.0u.n
73.3c.o
Base Type
39.3o.5
39.7a.4
50.4n.9
51.0p.a
52.9p.o
53.0p.o
81.6a.1
81.8p.1
Before
Abstract Type
29.5o.9
29.90.9
17.8n.2
48.30.2
49.5o.1
49.8o.1
78.0a.a
78.1
05/30/2022
Description Importance
30.9o.7
31.2a.7
47.7o.1
48.3o.1
55.5p.a
55.8o.s
85.3a.2
85.76.2
29.8p.6
30.4g.8
39.1p.2
39.9o.1
52.7o.s
53.3o.5
76.30.0
76.66.0
Macro Avg.
31.8p.2
32.3a.2
45.0o.1
45.7o.1
51.80.o
52.2n.0
78.8o.1
79.1..1
Default Base Type
3.6
24.7o.1
27.1
38.7p.2
41.80.2
35.4p.0
37.8o.o
48.3a.1
50.4o.1
29.3p.9
33.11.1
46.5o.1
50.0p.2
39.2p.1
41.4n.1
52.50.8
541.6p.7
After 05/30/2022
Abstract Type
18.9.0
21.91.a
36.8u.2
40.30.3
38.30
39.70.4
55.5a.a
56.80.4
Description
24.80.9
27.00.a
40.7o.o
42.90.0
36.4p.2
38.30.2
57.8o.1
60.3o.1
Importance
21.30.1
23.60.3
32.8u.4
35.40.3
32.3p
34.30.3
44.5u.a
46.88
Macro Avg.
23.8p.2
26.50.3
39.1n.o
42.1o.a
36.3p.1
38.30.1
51.7a.2
53.86.2
Table 14: Results of generalization study on end-to-end entity linking (EL). We report recalls calculated by title matching with ROUGE-L Fl threshold and 0.8 in this table. The partition "Before 05/30/2022" denotes mentions linked to the training ontology (seen entities), while "After 05/30/2022" denotes mentions linked to entities that are out of training ontology (unseen entities). Subscript instructions for each sample. GENRE is closed-word world generative IE baseline without the instruction- followiin ability, so we only report GENRE scores with the default instruction.
the deviation under three different rephrased

---

Open1E6
ChatGPT
ChatGPT w/Demo
PIVOINE-Ih
PIVOINE-Th
Instruction
P
P
Fl
R
Default
13.0
18.5 15.2 25.2115
17.8 36.9o.1 23.7p.a a
16.40.
37.80.2 22.0o.o 54.
55.8..0
54.06.o
65.70.1 68. 68. 66.30.3
Base Type
1o. 34.61.3 21.70.5
15.2o.1
43.00.3 22.0o.o o 56.6o.7 61.6o.s
58.3o.s
60.31 . 75.70.4 66. 70.9
Abstract Type
13.70.s 37.80.3 22.0g.s
14.6u.a
46.44.
56. 21.50.s
54.5u.1 55.0u.a
66.44
71.50.1 68.30.s
Deseription
16.6o.1 35.8a.a 22 22.0..o
12.0a.2
35.2a.s 17 17.50.s 52.3a.c
63.8p.c c 57. 0o.o
69 30.4 77. 7o. 73.0o.a
Importance
2 35.80.4 23.Tu.s
17.20
39.6u.1 23.5c.s 5-1.9u.3 57 37.9..4 55. 7u.s
62. 4u.1 68.86.2 65. 0o.d
Macro Avg.
166.9.2 36.2g.2 22.6o.3 15. 40.4g.2 2.. 21 21.3p. 54.99.4 58. 56.00.3 641.8.3 72.4p.o 67.90.2
Table 15: Main results of open relation extraction (Open RE). We report precision, recall, and Fl with the CaRB scoring based on the ROUGE-L matcher. Subscript scores are the deviation under three different rephrased instructions for each sample. OPENIE6 am openlE baseline without the instructionfolllwee ability, so we only report Open1156 scores with the default instruction.
Partition
Instruction
Open1E6 ChaiGPT
ChatGPT w/ Demo
PIVOINE-Ih
PIVOINE-7h
Default
21.9
35.1a.1
35.90.1
57.70.o
73.3p.o
Base Type
32.11.7
40.4o.8
63.6p.s
83.70.5
Before
Abstract Type
36.00.s
43.70.6
58.8o.1
78.4..1
05/30/2022
Description Importance
33.3u,s
33.0o.7
67.90.6
84.1u.1
34.2g.5
38.50.2
61.0o.s
77.40.2
Macro Avg.
34. la.2
38.30.0
61.80.3
79.40.1
Default
18.2
32.7u.2
34.0p.2
35.80.1
36.1o.1
Base Type
23.66 60.8
27.80.6
39.50.2
37.90.2
After 05/30/2022
Abstract Type Description
28.01.1
33.51.3
32.6p.v
36.40.4
24.8 24.8a.2
22.40.5
30.3o.4
35.30.2
Importance
26.2a.7
29.0o.6
34.1o.s
34.6u.1
Macro Avg.
27.1a.5
29.3o.5
34.50.1
36.1o.1
Table 16: Results generalization study on open relation extraction (Open RE). We report recall with the CaRB scoring based the ROUGE-L matcher. The partition "Before 05/30/2022" denotes mentions linked the training ontology (seen entities), while "After 05/30/2022" denotes mentions linked to entities that are out of training ontology (unseen entities). Subscript scores are the deviation under three different rephrased instructions for each sample. OPENIE6 is an openlE baseline without the inslloction-followingg ability, so we only report Openllé6 scores with the default instruction.
CharGPT
ChatGPT w/ Demo
PIVOINE-Ib
PIVOINE-7b R
Instruction
P
R
F1
P
R
F1
R
Fl
F1
Default
2.7o.0
3.80.a
3.10.
6.40.0
7.50.0
6.90.0
46.3o.1 37.9u.1
41.7a.1 61.30.0 55.30. 58.1
Base Type
13.0o.3
19.40.s
15.60.4
29.90.2
35.0p.4 32. 32.3p.a 46.0p.1 39.9a.2
42.7a.1 59.3a.2 57.60.1
58.4
Abstract Type
8.90.5
7.50.2
8,20.3
16.2o.s
18.2p.2 17.1o.3 42.To.2 35.60.o 38.8a.1 62.3a.3 53.9o.1 57.
Description
2.3a.1
4.40.1
3.1o.1
6.40.1 6.5o.1 6.40.1 44.90.1 43.8a.4 44. 64.1a.1
63.2 63.6o.1
Importance
2.6o.2
3.80.3
3.1o.2 5.90.0 7.20.1 6.5g.0 39.50.2 39.5a.3 39.5g.1 50.6g.o
53.4o.1 52.0o.o
Macro Avg. 7.50.1
8.80.a
7.90.a
14.4p.o 15.4p.2 14.8p.1 42 4n.o 38.7a.1 40.4a. 57.7a.o 55.3o.1 56.40.o
Table 17: Main results of fine-gre fine-grained entity typing (ET). We report precision, recall, and scores based on exact matching of type names in this table. Subscript scores the deviation under three different rephrased instructions for each sample.

---

Partition
Instruction
|ChaiGPT ChatGPT w/Demo PIVOINE-Ib PIVOINE-7b
Default
4.00.0
7.80.1
41.90.1
63.40.0
Base Type
21.1a.4
34.4u.7
46.2u.2
71.30.1
Before
Abstract Type
8.4a.2
19.20.1
40.0a.a
65.6p.3
05/30/2022
Description
5.0u.1
7.6u.1
50.6u.s
73.30.1
Importance
4.2g.5
7.60.3
47.10.3
65.5p.1
Macro Avg.
9.Ta.1
15.96.2
44.4o.1
66.8p.1
Default
2.6..0
5.90.3
17.70.1
13.80.0
Base Type
16.1a.s
36.20.0
27.2o.3
30.2p.o
After
Abstract Type
5.80.5
16.30.3
27.40.1
32.2p.s
05/30/2022
Description
2.T0.4
3.10.2
21.00.2
29.1o.1
Importance
2.70.4
6.00.9
12.7o.1
10.70.2
Macro Avg.
6.96.1
14.2o.1
22.80.0
21.1o.2
Table 18: Results of generalization study on entity typing (ET). We report recalls of entity types based on the exact matching of type names. The partition "Before 05/30/2022" denotes mentions linked to the training ontology (seen entities), while "After 05/30/2022" denotes mentions linked to entities that out of training ontology (unseen entities). Subscript scores are the deviation under three different rephrased instructions for each sample.
Partition
Instruction
ChatGPT ChatGPT w/Demo PIVOINE- PIVOINE-7b
Default
36.2p.2
46.3p.1
74.3o.0
83.40.
Base Type
38.9p.6
48.7u.2
67.4g.1
79.5u.1
Before
Abstract Type
40.61.5
56.21.4
67.50.4
81.30.1
05/30/2022
Deseription
57.8p.s
85.9p.6
75.8u.a
86.8u.1
Importance
38. 2p.6
46.5p.1
75.9a.2
84.60.1
Macro Avg.
40.2p.4
53.6u.2
71.4a.1
82.50.9
Default
36.80.3
45.00.2
51.6u.1
60.
Base Type
37.9p.9
44.61.1
54.7a.4
60.0o.a
After 05/30/2022
Abstract Type
28.To.3
44.1o.s
51.30.4
58.80.0
Description
47.7a.7
75.96.c
81.5a.a
93.1o.2
Importance
36.01.1
44.1o.s
45.00.s
57.90.3
Macro Avg.
36.0p.3
48.1p.2
55.7o.3
64.10.2
Table 19: Results of generalization study on entity description generation. We report average ROUGE-L Fl scores in this table. The partition "Before 05/30/2022" denotes mentions linked the training ontology (seen entities), while "After 05/30/2022" denotes mentions linked to entities that are out of training ontology (unseen entities). Subscript scores the deviation under three different rephrased instructions for each sample.
Partition
Instruction
ChaiGPT ChatGPT w/Demo PIVOINE-Ib PIVOINE-7b
Default
8.50.
1.loo
69.8a.
78.00.o
Base Type
9.10.4
7.702
71.3a.1
76.7a.1
Before
Abstract Type
10.1o.4
14.8p.2
67.3a.
78.7a.2
05/30/2022
Description
10.1g.
19.4u.s
65.6u.1
76.2u.1
Importance
10.1o.6
1.1o.3
75.1a.
81.60.1
Macro Avg.
10.1a1
8.40.0
69.7o.1
78.0o.a
Default
10.9p.6
1.80.1
16.9g.1
21-1aa
Base Type
14.61.a
11.91.8
22.7a.a
24.5a.a
After 05/30/2022
Abstract Type
17.12s
21.113
23.6u.1
27.5u.3
Description
15.71.6
37.41.2
18.90.1
17.90.a
Importance
14.01.4
1.8n.a
24.5a.2
28.Ta.2
Macro Avg.
14.3a.a
13.30.1
20.5a.2
22.7a.1
Table 20: Results of generalization study entity aliases generation. We report recalls of entity aliases based
exact matching in this table. The partition "Before 05/30/2022" denotes mentions linked to the training ontology (seen entities), while "After 05/30/2022" denotes mentions linked entities that are out training ontology (unseen entities). Subscript scores are the deviation under three different rephrased instructions for each sample.

---

ChatGPT
ChatGPT w/Demo
PIVOINE- lb (ET=I))((T-O)
PIVOINE-7h
Instruction
MD
IE(T=1) IE(T=0.8)
MD
IE(T=))
MD
MD
IB(T=I) (E(T=088)
Number
51.4
24.80.s
25.90.7
49.0p.1
37.50.2
39.1u.2
69.9..1
55.90.1
57.1
81.30.1
71.30.1
22.20.1
Number+
53.1o.s
25.80.3
27.1
50.3p.o
38.4o.
40.1n.o
69.4..2
55.20.2
56.90.2
80.8p.
71.2p.1 72.2o.1
Base Type
Number+
53.3
25.00.2
26.2o.1
51.1o.1
38.0p.
39.5p.1
72.40.3
57.46.2
59.00.2
82.0p.
73.6p.2
74.3p.2
Abstract Type
Macro Avg.
52.6u.3 25.2v.1
26.40.2
[50.lp.v
38 38.0v.1 39.6p.1
70. 56.2g.u 57.7a.a
[81.4..1
72.0u.1 72.90.1
Table 21: Results of partial extraction instructions on mention detection (MD) and information extraction (IE). We report precision for mention spans in MD and title matching with ROUGE-L Fl threshold 0.8 and in IE. The partition "Before 05/30/2022" denotes mentions linked to the training ontology (seen entities), while "After 05/30/2022" denotes mentions linked to entitics that are out of training ontology (unseen entities). Subscript scores are the deviation under three different rephrased instructions for each sample.
Category
Prompt
Default Base Type
Context: "Extract entities." ,n Please rephrase this context.
Context: "Extract entities in types (types]. mm (types) in the context a placeholder for a list of entity types. Please rephrase this context and keep [types] in the rephrased sentence. [types] should be put after the word "types""
Abstract Type
Context: "Extract entities in types (types J." "mnn (types] J in the context a placeholder for a list of entity types, Please rephrase this context and keep [types] in the rephrased sentence. (types) should be put after the word "types""
Description
Context: "Extract entities in following descriptions: [descriptions] (',n [descriptions) in the context is placeholder for a list of entity descriptions. Please rephrase this context and keep [descriptions) in the rephrased sentenec.
Importance
Context: "Extract the most importunt [number) entities." [number] in the context is placeholder for the number of entity. Please rephrase this context and keep [number] in the rephrased sentence. Context: "Extract (number) entities." "i n'nn [oumber) in the context is a placeholder for the number of entity. Please rephrase this context and keep (numher) in the rephrased sentence.
Number
Number+Base Type
Context: "Extract [number] entities in types (types)." [yppe))."nn) n'nn (types) in the context placeholder for list of entity types. (number) in the context is placeholder for the number of entities. Please rephrase this context and keep (types) and (number) in the rephrased sentence. [types] should be put after the word "types""
Number+Abstrrat Type Context: "Extract [numher] entities in types (types J. ''nn [types] the context a placeholder for list of entity types. (number) in the context is placeholder for the number of entities. Please rephrase this context and keep (types) and [number] the rephrased sentence. [types] should be put after the word "types""
Table 22: Details ChatGPT prompts we used to rephrase manually designed templates.
Type
Prompt
ChatGPT [context]: [context). \ infinstruction):: (instruction). An uPlease provide the response in the JSON format. The response should contains entities and triplets. Each entity has mention. title, list of types, description. and list of aliases. Each triplet has its head and tail mentions, and list of relations. Here example of the return JSON format: ("entities": : [[ ("mention": String, "title:: String. "type". List[String), "description" String, "aliases": [isti(r "triplets": (t"head": String. "tail": String, "relations": Stringl)))
Table 23: Details ChatGPT prompts we used to address instruction-folloiin open-world IE, (context), (instruc- tion] are placeholders for input context, instruction, and one-shot example, respectively.


Reviews:
[{"review_id": "dYJ3XvaFwu", "reviewer": "Reviewer_CAmu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback.\nWhile we initially named our task after OpenIE[1], a well-known research topic that extracts mentions and their relations in an open setting, we understand the reviewer's concern that the term IE has evolved to encompass a broader range of tasks. Therefore, we will modify our task as the entity profiling, representing the extraction of entity-related information. Similarly, we agree the open-world may be a misleading umbrella word though our initial purpose is to use it to distinguish our works with traditional closed-world IE. So we will modify the title as “Open-setting Entity Profiling” to make it more precise. Nevertheless, our work still has substantial contributions for exploring an open-set entity-related information extraction that can generalize to various semantics of instructions. \n\nAnd we also disagree with the reviewer’s claim that many existing works on open information extraction can achieve this goal. Previous works in OpenIE cannot handle instructions as varied as in our work. Recent works such as InstructUIE[2] also try to extend OpenIE to  handle user requests in multiple IE tasks but they are concurrent works of PIVOINE. We would sincerely appreciate it if the reviewer can provide further related works for reference.\nWe fully understand the reviewer’s concern about data leakage in LLM pretraining, so we have provided analysis on this issue in our manuscript. We consider the potential leakage may come from corpus or entities. First, we show the pretraining corpus used in BLOOM is before 06/20/2022 in the footnote in Page 5, while our evaluation set only contains corpus after 06/20/2022, which can be considered independent with the pretraining corpus. Second, we only consider extracting new entities after 06/20/2022 to minimize the risk of data leakage.\nWe agree comparing experiments with aligned LLMs such as Alpaca will further elaborate the effectiveness of our methods. We add results of the latest Vicuna-7b-v1.5 [3], which is an open-source instruction-tuning model trained on over 150k dialogues based on the foundation model LLaMa2.\nHere we present the main results of Vicuna-7b-v1.5, ChatGPT, and PIVOINE-7. Below is an extension of Table 1 in our manuscript.\n\n| Method | MD | EL(T=1) | EL(T=0.8) | ET | OpenRE | EIG(Desc.) | EIG(Alias) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Vicuna-7b-v1.5 | $30.6_{0.0}$ | $16.5_{0.0}$ | $17.2_{0.0}$ | $3.4_{0.0}$ | $20.0_{0.0}$ | $28.4_{0.0}$ | $10.8_{0.0}$ |\n| CHATGPT W/DEMO  | $51.1_{0.1}$ | $38.6_{0.0}$ | $39.9_{0.0}$ | $14.8_{0.0}$ | $21.4_{0.0}$ | $52.0_{0.0}$ | $13.0_{0.1}$ |\n| PIVOINE-7B  | $79.6_{0.0}$ | $69.8_{0.0}$ | $70.7_{0.0}$ | $56.4_{0.0}$ | $67.8_{0.0}$ | $80.2_{0.0}$ | $80.5_{0.0}$ |\n\nBelow is an extension of Table 2 in our manuscript.\n\n| Partition | Method | MD | EL(T=1) | EL(T=0.8 ) | ET | OpenRE | EIG(Desc.) | EIG(Aliases) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|Before 05/30/2022 | Vicuna-7b-v1.5 | $26.5_{0.0}$ | $14.9_{0.0}$ | $15.1_{0.0}$ | $3.3_{0.0}$ | $36.4_{0.0}$ | $29.4_{0.0}$ | $8.8_{0.0}$ |\n|After 05/30/2022 | Vicuna-7b-v1.5 | $26.7_{0.0}$ | $13.0_{0.0}$ | $14.3_{0.0}$ | $3.1_{0.0}$ | $25.5_{0.0}$ | $25.8_{0.0}$ | $10.5_{0.0}$ |\n\nResults of Vicuna-7b-v1.5 show PIVOINE can outperform existing powerful open-source instruction-tuning LLMs, which provides further evidence of effectiveness of our method. We will add more comprehensive comparison experiments, such as involving Alpaca, in the revised version of the manuscript.\n\n\n[1] Niklaus, Christina, et al. \"A survey on open information extraction.\" arXiv preprint arXiv:1806.05599 (2018).\n[2] Wang, Xiao, et al. \"InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction.\" arXiv preprint arXiv:2304.08085 (2023).\n[3] Zheng, Lianmin, et al. \"Judging LLM-as-a-judge with MT-Bench and Chatbot Arena.\" arXiv preprint arXiv:2306.05685 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) Good Writing: The paper is well-written, presenting the concepts and the ideas in a clear and concise manner, making it accessible to a broad readership.\n\n(2) New Problem: The authors have targeted an relatively unexplored area, considering user instructions for open IE. This novel focus has the potential to offers new opportunities for research.\n\n(3) New Dataset: The constructed dataset is a substantial contribution, providing a comprehensive corpus and diverse instructions specifically tailored for the proposed task. This dataset can serve as a valuable resource for future research.\n", "reasons_to_reject": "(1) Overclaimed task setting: this paper claims the task as open world information extraction, but actually it only focuses on the entity-related tasks. It cannot even handle some existing well-defined ie tasks (like slot filling, coreference resolution, and table extraction), let alone solve more open-scenario ie problems. Thus, it’s not convincing to claim the proposed task as open-world. \n\nAdditionally, this paper claims that the novelty of open-world IE is considering user instructions and going beyond a predefined ontology of entities and relations. However,  many existing work on open information extraction can already achieve this goal. Thus,  this point is kind of overclaimed. \n\n(2) Data leakage problem: Since the dataset is collected from Wikipedia, there might be a risk of data leakage if the large language models have already been trained on the test data. This could lead to overly optimistic performance metrics and hinder the generalization of the model to truly unseen data.\n\n(3) Missing comparing experiment: this paper targets on instruction tuning for IE. There are existing powerful open-source instruction tuning models like Alpaca [1]. Without comparing with these models, it would not be persuasive enough about the effectiveness of the proposed method. \n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper tackles the new problem of entity-centric Information Extraction, which goes beyond the conventional open IE tasks by considering user instructions. To handle this task, this paper creates a substantial instruction tuning dataset and s fine-tunes a large language model named PIVOINE on this dataset. The results demonstrate PIVOINE's superiority over traditional methods and ChatGPT-based baselines in handling unseen instructions and out-of-ontology cases.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fikY0q4A42", "reviewer": "Reviewer_FP8C", "summary": "", "questions": "Please refer to Reasons To Reject.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful suggestions.\n\n1. Data leakage issues are one of the main challenges in our work so we design a very delicate evaluation set to minimize the risk of data leakage. First, as we discussed in Section 4, the pretraining corpus used by BLOOM only contains Wikipedia articles before 06/20/2022 and the same of our training set, while our evaluation set only contains articles after 06/20/2022. With such a method we minimize the data leakage from the pretrain corpus and training set. Furthermore, we only evaluate our method on those new entities incorporated into Wikidata after 06/20/2022, which were rarely involved in any Wikipedia corpus before 06/20/2022. Building an evaluation set with time difference is considered as a promising way to evaluate LLMs in even more general purpose [1].\n\n2. Our contributions are not limited to training an LLM for entity profiling but also exploring and providing a comprehensive model, dataset, and evaluation recipe of a more open-setting information extraction task that is not well-studied in previous IE research. Therefore, we believe this work would provide further impact to the community based on these substantial contributions.\n\n3. We totally understand the reviewer's requests about evaluating more open-source LLMs on our task though LLaMa and ChatGLM are concurrent works of PIVOINE. We consider extending our method from BLOOM to more open-source LLMs as one of our valuable future works. Furthermore, we add results of the latest Vicuna-7b-v1.5 [3], which is an open-source instruction-tuning model trained on over 150k dialogues based on the foundation model LLaMa2.\nHere we present the main results of Vicuna-7b-v1.5, ChatGPT, and PIVOINE-7. Below is an extension of Table 1 in our manuscript.\n\n| Method | MD | EL(T=1) | EL(T=0.8) | ET | OpenRE | EIG(Desc.) | EIG(Alias) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| Vicuna-7b-v1.5 | $30.6_{0.0}$ | $16.5_{0.0}$ | $17.2_{0.0}$ | $3.4_{0.0}$ | $20.0_{0.0}$ | $28.4_{0.0}$ | $10.8_{0.0}$ |\n| CHATGPT W/DEMO  | $51.1_{0.1}$ | $38.6_{0.0}$ | $39.9_{0.0}$ | $14.8_{0.0}$ | $21.4_{0.0}$ | $52.0_{0.0}$ | $13.0_{0.1}$ |\n| PIVOINE-7B  | $79.6_{0.0}$ | $69.8_{0.0}$ | $70.7_{0.0}$ | $56.4_{0.0}$ | $67.8_{0.0}$ | $80.2_{0.0}$ | $80.5_{0.0}$ |\n\nBelow is an extension of Table 2 in our manuscript.\n\n| Partition | Method | MD | EL(T=1) | EL(T=0.8 ) | ET | OpenRE | EIG(Desc.) | EIG(Aliases) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n|Before 05/30/2022 | Vicuna-7b-v1.5 | $26.5_{0.0}$ | $14.9_{0.0}$ | $15.1_{0.0}$ | $3.3_{0.0}$ | $36.4_{0.0}$ | $29.4_{0.0}$ | $8.8_{0.0}$ |\n|After 05/30/2022 | Vicuna-7b-v1.5 | $26.7_{0.0}$ | $13.0_{0.0}$ | $14.3_{0.0}$ | $3.1_{0.0}$ | $25.5_{0.0}$ | $25.8_{0.0}$ | $10.5_{0.0}$ |\n\nResults of Vicuna-7b-v1.5 show PIVOINE can outperform existing powerful open-source instruction-tuning LLMs, which provides further evidence of effectiveness of our method. We will add more comprehensive comparison experiments, such as involving Alpaca, in the revised version of the manuscript.\n\n\n\n[1] Jang, Joel, et al. \"TemporalWiki: A lifelong benchmark for training and evaluating ever-evolving language models.\" arXiv preprint arXiv:2204.14211 (2022).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper emerges as a promising solution to tackle the open-world challenge in IE. \n2. The paper constructs a dataset for Open-world IE enriched with a comprehensive corpus, extensive annotations, and diverse instructions. \n3. Experimental setting is very comprehensive and insightful.\n4. The paper is well-organized and clearly written.\n", "reasons_to_reject": "Though the paper is well-motivated and conducts comprehensive experiments to evaluate the proposed model, I would like to provide some suggestions as follows:\n1. I would like to know how to prove that the proposed method does not have data leakage issues. Maybe the INSTRUCTOPENWIKI has already seen those unknown entities, which may obviously increase the performance.\n2. The method is relatively simple and mainly based on feature engineering, which may limit the interest in research community.\n3. More open-source large models such as Lama, ChatGlm could also be considered.\n", "questions_to_authors": "Please refer to Reasons To Reject.", "paper_topic_and_main_contributions": "This work considers the problem of Open-world Information Extraction (Open-world IE) and reformulates open-world IE into auto-regressive generation by linearizing the structure triples into the JSON format. Construct a substantial instruction tuning dataset for this task and finetune pre-trained BLOOM model to obtain PIVOINE. The experiments demonstrate that PIVOINE significantly outperforms previous methods, displaying impressive generalization capabilities on both unseen instructions and out-of-ontology cases. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "NA"}, "extra_text": {}}, {"review_id": "M2QTMKFltC", "reviewer": "Reviewer_YbHk", "summary": "", "questions": "A. How are the eight popular categories of instructions selected?  \n\nB. Can you clarify what is \"entity priorities in Wikipedia\"? Page views?\n\nC. Is unseen ontology equivalent to an unseen entity? For example, 2023 ATP Tour was referred to as an unseen entity, but related entities such as the 2022 APT Tour could be in the training set. 2022 APT Tour would be an entity of the same type in the ontology.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's insightful suggestions.\n\nWhile we initially named our task after OpenIE[1], a well-known research topic that extracts mentions and their relations in an open setting, we understand the reviewer's concern that the term IE has evolved to encompass a broader range of tasks. Therefore, we will modify our task as the entity profiling, representing the extraction of entity-related information. Similarly, we agree the open-world may be a misleading umbrella word though our initial purpose is to use it to distinguish our works with traditional closed-world IE. So we will modify the title as “Open-setting Entity Profiling” to make it more precise. Nevertheless, our work still has substantial contributions for exploring an open-set entity-related information extraction that can generalize to various semantics of instructions. \n\nWe appreciate the suggestions for further exploration. Our contributions are not limited to presenting a fine-tuned task-focused LM that can outperform general proprietary chabot in zeroshot setting (which is also questionable whether ChatGPT is indeed zero-shot in our task). We explore and provide a comprehensive model, dataset, and evaluation recipe of a more open-setting information extraction task that is not well-studied in previous IE research. We consider ChatGPT as a strong baseline as there is no previous IE works that are as open as our work in entity profiling.\n\nResponse to Questions:\n\n(A) We build the eight instructions based on the targeted attributions and different levels of “open-domain” in real-world requests. For example, descriptions are the most open-domain and fine-grained category while the base type can be considered as the most coarse grained request.\n\n(B) Entity priorities are calculated based on the in-link and out-link on Wikidata to reveal the popularity of entities. The detailed metric is described in Section 4.1 in [1].\n \n(C) Yes, the unseen ontology mainly refers to unseen entities in our context.\n\n[1] Andrew Chisholm and Ben Hachey. 2015. Entity Disambiguation with Web Links. Transactions of the Association for Computational Linguistics, 3:145–156.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Instruction-tuning for IE is relatively understudied compared to other NLP tasks such as QA or summarization. \n- The weakly-supervised dataset InstructOpenWiki could be a useful resource to help train models for entity-related tasks.\n- Error analysis of different failure cases is interesting and helps break down the challenges in this task.", "reasons_to_reject": "- The new task name is misleading and it is questionable to what degree is the task \"open-domain\". \n\nThe paper reads \"we introduce Open-world Information Extraction (Open-world IE) to accommodate broad and diverse requests related to entity profiles surpassing predefined ontologies’ limits.\"  In Table 1, the example shows that the task is essentially entity linking, typing and relation extraction for the target entity. The task could simply be introduced as \"entity profiling\". Using the umbrella term \"Open-world IE\" is misleading since the task does not involve events or coreference, which are also important tasks under IE. \n\nOne of my main concerns is whether the task is truly \"open-domain\".  The definition of entity linking (to link a mention to an entry in a knowledge base) means that it is closed domain in nature. The only difference between GENRE and PIVOINE is that they link to different snapshots of Wikipedia. Both are closed-domain.  For entity typing and relation extraction, note that Wikidata is closed-domain and the dataset is built through linking to Wikidata. While Wikidata has a lot of entity types  (20K as reported for InstructOpenWiki), this is on the same scale as UFET [1]. Moreover, if the entity types were truly open-domain, then F1 would not be an appropriate evaluation metric as it fails to capture similarities between types such as \"singer\", \"musical artist\", and \"musician\". \nFor relations, there are only several hundred relations in the dataset. The only open-domain component seems to be the entity description. \n\n[1] Ultra-Fine Entity Typing (Choi et al., ACL 2018) \n\n-  The findings of the paper are not insightful. \nExperiments show that fine-tuning an LLM for this instruction-guided IE task works better than non-LLM which has no instruction-following ability and ChatGPT out of the box. This conclusion seems quite obvious. (Before the LLM era, if someone said that their supervised system worked better than their zero-shot / one-shot system it would not be a meaningful finding.)  \nI would encourage the authors to go one step further and investigate aspects like \"how much data is needed\" and \"why this data works\". The analysis in Section 5.3  already provides some leads for investigation, for example, if failing to produce a valid JSON output is a problem, then can we solve this part directly?", "questions_to_authors": "A. How are the eight popular categories of instructions selected?  \n\nB. Can you clarify what is \"entity priorities in Wikipedia\"? Page views?\n\nC. Is unseen ontology equivalent to an unseen entity? For example, 2023 ATP Tour was referred to as an unseen entity, but related entities such as the 2022 APT Tour could be in the training set. 2022 APT Tour would be an entity of the same type in the ontology.", "paper_topic_and_main_contributions": "This paper proposes a new task \"open-world information extraction\" and constructs a dataset InstructOpenWiki for instruction-tuning models to solve this task. Although the task name seemingly includes IE as a whole, it is essentially extracting entity profiles.\nThey fine-tune an open-source LLM BLOOM on InstructOpenWiki and show that it outperforms non-LLM baselines and vanilla ChatGPT on the task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Citation for GENRE paper is wrong. Should be De Cao, Nicola, Gautier Izacard, Sebastian Riedel and Fabio Petroni. “Autoregressive Entity Retrieval.” (ICLR 2021)"}, "extra_text": {}}, {"review_id": "P1tzzK2YUy", "reviewer": "Reviewer_VgsG", "summary": "", "questions": "A. How is the evaluation of cross-instructions conducted? The paper shows that the Number + Base Type and Number + Abstract Type instructions have high error rates (Fig 4). Could you clarify what is considered wrong in these cases? Is it the extracted numbers or the types?\n\nB. Given the relation extraction data are created by aligning Wikipedia with Wikidata with distant supervision, some concerns about the dataset were raised. In prior work, Gao et al., (2021) found that distant supervision often brings a lot of noise and causes wrong relations labels. In addition, a pair of head and tail entities might also be related with multiple relations. Can you elaborate on how the relation is being selected and how the dataset's quality is being ensured?\n\nGao et al., (2021) : Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction \n\nC: The concept of generating entity profiles is interesting,  however, the evaluations are still conducted using single evaluation metrics, with no comprehensive evaluation of each entity. Is it possible to provide overall score for on entity level?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for great questions. We first would like to clarify the concern about unseen instructions. As we described in Line 400-407, unseen instructions are those instructions with (1) unseen entities and (2) unseen query semantics. However, the reviewer actually raises another definition of unseen instructions, which refers to unseen tasks such as cross instructions. We fully agree such scenarios are also a new challenge of generalization abilities, however, exploration of unseen instructions in our definition can also reveal model generalization abilities in terms of instruction semantics within each task. Therefore, we will add this discussion and provide clearer definition of unseen instructions.\n\nResponses to Questions:\n\n(A) We require both the correct number and type when evaluating in cross-instruction settings. We notice our models tend to ignore the number constraints and extract more entities in these tasks.\n\n(B) First, we use hyper-links in wiki to conduct entity linking in building our dataset, which is checked and frequently revised by contributors of wikipedia. Besides, we have considered multi-relation entity pairs in our dataset. And we train LLMs to generate a list of relations during prediction.\n\n(C) Although we split the profiling into six tasks, the evaluation metrics are calculated in kind of “entity-level” as the reviewer suggested. For example, a correct entity linking requests correctly detection of the corresponding entity mentioned first. We can also add more comprehensive entity-level overall scores in the revised version of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is the first work in Open-world IE that demonstrates the ability of large language models to be customized to generate complete entity profiles, as opposed to focusing on single tasks. This approach will facilitate future research in the direction of customized and complete OpenIE.\n\nThe authors introduce the INSTRUCTOPENWIKI dataset and the PIVOINE model, providing valuable resources for future research in this area. They conduct extensive experiments that offer a comprehensive evaluation of the model's performance, demonstrating its effectiveness in following instructions and extracting entity profiles.\n\nThe authors propose a delicate method that uses the time difference between Wiki dumps to construct a strictly open-world test set. This method minimizes potential data leakage and maximizes the completeness of the training set’s annotations.", "reasons_to_reject": "While the authors claim that the model can follow unseen instructions, the definition of unseen instructions seems to primarily focus on unseen entities, and the model fails to follow cross-instructions, which are also unseen. The extent of what is considered unseen is debatable. For example, the \"2023 ATP Tour\" example in Figure 1, although it doesn’t occur in the training data, “2022 ATP Tour” is included in the training data. This raises concerns that the model may simply mimic what’s encoded in the training data, making the comparison against models that are untrained (e.g., ChatGPT) unfair.", "questions_to_authors": "A. How is the evaluation of cross-instructions conducted? The paper shows that the Number + Base Type and Number + Abstract Type instructions have high error rates (Fig 4). Could you clarify what is considered wrong in these cases? Is it the extracted numbers or the types?\n\nB. Given the relation extraction data are created by aligning Wikipedia with Wikidata with distant supervision, some concerns about the dataset were raised. In prior work, Gao et al., (2021) found that distant supervision often brings a lot of noise and causes wrong relations labels. In addition, a pair of head and tail entities might also be related with multiple relations. Can you elaborate on how the relation is being selected and how the dataset's quality is being ensured?\n\nGao et al., (2021) : Manual Evaluation Matters: Reviewing Test Protocols of Distantly Supervised Relation Extraction \n\nC: The concept of generating entity profiles is interesting,  however, the evaluations are still conducted using single evaluation metrics, with no comprehensive evaluation of each entity. Is it possible to provide overall score for on entity level?", "paper_topic_and_main_contributions": "The paper addresses the challenge of Open World Information Extraction (IE) through instruction tuning, presenting a new concept of generating comprehensive entity profiles. Unlike previous work that focuses on specific tasks (e.g., NER, entity linking) within IE, this paper integrates six relevant tasks to create entity profiles. The primary contributions of the paper include the development of the INSTRUCTOPENWIKI, a large-scale instruction-following open-world IE dataset, and the creation of the PIVOINE model, an instruction tuning model based on BLOOM. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. In line 182, there appears to be a missing comma at the end.\n2. The readability of the paper can be improved by clearly indicating the location of figures and tables, especially when they are in the appendix. Many of the tables and figures are located in the appendix.\n3. It would be helpful to elaborate on why focusing on \"desired targets\" is important and clarify the meaning of \"desired targets\" (mentioned in lines 058-063) in the introduction."}}]

Author rebuttal:

Meta-review: This paper presents INSTRUCTOPENWIKI, a large-scale instruction-following open-world IE dataset, and the creation of the PIVOINE model, an instruction tuning model based on BLOOM. The reformulation of open-world IE into a linearized JSON format is a promising solution to the task, though the open-world naming is overclaimed and misleading. The main concerns are with data leakage, though this issue seem to be addressed by the authors in the rebuttal.
Decision: Accept-Findings