Instruct and Extract: Instruction Tuning for On-Demand Information Extraction
Authors: Yizhu Jiao
Ming Zhong
Sha Li
Ruining Zhao
Siru Ouyang
Heng Ji
Jiawei Han
Venue: EMNLP 2023 Main (2023)
Keywords: information extraction
instruction-tuning
language model
open scenario
TL;DR: We propose a novel paradigm called On-Demand Information Extraction and the corresponding dataset, InstructIE, that enables personalized extraction of desired content from text.

Abstract: Large language models with instruction-following capabilities open the door to a wider group of users. However, when it comes to information extraction – a classic task in natural language processing – most task-specific systems cannot align well with long-tail ad hoc extraction use cases for non-expert users. To address this, we propose a novel paradigm, termed On-Demand Information Extraction, to fulfill the personalized demands of real-world users. Our task aims to follow the instructions to extract the desired content from the associated text and present it in a structured tabular format. The table headers can either be user-specified or inferred contextually by the model. To facilitate research in this emerging area, we present a benchmark named InstructIE, inclusive of both automatically generated training data, as well as the human-annotated test set. Building on InstructIE, we further develop an On-Demand Information Extractor, ODIE. Comprehensive evaluations on our benchmark reveal that ODIE substantially outperforms the existing open-source models of similar size.

Instruct and Extract: Instruction Tuning for On-Demand Information Extraction
Yizhu Jiao, Ming Zhong, Sha Li, Ruining Zhao, Siru Ouyang, Heng Ji, Jiawei Han University of Illinois Urbana-Champaige yizhuj2@i www.eiininoieeeee
Abstract
Instruction: T personally prefer eating fruits. Extract some key features of the fruits.
Large language models with instruction- following capabilities open the door a wider group of users. However, when it comes to information extraction - a classic task natu- ral language processing most task-specific systems cannot align well with long-tail ad hoc extraction use cases for non-expert users. To address this, propose novel paradigm, termed On- Demand Information Extraction, to fulfill the personalized demands of users. Our task aims to follow the instructions to extract the desired content from the associ- ated text and present in a structured tabular format. The table headers can either be user- specified or inferred contextually by the model. To facilitate research in this emerging area, present benchmark named INSTRUCTIE, in- clusive of both automatically generated train- ing data, well as the aunan-annotated test set. Building on INSTRUCTIE, we further de- velop an On-Demand Information Extractor, ODIE. Comprehensive evaluations on our benchmark reveal that ODIE substantially out- performs the existing open-source models of similar size. Our code and dataset are released on http:///tttuuu.omyyjjooo//// ----nmndd-----
Text:
Strawberries are popular fruit known for their vibrant red color and sweet, juicy flavor.
One of the most widely consumed fruits, apples come in various colors, including red...
Bananas ure elonguted, slightly curved fruits that have thick, protective peel and soft, sweet flesh
Extracted Table:
Fruit
Shape
Taste
Nutrients
Heart- shaped
Sweet, juicy
Vitamin C,
Strawberries
antioxidants
Crisp,
Dictary fiber,
Apples
Round
sweet
vitumin C
Elongated. curved
Soft.
Potassium,
Bananas
vitamin B6
Example of on-demand information extraction. The header extracted table can be specified by the user, or inferred by the model.
Figure 1:
be covered by existing ontologies and the format might not match any existing task (relation extrac- tion usually involves only two entities and event extraction requires a trigger). Therefore, there is an unfulfilled demand for a more flexible paradigm to extract structured information.
1 Introduction
For addressing the shortcomings of conventional information extraction systems for average users, the burgeoning field of Large Language Models (LLMs) offers a promising direction. The latest generation LLMs is typically subjected to an essential step known as instruction tuning to cater to explicit user commands (Ouyang et al., 2022: OpenAI, 2023). In this stage, models are trained using datasets with specific instructions and the expected responses, which improves LLMs in un- derstanding and reacting to various human queries in natural language. Instruction tuning can be seen as form of extreme multi-task learning where each input-output pair is a different task, or meta- learning where the model learns to adapt using
Information extraction has conventionally been di- vided into set of well-defined sub-tasks, includ- ing named entity recognition (Sang and Meulder, 2003; Levow, 2006; Weischedel et al., 2013), re- lation extraction (Zhang et al., 2017; Han et al., 2018; Yao et al., 2019), event extraction (Deng et al., 2020; Wang et al., 2020; Zhan et al., 2023) and so on. These components serve as the basis for building complex systems such virtual as- sistants and news monitoring systems. However, an average user might also have information ex- traction needs (as shown in Figure 1) that do not align well with any of the well-defined tasks: the information elements (shape and taste) might not

---

the instructions. As a result, these models acquire zero-shot learning ability which emerges as natural interactions with non-expert users.
various online sources in privacy-sensitive cases, generated by the best-performing language model, GPT-4 (OpenAl, 2023). The dataset encompasses different levels of difficulty part of the instruc- tions further demands comprehensive reasoning and summarization abilities.
Instruction-follloingg models unlock access to a broader user base, simultaneously uncovering the diverse requirements in IE scenarios. In light of this, we propose novel IE paradigm, termed On-Demand Information Extraction, in this paper. Our task designed respond to a user's unique instruction and the related text by extracting the sought-after information and presenting it in a user- friendly, structured table format (see Figure 1). It goes beyond the constraints of predefined task set- tings or ontologies, i.e., the header of the output table can be either personalized by the user or in- ferred by the model itself from the given text and the instruction. This provides users the flexibility offer instructions with varying levels of speci- ficity, thereby customizing the output to suit their individual needs. Moreover, On-Demand Informa- tion Extraction is not limited to specific domain, making it a highly scalable and adaptable solution for wide range of applications.
On top of INSTRUCTIE, we develop our model, ODIE (On-Demand Information Extractor), which founded on the LLaMA-7B (Touvron et 2023) and LoRA (Hu al., 2022) techniques. Further- more, we establish an extensive test bed specifi- cally for this task. Compared to the performance of powerful open-source instruction-following mod- els, ODIE brings substantial improvements in the accuracy of the extracted headers and contents cording to automatic metrics and human evalua- tions. Ablation studies also verify the effectiveness of our proposed filtering method. These findings underscore the potential of INSTRUCTIE and ODIE to facilitate research in on-demand IE scenarios.
2 Related Work
2.1 Information Extraction
To benchmark this new task, we construct an instruction-tuning dataset for on-demand informa- tion extraction, named INSTRUCTIE. It com- prised of 14,579 training pairs and 150 manually curated test samples, serving as a supplement to the existing open-source instruction-tune tunnnng collections. To compile training data and boost the model's capacity adhere specific instructions, we ploy an automatic generation process via ChatGPT, which generates a range of instructions ensure wide coverage across diverse domains. We fur- ther incorporate a Ofain-of-Thoughh prompting ap- proach (Wei et al., 2022) into our data generation process, enabling us to investigate the effects of elaborating the "thought process" before extracting tables for on dendemand IE use cases. More impor- tantly, propose multi-faceted validation meth- ods to filter out low-quality samples, ensuring that the synthetic data curated from four perspectives: validity, informativeness, consistency with the in- struction, and faithfulness to the text.
Information extraction is conventionally repre- sented by sub-tasks such as named entity recog- nition (Lample et al., 2016; Cao et al., 2019; Lin et al., 2019; Huang et al., 2021; Lin et al., 2020; Yu and Ji, 2023), relation extraction (Yu et al., 2017; Sui et al., 2020), event extraction (Ji and Grishman, 2008; Yu et al., 2021; Zhan et al., 2023; Jiao et al., 2023). However, there is a clear gap between the task-specific systems in existing studies and the requirements in open IE scenarios (Li et al., 2023). To cater to a wider group of average users (Ouyang et al., 2023), we propose a novel on-demand infor- mation extraction task in this paper. Although our output format mirrors the text-to-table setting (Bao et al., 2018; Wu et al., 2022), our task diverges sig- nificantly from previous work in two key aspects: I) our task hinges user instructions, enabling per- sonalized extraction, and 2) the headers of the table are confined to pre-defined types. Instead, they can either be defined by the user or independently inferred by the model based on the context.
Additionally, to evaluate language models em- pirically, introduce a set of manually annotated test data from scratch. This dataset showcases wide range of instructions spanning extremely di- verse domains, providing clear reflection of real user requirements. The background text utilized in this dataset is carefully collected by retrieving from
2.2 Instruction Tuning
Instruction tuning provides promising solution for finetuning LLMs to better understand and re- spond human requests that are expressed in nat- ural language (Ouyang et al., 2022; Sanh et al., 2022). The success of instruction tuning relies

---

Step Fixed Instruction Generation
Step 2: Text Gemeration
Step 3: Open Instrection Generation
Fixed
Intruciio: position
Fited
We seeking highty eggained creatve Marketing
Iestraction: hnsnet all key indortation from. job poslings
salaz
Goodroon with the plaasing evecution of our maiketing cumpaigta...
Seed Exumples
Step 4: Instruction Paraphrase
6i Instance Filtering
Step 5:
Table Generation
Open Flard
Frred
Indrartion: Can you help cntnect postion valary fnom paiing?
from
We exdere highty reganined diseive Marketin Chendinator su.ecal.....
Ouipul:
Instrcttoo: waxt key informaioon from jos postings
Figure Overall framework of training data generation.
on diverse and representative instruction datasets, which help prepare language models for potential downstream usage (Wang et al., 2023). Existing instruction-tuning datasets often collected via crowdsoureing (Mishra et al., 2022: Wang al.. 2022b; Databricks, 2023) or via distillation from LLMs (Wang et al., 2022a; Honovich et al., 2022: Taori et al., 2023; Peng et al., 2023). As outlined in Wang et al. (2023), the current landscape of open- source datasets for instruction tuning is predomi- nantly composed of open-ended conversations, rea- soning, and coding tasks. Consequently, they fail adequately prepare the model to follow instruc- tions related information extraction. Hence, is imperative to develop benchmark that is represen- tative of instruction based information extraction in order to advance the exploration in this field.
types: fixed instruction and open instruction. For fixed instruction, the desired headers for the output table are clearly defined, and the model's task is to pull out the relevant content from the text. In contrast, open instruction presents a more demand- ing scenario where the model needs to first infer table headers based on the context, and then extract corresponding information.
3.2 Automatic Generation of Training Data
To enhance the instruction-followin capability of LLMs in on-demand IE, high-quality training data is of paramount importance. As shown in Figure 2, we devise an automatic pipeline via ChatGPT, encompassing six steps as follows. All the used prompts are listed in Appendix.
Fixed Instruction Generation. As instruction is the determining factor in the on-demand use case, we start with fixed instruction collection. Con- cretely, we provide five manually labeled demon- strations and harness the power of in-context learn- ing (Brown et al., 2020) guide the model in dis- cerning what constitutes fixed instruction. In principle, the fixed instruction should specify the header to be extracted, and the type of the text, e.g., "Extract position and salary from job postings". To promote diversity. we require Chat- GPT to generate 10 different instructions along with distinct domains for each iteration.
3 INSTRUCTIE Dataset
In this section, first formulate the on-demand in- formation extraction task. Following this, we delve into the details of automatically generating training data for INSTRUCTIE, as well as the process of human annotations for the test set.
3.1 Task Formulation
Formally, given a user instruction I and associated background text X, the model M is designed to extract pertinent information and organize it into a structured table T. In the table, the first row repre- sents the table header and the rest is the extracted content. To better align with real user needs, note that the input instructions are not required to ex- plicitly specify the header to be extracted. With this criterion, instructions are categorized into two
Background Text Generation. With given fixed instruction and domain, the subsequent step involves generating the related background text. We also provide demonstrations and specify in the prompt that the generated text should cover the des-

---

ignated header and conform the given text type and domain. For instance, for the fixed instruction in the previous paragraph, ChatGPT is supposed to generate multiple job postings containing different positions and salaries.
cation and filtering methods are crucial. Since the final output of the entire pipeline is the generated table, serves as a comprehensive reflection of the quality of each preceding step. As a result, we cen- ter our attention on the generated table to carefully craft a filtering mechanism across four dimensions. (I) Validity. This checks the validity of the pipeline output, determining whether it conforms to the tabular format. Any instances that don't 't meet this format are filtered out.
Open Instruction Generation. After creating the background text, our goal is to produce corre- sponding open instructions that stand apart from the fixed instruction. An open instruction does not specify the type of header, i.e., "position" and "salary", but rather employs relatively vague re- quirements, such as "key information", mirror- ing part real-world usage. We find that open instructions directly converted from fixed instruc- tions bear strong similarity. Hence, we discard fixed instructions as input, allowing ChatGPT to generate distinct open instructions using only text.
(2) Informativeness. We necessitate that the gen- erated table comprises an adequate number of rows and columns, without containing excessive empty cells, to ensure it offers sufficient information. (3) Consistency with instruction. For each fixed instruction, the header extracted by the model should strictly align with the provided instruction. To check for consistency with instruction, we for- mulate it as a Natural Language Inference (NLI) problem. Essentially, Given each header H, we construct sentence such "extract H from the text" as the hypothesis while the instruction serves as the premise. Taking these two as the input, we adopt neural evaluator (Zhong et al., 2022) cal- culate the factual consistency score. If the average consistency score is below a predefined threshold, the table will be dropped.
Instruction Paraphrasing. To equip the model with the ability better comprehend user instruc- tions across a diversity of styles, we incorporate an additional paraphrasing step. We delineate four styles: comprehensive query. casual interaction, direct command, and professional request. Each instruction is randomly assigned style, and Chat- GPT is tasked with paraphrasing it accordingly. It's also emphasized in the prompt that the paraphrased instruction must retain its key elements, specifically the header and the type of text.
(4) Faithfulness to text. For every extracted table, each cell's content should correspond faithfully to the provided background text. We still take this as an NLI task, i.e., given cell C, utilize its header H build a hypothesis such as "The H is C Meanwhile, the background text is regarded as the premise. The same neural evaluator calculates the scores, and a threshold is established to select instances that demonstrate high faithfulness.
Table Generation. Given a pair of instruction and background text, the model takes these as in- put and extracts the corresponding information into a tabular format. Considering the different cate- gories of instructions, we set distinct expectations for the table headers. For fixed instructions, we anticipate that the model adheres strictly to the in- struction, extracting only the headers specifically mentioned. Conversely, for open instructions, we aim for an output inclusive of all relevant details. Therefore, the model is tasked with generating as many columns as possible to ensure comprehensive instruction execution. For each input, we generate two versions of the output: 1) Direct, which rep- resents the direct output of the table without any accompanying text; and 2) CoT, which denotes the introduction of a Shain-of-Thoughh method (Wei al., 2022), allowing the model articulate its thought process prior to table extraction.
3.3 Manual Annotation for Test Data
We create test set by hand to evaluate how well LLMs perform in the on-demand information ex- traction task. In the annotation process, GPT-4 (OpenAI, 2023) is employed to assist human an- notators, ensuring wide-ranging collection. The specific process and details follows.
Candidate Domain Creation. To achieve diver- sity in test data, incorporate the vast knowledge base of GPT-4 enable human annotators to think beyond conventional domains and consider more specialized fields. Specifically, we prompt GPT-4 to generate 1,000 instructions across diverse do- mains as a candidate pool. Then we hand-pick
Verification and Filtering. To uphold the high quality of generated instances, meticulous verifi-

---

150 samples that are not only representative but also align with the genuine needs of different user groups Notably, these generated instructions are not part of the test set but are used as cues to kick off the gata-gath phase.
Train
Data
Direct
CoT
Overall
Test
Instruction
7,483
7,096
14,579
150
Open Instruction
3,773
3.676
7,449
Fixed Instruction
3,710
3,420
7,130
114
Text
4,507
4,380
4,751
150
Retrieved Text
0
0
0
119
# Generated Text
4,507
4,380
4,751
31
Background Text Collection. Based on candi- date instructions, the human annotators proceed to collect the relevant text. The principle is to re- trieve real text from the web that meets the require- ments. However, for several candidate instructions involving privacy issues (e.g., medical records), we use GPT-4 to generate synthetic data instead. between
Domain
82
82
83
61
Instr.
20.4
20.3
20.4
Ave. Text Len.
281.2
277.8
279.6
310.8
# Ave. Table Cell
20.8
23.2
22.0
17.1
# Ave. Table Row
3.6
3.7
3.6
4.7
# Ave. Table Column
6.2
6.7
6.5
4.1
Easy Level
Medium Level
The length of the text is specified 100 and 1,000 words and should contain structured information suitable for extraction.
# Hard Level
38
Table Statistics our dataset, InstructIE. # denotes the number. Ave. denotes the average value. Len. denotes the length words.
Instruction Annotation. Given the collected text, annotators are required to write correspond- ing instructions. Standard instructions should be in line with the user input, and diverse in descrip- tion style. The information required for extraction should found in the original text. Each instruc- tion is limited to 200 words, and the annotator can determine whether it is an open or fixed instruc- tion before annotating based on the content of the text. In addition, the need for appropriate reason- ing summarization abilities when extracting is permitted, if it is consistent with real-world usage (Concrete examples in Appendix E).
categorization: typically, easy-level instances have fixed instructions and small- or modcrately-sizer groundtruth tables. Those labeled as medium often include the fixed instructions to extract large-sized and complex tables or a small portion of open in- structions. Instances classified as hard generally necessitate the reasoning summarization abilities and incorporate most of the open instructions.
Regarding the final data format, beyond the struction, text, and table, we also provide the sub- sequent human-annotated information:
(1) "Domain" represents the area or field that the
Table Annotation. To improve the efficiency of table annotation, we utilize GPT-4 to generate three tables by setting temperature to 1.0 to serve as ref- erences for annotators for each input. Annotators can modify or integrate these reference tables into the final result. The content presented in the table should be exhaustive, aligned with the user instruc- tion, and accurately reflect the given background text. For information that is not specified in the original text, "N/A" is uniformly used in the table.
data instance pertains to.
(2) "Category" denotes whether the instruction
classified as open or fixed.
(3) "Souree type" characterizes the source of background text, classifying it as either "retrieve" sourced from real-world text, or "generate" produced by GPT-4.
(4) "Difficulty level" indicates the complexity of the extraction for the current sample from the human perspective, and is divided into three levels: "casy", "medium". and "hard".
More Annotation Details. For the entire pro- cess, we invite graduates and undergraduate student with research experience in the field of IE for data annotation. Each instance is annotated by an annotator and then reviewed by a reviewer. The annotator and reviewer discuss together and make necessary modifications until they reach an agreement. After finalizing the annotation, all an- notators are gathered to categorize each instance into three difficulty levels, including easy, medium, and hard. We take the following criteria for this
3.4 Dataset Analysis
INSTRUCTIE dataset consists of 14,579 samples for training and 150 for testing, with the statistics displayed in Table 1. Notably, the scale of our test aligns with existing instruction-tuning bench- marks. For instance, AlpacaFarm (Dubois et al., 2023), a collection of existing instruction-tuning test sets, encompasses 90 instructions from the Vi- cuna evaluation (Chiang et al., 2023), 129 from

---

literature
health
meteorology
Text Length
medicine
environment
archacology
climate
technology
law
psychology
sociology
1100 1200
polities
media
Instruction Length
urban
biology
astronomy
environment
fashion
ecology
ethics
culture
education
security
ngrieulture
5
10
15
25
30
35
50
55
60
economies
sports
zoology
culinary
history
Table Cell
Figure 3: Representative domains testing data.
15
20
25
45
50
55
Testing
Training
Anthropic (Bai et al., 2022), 156 from the Koala evaluation (Geng et al., 2023), 188 from Open As- sistant (Köpf et al., 2023), and 252 from the evalua- tion of self-instruct (Wang et al., 2022a). However, these benchmarks primarily focus on open-ended dialogues, reasoning, and coding tasks, and hence, INSTRUCTIE can supplement these sources by fo- cusing on the aspect of information extraction.
Figure 4: Distribution of training and testing data.
and finally present the detailed results.
4.1 Model Training ODIE
In particular, our training set comprises 7,483 direct instances and 7,096 CoT instances. The dataset, entirely composed of model-generated in- structions and texts, extensively spans 84 domains (listed in Appendix A). Our analysis reveals that the tables have an average of cells after filter- ing. Interestingly, on comparing the direct and CoT methods, we observe that the tables produced via CoT approach contain an average of 3 more cells than those generated directly.
To establish a model with instruction-followingg capability on IE, we finetune LLaMA-7B (Tou- vron et al., 2023) with LoRA (Hu et al., 2022), a parameter-efficient fine-tuning technique, on the training set of INSTRUCTIE to obtain ODIE. We format the datasets to follow chatbot-style schema to allow interactions between the user and the lan- guage model (a.k.a. "assistant") into one input sequence. Concretely, each instance begins with the systsyste semttm]? |>" token, accompanied by distinct system prompts for both Direct and CoT methods! This is succeeded by the "<|user token and the input instruction, after which the model's 's response follows the "< "<|assistant |>" token. During train- ing, we compute the cross entropy loss only tokens after < //ssisttnt|>>
As for the testing data, the instructions are cate- gorized into two types 36 open instructions and 14 fixed instructions. We retrieve 119 real texts, alongside 31 generated texts areas that are sen- sitive to privacy. As illustrated in Figure 3, 150 testing cases span broad range of 61 domains. The manually annotated tables contain 17 cells on average. Figure 4 compares the distribution in text length, instruction length, and the average number of cells in tables between the training and testing data. Moreover, the test set of INSTRUCTIE has exhibits a well-balanced distribution across three levels of difficulty, comprising 56, 55, and 39 in- stances respectively.
For a fair comparison, we adopt identical train- ing paradigms, hyperparameters, and backbone models for training ODIE and all other baseline models. The only exception is the number of epochs, the optimal number of training steps required to achieve peak performance varies de- pending on the size of the training data. More details about training can be found in Appendix D.
4 Experiment
'System prompt for the CoT method is "You are a helpful assistant. Follow the user instruction to output paragraph as the explanation and extract information from the given text into a concise markdown table." For the direct method, "output paragraph as the explanation" is removed.
this section, we first introduce our instruction- tuned model, thendescribe the experimental setup

---

4.2 Evaluation Metrics
Category
Models
Fixed
Open
Overall
Since the on-demand information extraction task is formulated as the text-to-table generation, we divide the evaluation into two parts: evaluating the table headers and table contents. Table headers reflect how well the model understands user in- structions. We adopt soft matching strategy (Jiao al., 2022) by using SentenceBERT (Reimers and Gurevych, 2019) to calculate the cosine similarity as the semantic similarity score. The table con- tents reflect the quality of extraction. We use the ROUGE-L F score (Lin, 2004) to evaluate the gen- erated output. We also conduct human evaluations to provide a comprehensive assessment.
Open-Source Models
ALPACA
65.89
45.69 49.26
59.80
TOLU
77.78
69.44
Our Models
ODIE-DIRECT
83,59
51.67
73.82
Filtering
83.54
51.37
73.61
ODIE-COT
72.32
54.17
66.81
Filtering
68.97
53.01
64.11
Proprictary Models CHATGPT GPT-4
81.69
57.86 57.78
74.49
82.06
74.47
Table 2: Experimental results for header evaluation. The metric is F1 (%) the soft matching score.
demonstrates an even better performance in the con- text of open instructions, surpassing ODIE-DIRECCY by 2.5% while it shows weaker performance on fixed instructions. Based careful observation, speculate that this is because the CoT stim- ulates the model to think more dynamically and adaptable, which particularly beneficial for open instructions that can have different formulations and contexts. However, when comes to fixed instructions. CoT might grant the model excessive flexibility, exacerbating the issue of violating the in- struction and generating additional headers (Figure 8 in the Appendix shows case study comparing ODIE- DIREET and ODIE-COT).
4.3 Evaluated Models
We compare three categories of models: open- source instruction-folloviin models, proprietary models, and models trained on INSTRUCTIE.
Public Instruction-Followinn Models. ALPACA and TČL.U are state-of-the-an instruction-followin models, with instruction tuning on 52K and 512K data collections (Databricks, 2023; Longpre et al., 2023; Köpf al., 2023; Peng al., 2023; Chaud- hary, 2023), respectively. TüLU, in particular, stands out as the best-performing model the 7B size (Wang et al., 2023). We acquire these two baseline models by training on publicly accessible datasets using LoRA technique.
4.5 Table Content Evaluation
Proprietary Models. We select the two best LLMs, CHATGPT and GPT-4 (OpenAI, 2023) for comparison. Since their training data is not dis- closed and the model size much larger than ours, we mainly use their performance as the reference.
Table 3 presents evaluation results for table content analysis, showing that our ODIE models outper- form open-source basclines substantially, Notably, ODIE-DIRECT performs 5.4% better than the best baseline model, TÜLU. Among ODIE mod- els, those with multi-facet filtering bring least 2.6% higher performance than versions without fil- tering, validating the effectiveness of our proposed method. However, despite these improvements, all open-source models, including ours, are not on par with state-of-the-ard proprietary models in the table content evaluation.
Depending whether the CoT method is added during training, include two and ODIE-
Our Models.
versions of our model COT. Additionally, we report two variants obtained by removing the filtering process of training data.
4.4 Table Header Evaluation
We report the soft matching scores for both fixed and open instructions in Table 2. Our model showcases highly competitive performance: ODIE- DirecT outperforms the best open open- souncsource bascline by 4%, and nearly matches the performance of GPT models despite the significant difference in model sizes. This outcome verifies that our mod- cls effectively develop the instruction-folloiing capability for on-demand IE. Notably, ODIE-CoT
We further conduct an in-depth analysis examin- ing various models across three aspects: instruction categories, text sources, and difficulty levels.
Instruction category: All models perform con- siderably better on fixed instructions opposed to open instructions. This is expected since open in- structions demand more sophisticated text analysis and extraction abilities.
Text source: Models find the retrieved context

---

Difficulty Medium
Category
Source Retrieve
Models
Easy
Hard
Fixed
Open
Generate
Overall
Data
Open-Source Models
ALPACA
26.27
20.08 39.15
22.72 38.68
25.29
16.07
21.18 39.59
23.08
TOLU
43.69
42.55
34.94
45.08
40.72
512K
Our Models
DDIE-DIRECT
48.01
45.38
43.71
47.19
41.92
49.49
45.00
45.93
7.5K
Filtering
46.31
39.85
42.44
44.42
38.23
46.71
41.95
42.93
7.5K
ODIE-COT
44.47
39.99
41.73
43.02
39.25
49.01
40.32
42.12
7.1K
Filtering
41.26
36.59
41.20
38.93
37.87
39.53
7.1K
Proprictary Models
CHATGPT 52.45
50.56
51.07
53.21
45.66
55.97 65.89
50.21 57.28
51.40
GPT-4
60.78
55.76
61.24
61.51
51.29
59.06
Table 3: Experimental results for table content evaluation. The metric is F1 score of ROUGE-L (%).
more challenging, as the generated text usually follows a more structured format, which makes easier to identify the key information.
Table Header
100%
163
60%
206
Difficulty level: Most models tend perform best on easy instances, yet intriguingly, part of the models exhibit better performance in hard-level samples compared to medium-level ones. After manual analysis, we attribute this to the fact that medium-level instances usually require extracting complex and large-size tables, where models often fail to capture all essential elements in the text, re- sulting in lower scores. Conversely, hard-level sam- ples demand reasoning or summarization ability for specific rows, columns, or cells. For instance, when extracting the information from recipes, mod- els may struggle with calculating the cooking time of all the steps, but still produce a reasonable ta- ble. It would not cause significant impacts on the score. This discrepancy between human judgment and model performance calls for effort the more fine-grained evaluation metrics.
130
86
86
41
Alpaca
TÍR
ODEE
GPT4
Wrong
Partly Correct
Correct
(a) Evaluation on table headers
Table Content
100%
1449
60%
99
110
115
20%
142
47
0h
Alpaca
Correct and satisfying
Acceptable content with minor imperfection
Relavant content but has significant errors
Irrelevant Invalid
(b) Evaluation for table headers
Figure 5: Human evaluations on headers and contents.
4.6 Human Evaluation
For human evaluation, we ask three annotators of the instructions to evaluate model outputs. The evaluators are asked to rate the output of four instluction-followiing models based on whether the extracted information is accurate. In line with tomatic evaluations, content of the table separately. The details and the annotation platform are presented in Appendix C.
in terms of correct and partly-correct scores. Con- versely, ALPACA struggles to follow instructions, frequently producing invalid results.
evaluate the header and
For the table contents, we set up four-level rating standards, including (1) Rating-A: valid and satisfy- ing, (2) Rating-Bi BB acceptable but has minor errors or imperfections, (3) Rating- C: relevant to the in- struction with significant errors, and (4) Rating-Di D: irrelevant completely invalid. As shown in Fig- ure 5(b), GPT-4's much larger model size allows to excel beyond the other models, while ODIE continues to display clear benefits over ALPACA and TÜLU, particularly in Rating A and
For the table headers, we design three-level rat- ing system for the models' outputs: (1) Rating-A: correct, (2) Rating-B: partly correct, or (3) Rating- C: completely wrong. Figure 5(a) illustrates the performance four models, with ODIE showing comparable results to GPT-4, both receiving only 9 votes Rating C. ODIE also outperforms TüLU

---

Metries
Pearson
Spearman
Kendall
Limitations
[Header] Exact Match.
0.640
0.609
0.494
While this paper contributes to the research field by introducing the On-Demand Information Extrac- tion task and constructing the INSTRUCTIE dataset, it still has the following limitations:
[Header] Semantic Sim.
0.817
0.769
0.637
[Content] Exact Match.
0.338
0.375
0.272
[Content] Semantic Sim.
0.764
0.705
0.558
[Content] Rougel.
0.713
0.704
0.554
Table 4: Correlation Analysis. To further study different metrics our proposed task, we analyze the correlation between these automatic metrics and human evaluations. Here Exact Match. means exact matching and Semantic Sim. means semantic similarity. The analysis results in- cludes the Pearson, Spearman, and Kendall coefficients.
Model Size and Data Constraints: The ex- periments presented in this paper primarily focus on the utilization of the 7B model. Due to the limited computing resourees, an ex- ploration into the impact of varying model sizes and the potential benefits of using larger datasets could not be conducted. It remains open question how scalability in terms of model size and data volume would affect the performance and efficiency of the On-Demand Information Extraction task.
4.7 Evaluation Metric Analysis
To make our evaluation comprehensive, we study two automatic evaluation metrics on table head- ers (Exact Match and Semantic Similarity) and three ones on table content (Exact Match, Seman- tic Similarity, and ROUGEL). The experiments results can be found in Tables 7-10 in the appendix, which show similar trend. On these different met- rics, our methods can still obviously outperform the open-source baselines while being comparable with GPT4 on table header generation.
2. Combination of Direct and CoT Methods: In our experimental analysis, the Direct and CoT method are discussed and evaluated sep- arately in the On-Demand Information Extrac- tion task. However, the potential synergistic effeets combining both methods have not been investigated. It could possibly yield in- sights into different dimensions of the task or further improve the model performance.
To discover the suitable automatic evaluation metric for our proposed task, we further analyze the correlation between these automatic metrics and human evaluations. Table 4 lists the analy- sis results, including the Pearson, Spearman, and Kendall coefficients. The results indicate that, for table header, two studied metrics are highly cor- related with human evaluation. But for table con- tent, RougeL and semantic similarity can indeed be more reliable metrics for this task compared with exact matching due to significantly high correlation. In this paper, we show Rougel. for its wide use in prior instruction-tuning work (Wang et al., 2023).
The evaluation metrics used in the current experiments are primarily detecting the overall similarity between the model outputs and the groundtruth. However, given the flexible nature of table structures, it is imperative to have evaluation metrics that can assess the accuracy and quality of table construction and information organization in a more fine- ine-grained manner. Developing more effective and precise evaluation metrics is nec- essary to robustly evaluate different aspects of our On-Demand Information Extraction task.
Evaluation Metries:
5 Conclusion
4. Contextual Inference and Complex Instruc- tions:
As highlighted in the conclusion, the current model has room for improvement contextually inferring table structures and pro- cessing complex instructions. This limitation can affect the utility and user experience, par- ticularly for users who not domain experts and may know how to frame their queries optimally. Enhancing the model's capabilities in these areas essential for ensuring that the On- -e-Demand IE task is accessible and friendly for a wide range of users.
This paper introduces a new task, On-Demand In- formation Extraction to fulfill users' personalized needs by extracting content based on instructions and organizing it in a table with user-specified or modcl-inferred headers. To benchmark this new task, we construct comprehensive dataset IN- STRUCTIE including synthesized training data and human-annotated test data. Our developed model ODIE outperforms existing sounesource models in extensive experiments.

---

In light of these limitations, future work should focus on exploring the impact of model size and data scale, investigating the combination of differ- ent data types, developing more nuanced evaluation metrics, and improving the model's ss ability to infer context and handle complex instructions. These efforts are crucial in advancing the On-Demand Information Extraction systems and making them more accurate and widely applicable.
Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback.
Junwei Bao, Duyu Tang, Nan Duan, Zhao Yan, Yuanhua Lv, Ming Zhou, and Tiejun Zhao. 2018. Table-to- text: Describing table region with natural language. In Proceedings the AAAI conference on artificial intelligence, volume 32.
Ethics Statement
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz. Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish. Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Anmual Conference on Neural Information Process- ing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual.
In conducting the research presented in this pa- per, we adhere to ethical standards and principles ensure the integrity and validity of our work. Specifically, the dataset INSTRUCTIE constructed for this research is developed with utmost care to ensure that no personal or sensitive information is included. The buman-annolated test data is col- lected and used in compliance with relevant ethical guidelines. Additionally, during the data collection process, we verify the licenses of the data source websites to ensure that our use of the data sticks the terms and conditions stipulated by the data providers. Moreover, during the development of the dataset, we mitigate any biases that may arise, ensuring that the data is representative and does not favor any particular group or perspective.
Yixin Cao, Zikun Hu, Tat-Seng Chua, Zhiyuan Liu, and Heng Ji. 2019. Low-resource name tagging learned with weakly labeled data. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th Interna- tional Joint Conference on Natural Language Pro- cessing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 261-270. Association for Computational Linguistics.
Acknowledgements
Research was supported in part by US DARPA KAIROS Program No. PA8750-19-2-100 and INCAS Program No. HR001121C0166 National
Sahil Chaudhary, 2023. Code alpaca: An instruction- following llaaa model for code generation. https: //gi aegithuubccom www.ccom//aaiii. 12801 14/ cocaaaapaa..
Science Foundation IIS- 19-515,,,,,, 11S-17-41317, and IIS 17-04532, and the Molecule Maker Lab Institute: An AI Research Institutes program sup- ported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environment (I-GUIDE) by NSF under Award No. 2118329. Any opinions, findings, and conclusions or recommendations ex- pressed herein are those of the authors and do not necessarily represent the views, either expressed or implied, of DARPA or the U.S. Govermment.
Wei-Liin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang. Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. 2023. Vieuna: An open- source chatbot impressing gpt-4 with 90%* cchttgtt quality. Blog post.
Databricks. 2023. language model trained on the databricks ma- chine learning platform. databrickslabs,
Databricks' dolly, large
Shumin Deng, Ningyu Zhang, Jiaojian Kang. Yichi Zhang, Wei Zhang, and Huajun Chen. 2020. Meta- learning with dynamic- dymamii-memory- memory-basee prototypical network for few-shot event detection. In WSDM '20: The Thirteenth ACM International Conference on Web Search and Data Mining. Houston, TX, USA, February 3-7, 2020, pages 151-159. ACM.
References
Yuntao Bai, Andy Jones, Kamal Ndousse. Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Haafield-Dodds, Danny Hernandez, Tristan Hume,
Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang. Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy

---

Liang, and Tatsunori B. Hashimoto, 2023. Alpaca- farm: A simulation framework for methods that learn from human feedback. CoRR, abs/2305. 14337777.7777
David Glushkov, Amav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. 2023. Openassistant conversations de- mocratizing large language model alignment. CoRR, abs/2304.07327.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wal- lace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. Koala: A dialogue model for academic re- search. Blog post.
Guillaume Lample, Miguel Ballesteros, Sandeep Sub- ramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016, pages 260-270. The Association for Computational Lin- guistics.
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao,
Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-ar evaluation. In Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31- November 4, 2018, pages 4803-4809. Association for Computational Linguistics.
Gina-Anne Levow. 2006. The third international chi- nese language processing bakeoff: Word segmenta- tion and named entity recognition. In Proceedings of the Fifth Workshop on Chinese Language Processing, SIGHAN@ COLING/ACL 2006, Sydney, Australia, July 22-23, 2006, pages 108 117. Association for Computational Linguistics.
Or Honovich, Thomas Scialom, Omer Levy, and Timo Schick. 2022. Unnatural instructions: Tuning lan- guage models with (almost) no human labor. CoRR, abs/2212.09689.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net,
Bo Li. Gexiang Fang, Yang Yang, Quansen Wang. Wei Ye, Wen Zhao, and Shikun Zhang. 2023. Evaluating chatgpt's information extraction capabilities: An as- sessment of performance, explainability, calibration, and faithfulness. CoRR, abs/2304. 11633.
Jiaxin Huang, Chunyuan Li. Krishan Subudhi, Damien Jose, Shobana Balakrishnan, Weizhu Chen, Baolin Peng, Jianfeng Gao, and Jiawei Han. 2021. Few- shot named entity recognition: An empirical baseline study. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021. Virtual Event Punta Cana, Domini- can Republic. 7-11 November, 2021, pages 10408- 10423, Association for Computational Linguistics.
Cein-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Ying Lin, Heng Ji. Fei Huang, and Lingfei Wu. 2020. A joint neural model for information extraction with global features. In Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, ACL 2020, Online, July 5-10, 2020, pages 7999-8009 Association for Computational Linguis- tics.
Heng Ji and Ralph Grishman. 2008 Refining event extraction through cross-document inference. ACL 2008, Proceedings of the 46th Annual Meeting of the A Association for Computational Linguistics, June 15- Association for Computer Linguistics.
Ying Lin, Liyuan Liu, Heng Ji. Dong Yu, and Jiawei Han. 2019. Reliability-wware dynamic feature com- position for name tagging. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL. 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 165-174. Association for Computational Linguistics.
20, 2008, Columbus, Ohio, USA, pages
The
Yizhu Jiao, Sha Li, Yiqing Xie, Ming Zhong, Heng Ji, and Jiawei Han. 2022. Open-vocabulary argument role prediction for event extraction. In Findings of the Association for Computational Linguistics: EMNIP 2022, pages 5404-5418, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Lc, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. CoRR, abs/2301.
Yizhu Jiao, Ming Zhong, Jiaming Shen, Yunyi Zhang. Chao Zhang. and Jiawei Han. 2023. Unsupervised Pro-
event chain mining from multiple documents. ceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 4 May 2023, pages 1959. ACM.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannanch Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27. 2022, pages 3470-3487, Association for Com- putational Linguistics.
Andreas Köpf. Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri,

---

OpenAl. 2023. Gpt-4 technical report. abs/2303.08774.
ArXiv,
Rohan Taori, Ishaan Guirajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto, 2023. Stanford alpaca: An followiinnn llama model. https:// gi thub. www.:/cccccaaaaaaaeeeeee wtt.//tttttttttttetttooe ttttttttsu-aaaa hat/sssulaaassseee http//staffodddddaee
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der. Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NeurlPS.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet. Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023, Llama: Open and efficient foundation language models. CoRR. abs/2302.1399
Siru Ouyang. Shuohang Wang, Yang Liu, Ming Zhong, Yizhu Jiao, Dan Iter, Reid Pryzant, Chenguang Zhu, Heng Ji, and Jiawei Han. 2023. The shifted and the overlooked:
Xiaozhi Wang, Ziqi Wang, Xu Han, Wangyi Jiang, Rong Han, Zhiyuan Liu, Juanzi Li, Peng Li, Yankai Lin, and Jie Zhou. 2020. MAVEN: A massive general do- main event detection dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Longuage Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020. pages 1652-1671. Association for Computational Linguistics.
a task-oriented investigation user-gpt interactions. In the Association for Computational Linguistics:
EMNLP 2023. Association for Computational Lin- guistics.
Baolin Peng, Chunyuan Li. Pengcheng He, Michel Gal- ley, and Jianfeng Gao. 2023. Instruction tuning with GPT-4. CoRR, abs/2304.03277.
Yizhong Wang, Hamish Tvison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi. 2023. How far can camels go? exploring the state of instruction tuning on open resources. CoRR, lbs/2306.04551.
Nils Reimers and Iryna Gurevych. 2019. Sentence- Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, FEMNLP-IJCNT.P 2019, Hong Kong, China, November 3-7, 2019, pages 3980-3990. Association for Computational Linguistics.
Yizhong Wang. Yeganeh Kordi, Swaroop Mishra, Al- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- nanch Hajishirzi. 2022a. Self-instruct: Aligning lan- guage model with self generated instructions. CORR, abs/2212.10566
Brik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language- independent named entity recognition. In Proceed- ings of the Seventh Conference on Natural Language Learning, CoNLL 2003, Held in cooperation with HILT-NAACI. 2003, Edmonton, Canada, May 31- June 1, 2003, pages 142-147. ACI..
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, An- jana Arunkumar, David Stap, Eshaan Pathak, Gi- annis Karamanolakis, Haizhi Gary Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuz- nia, Krima Doshi, Kuntal Kumar Pal, Maitreya Pa- tel, Mehrad Moradshahi, Mihir Parmar, Mirali Puro- hit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravschaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A. Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b. Suppernnaturllinsstuutiinn Generalization via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2022, Abu Dhabi, United Arab Emirates. December 7-11, 2022, pages 5085-5100 Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari. Canwen Xu, Urmish Thakker. Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abhcesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Teehan, Teven Le Seao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shor task gener- alization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. 2022. Chain-of-thought prompt- ing elicits reasoning in large language models. In NewrIPS.
Dianbo Sui, Yubo Chen, Kang Liu, Jun Zhao, Xian- grong Zeng, and Shengping Liu. 2020. Joint entity and relation extraction with set prediction networks. CoRR, abs/2011.01675,
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Ed- uard Hovy, Sameer Pradhan, Lance Ramshaw, Nian- wen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran-

---

chini, et al. 2013, Ontonotes release 5.0 Idc201 3t19. Linguistic Data Consortium, Philadelphia, PA, 23.
sallety
real-estate
management
energy
heepitality recreation
politics
Xueqing Wu, Jiacheng Zhang, and Hang Li. 2022. Text- to-table: A way of information extraction. In Proceedings of the 60th Anmual Meeting of the As- sociation for Computational Linguistics ( Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 2518-2533, Association for Computa- tional Linguistics.
technology
service
employment
environment
medicine
science
health
tourlem
art
ccology
product
electronies
culture
industry
hiology
retail
Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. Docred: A large-scale document-level relation extraction dataset. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Pa- pers, pages 764-777. Association for Computational Linguistics.
emergency
law
communication
history
Figure 6: Representative domains of training data.
Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 2023-2038. Association for Computa- tional Linguisties.
Dian Yu, Lifu Huang, and Heng Ji. 2017. Open rela- tion extraction and grounding. In Proceedings of the Fighth International Joins Conference on Natural Language Processing, IJCNLP 2017, Taipei, Taiwan, November 27- December 2017 Volume 1: Long Papers, pages 854-864. Asian Federation Natural Language Processing.
A Domain of Training Data
In our INSTRUCTIE dataset, all the instructions. and texts in this dataset model-ggnerated and extensively involve 84 domains (see Figure 6), such as management, ecology, marketing, health, astron- omy, metcorology, beauty, design, linguistics, and hospitality.
Pengfei Yu and Hong Ji. 2023. Shorten the long tail for rare entity and event extraction. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023, pages 1331-1342. Association for Computational Linguis- tics.
B Prompting Templates for Data Generation
The INSTRUCTIE dataset relies on a number of prompting templates in order to elicit the genera- tion from language models for training data. Here provide five templates for 1) fixed instruc- tion generation, 2) background text generation, 3) open instruction generation, 4) instruction para- phrasing, and 5) table generation (Table 5 and Table 6). Note that during the stage of test data annota- tion, we use the same prompts to generate instruc- tion hints for Candidate Domain Creation, generate texts for Background Text Collection, and generate reference tables for Table Annotation.
Pengfei Yu, Heng Ji, and Prem Natarajan. 2021. Life- long event detection with knowledge transfer. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event Punta Cana, Dominican Repub- lic, 7-11 November. 2021, pages 5278-5299 Associ- ation for Computational Linguistics.
Qiusi Zhan, Sha Li, Kathryn Conger, Martha Palmer, Heng Ji, and Jiawei Han. 2023. GLEN; general- purpose event detection for thousands of types. CoRR, abs/2303.09093.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. 2017. Position-awand attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 35-45, Association for Computational Linguis- tics.
C Human Evaluation Details
C.1 Human Evaluation Setup
Here we provide more details for the human eval- uation described in the experiment for rating the models' responses to the 150 uscr-oriented informa- tion extraction instructions. Human evaluation in done using a open source annotation tool, doccano To ensure faithful and reliable evaluation, asked
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multi- dimensional evaluator for text generation. In Pro- ceedings of 2022 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2022,

---

To measure how reliable our human evaluation is, we calculate the inner-rater agreement among our three evaluators. We first report the Flciss's s kappa value?, which is commonly used to measure inter- rater agreement for categorical items. When calcu- lating this, we separately handle the table headers and contents. For the table headers, treat the 3-level rating as categorical variable, leading to kappa value of 0.55, which is moderate agree- ment according to common practice. Furthermore, we also calculate the agreement of our evaluators on 4-level rating for table contents, 0.49, which Stant Anmotaatio
Corkent intruction bether- .gghiertrgrteeerrrmm..
Cintart Inteortarindll
"fnstruction
To shape political campaign stretegy and policy decisions, need insights public opirion Please extract data from these polls regarding the factora that workers consider important jeb, along with the parcentage of worken who rank each factor an the moat important.
*Tex"*
makes
factors
experience: pay,
ratiemmet benefits,
amount vacation, optionn for remote work and opportuntiie for advancement. When soked to rank the most important footors job, parcent pul pay the top Having a good boss cornes secund, percent wurkers narking it most important, About percurt about benefits while than percent destre sufficient amount vacation
"Table** Factor
Percentage e of workers who rank as the most important)
Pay
14%
154%
Retirement benefits 10%%
AAmunt vacation 885
Figure 7: Annotation interface for human evaluation. The predictions from different models present in random order and the model information being anonymized. Our expert evaluators are required to read the instruction and input, refer to the target, and then select the rating for the model's outputs from three options for the table headers and fours options for the table contents
three authors of these instructions (and of this pa- per) to judge model predictions. These three evalua- tors coordinated the standards for the rating system before starting annotation and then each of them rated all the instances independently. They were presented with the instruction, the background text, the target output (as reference), and the model outputs. Model responses listed in random or- der, with all the model information anonymized. Figure 7 provides a screenshot of the annotation interface. The reported performance in this paper is based on the results from all evaluators.
also indicates moderate agreement.
D Implementation Details
D.1 Training Data Generation
In this part, we introduce the parameters and details involved in the phase of generating training data. In our pipeline, for the first step. Fixed Instruction Generation, we prompt ChatGPT generate 10 different fixed instructions each iteration, doing this a total of 500 times, resulting 5000 instruc- tions along with their corresponding domains. Each domain here is represented by single word. In the second step. Background Text Generation, we generate piece of text corresponding each in- struction, resulting in 5000 pieces of text. In the third step, Open Instruction Generation, we gener- ate an open instruction for each piece of text. By combining the two types of instructions, we ob- tain a total of 10,000 training pairs. In the fourth step, Instruction Paraphrasing, we sample ten in- structions each time, regardless of whether they are fixed open, and pass them together to ChatGPT for paraphrasing. ChatGPT is required to output the same number of instructions. If the number of outputs from GPT does not match, the results of that round will be discarded. This means that we query ChatGPT more than 1000 times total.
C.2 Human Evaluation Agreement
httpp://www.wkkppod....oo//hhhhs kappa

---

In the fifth step. Table Generation, we generate table for each of the 10,000 training pairs. After the above steps, we obtain 10,000 raw direct and CoT instances. the sixth step, Verification and Filter- ing, considering Validity, 438 direct data instances are filtered out. To ensure Informativeness, we re- quire the sum of the number of rows and columns in the table to be greater than 3, and the number of columns to be more than 1. Additionally, the number of "N/A" in the table should be less than 4. Tables that do not meet these requirements, to- taling 653, are discarded. To ensure Consistency and Faithfulness, we set a threshold equal to 0.5. Only the tables with average scores greater than the threshold are retained. These two steps filtered out 552 and 874 direct data instances respectively. As for CoT, these four filtering strategies remove 961, 250, 696, and 997 instances respectively.
ODIE-DIRECT can extract decent table while ODIE-COT violates the instruction and generates five irrelevant headers, including the type of inci- dent, affected company, data, location, and other details. We speculate that this is because the CoT stimulates the model to think more dynamically and adaptably, which is particularly beneficial for open instructions that can have different formula- tions and contexts.
Figure 9, 10, and 11 respectively show three examples of different difficulty levels. The first example is at the easy level, which involves fixed instruction and small-sized groundtruth tables. Ac- cording to the model outputs, we can see that most of the four models can extract the mostly-correct in- formation. The second medium example is fixed instruction extract relatively complex table. this case, find that the two open-source baselines miss some rows or columns. In contrast, our model can extract more useful in- formation. But even for GPT4, there is a pattern in its output that does not match the groundtruth. The third example is the hard case, whose instruction requires commonsense reasoning, which requires the model to understand the color of food and find out the orange ones. According to the experimen- tal results, all models fail exclude non-orange Strawberry Applause. But in general, they were able to extract the recipe's name, the ingredients, and the approximate cooking time.
size- and
D.2 Model Training
We use LLaMA-7B (Touvron et al., 2023) as the backbone model and finetune it with the LoRA approach (Hu et al., 2022) for all the models. Dur- ing training, we configure the batch size to 64 and the maximum learning rate to 3e-4 with 0.03 warmup ratio. For all the experiments, the LoRA set 16, and we apply a dropout rate of 0.05. We keep these hyperparameters the same for fair comparison. Therefore, the only differences among ALPACA, TULU, and ODIE lie in the instruction- tuning data utilized for training and the number of training epochs. Considering the difffernnt amounts of training data, we train ALPACA 5 epochs (52K data), TÜLU 2 epochs (512K data), and ODIE 20 epochs (7K data) respectively achieve the best performance INSTRUCTIE.
During the inference process, we also adhere to the same set of parameters: a temperature of 0.1, top_p P of 0.75, top_k k of 40, 4 beams, and a maximum generation length of 2,048.
E
Case Study
Our case study is conducted from two perspectives, the type of the model and the difficulty level of the dataset. Figure 8 shows the output of our two models, ODIE- DIRECT and ODIE-COT, for the same instruction. The instruction is fixed, so expect the model to follow the instruction and ex- tract the information specified by the instruction, which are patterns and indicators. According to the observation of model outputs, we can find that

---

Instruction: Please conduct analysis of the logs and security incident reports. Our objective is identify patterns and indicators that suggest the presence of cyher threats..
Text:
Rapid7 analyzed a sample webshell payload associated with successful exploitation. The webshell code would first determine if the inbound request contained header named X-somnnck-ommmnt,, and would retur a 404 "Not Found' error if the header was not populated with specific password-like value. As of June 1, 2023, all instances of Rapid7-observed MOVEit Transfer exploitation involve the presence of the file human2.aspx in the wwwroot folder of the MOVEit install directory (human.aspx the native aspx file used by MOVEit for the web (errfuce).Successse exploitation allows attuckers to downloud and steal sensitive information such list of stored files, usernames who uploaded the files, paths, configured Azure Blob Storage accounts, data from Azure Blob Storage containers, data from servers, so on. The attackers can also insert and delete new random named MOVEit Transfer user with the login name 'Health Check Service' and create new MySQL sessions. Progress Software advising MOVEit customers to check for indicators of unauthorized access over "at least the past 30 days", as well as other remediation recommendations.
P
Model Output of ODIE-Direct:
Pattern
Indicator
Determines X--iLLok-Coomenn header populated with specific password- like value before
Webshell payload
returning 404 'Not Found error
Presence of human2. aspx fill in wwwroot folder of MOVEi Eit install directory, allows attackers to downloud and steal sensitive information, insert and delete new randorn momed MOVEi Transfer
MOVEit Trunsfer exploitation
user, create new MySQL sessions
Advice from Progress Software
Check for indicators of unauthorized access least the 30 days", as well as other remediation recommendations
Model Output of CoT:
identify patterns indicators of eyber threats from the logs and security incident reports, we need extract information reluted to cyber attucks, data breaches, malware infections, and other lypes of security incidents. We will structure this information in tubular format that clearly shows the type of incident, the affected company or organization, the date and time of the incident, the location of the incident, and any other relevant details that may suggest the presence of a eyber threut.
Affected Company/O rganization
Date and Time
Type of Incident
Location
Details
Successful exploitation allows attackers to download and steal sonsitive information such as stored files, usemames who uplouded the files, file paths, comfugured Azure Storage accounts, from Azure Storage containers, data from and delete new random named MOVEit Transfer user with the login name 'Health Chock Service' and create now MySQL sessions.
Cyber
Progress
N/A
N/A
Attack
Software
and The attackers also insert
Figure 8: Case study on comparing ODIE-DIRECT and ODIE-COT. For the fixed setting. ODIE-DIRECT is better following instructions, which aims to extract the pattern and indicator. But ODIE-COT is facing the exacerbating issue of violating the instruction and generating five irrelevant headers, including the type of incident, affected company, data, location, and other details.

---

Instruction: Retrieve product information such as product name, target audience, key feature, material, and price from e-commeerve websites. We need this data urgamized für cutegorization and comparison purposes.
Text:
Gildan Cotton Crew Nock T-Shirts are anything but basic. Crafted with soft, breathable cotton and moisture wicking technology to keep you cool and comfortable no matter what's on the agenda. Perfect for layering or lounging you thank us later. Price:S18.99.
C4 Energy Drink now available in delicious und sugar-free carbonated forn. This drink features 200mg of citrulline malate every serving. Carbonated Original the Go packs the legendary energy, endurunce and pumps that you know love C4, Available 18 puck two flavors, Frozen bombsicle and Strawberry Satermelon.PriceSS55
caffeine, CarnoSyn beta-alanine
Groundtruth:
Product Name
Target Audience
Key Feature
Material
Price
Gildaan Cotlon Crew Neck T-Shirts
N/A
Moisture wicking technology
Cotton
$18.99
Energy Drink
N/A
200mg caffeine, beta-olanine
Carbonated drinks
$33.59
Model Output of Alpaca:
Product Name
Target Audience
Key Feature
Material
Price
Gildan Cotton Crew Nock T-Shirts
Everyone
Soft, breathable
Moisture wicking
$18.99
C4 Energy Drink
Everyonc
Sugar-frce
Carbonsted
$33.59
C4 Energy Drink
Everyone
Sugar-fice
Carbonated
$33.59
Model Output TŪlu:
Product Name
Target Audience
Key Feature
Material
Price
Gildan Colton Crew Neck T-Shirts
Clothing
Sofl, breathable cotton
Colton
$18.99
200mg caffeine, CarnoSyn beta- alanine, citrulling malate
C4 Energy Drink
Energy drink
Carbonated drinks
$33.59
Model Output ODIE:
Product Name
Targel Audience
Key Feature
Material
Price
Anyone looking
Soft, brcathuble moisture wicking
Gildan Cotton
comfortable Inyering
Cottonn
$18.99
Crew Neck T-Shirts
lounging
technology
Anyone looking for sugar- free carbonated
200mg of caffeine,
Carbonated water, sugar-free carbonated water, maltodextrin, citric acid, natural
C4 Energy Drink
CarnoSyn beta-alanine
$33.59
energy drink
citrulline malate
Model Output GPT4:
Product Name
Key Feature
Material
Price
Gildan Crew Neck T-Shirts
Moisture wicking tochnology
Cotton
$18.99
C4 Energy Drink
All
200mg of caffeine. beta-alanine
Carbonated drinks
$33.59
Figure Case study on easy-level instruction.

---

Instruction: Pleuse conduct unalysis of the logs and security incident reports. Our objective is to identity pullemss and indientors that suggest the presence of cyber threats..
Text:
Rapid7 analyzed sample webshell payload associated with successful exploitation. The webshell code would first determine if the inbound request contained header named X-siT.ock- Commmmn,n,, and would return 404 'Not Found' error if the header not populated with specific password-like value. As of June 2023, all instances of Rapid7-observed MOVEit Transfer exploitation involve the presence of the file human2.aspx in the wwwrool folder of the MOVEit install directory (human.aspx is the native aspx file used by MOVFit for the web Succeess) exploitation allows ullackers to download and steal sensitive infonmation such as list of stored files, usernames who uploaded the files, file paths, configured Azure Blob Storage accounts, data from Azure Blob Storage containers, data from servers, and so on. The attackers can also insert and delete a new random named MOVEit Transfer user with the login name 'Health Check Service' and create MySQL sessions, Prugress Software is advising MOVEit customers check for indicators of unauthorized access over "at least the past days". as well as other remediation recommendations.
Groundtruth:
pattern
Indicator
Webshell payload
Presence of X-siLook-CCommaee header with password- value
MOVEi Transfer exploitation
File human2.aspx in wwwroot folder of MOVEi install directory
Stealing sensitive information
List stored files, uscrnames, file paths, etc.
Unauthorized access
Health Check Service as random named MOVEit Transfer user
New MySQL sessions
Crention of new MySQL sessions by attackers
Model Output of Alpaca:
Threat Patterns
Presence of "x-silock-Coomen headerin- 404 Not Found error
File human2.aspe aspx in wwwroot folder'n- Successful exploitation
Attackers cun download and sensitive information
Attackers can insert and delete a new rundom MOVIa Transfer userin- Advisory to check for indicators of unauthorized access over "at least the past days
Model Output of TŪLU:
Cyber Threat
Indicator
Webshell Payload
K-siLock-Coommeee header with specifie password-lie like value
MOVEit Transfer Exploitation
Preseuce of aspp file in wwwroot folder of MOVEit install directory
Model Output of ODIE:
Pattern
Indicator
Determines if X-si mocckkCCmmnnt header is populated with specific password- like value hefore
Webshell payload
returning 404 'Not Found" error
Presence of human22.sppp aspx file in wwwroot folder of MOVEit install directory, allows attackers to
MOVEit Transfer
download and stcal sensitive information, insert and delete new random named MOVEit Transfer
exploitation
user, and create new MySQL sevsions
Advice from Progress
Check for indicators of unauthorized access "at least the past 30 days", well other remediation recommendations
Software
Po9
Model Output of GPT4:
Pattern
Indicutor
Inbound request containing header named X-siI. boil.ock-Coomenn. returns 404 'Not l'ound" error not populated with specific password-like value
Webshell payload
Instanees of Rkapid--bsserve MOVEit Transfer exploitation involve the presence of the file human2. aspx in the wwwroot folder of the MOVTi install directory
File Presence
Successfiul exploitation allows al altuckers to downloud and steal semsitive information such as list of stored files, usernames who uploaded the files, file paths, configured Azure Blob Storage accounts, data from Azure Blob Storage containers, data from servers, etc.
Stolen Information
Attackers can insert and delete a new randem named MOVEit Transfer user with the login name Health Check Service' and create new MySQI. sessions
Unauthorized sccess
Remediation Recommendations
Progress Software advises MOVEit customers to check for indicators of unauthorized access over at least the past 30 days
Figure 10: Case study medium-levd instruction.

---

Instruction: What are the recipes can make for my picky toddler who only cat food with orange color? Provide the recipe, ingredients, and cooking time.
Text:
PUMPKIN BREAD: Preheut oven to 325 degrees Fahrenheit (165 degrees Celsius) and grease 9x loaf pun. Tn large bowl, beat the oil and honey together together with a whisk. Add the eggs and whisk until blended. (If your coconut oil solidifies on contact with cold ingredients, simply let the bowl rest in warm place for few minules, like on top of your stove, or warm it for about seconds in microwave.) Add the pumpkin purée, milk, pumpkin spice, baking soda, vanilla and salt, and whisk to blend. Lastly, switch to a big spoon and stir in the flour, just until combined. Some lumps are ok! If you's re udding any additionul mix-ins, gently fold them in now. Pour the batter into your greased loaf pan and sprinkle lightly with cinnamon. If you'd d like pretty swirled effect, run the tip of a knife across the batter in zig-zag pattern. Bake for 55 to 60 minutes, until toothpick inserted into the center comes out clean (typically, i haven't added any mix-ins, my bread is done at 55 minutes; if have added mix-ins, it needs closer to minutes). Let the bread cool the loaf pan for 10 minutes, then carcfully transfer the bread to a vooling rack to cool for 20 minutes before slicing.
TURKEY - CARROT ROLL-UP: Place the tortilla on a cutting bourd. Spread the hummus over the entire surface of the tortilla, On the bottom 2/3 of the tortilla, sprinkle the carrots, Pat down to make them stick in the hummus. Lay the turkey lunch meat over the carrots, but avoid the top 1/3 of the tortilla. Starting with the bottom of the tortilla, with the currots and turkey, tightly roll the tortilla all the way the lop. The lop 1/3 section of hummmus will act like glue to help the tortilla stay in place. Cut into 1-inch sections. The whole process needs 10 min.
Strawberry Applause: In a large stockpot, bring the cherries, water, cinnamon, and vanilla to a boil, Reduce to a simmer over medium-low heat, cover, and cook for 15 minutes, stirring occasionally. Add in the strawberries and stür. Cover and then cook for another 15 minutes. Let cool slightly. For a chunky sauce, mash the cherries and strawberries with the back of wooden spoon. For smoother sauce, transfer all ingredients into blender or food processor und puree for 30-60 seconds or until you achieve your desired consistency. I pureed mine for 45 seconds. If pureeing for baby, puree for 1-2 minutes or until very smooth, Transfer to an air-tight container and serve.
Groundtruth:
Recipe Name
Ingredients
Cooking Time
PUMPKIN BREAD
oil, honey, pure milk, pumpkin spice, baking soda, vanilla, salt, flour
80 min
TURKEY CARROT ROLL-UP
Tortilla, hummus, shredded carrots, turkey lunch meat
10
Model Output of Alpaca:
Recipe Name
Ingredients
Cooking Time
Pumpkin Bread
cup pumpkin purčc
5S-60 minutes
Turkey Carrot Ro
Tortilla
20 min
Strawberry Applause
Cherries, water, cinnamon, vanilla
minutes
B09
Model Output of TŪlu:
Recipe Name
Ingredients
Cooking Time
Pumpkin Bread
Oil, honey, eggs, pumpkin purée, milk, pumpkin spice, baking soda, vanilla, salt
55-60 minutes
Turkey Carrot Roll-Up
Tortilla, hummus, carrots, turkey lunch meat
min
Strawberry Applause
Cherries, water, cinnamon, vanilla
15 minutes
Model Output of ODIE:
Recipe Name
Ingredients
Cooking Time
Oil, honey, eggs, pumpkin purée, milk, pumpkin spice, baking soda, vanilla, sult, flour
Pumpkin Bread
55-60 minutes
Turkey Carrot Roll-Up
Tortilla, hummus, currots, turkey lunch meat
10 min
15 minutes + mashing/purese time
Strawberry Applause
Cherries, water, cinmanon, vanilla, strawberries
P
Model Output of GPT4
Recipe Name
Ingredients
Cooking Time
Oil, honey, egus, pumpkin purée, milk, pumpkin spice, baking soda, vanilla, salt, flour, cinnamon
Pumpkin Bread
55-60 minates
Turkey Carrot Roll-Up
Tortilla, hummus, currots, turkey lunch meat
No cooking required
Strawberry Applause
Cherries, water, cinnumon, vanilla, strawberries
30 minutes
Figure 11: Case study on hard-level instruction with reasoning.

---

Step
Prompt
want to generate some real-world examples of information extraction that users would let the Al help with.
Specifically, example should contain the following two items:
1. Instruction: [User input, usually refers extracting some desired information from a given text.]
2. Domain: [The domain to which the user query belongs.]
The following several examples:
Example I:
Instruction:
Fixed Instruction
Domain:
Example 2:
Generation
Instruction:
Domain:
Following format of examples ahove, would like you help me generate more new examples that meet following requirements:
These examples should various domains.
These examples should described different styles.
3. The generated domains not overlap with the above example.
Give information extraction instruction, aim to generate some real-world example text from which the information be extracted.
Specifically, each instruction includes the domain of background text and the type of extracted information. So would like the generuted include the information that needs to be extracted.
follow the domain targeted by the instruction and explicitly
The following are several examples:
Background Text Generation
Example
Instruction:
Example
Instruction:
Text:
Following format of examples above, would like you to help me generate the text following instruction:
Instruction:...
Give background generate instruction which mentions extracting the information from it. But don't point what kind of information should extracted.
The following several examples:
Example
Text:
Open Instruction Generation
Instruction:
Example 2:
Text:
Instruction:
Following the format of the examples above, would like you to help me generate instruction following text:
Text:
Given instructions, paraphrase them hy in different descriptive ways and make them like professional request keep key elements. So the outputs should paraphrased instructions. Remember output index newline.
Instruction Paraphrasing
Sentence 1:
Sentence
Sentence 3:
Sentence
Table 5: Prompts used for training data generation.

---

Step
Prompt
Given an information extraction instruction and hackground text, extract the information as markdown table. If the instruction specifies the type information to be extracted, ensure to follow the instruction. Otherwise, let this table include as many columns as possible. And keep the content brief.
The following are several examples:
Example
Instruction:
Table Generation
Text:
Table:
(Direct)
Example 2:
Instruction:
Text:
Table:
Following the format of the examples above, I would like you to help me extract the table for the following instruction and text: Instruction:...
Text:
Given an information extraction instruction and the background text, extract the information as markdown table and produce a paragraph as the explamation. If the instruction specifies the type of information to be extracted, ensure to follow the instruction. Otherwise, let this table include as many columns as possible, And keep the content brief.
The following are several examples:
Example 1:
Instruction:
Text:
Explanation:
Table Generation
Table:
Example 2:
(Co'T)
Instruction:
Text:
Explanation:
Table:
Following the format of the examples above, would like you help extract the table for the below instruction and Please adopt a step-by-step approach: generate comprehensive explanation the first step, followed by table extraction the second step:
Instruction:
Text:
Table Prompts for two methods table generation: Direct and CoT.
Difficulty Medium
Category Fixed
Source
Models
Easy
Hard
Open
Generate
Retrieve
Overall
Data
Open-Source Models
ALPACA
27.42
27.61 29.80
25.19 30.90
30.90 33.20
15.29 18.62
27.16 30.65
27.11 30.07
TOLU
29.45
27.98
512K
Our Models
ODIE-DIRECH
34.06 28.70
32.98
31,20
37.54
18. 98
31.79
33.02
32.76
7.5K
ODIE-COT
26.09
27.99
29.81
21.13
25.24
28.33
27.60
7.1K
Proprictary Models
CHATGPT 35.03 33.41
28.98 30.63
32.71
35.04 35.63
23.29 24.27
29.22 30.25
33.11 33.03
32.21 32.50
GPT-4
33.80
Table 7: Full results of table header evaluation.
The metric is F1 (%) of the exact matching score.

---

Difficulty Medium
Category
Source Retrieve
Models
Easy
Hard
Fixed
Open
Generate
Overall
Data
Open-Source Models
ALPACA
64.66
59.31 66.73
54.53
65.89
45.69
59.57
59.90 69.47
59.80
TULU
74.18
67.02
77.78
49.26
69.39
69.44
512K
Our Models
78.26
74.21
67.97
83.59
51.67
72.41
74.22
73.82
7.5K
Filtering
80.08
72.72
67.02
83.54
51.37
72.82
73.83
73.61
7.5K
ODIE-CoT
72.54
63.19
64.45
72.32
54.17
64.70
67.54
66.81
7.1K
Filtering
69.35
58.88
64.81
68.97
65.12
64.11
7.1K
Proprietary Models
CHATGPT 80.41 77.77
70.53 70.58
72.45 75.51
81.69 82.06
57.86 57.78
75.02 78.36
74.41 73.29
74.49
GPT-4
74.47
Table 8: Full results of table header evaluation. The metric is F1 (%) of the soft matching score.
Difficulty Medium
Category
Source
Models
Easy
Hard
Fixed
Open
Generate
Retrieve
Overall
Data
Open-Source Models
ALPACA
10.13
7.35 6.62
11.50 11.56
10.15
7.13
10.03
9.11 7.68
9.33
52K
TÜLU
7.75
7.10
12.83
9.17
7.96
512K
Our Models
ODIE DIRECT
17.74 14.22
15.35
14.28
15,59
16.19
16.42
15.80
ODIE-COT
10.98
10.73
11.41
13.39
11.64
11.96
11.90
7.1K
Proprietary Models
CHATGPT
16.11 15.21
16.38 12.52
12,96
15.34
14.76 13.89
12.47 12.04
16. 13
15.23
GPT-4
11.88
12.97
13.57
13.22
Table 9: Full results of table content evaluation.
The metric is F (%) of the exact matching score.
Difficulty Medium
Category
Source
Models
Easy
Hard
Fixed
Open
Generate
Retrieve
Overall
Data
Open-Source Models
ALPACA
$2.09
40.00 60.05
49.57 63.50
48.32
41.73 57.83
48.08 65.41
46.45 62.34
46.86 63.14
52K
TULU
66.77
64.57
512K
Our Models
ODIE-DIRECH DIRECT
67.28
68.18
64.23
69.94
56.52
65.49
67.05
66.68
7.5K
ODIE ODIE-COT
63.67
60.76
65.83
63.97
60.59
67.79
61.78
63.21
7.1K
Proprietary Models
CHATGPT 71.71 75.70
69.48 73.09
70.65
72.61 76.67
64.50
70.72 74.45
70.59 74.83
GrT-4
75.99
76.08
Table 10: Full results of table content evaluation. The metric F, (%) of the semantic similarity score.


Reviews:
[{"review_id": "dFmiJ00UqJ", "reviewer": "Reviewer_REDP", "summary": "", "questions": "- As mentioned in line 232-243, the open instructions are generated given the background text, would this setting guarantee most of the fields has related information mentioned in the background text. Are they any arrangement to make sure the instruction tuning dataset contains text with N/A ground-truth structure?\n- What would be the performance of supervised models that are directly fine-tuned on table extraction datasets? It would be nice to have a comparison to provide a better understanding of the capability of zero-shot LLM.\n- It’s great to see the proposed dataset covers a lot of domains. How are those domains annotated? Is the domain categorization from some existing ontology/definition?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and for acknowledging the importance of our task formulation, resource generation, and empirical results. We're encouraged that you find our approach and datasets to be valuable contributions to the field of information extraction. Your feedback affirms the utility and effectiveness of our model and methodology. We appreciate the opportunity to clarify and address the concerns you've raised. Below, we provide a point-by-point response to the issues mentioned.\n\n**R1. About Task Formulation**\n\nThank you for raising your concern. By claiming on-demand information extraction, we aim to equip the average users with the ability to extract the desired information by specifying a natural language instruction.  \nRegarding the intention behind a table output format, it’s indeed to present extracted information in a manner that's immediately accessible and comprehensible to end-users. The users can customize the table based on the type of information they seek to fulfill their demands. This eliminates the constraints of traditional systems that abide by a rigid format. \nHowever, we still admit the current task formulation still has room for improvement in processing more complex structures as the extracted output for fulfilling more challenging user needs. \n\n**R2. About Instruction Tuning vs. Fine-Tuning**\n\nWe appreciate this perspective and conducted an experiment accordingly. Specifically, we train the LLaMA 7b model without any specific user instructions and provide just a general system instruction, “extract a table from the text”. To ensure the comprehensive evaluation, we evaluate the model outputs on five different metrics, including two for table header and three for table content (please find more explanations about evaluation metrics in the third answer). The experimental results are listed below in the table.  It shows that the direct version of our full model significantly outperforms the variant trained without user instructions, which indicates distinct advantages of instruction tuning. It provides the model with explicit prompts about the desired output structure, enabling better alignment with user requirements like customized table headers or specific expected table content. \n\n| Method          | Header  |   | Content |     |     |\n|-----------------|------------------------|------------------------|-----------------|-------------------------|-------------------------| \n|     |  **Exact matching**  |  **Semantic Similarity** | **Exact matching** | **Semantic Similarity** | **RougeL** |\n| **ODIE**            | **32.76**                   | **73.82**                   | **15.80**           | **66.68**                    | **45.93**                    | \n| w/o instruction | 28.17                   | 62.35                   | 7.53            | 53.72                    | 26.90                    | \n\n\n**R3. About Evaluation Metrics**\n\n__Initial Choice of ROUGE-L__: ROUGE-L was initially selected for its ability to measure n-gram overlap, offering a more flexible evaluation than exact-match metrics. Furthermore, its wide use in previous instruction-tuning works [1, 2] also ensures our work is comparable to prior research. \n\n__Incorporating two new metrics__: To address this limitation, we adopt two more metrics, which are exact matching and semantic similarity , alongside ROUGE scores in our evaluation. Specifically, exact matching requires two table cells to be exactly the same while semantics similarity uses SentenceBERT [3] to calculate the cosine similarity as the semantic similarity score between two cells. Both metrics are micro F1 scores. \n\n__Experimental results__: The results are shown below in the three tables for both table header and content (Semantics similarity for table headers has been included in our submission). The results demonstrate that our proposed model continues to achieve superior performance under these new metrics. \n\n__(1) Exact matching for table header__\n\n| Models              | Difficulty |   |   | Category |  | Source Type |  | Overall    |\n|---------------------|------------------|--------------------|------------------|-------------------------|------------------------|------------------------|------------------------|------------|\n|               | **Easy** | **Medium** | **Hard** | **Fixed Header** | **Open Header** | **Generate** | **Retrieve** | **Overall**    |\n| Open-Source Models  |                  |                    |                  |                         |                        |                        |                        |             |\n| alpaca              | 27.42            | 27.61              | 25.19            | 30.90                   | 15.29                  | 26.80                  | 27.16                  | 27.11       |\n| TULU                | 29.45            | 29.80              | 30.90            | 33.20                   | 18.62                  | 27.98                  | 30.65                  | 30.07       |\n| **Our Models**          |                  |                    |                  |                         |                        |                        |                        |             |\n| **ODIE-direct**        | **34.06**            | **32.98**              | **31.20**            | **37.54**                   | **18.98**                  | **31.79**                  | **33.02**                  | **32.76**       |\n| **ODIE-cot**            | 28.70            | 26.09              | 27.99            | 29.81                   | 21.13                  | 25.24                  | 28.33                  | 27.60       |\n| Proprietary Models  |                  |                    |                  |                         |                        |                        |                        |             |\n| Turbo               | 35.03            | 28.98              | 32.71            | 35.04                   | 23.29                  | 29.22                  | 33.11                  | 32.21       |\n| GPT4                | 33.41            | 30.63              | 33.80            | 35.63                   | 24.27                  | 30.25                  | 33.03                  | 32.50       |\n\n\n\n__(2) Exact matching for table content__\n| Models              | Difficulty |   |   | Category |  | Source Type |  | Overall    |\n|---------------------|------------------|--------------------|------------------|-------------------------|------------------------|------------------------|------------------------|------------|\n|               | **Easy** | **Medium** | **Hard** | **Fixed Header** | **Open Header** | **Generate** | **Retrieve** | **Overall**    |\n| Open-Source Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| Alpaca              | 10.13            | 7.35               | 11.50            | 10.15                   | 7.13                   | 10.03                  | 9.11                   | 9.33       |\n| Tulu                | 7.75             | 6.62               | 11.56            | 7.10                    | 12.83                  | 9.17                   | 7.68                   | 7.96       |\n| **Our Models**          |                  |                    |                  |                         |                        |                        |                        |            |\n| **ODIE-Direct**         | **17.74**            | **15.35**              | **14.28**            | **15.59**                   | **16.19**                  | **13.83**                  | **16.42**                  | **15.80**      |\n| **ODIE-CoT**            | 14.22            | 10.98              | 10.73            | 11.41                   | 13.39                  | 11.64                  | 11.96                  | 11.90      |\n| Proprietary Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| turbo               | 16.11            | 16.38              | 12.96            | 15.34                   | 14.76                  | 12.47                  | 16.13                  | 15.23      |\n| GPT4                | 15.21            | 12.52              | 11.88            | 12.97                   | 13.89                  | 12.04                  | 13.57                  | 13.22      |\n\n\n__(3) Semantic similarity for table content__\n| Models              | Difficulty |   |   | Category |  | Source Type |  | Overall    |\n|---------------------|------------------|--------------------|------------------|-------------------------|------------------------|------------------------|------------------------|------------|\n|               | **Easy** | **Medium** | **Hard** | **Fixed Header** | **Open Header** | **Generate** | **Retrieve** | **Overall**    |\n| Open-Source Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| Alpaca              | 52.09            | 40.00              | 49.57            | 48.32                   | 41.73                  | 48.08                  | 46.45                  | 46.86      |\n| Tulu                | 66.77            | 60.05              | 63.50            | 64.57                   | 57.83                  | 65.41                  | 62.34                  | 63.14      |\n| **Our Models**          |                  |                    |                  |                         |                        |                        |                        |            |\n| **ODIE-Direct**         | **67.28**            | **68.18**             | 64.23            | **69.94**                   | 56.52                  | 65.49                  | **67.05**                  | **66.68**      |\n| **ODIE-CoT**            | 63.67            | 60.76              | **65.83**            | 63.97                   | **60.59**                  | **67.79**                  | 61.78                  | 63.21      |\n| Proprietary Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| turbo               | 71.71            | 69.48              | 70.65            | 72.61                   | 64.50                  | 70.20                  | 70.72                  | 70.59      |\n| GPT4                | 75.70            | 73.09              | 75.99            | 76.67                   | 69.13                  | 76.08                  | 74.45                  | 74.83      |\n\n\n__Correlation Analysis__: To further study different metrics on our proposed task, we analyze the correlation between these automatic metrics (Exact Match, Semantic Similarity, ROUGEL) and human evaluations. The analysis results are listed below, including the Pearson, Spearman, and Kendall coefficients. The results indicate that, for table header, these three metrics are highly correlated with human evaluation. But for table content, semantic similarity and RougeL can indeed be more reliable metrics for this task compared with exact matching.  \n\n| Metrics     | Pearson  | Spearman | Kendall  |\n|------------------|----------|----------|----------|\n| [Header] Exact Matching      | 0.640    | 0.609    | 0.494    |\n| [Header] Semantic Similarity      | **0.817**    | **0.769**    | **0.637**    |\n| [Header] RougeL    | 0.694    | 0.735    | 0.608    |\n| [Content] Exact Matching     | 0.338    | 0.375    | 0.272    |\n| [Content] Semantic Similarity     | **0.764**    | **0.705**    | **0.558**    |\n| [Content] RougeL   | 0.713    | 0.704    | 0.554    |\n\n\nWe sincerely appreciate the reviewers for highlighting the evaluation issue, allowing us to conduct a thorough study that provides more detailed insights and resolves ambiguities stemming from the use of a single metric. These new findings will be definitely added in the next paper version. We will be happy to discuss at the platform if the reviewers have further questions or comments.\nAs mentioned in the limitation section, we admit the challenges of proposing comprehensive evaluation metrics for on-demand information extraction. Proposing better metrics is a critical aspect that we aim to explore in our future work. \n\n\n\n**R4. About Empty Cell for Open Instructions**\n\nThank you for this query. \n\nAn open instruction does not specify the type of header, i.e., “position” and “salary”, but rather employs relatively vague requirements, such as “key information” (as mentioned in Lines 235-239). For such an instruction, the model aims for an output inclusive of all relevant details from the input text  (Lines 260-264). Thus, most fields derive directly from the text. That said, we do allow tables to have empty cells for instances where certain information might be missing from the input (Lines 286-289).  According to statistics of our test data, there are a total of 1950 table cells, of which 103 are N/A, making up 5% of the total.\n\nTo guarantee that the groundtruth tables are informative, we have implemented filtering mechanisms for both the generated training and annotated test data. This ensures that tables with excessive empty cells (indicative of missed information) are not part of our dataset, as detailed in Lines 286-289. \n\n\n\n\n**R5. Compare with supervised table extraction method**\n\nThanks for asking this question. We further conduct experiments on a supervised table extraction approach, text2table [4], from ACL 2022. Specifically, we adopt the public implementation of this bart-based model trained with the default setting on the WikiTableText dataset and tested with table constraints on our test data. The experiment results are shown below, which demonstrate our method obviously outperforms this baseline. We think it’s because our method can benefit from customized table structures and contents following flexible user instructions.  \n\n| Method          | Header  |   | Content |     |    \n|-----------------|------------------------|------------------------|-----------------|-------------------------|\n|     |  **Exact matching**  |  **Semantic Similarity** | **Exact matching** | **Semantic Similarity** | \n| **ODIE**            | **32.76**                   | **73.82**                   | **15.80**           | **66.68**                    | \n| Text2table | 15.02                   | 39.73                   | 2.01            | 13.32                    | \n\n\n\n**R6. About Domain Annotations**\n\nWe're glad you appreciate the diversity of our dataset. For the training data, domains are autonomously generated by ChatGPT in separate batches, with an emphasis on ensuring each batch contains distinct domains for wide-ranging coverage (Lines 209-221). As for the test data, leveraging GPT-4's vast knowledge, we initially generated a pool of 1,000 instructions spanning diverse domains. From this, annotators meticulously selected 150 samples that are not only representative but also align with the genuine needs of different user groups (as described in Lines 319-330).\n\n\n**Reference**: \n\n[1] Yizhong Wang*, Swaroop Mishra*, Pegah Alipoormolabashi, Yeganeh Kordi et al.Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. EMNLP 2022.\n\n[2] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022a. Self-instruct: Aligning language model with self generated instructions. ACL 2023.\n\n[3] Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. EMNLP-IJCNLP 2019.\n\n[4] Xueqing Wu, Jiacheng Zhang, Hang Li. Text-to-Table: A New Way of Information Extraction. ACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Task formulation, resources and modeling for a more flexible information extraction formulation are quite needed\n- Automatically generated instruction tuning dataset and manually created test set with 150 instances would be helpful\n- Empirical results show that tuning on the generated dataset would improve the IE results on the table extraction evaluation dataset", "reasons_to_reject": "- The task formulation of “dynamic table extraction with the more flexible header“is not quite matched with the claim of “on-demand information extraction“, it would be necessary to define the scope better\n- Not sure how much gain is from instruction tuning compared to simply fine-tuning with task-specific data points without instruction info. As we can expect fine-tuning a LM on task data (even without instruction) can still lead to some performance improvement. A comparison with fine-tuning on task instances without instruction would be helpful to quantify such a gain\n- Using ROUGE score to evaluate IE results leads to an unsatisfactory setting. As the legal values of certain table fields could be very similar, but incorrect. If the exact match results could be reported it would be very helpful to clear such ambiguity.", "questions_to_authors": "- As mentioned in line 232-243, the open instructions are generated given the background text, would this setting guarantee most of the fields has related information mentioned in the background text. Are they any arrangement to make sure the instruction tuning dataset contains text with N/A ground-truth structure?\n- What would be the performance of supervised models that are directly fine-tuned on table extraction datasets? It would be nice to have a comparison to provide a better understanding of the capability of zero-shot LLM.\n- It’s great to see the proposed dataset covers a lot of domains. How are those domains annotated? Is the domain categorization from some existing ontology/definition?", "paper_topic_and_main_contributions": "This paper proposes to curate instruction tuning resources and train a instruction following model to perform “on-demand information extraction“ task (which is a table extraction task with a dynamic table head requirement). The authors propose an automatic pipeline to generate (instruction, text, table) data points for instruction tuning, and tune a LLaMA model using the newly generated InstructIE dataset. They also propose a manually curated evaluation dataset with 150 samples to evaluate the performance in terms of head and table content. Evaluation results show that the proposed model achieves improvement compared with open-source LLM, which is the base model before fine-tuning on the new instruction tuning dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FC9xeYTqMT", "reviewer": "Reviewer_Z3zM", "summary": "", "questions": "A. You make much of the multi-domain aspect of the experimental data.\nHow are domains defined?  I believe you explicitly specify the domain in\ngenerating synthetic data.  Is this the only domain \"knob\" you employ?\n\nB. Could you say more about the retrieval of real texts and how those\nwere sorted by domain?\n\nCl Your description of the human evaluation is a little vague.  I'm\ninferring that a complete datapoint in that evaluation was a complete\nmodel output (all fields and extractions).  Does \"correct and\nsatisfying\" mean the model got all headers and fields exactly correct,\nor just \"correct enough for my purposes\"?\n", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review, which recognizes the importance of our proposed task. We appreciate your acknowledgment of the replicability of our training procedures and the potential for broader community engagement. Your comments on the accessibility of using a smaller model like Llama 7B are particularly encouraging, and we're pleased you see the value in that. We value your insights and have addressed the concerns you raised. Below, we provide a point-by-point response to the issues mentioned.\n\n**R1. About application scenarios**\n\nWe deeply appreciate your acknowledgment of our motivation. We think the proposed task, On-Demand Information Extraction, has the vast applicability across industries and its potential to streamline tasks, making information extraction more efficient and user-centric. \n\nFor example,  doctors can pinpoint needed health metrics from a patient's history, like \"medications and dosages over the last year for the patient,\" for quicker, more targeted information. Also, legal professionals can speed up reviews by specifying queries like \"extract clauses on breach of the contract penalties.\"\n\nIn these real world scenarios, existing IE approaches cannot be directly applied to and handle them because these approaches tend to extract exhaustively and rely on predefined ontologies. But equipped with our setting, an average user can just input a natural language instruction to define and extract the desired information from texts. \n\n\n\n\n**R2. About Evaluation Metrics**\n\nWe sincerely appreciate the reviewer for highlighting the evaluation issue, allowing us to conduct a thorough study that provides more detailed insights and resolves ambiguities stemming from the use of a single metric. \n\nSpecifically, we supplemented exact matching and semantic similarity (same as cosine similarity) as an evaluation metric for both header and content, as recommended by other reviewers. The results validate our proposed model's commendable performance even with these stringent metrics. \n\n__(1) Exact matching for table header__\n\n| Models              | Difficulty |   |   | Category |  | Source Type |  | Overall    |\n|---------------------|------------------|--------------------|------------------|-------------------------|------------------------|------------------------|------------------------|------------|\n|               | **Easy** | **Medium** | **Hard** | **Fixed Header** | **Open Header** | **Generate** | **Retrieve** | **Overall**    |\n| Open-Source Models  |                  |                    |                  |                         |                        |                        |                        |             |\n| alpaca              | 27.42            | 27.61              | 25.19            | 30.90                   | 15.29                  | 26.80                  | 27.16                  | 27.11       |\n| TULU                | 29.45            | 29.80              | 30.90            | 33.20                   | 18.62                  | 27.98                  | 30.65                  | 30.07       |\n| **Our Models**          |                  |                    |                  |                         |                        |                        |                        |             |\n| **ODIE-direct**        | **34.06**            | **32.98**              | **31.20**            | **37.54**                   | **18.98**                  | **31.79**                  | **33.02**                  | **32.76**       |\n| **ODIE-cot**            | 28.70            | 26.09              | 27.99            | 29.81                   | 21.13                  | 25.24                  | 28.33                  | 27.60       |\n| Proprietary Models  |                  |                    |                  |                         |                        |                        |                        |             |\n| Turbo               | 35.03            | 28.98              | 32.71            | 35.04                   | 23.29                  | 29.22                  | 33.11                  | 32.21       |\n| GPT4                | 33.41            | 30.63              | 33.80            | 35.63                   | 24.27                  | 30.25                  | 33.03                  | 32.50       |\n\n\n\n__(2) Exact matching for table content__\n| Models              | Difficulty |   |   | Category |  | Source Type |  | Overall    |\n|---------------------|------------------|--------------------|------------------|-------------------------|------------------------|------------------------|------------------------|------------|\n|               | **Easy** | **Medium** | **Hard** | **Fixed Header** | **Open Header** | **Generate** | **Retrieve** | **Overall**    |\n| Open-Source Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| Alpaca              | 10.13            | 7.35               | 11.50            | 10.15                   | 7.13                   | 10.03                  | 9.11                   | 9.33       |\n| Tulu                | 7.75             | 6.62               | 11.56            | 7.10                    | 12.83                  | 9.17                   | 7.68                   | 7.96       |\n| **Our Models**          |                  |                    |                  |                         |                        |                        |                        |            |\n| **ODIE-Direct**         | **17.74**            | **15.35**              | **14.28**            | **15.59**                   | **16.19**                  | **13.83**                  | **16.42**                  | **15.80**      |\n| **ODIE-CoT**            | 14.22            | 10.98              | 10.73            | 11.41                   | 13.39                  | 11.64                  | 11.96                  | 11.90      |\n| Proprietary Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| turbo               | 16.11            | 16.38              | 12.96            | 15.34                   | 14.76                  | 12.47                  | 16.13                  | 15.23      |\n| GPT4                | 15.21            | 12.52              | 11.88            | 12.97                   | 13.89                  | 12.04                  | 13.57                  | 13.22      |\n\n\n__(3) Semantic similarity for table content__\n| Models              | Difficulty |   |   | Category |  | Source Type |  | Overall    |\n|---------------------|------------------|--------------------|------------------|-------------------------|------------------------|------------------------|------------------------|------------|\n|               | **Easy** | **Medium** | **Hard** | **Fixed Header** | **Open Header** | **Generate** | **Retrieve** | **Overall**    |\n| Open-Source Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| Alpaca              | 52.09            | 40.00              | 49.57            | 48.32                   | 41.73                  | 48.08                  | 46.45                  | 46.86      |\n| Tulu                | 66.77            | 60.05              | 63.50            | 64.57                   | 57.83                  | 65.41                  | 62.34                  | 63.14      |\n| **Our Models**          |                  |                    |                  |                         |                        |                        |                        |            |\n| **ODIE-Direct**         | **67.28**            | **68.18**             | 64.23            | **69.94**                   | 56.52                  | 65.49                  | **67.05**                  | **66.68**      |\n| **ODIE-CoT**            | 63.67            | 60.76              | **65.83**            | 63.97                   | **60.59**                  | **67.79**                  | 61.78                  | 63.21      |\n| Proprietary Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| turbo               | 71.71            | 69.48              | 70.65            | 72.61                   | 64.50                  | 70.20                  | 70.72                  | 70.59      |\n| GPT4                | 75.70            | 73.09              | 75.99            | 76.67                   | 69.13                  | 76.08                  | 74.45                  | 74.83      |\n\n\n__Correlation Analysis__: To further study different metrics on our proposed task, we analyze the correlation between these automatic metrics (Exact Match, Semantic Similarity, ROUGEL) and human evaluations. The analysis results are listed below, including the Pearson, Spearman, and Kendall coefficients. The results indicate that, for table header, these three metrics are highly correlated with human evaluation. But for table content, semantic similarity and RougeL can indeed be more reliable metrics for this task compared with exact matching.  \n\n| Metrics     | Pearson  | Spearman | Kendall  |\n|------------------|----------|----------|----------|\n| [Header] Exact Matching      | 0.640    | 0.609    | 0.494    |\n| [Header] Semantic Similarity      | **0.817**    | **0.769**    | **0.637**    |\n| [Header] RougeL    | 0.694    | 0.735    | 0.608    |\n| [Content] Exact Matching     | 0.338    | 0.375    | 0.272    |\n| [Content] Semantic Similarity     | **0.764**    | **0.705**    | **0.558**    |\n| [Content] RougeL   | 0.713    | 0.704    | 0.554    |\n\n\nThese new findings will be definitely added in the next paper version. We will be happy to discuss at openreview platform if the reviewers have further questions or comments.\n\nAs mentioned in the limitation section, we admit the challenges of proposing comprehensive evaluation metrics for on-demand information extraction. Proposing better metrics is a critical aspect that we aim to explore in our future work. \n\n\n\n\n**R3. About Future Direction**\n\nThanks for pointing out these questions. We think there are at least 5 promising directions based on this work (three of them mentioned in the limitation section). \n*Hybrid Method Combination*: We've assessed Direct and CoT separately but haven't studied their synergistic effects, which could enhance performance and provide new insights.\n*textbf{Evaluation Metrics*: Current metrics focus on similarity to ground truth. A more granular set of metrics is needed for table quality and organization.\n*Contextual Inference and Complex Instructions*: Room exists for improvement in contextual understanding and handling intricate queries, to make the system more accessible.\n*Exploration of Complex Output Formats*: Traditional methods use fixed output structures; investigating more complex and flexible formats can cater to varied user needs.\n*Integration with Other Systems: On-Demand IE can enhance other systems, such as database management tools, and virtual assistants. Future work should target seamless integration via standardized APIs and performance tweaks.\n\n\n\n**R4. About Choosing Domain**\n\nWe're glad you appreciate the diversity of our dataset. For the training data, domains are autonomously generated by ChatGPT in separate batches, with an emphasis on ensuring each batch contains distinct domains for wide-ranging coverage (as mentioned in LINES 209-221). As for the test data, leveraging GPT-4's vast knowledge, we initially generated a pool of 1,000 instructions spanning diverse domains. From this, annotators meticulously selected 150 samples that are not only representative but also align with the genuine needs of different user groups. These domain labels are further revised and merged considering label semantics. (as described in LINES 319-330).\n\n\n\n**R5. About Retrieval of Real Texts**\n\nGiven a domain, we first manually identified a representative type of text for that domain, like legal documents for law. Using a search engine like Newbing, we accessed the desired text type, ensuring they contained ample and suitable information elements for extraction. We also ensured text lengths between 100 and 1,000 words to meet the large language models' input length restrictions.\n\n\n**R6. About Human Evaluation**\n\nIn our human evaluation, \"correct and satisfying\" implies that the extracted information was both semantically accurate and aligned with the user's query. It doesn't necessitate an exact match for every header and field but ensures the output effectively meets the user's intent. For example, it would be regarded as correct if the groundtruth is “2 min” but the model output is “2 minutes”.  Because we think capturing the semantic essence is flexible and paramount compared with rigid exact matches, especially in diverse real-world scenarios.\n\n**Reference**\n\n[1] Yizhong Wang*, Swaroop Mishra*, Pegah Alipoormolabashi, Yeganeh Kordi et al.Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks. EMNLP 2022.\n\n[2] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022a. Self-instruct: Aligning language model with self generated instructions. ACL 2023.\n\n[3] Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. EMNLP-IJCNLP 2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think the contributions of the paper are substantial.  Early\nevidence seems to suggest that instruction-trained LLMs are not\nparticularly strong in IE tasks, one possible reason being that they\nhaven't been exposed to tasks of the right type.  The paper tends to\nconfirm this hypothesis and offers a replicable procedure (including a\nnew dataset of IE instructions?) to correct this shortcoming.\n\nThe paper is well written, with good separation between paper body and\nappendix.  The training procedure nicely addresses difficulties that\ncan be encountered in working with synthetic text as training data, a\nprocedure that is clearly described.  It should be possible for other\nresearchers to replicate the effects observed in the paper (if not the\nspecific outcomes) and build on this research.  I believe that many\nwill want to do so.\n\nIt is appealing that the work was done with a \"small\"\ninstruction-trained LLM.  An ongoing pain point in NLP research is the\nreliance on proprietary models for some of the more interesting\nresearch directions.  Llama 7B can be run on a beefy laptop, making it\nfeasible for the community to adapt and extend this work.\n", "reasons_to_reject": "Although I appreciate the paper's motivation to provide the benefits\nof IE to end users, I don't really see the need to advocate for a new\ntask.  I have difficulty imagining scenarios in which end users would\nwant to use such a capability.  IE is typically a means to many\ndifferent ends, making technicians the main constituency likely to be\ninterested in improvements to IE technology.  Thus, I think the\ncentral emphasis of the paper is off-target.  This is notable work, in\nmy opinion mainly because it helps the field out of ruts it's become\ntoo comfortable with, with a heavy focus on particular IE domains over\nothers and a conceptualization (NER, relation extraction, event\nextraction) that made sense initially, but may now be an impediment to\nprogress.  The paper offers a direction that enables rapid deployment\nof IE to underserved domains, of which there is probably an infinite\nsupply.\n\nBecause of this perspective, I found the performance evaluations\nsomewhat underwhelming, with their use of soft matching, SBERT\nsimilarity, ROUGE, etc.  These metrics make sense for more qualitative\ntasks, where goodness is difficult to characterize precisely, but not\nso much for IE.  (It's probably reasonable to evaluate the open\nvariant of the paper's task, which I do find interesting, in such a\nfashion.) The paper acknowledges this shortcoming in the Limitations\nsection (which is thoughtfully written).  I think this flaw is\nmaterial.  It is really hard to tell how this work moves the needle\nwith respect to the baseline of supervised extraction.  I think it\ndoes, but have no good evidence for that opinion.\n\nThe paper doesn't offer a lot of ideas about where to go next.  What\nare useful directions, now that we have a model like this?  Can you\nimprove its accuracy somehow?  Or are you heading in the direction of\napplications, as implied by the user-facing task you define.  There\nare a couple \"future work\" ideas masquerading as Limitations, such as\nmore rigorous accuracy evaluations and some kind of interleaving of\ndirect and CoT approaches.  These ideas and any other next steps merit\nmention in the body of the paper.\n", "questions_to_authors": "A. You make much of the multi-domain aspect of the experimental data.\nHow are domains defined?  I believe you explicitly specify the domain in\ngenerating synthetic data.  Is this the only domain \"knob\" you employ?\n\nB. Could you say more about the retrieval of real texts and how those\nwere sorted by domain?\n\nCl Your description of the human evaluation is a little vague.  I'm\ninferring that a complete datapoint in that evaluation was a complete\nmodel output (all fields and extractions).  Does \"correct and\nsatisfying\" mean the model got all headers and fields exactly correct,\nor just \"correct enough for my purposes\"?\n", "paper_topic_and_main_contributions": "The paper seeks to democratize information extraction (IE), addressing\na task it calls \"on-demand information extraction,\" essentially a\nzero-shot document-level table construction task based on textual\nprompts provided by putative end users.  The paper describes a system\ncapable of performing two variants of this task, one in which the\nextraction targets (fields or columns) are specified, and one in which\nthey are not.  In development of this system, the paper creates a new\ndata source of synthetic extraction tasks spanning a large number of\ndomains and describes the process by which it is used for instruction\ntraining of a small model in the GPT family (Llama 7B).  Extensive\nexperiments benchmark the system against a range of alternatives in\nthe same family, showing that it outperforms models not trained in the\nsame fashion, including much larger models--although performance tends\nto lack behind ChatGPT and GPT4.\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "A more succinct statement of contributions would be helpful.  If you\nintend to release InstructIE, so state.  I suspect it would excite\ninterest.\n\nShould the (b) label in Figure 5 read \"Evaluation for table contents\"?\n"}}, {"review_id": "Ny0KSZO0Wk", "reviewer": "Reviewer_i6Bg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review and for recognizing the importance of putting user demand at the forefront in on-demand information extraction. We appreciate your positive remarks on the novel task setting and our efforts in constructing a dedicated instruction-tuning dataset to support and benchmark the ODIE task. We value your insights and have addressed the concerns you raised. Below, we provide a point-by-point response to the issues mentioned.\n\n**R1. About Task Format**\n\nWe appreciate your feedback on the relational table format. We chose the relational table format as it is most intuitive to end-users and many standard IE tasks (such as NER and relation extraction) can be expressed in this format. This output format can also be used to represent ad-hoc user needs such as a combination of entities, entity attributes and relations, a task that would require a complex pipeline of IE components according to current task definitions. \n\nHowever we do acknowledge that our format does not encompass all current IE tasks in a convenient way. For event extraction with multiple event types, if the ontology is small, we can represent the output using a table with headers such as “event type, trigger, role 1, argument 1, role 2, argument 2…” and each row would be an event instance.  Another option would be to run our model multiple times, each with a different event type. We will mention this limitation in the paper and be more precise about the scope of our model. \n\n\n\n**R2. About Evaluation Metrics**\n\nThanks for pointing out the concern. \n\n**Clarification on the adopted metrics**: We would like to clarify that the cosine similarity score in our paper is permutation-invariant for columns and rows. Because each table cell from the groundtruth first is aligned with one cell in the model outputs according to semantic similarity, and then we average the maximum similarity scores for all cells as the final scores. Meanwhile, ROUGE-L score is permutation-invariant for rows since it’s computed for each row. Similarly, each row from the groundtruth is aligned with one row in the model outputs for score calculation. \n\n**Supplementing more metrics**: Meanwhile, as recommended by other reviewers, we supplemented exact matching and semantic similarity (same as cosine similarity) as an evaluation metric for both header and content. The results validate our proposed model's commendable performance even with these stringent metrics. \n\n__(1) Exact matching for table header__\n\n| Models              | Difficulty |   |   | Category |  | Source Type |  | Overall    |\n|---------------------|------------------|--------------------|------------------|-------------------------|------------------------|------------------------|------------------------|------------|\n|               | **Easy** | **Medium** | **Hard** | **Fixed Header** | **Open Header** | **Generate** | **Retrieve** | **Overall**    |\n| Open-Source Models  |                  |                    |                  |                         |                        |                        |                        |             |\n| alpaca              | 27.42            | 27.61              | 25.19            | 30.90                   | 15.29                  | 26.80                  | 27.16                  | 27.11       |\n| TULU                | 29.45            | 29.80              | 30.90            | 33.20                   | 18.62                  | 27.98                  | 30.65                  | 30.07       |\n| **Our Models**          |                  |                    |                  |                         |                        |                        |                        |             |\n| **ODIE-direct**        | **34.06**            | **32.98**              | **31.20**            | **37.54**                   | **18.98**                  | **31.79**                  | **33.02**                  | **32.76**       |\n| **ODIE-cot**            | 28.70            | 26.09              | 27.99            | 29.81                   | 21.13                  | 25.24                  | 28.33                  | 27.60       |\n| Proprietary Models  |                  |                    |                  |                         |                        |                        |                        |             |\n| Turbo               | 35.03            | 28.98              | 32.71            | 35.04                   | 23.29                  | 29.22                  | 33.11                  | 32.21       |\n| GPT4                | 33.41            | 30.63              | 33.80            | 35.63                   | 24.27                  | 30.25                  | 33.03                  | 32.50       |\n\n\n\n__(2) Exact matching for table content__\n| Models              | Difficulty |   |   | Category |  | Source Type |  | Overall    |\n|---------------------|------------------|--------------------|------------------|-------------------------|------------------------|------------------------|------------------------|------------|\n|               | **Easy** | **Medium** | **Hard** | **Fixed Header** | **Open Header** | **Generate** | **Retrieve** | **Overall**    |\n| Open-Source Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| Alpaca              | 10.13            | 7.35               | 11.50            | 10.15                   | 7.13                   | 10.03                  | 9.11                   | 9.33       |\n| Tulu                | 7.75             | 6.62               | 11.56            | 7.10                    | 12.83                  | 9.17                   | 7.68                   | 7.96       |\n| **Our Models**          |                  |                    |                  |                         |                        |                        |                        |            |\n| **ODIE-Direct**         | **17.74**            | **15.35**              | **14.28**            | **15.59**                   | **16.19**                  | **13.83**                  | **16.42**                  | **15.80**      |\n| **ODIE-CoT**            | 14.22            | 10.98              | 10.73            | 11.41                   | 13.39                  | 11.64                  | 11.96                  | 11.90      |\n| Proprietary Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| turbo               | 16.11            | 16.38              | 12.96            | 15.34                   | 14.76                  | 12.47                  | 16.13                  | 15.23      |\n| GPT4                | 15.21            | 12.52              | 11.88            | 12.97                   | 13.89                  | 12.04                  | 13.57                  | 13.22      |\n\n\n__(3) Semantic similarity for table content__\n| Models              | Difficulty |   |   | Category |  | Source Type |  | Overall    |\n|---------------------|------------------|--------------------|------------------|-------------------------|------------------------|------------------------|------------------------|------------|\n|               | **Easy** | **Medium** | **Hard** | **Fixed Header** | **Open Header** | **Generate** | **Retrieve** | **Overall**    |\n| Open-Source Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| Alpaca              | 52.09            | 40.00              | 49.57            | 48.32                   | 41.73                  | 48.08                  | 46.45                  | 46.86      |\n| Tulu                | 66.77            | 60.05              | 63.50            | 64.57                   | 57.83                  | 65.41                  | 62.34                  | 63.14      |\n| **Our Models**          |                  |                    |                  |                         |                        |                        |                        |            |\n| **ODIE-Direct**         | **67.28**            | **68.18**             | 64.23            | **69.94**                   | 56.52                  | 65.49                  | **67.05**                  | **66.68**      |\n| **ODIE-CoT**            | 63.67            | 60.76              | **65.83**            | 63.97                   | **60.59**                  | **67.79**                  | 61.78                  | 63.21      |\n| Proprietary Models  |                  |                    |                  |                         |                        |                        |                        |            |\n| turbo               | 71.71            | 69.48              | 70.65            | 72.61                   | 64.50                  | 70.20                  | 70.72                  | 70.59      |\n| GPT4                | 75.70            | 73.09              | 75.99            | 76.67                   | 69.13                  | 76.08                  | 74.45                  | 74.83      |\n\n\n__Correlation Analysis__: To further study different metrics on our proposed task, we analyze the correlation between these automatic metrics (Exact Match, Semantics Similarity, ROUGEL) and human evaluations. The analysis results are listed below, including the Pearson, Spearman, and Kendall coefficients. The results indicate that, for table header, both of these three metrics are highly correlated with human evaluation. But for table content,  semantic similarity and RougeL can indeed be more reliable metrics for this task compared with exact matching.  \n\n| Metrics     | Pearson  | Spearman | Kendall  |\n|------------------|----------|----------|----------|\n| [Header] Exact Matching      | 0.640    | 0.609    | 0.494    |\n| [Header] Semantic Similarity      | **0.817**    | **0.769**    | **0.637**    |\n| [Header] RougeL    | 0.694    | 0.735    | 0.608    |\n| [Content] Exact Matching     | 0.338    | 0.375    | 0.272    |\n| [Content] Semantic Similarity     | **0.764**    | **0.705**    | **0.558**    |\n| [Content] RougeL   | 0.713    | 0.704    | 0.554    |\n\nWe sincerely appreciate the reviewers for highlighting the evaluation issue, allowing us to conduct a thorough study that provides more detailed insights and resolves ambiguities stemming from the use of a single metric. These new findings will be definitely added in the next paper version. We will be happy to discuss at openreview platform if the reviewers have further questions or comments.\nAs mentioned in the limitation section, we admit the challenges of proposing comprehensive evaluation metrics for on-demand information extraction. Proposing better metrics is a critical aspect that we aim to explore in our future work.\n\n\n\n\n**R3. Table contents being exact text pieces**\n\nIn our problem setting, we did not enforce that table contents must be spans of the original text. Our instructions (especially those in the hard set) require some amount of reasoning and summarization over the text so the final result might not be available as a text span.\n\n\n\n**R4. Rating System in Human Evaluation**\n\nWe understand the need for clarity in this aspect. In our setting, \"correct\" and \"partly correct\" are not strictly bound to exact matches. We allow the extracted contents to be semantically aligned with the original text, acknowledging that different phrasings might convey equivalent meanings. \nFor example, it would be regarded as correct if the groundtruth is “2 min” but the model output is “2 minutes”.  Because we think capturing the semantic essence is flexible and paramount compared with rigid exact matches, especially in diverse real-world scenarios.\n\n\n\n\n**R5. Size of the Test Set**\n\nThanks for raising your concern. Our 150 test instances involve 61 various domains. These instances are both representative and resonate with the genuine requirements of varied user groups. Additionally (as mentioned in LINES 403-414), the scale of our test set aligns with existing instruction-tuning benchmarks, such as 129 in Anthropic, 156 in the Koala evaluation, and 188 in Open Assistant.\n\n\n**R6. Availability of Dataset**\n\nWe have submitted the dataset as Supplementary Materials, along with different model outputs and human evaluation results.  Please feel free to take a look if you are interested. Also, the data, model outputs, and human evaluations will be open-sourced after notification.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper focuses on an important research question: on-demand information extraction, which puts user demand ahead. The schemas of the extraction are explicitly specified or implicitly inferred from user instructions. This is a more realistic and applicable task setting in the era of large language models compared to the traditional IE settings where the schema is predefined.\n2. This paper proposes a large instruction-tuning dataset to support and benchmark the ODIE task, including synthesized training data (14k) and human-annotated test data (150).\n", "reasons_to_reject": "Though well-motivated, the task formulation is not well-defined and should be justified:\n1. The output format is formulated as a relational table, where the headers are the target attributes and one row refers to one record. However, there are some output cases that this output format can't cover. For example, the multi-event extraction may outputs events of different event types, resulting in multiple relational tables. I suggest the authors clearly define the scope of the extraction or define a more flexible output format.\n2. The cosine similarity (to evaluate the table header) and the ROUGE-L (to evaluate the table content) are both inappropriate. One intuition is that the relational tables are permutation-invariant for columns and rows, but neither of the two metrics is.\n3. It's not sure whether the extracted items in the table content should be exact text pieces in the context (same as the traditional IE settings).\n\nMINOR ASPECTS\n\n1. The rating system of human evaluation is vague. For example, what are \"correct\"/\"partly correct\" table headers? If the \"correct\" means the exact match, there would be no need for manual assessment.\n2. The 150 test samples are insufficient compared to the training set spanning a broad range of 83 domains.\n3. Will the INSTRUCTIE dataset be publicly available?\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a new task, On-Demand Information Extraction (ODIE), to fulfill the personalized needs of users. ODIE aims to extract the desired content and present it in a structured tabular format by following user instructions. The table headers are either user-specified (the \"fixed\" setting) or model-inferred (the \"open\" setting). A novel dataset named INSTRUCTIE is constructed, consisting of synthesized training data (14k) and human-annotated test data (150). The LLaMA-7B finetuned on the training data outperforms existing open-source models but is still inferior to proprietary models (ChatGPT and GPT-4). The main contributions lie in the novel task setting and the instruction-tuning dataset. However, it's not sure whether the dataset will be publicly available.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "● Line 208: missing appendix.\n\n● Line 250: what is a \"style\"?\n\n● Line 295: Given -> given\n\n● Line 441: thendescribe -> then describe\n\n● Figure 4: missing the y-axis.\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a prompt-based approach to IE where users extract tables of text from documents by providing instructions.  A Llama-7B model is fine-tuned on a dataset of instructions, input documents and output tables generated by ChatGPT.  The approach that is taken seems to be similar to Alpaca, however the model is specifically targeting IE based on user instructions.

The paper is well written, and the experiments seem convincing.  Reviewers raised some concerns, including the motivation for the new task and regarding the chosen evaluation metrics.
Decision: Accept-Main