IEPILE: Unearthing Large-Scale Schema-Based Information Extraction Corpus (2024)

Abstract: Large Language Models (LLMs) demonstrate remarkable potential across various domains; however, they exhibit a significant performance gap in Information Extraction (IE). Note that high-quality instruction data is the vital key for enhancing the specific capabilities of LLMs, while current IE datasets tend to be small in scale, fragmented, and lack standardized schema. To this end, we introduce IEPILE, a comprehensive bilingual (English and Chinese) IE instruction corpus, which contains approximately 0.32B tokens. We construct IEPILE by collecting and cleaning 33 existing IE datasets, and introduce schema-based instruction generation to unearth a large-scale corpus. Experimentally, IEPILE enhance the performance of LLMs for IE, with notable improvements in zero-shot generalization. We open-source the resource and pre-trained models, hoping to provide valuable support to the NLP community.

text: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 127–146 August 11-16, 2024 ©2024 Association for Computational Linguistics
Large Language Models (LLMs) demonstrate remarkable potential across various domains; however, they exhibit a significant performance gap in Information Extraction (IE). Note that high-quality instruction data is the vital key for enhancing the specific capabilities of LLMs, while current IE datasets tend to be small in scale, fragmented, and lack standardized schema. To this end, we introduce IEPILE, a comprehensive bilingual (English and Chinese) IE instruction corpus, which contains approximately 0.32B tokens. We construct IEPILE by collecting and cleaning 33 existing IE datasets, and introduce schema-based instruction generation to unearth a large-scale corpus. Experimentally, IEPILE enhance the performance of LLMs for IE, with notable improvements in zero-shot generalization. We open-source the resource and pre-trained models, hoping to provide valuable support to the NLP community.

heading: 1 Introduction
text: Large Language Models (LLMs) have achieved significant breakthroughs in multiple Natural Language Processing (NLP) tasks (Du et al., 2022; Touvron et al., 2023b; Jiang et al., 2023; Zhao et al., 2023; Pu et al., 2023; Yang et al., 2024; Wu et al., 2023; Wang et al., 2023c; Fei et al., 2024). However, recent studies (Li et al., 2023a; Ma et al., 2023; Xu et al., 2023; Wadhwa et al., 2023; Wan et al., 2023; Gao et al., 2023; Li et al., 2023b; Jiao et al., 2023; Huang et al., 2023; Wang et al., 2024) indicate a significant performance gap in the task of Information Extraction (IE) when utilizing LLMs. (Lee et al., 2022a; Gao et al., 2023) further illustrate that the major reason may lie in limited highquality, large-scale data corpus. Concretely, most IE datasets are often limited in size, scattered in
* Equal Contribution. † Corresponding Author.
distribution, and lack standardization in schema1. Faced with these limitations, there is an urgent need to collect instruction data in a unified and automated manner to build a high-quality, large-scale IE corpus. To this end, we collect and clean various existing IE datasets to obtain a comprehensive bilingual IE instruction dataset named IEPILE2. During the corpus construction, we find existing methods for constructing IE instruction data suffer from two issues for generalizable IE: 1) Schema Query Disparity: There may be inconsistency in the number of schema queries within instruction between training and evaluation which can harm model generalization; 2) Semantic Confusion: The co-occurrence of semantically similar schemas within instructions may confuse the model. Thus, we introduce a schema-based instruction generation strategy. We first construct a hard negative schema dictionary to promote the more frequent occurrence of semantically similar schema in instructions. Then, we introduce batched instruction generation, dynamically limiting the number of schemas queried in each instruction to split_num, which not only addresses the issue of performance degradation due to inconsistent numbers of schema queries during training and evaluation, but also enhances the robustness when dealing with semantically confusing schema. Finally, we obtain IEPILE which contains approximately 0.32B tokens.
By fine-tuning a selection of the latest prominent models (Yang et al., 2023; Touvron et al., 2023b; Bai et al., 2023) on the IEPILE dataset, we show that LLMs with IEPILE can yield better zeroshot performance than baselines. This achievement not only verifies the effectiveness of the IEPILE dataset but also provides a framework for creating IE datasets in other domains.
1We refer to the schema as pre-defined types of entities, relations, events (arguments and roles), etc.
2IEPILE adhere to the CC BY-NC-SA 4.0 license except for ACE2005 which adheres to the LDC User Agreement.
127

heading: 2 IEPILE
text: In this section, we introduce the construction of IEPILE and provide details in Appendix B.

heading: 2.1 Data Collection and Cleaning
text: To broadly cover various domains and meet the practical demands, we collect datasets necessary for IE from multiple data sources. Our corpus mainly involves bilingual data (Chinese and English) and focuses on three principal categories of IE tasks: Named Entity Recognition (NER), Relation Extraction (RE), and Event Extraction (EE). In total, we gather 26 English datasets and 7 Chinese datasets. We also employ standardization procedures to maintain data quality and format uniformity, involving format unification, instance deduplication, and the exclusion of low-quality data.

heading: 2.2 Schema-Based Instruction Generation
text: We concentrate on instruction-based information extraction (IE), a methodology that incorporates three crucial elements to compose an instruction: 1) Task Description, a template utilized to distinguish between different IE tasks; 2) Input Text, the source text to be extracted; and 3) Schema sequence, which defines the information that the model is supposed to extract, including entity types, relations, events, etc. Among these, the schema sequence is critical as it reflects the specific extraction requirements and is dynamically variable. Therefore, the construction of the schema sequence within an instruction holds critical significance.
Positive and Negative Schema Mechanism in Instructions. Firstly, we define schemas that actually exist within the input text as positive schemas and those that do not appear as negative schemas.
As illustrated in Figure 1, the “location contains” present in the annotation is a positive schema, while all other schemas from the predefined label set L are negative schemas. Traditional IE frameworks, which are treated as sequence labeling tasks, take text as input and produce a label for each token as output, without involving the concept of positive or negative schemas within the model’s input. However, in the era of generative IE, represented by models like UIE (Lu et al., 2022a), introduce the concept of integrating a schema sequence (refers to as Structural Schema Instructor, or SSI) in the model’s input to guide its output, restricting the range of output to the SSI. The method necessitates including the entire predefined label set of a dataset as the SSI to guide the model’s output during inference. As a result, if the SSI during the training contains only positive schemas, the model will tend to generate corresponding answers for every label within the SSI during inference. Therefore, to make the model explicitly reject generating outputs for negative schemas, it is necessary to incorporate negative schemas into the SSI.
In this paper, the schema sequence included in the instructions follows the concept of SSI. However, we observe that existing research (Wang et al., 2023b; Xiao et al., 2023) tends to adopt a rather crude schema processing strategy when constructing instructions, meaning that all schemas within a predefined label set are used to build the instructions. This approach potentially entails two significant issues: 1) Inconsistency in the number of schema queries within instruction between training and evaluation. For example, the model’s performance will decrease if it is trained on about
Algorithm 1 Schema-Based Instruction Generation Require: Text S, Predefined label set L, Positive schema set
Pos_L, Number of schemas to split split_num Ensure: Set of Instructions
Step 1: Initialize Hard Negative Schema Dictionary K for all schema in L do K[schema]← SEMANTIC-SIMILAR(schema,L) end for Step 2: Obtain Hard Negative Schemas Hard_L← ∅ for all schema in Pos_L do
Hard_L← Hard_L ∪ K[schema] end for Other_L← L− Pos_L−Hard_L Other_L← RANDOM-SELECT(Other_L, split_num)
Neg_L← Hard_L ∪Other_L L′ ← Neg_L ∪ Pos_L Shuffle L′ to obtain a randomized sequence Step 3: Batched Instruction Generation Instructions← [] num_batches← ⌈ |L′|
split_num⌉ for i← 1 to num_batches do
Batch← SEQUENTIAL-SELECT(L′, split_num, i)
Instructions ← Instructions ∪ GENERATE-INSTRUCTION(Batch)
end for
20 schema queries but tested with either 10 or 30, even if the training and evaluation schemas are similar in content. 2) Inadequate differentiation among schemas in the instructions. For example, semantically similar schemas like “layoffs”, “depart” and “dismissals”, may present co-occurrence ambiguities that could confuse the LLMs. Such schemas should co-occur more frequently within the instruction. Therefore, we introduce: 1) Hard Negative Schema Construction; and 2) Batched Instruction Generation. Detailed information can be found in Figure 1 and Algorithm 1.
Hard Negative Schema Construction. As illustrated in Figure 1, assume that dataset D possesses a predefined label set L. For a given text S, the schemas present in its annotation constitute the positive schema set Pos_L, while others form the negative schema set Neg_L. In our analysis, we discover that the primary cause of model mistakes stems from the semantic ambiguity of the schema. In traditional approaches, the Neg_L is simply defined as L − Pos_L. However, they overlook a critical aspect: it is important to pay special attention to negative schemas that are semantically similar to positive schemas. Inspired by the theory of contrastive learning, we propose the concept of a hard negative schema dictionary K, where each key represents a unique schema and
the associated value is a collection of schemas that are semantically similar to the key schema. The hard negative schemas are constructed by querying GPT-4 and manually reviewing them. Based on this, we define the hard negative schema set as Hard_L = K[Pos_L], and the other negative schema set as Other_L = L−Pos_L−Hard_L. The final Neg_L is constituted by Hard_L and a small subset of Other_L. Through this strategy, we not only present semantically similar schemas more frequently within the instruction but also reduce the number of training instances without sacrificing model performance.
Batched Instruction Generation. Subsequently, we obtain the final schema set L′ = Pos_L + Neg_L. We employ a batched instruction generation method, dynamically limiting the number of schemas inquired in each instruction to the number of split_num, which ranges between 4 and 6. Therefore, L′ will be divided into |L′|/split_num batches for querying, with each batch querying split_num schemas. Consequently, even if the number of schemas inquired during the evaluation phase differs from that of training, the batched mechanism allows us to distribute the inquiries across split_num schemas, thereby mitigating the decline in generalization performance.

heading: 2.3 Data Statistics
text: Based on the aforementioned methods, we obtain the IEPILE dataset, which includes roughly 2 million instruction entries and approximately 0.32B to-
kens (utilizing the Baichuan2 tokenizer). Figure 2 displays the distribution of domains and source datasets within the IEPILE, including 33 datasets spanning multiple domains such as general, news, finance, and biomedical. Additionally, Table 12 provides examples of instructions and outputs for 3 different tasks within the IEPILE.

heading: 3 Experiments
text: Based on IEPILE, we fine-tune several latest prominent models, then compare their zero-shot generalization capabilities against a range of baseline models. Results of the full supervision evaluation and training details are described in Appendix C.

heading: 3.1 Experimental Settings
text: Evaluation Metrics: We employ span-based Micro-F1 as the metric for measuring model performance. Baselines: We select a range of strong models for comparative analysis, which include UIE (Lu et al., 2022a), LLaMA2-13B-Chat (Touvron et al., 2023b), Baichuan2-13B-Chat (Yang et al., 2023), Qwen1.5-14B-Chat (Bai et al., 2023), Mistral-7B-Instruct-v0.2 (Jiang et al., 2023), ChatGPT (Ouyang et al., 2022), GPT-4 (OpenAI, 2023), LLaMA3-8B-Instruct, InstructUIE (Wang et al., 2023b), YAYI-UIE (Xiao et al., 2023). Zeroshot Benchmark: We collect 13 datasets that are not present in the training set. OneKE: Additionally, we perform full-parameter fine-tuning of the alpaca2-chinese-13B model utilizing IEPILE and other proprietary information extraction datasets. This paper also reports its results; for more detailed information, please refer to Appendix C.2.

heading: 3.2 Main Results
text: In Tables 1 and 2, we report the zero-shot performance across three tasks and two languages. Overall, after training with the IEPILE, the models achieve better results in the majority of tasks. We believe the success is due to the hard negative schema construction and batched instruction generation strategy, which can mitigate the train-eval mismatch and semantic ambiguity for the diverse schema. We also observe that IEPILE-models are slightly behind GPT-4 in English NER. We hypothesize that the marginal gap may be attributed to GPT-4’s exposure to a vast corpus of similar data during its training. Moreover, it is essential to note that InstructUIE focuses on English data while IEPILE incorporates both English and Chinese data. This disparity in data may influence the capability of the model in English, potentially reducing the performance. Additionally, OneKE achieves the best results in nearly all zero-shot evaluation tasks. We attribute this success to the enhancements brought by full parameter fine-tuning. 3.3 Analysis Inconsistency in the Number of Schema Queries Hurt Generalization. We investigate the impact on model performance when different numbers of schema queries are used during the training and
evaluation. We train the Baichuan2 using fullschema instructions on 3 datasets: Ontonotes (18 schemas), DuIE2.0 (49 schemas), and ACE2005 (33 schemas). For the evaluation, we test the model using two strategies: one with the full set of schema queries and another with a fixed set of 10 schema queries. The results depicted in Figure 3 (a) indicate that the mismatch in the number of schema queries during evaluation significantly reduces the model’s performance. Further analysis of the model’s outputs reveals that the model always tends to generate outputs for each inquiry. We hypothesize that the number of schema queries is one of the key factors affecting the generalization ability. The model needs to first adapt to the number of schema inquiries that are rare during the training and then adapt to the unseen schema.

heading: Inadequate Differentiation Among Schemas
text: Lead to Semantic Similar Confusion. We also evaluate the impact of removing the “Hard Negative Schema Dictionary” on the performance of Baichuan2-IEPILE, with particular attention to schemas that are hard to differentiate. According to the results in Figure 3 (b), we notice that the hard negative schema dictionary plays a relatively limited role in the NER task, which may be due to the clear boundaries inherent to entity recognition. However, the utilization of the hard negative schema dictionary notably enhances model performance in the DuIE2.0 and DuEE1.0 datasets. We observe that semantically similar and easily confused schemas frequently appeared in the model’s outputs, such as predicting “dismissal” and “resignation” in the event of “layoff”. Therefore, processing instructions that are semantically prone to confusion poses significant challenges, and the hard negative schema dictionary plays a crucial role in bolstering model robustness and improving the accuracy of predictions.

heading: 4 Conclusion and Future Work
text: In this paper, we introduce IEPILE, by collecting and cleaning existing Information Extraction (IE) datasets and utilizing a schema-based instruction generation strategy. Experimental results indicate that IEPILE can help enhance the zero-shot generalization capabilities of LLMs in instruction-based IE. In the future, we will continue to maintain the corpus and try to integrate new resources including open-domain IE, and document-level IE.

heading: Limitations
text: From the data perspective, our study primarily focuses on schema-based IE, which limits our ability to generalize to human instructions that do not follow our specific format requirements. Additionally, our work is limited to two languages and does not address Open Information Extraction(Open IE), though we plan to extend to more languages and Open IE scenarios in the future. From the model’s perspective, our research evaluates limited models, along with a few baselines due to the computation resources. Theoretically, IEPILE can be applied to any other LLMs, such as ChatGLM (Du et al., 2022) and Gemma (Mesnard et al., 2024).

heading: Ethical Considerations
text: In this paper, we strictly adhered to the standards and principles of ethics. All data collected are from publicly available materials, ensuring the transparency and legality of the research. We thoroughly review the data, verifying the legitimacy of their sources and compliance with their usage, thus avoiding any infringement on personal privacy or involvement with unauthorized information.

heading: Acknowledgements
text: We would like to express our sincere gratitude to the anonymous reviewers for their thoughtful and constructive feedback. This work was supported by the National Natural Science Foundation of China (No. 62206246, No. NSFCU23B2055, No. NSFCU19B2027), the Fundamental Research Funds for the Central Universities (226-2023-00138), Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Yongjiang Talent Introduction Programme (2021A-156-G), and Information Technology Center and State Key Lab of CAD&CG, Zhejiang University. This work was supported by Ant Group and Zhejiang University - Ant Group Joint Laboratory of Knowledge Graph.

heading: A Related Work
text: A.1 Information Extraction Datasets Large-scale pre-trained corpora are crucial for the effectiveness of LLMs, providing a wealth of knowledge and a foundation for language comprehension. At the same time, the annotated data for information extraction (IE) also holds its importance. Although the field of IE has accumulated a considerable amount of annotated data (Walker et al., 2006; Riedel et al., 2010; Sang and Meulder, 2003; Luan et al., 2018; Gui et al., 2023), these datasets are often limited in size, scattered in distribution, and lack standardization in schema. Faced with these limitations, there is an urgent need for generating instruction data through unified and automated methods to bridge the gap presented by the current absence of centralized, large-scale IE instruction datasets. In this paper, we concentrate on instruction-based IE scenarios. We develop a comprehensive, schema-rich instruction dataset for IE by collecting and cleaning existing IE datasets, called IEPILE. IEPILE is designed to enhance the adaptability and processing capabilities of LLMs for different IE tasks, simultaneously strengthening their generalization skills to extract from new domains and schemas. A.2 Information Extraction Models Recently, LLMs (Brown et al., 2020; Ouyang et al., 2022; Touvron et al., 2023a,b) demonstrate their exceptional versatility and generalization capabilities across a variety of downstream tasks (Vilar et al., 2023; Hegselmann et al., 2023). Particularly in the domain of IE, these models have the potential to tackle many challenges previously encountered in research (Zheng et al., 2017; Li et al., 2020a; Paolini et al., 2021; Lu et al., 2022b; Lou et al., 2023; Chen et al., 2022b, 2024), such as adaptability issues when dealing with unseen labels. Some studies (Wei et al., 2023; Wang et al., 2023a; Xie et al., 2023) make significant performance gains in low-resource settings by designing prompt-based frameworks and leveraging models like ChatGPT for in-context learning. Moreover, research efforts such as InstructUIE (Wang et al., 2023b), PIVOINE (Lu et al., 2023), and YAYI-UIE (Xiao et al., 2023), which employ instruction-tuning of open-source LLMs, also achieve notable successes on IE. Additional research explore areas such as prompt learning (Zhang et al., 2023a), guidelines (Sainz et al., 2023) and synthetic dataset (Amalvy et al., 2023). Despite these advancements, cur-
rent models fine-tuned with instruction data face a major challenge: the coarse schema handling strategies in constructing instructions could potentially impair the models’ capacity for generalization.

heading: B Construction Details of IEPILE

heading: B.1 Data Collection and Clean
text: Data Collection To comprehensively cover various domains and meet the practical demands of information extraction (IE), we collect IE datasets from multiple sources. IEPILE dataset mainly involves bilingual data (Chinese and English) and three IE tasks: Named Entity Recognition (NER), Relation Extraction (RE), and Event Extraction (EE). The English part mainly comes from the benchmark dataset IEINSTRUCTIONS (Wang et al., 2023b), while the Chinese data is similar to the Chinese datasets mentioned in the YAYI-UIE (Xiao et al., 2023). It should be noted that our Chinese dataset collection is conducted concurrently with the aforementioned research.
Specifically, the NER datasets include fifteen English datasets such as ACE2005 (Walker et al., 2006), AnatEM (Pyysalo and Ananiadou, 2014), BC2GM (Kocaman and Talby, 2020), BC4CHEMD (Kocaman and Talby, 2020), BC5CDR (Zhang et al., 2023b), CoNLL2003 (Sang and Meulder, 2003), FabNER (Kumar and Starly, 2022), FindVehicle (Guan et al., 2023), GENIA-Ent (Kim et al., 2003), HarveyNER (Chen et al., 2022a), MIT Movie (Liu et al., 2013), MIT Restaurant (Liu et al., 2013), MultiNERD (Tedeschi and Navigli, 2022), NCBI-Disease (Dogan et al., 2014), Ontonotes (Pradhan and Xue, 2009), and three Chinese datasets including MSRA (Levow, 2006), Resume NER (Zhang and Yang, 2018), CLUE NER (Xu et al., 2020). The RE task encompasses eight English datasets including ADE Corpus (Gurulingappa et al., 2012), CoNLL2004 (Carreras and Màrquez, 2004), GIDS (Jat et al., 2017), KBP37 (Zhang and Wang, 2015), NYT (Riedel et al., 2010), NYT11-HRL (Takanobu et al., 2019), SciERC (Luan et al., 2018), SemevalRE (Hendrickx et al., 2010), and two Chinese datasets, CMeIE (Luan et al., 2018), DuIE2.0 (Hendrickx et al., 2010). The EE task covers three English datasets: ACE2005 (Walker et al., 2006), CASIE (Satyapanich et al., 2020), PHEE (Sun et al., 2022), and two Chinese datasets, DuEE1.0 (Satyapanich et al., 2020), DuEE-fin (Sun et al., 2022). These datasets span various domains such
as general, medical, financial, and more. For more detailed statistical information, please refer to Tables 9, 10 and 11.
Data Cleaning During the data cleaning process, we address each dataset individually. Firstly, we calculate the text overlap within each dataset’s training, validation, and test sets. If a text is discovered to have multiple occurrences within the same file accompanied by inconsistent annotations, we exclude all corresponding instances from the dataset. Secondly, we compare the text overlap between training, validation, and test sets. If texts from the test set appear previously in the training or validation sets, we would exclude these instances from the training and validation sets. Furthermore, we formulate three heuristic rules to eliminate lowquality and meaningless data:
1) Non-alphabetic characters comprising more than 80% of the text;
2) Text length under five characters without any labels;
3) A high prevalence of stopwords such as ‘the,’ ‘to,’ ‘of,’ etc., exceeding 80%.
We believe that the aforementioned cleaning measures will positively affect model training and enhance its performance. Moreover, our efforts unify data formats across various tasks and conduct a thorough audit of each dataset, creating detailed data records that include the volume of data, domains, schemas, and other information. Figure 4 is an example of a data record for Ontonotes.

heading: B.2 Schema-Based Instruction Generation
text: Hard Negative Schema Construction. As illustrated in Figure 1, assume that dataset D possesses a predefined label set L. For a given text S, the schemas present in its annotation constitute the positive schema set Pos_L, while others form the negative schema set Neg_L. Inspired by the theory of contrastive learning, we construct a hard negative schema dictionary K, where each key represents a unique schema and the associated value is a collection of schemas that are semantically similar to the key schema. Consequently, the set of hard negative schema, Hard_L, is defined as K[Pos_L]. However, if Neg_L is composed solely of Hard_L, it would lack a sufficient number of negative instances for the model to learn effectively. Therefore, we define another set of negative schemas, Other_L = L −Hard_L − Pos_L. Ultimately, the Neg_L is composed of Hard_L and a small number of Other_L (roughly split_num). The
rationale behind the development of these hard negatives is two-fold: firstly, to induce a more frequent co-occur of semantically similar schemas within the instructions, and secondly, to reduce the volume of training instances without sacrificing the model’s performance. In the context of a dataset comprising 48 schemas with a given split_num of 4, traditional mode would dictate the creation of 12 unique instructions per data point. However, through the integration of hard negatives, this requisite can be substantially minimized to a mere 3 instructions.
Batched Instruction Generation. Subsequently, we obtain the final schema set L′ = Pos_L + Neg_L. During the instruction generation phase, the role of schemas is critically vital, as it reflects the specific extraction requirements and is dynamically variable. Traditional practices typically integrate the full schema set into the instruction. However, in this study, we employ a batched instruction generation method, dynamically limiting the number of schemas inquired in each instruction to the number of split_num, which ranges between 4 to 6. Therefore, L′ will be divided into |L′|/split_num batches for querying, with each batch querying split_num schemas. Consequently, even if the number of schemas inquired during the evaluation phase differs from that of training, the batched mechanism allows us to distribute the inquiries across split_num schemas, thereby mitigating the decline in generalization performance.
Selection of split_num. In the determination of the optimal range for split_num, our methodology integrates empirical results with an in-depth analysis of dataset characteristics. For a dataset containing N different labels, the theoretical value of split_num should fall within the interval [1, N]. Addressing datasets with heterogeneous label counts, our objective is to identify a split_num value that offers broad applicability across numerous datasets, thus ensuring this value serves as a common divisor for the majority of dataset label counts. For instance, for Named Entity Recognition datasets, we set split_num to 6; for Relation Extraction and Event Extraction datasets, we establish split_num at 4. We also observe that when split_num is 1, the ratio of positive to negative samples significantly impacts model performance, and the corresponding number of training samples becomes vast, affecting efficiency adversely. More crucially, we believe that enumerating multiple schemas in instructions aids the model in more effectively learning to distinguish and identify various schemas, thereby enhancing model performance.
Furthermore, to enhance model robustness and its clear understanding of the dynamically changing schema sequences in instructions, we set the actual number of schema splits within a dynamic range of [split_num // 2, split_num + split_num // 2]. Specifically, if the number of schemas in the last batch is less than half of split_num, it is merged with the previous batch; otherwise, it stands as an independent batch.
Instruction Format The instruction format of IEPILE adopts a structure akin to JSON strings, essentially constituting a dictionary-type string. This structure is comprised of three main components: (1) “instruction”, which is the task description outlining the objective of the instruction’s execution; (2) “schema”, a list of labels that need to be extracted; (3) “input”, the source text from which information is to be extracted. Examples of instructions corresponding to various tasks can be found in Table 12.

heading: B.3 Data Statistics
text: Based on the aforementioned methodologies, we construct a high-quality information extraction instruction dataset known as IEPILE. This dataset contains approximately two million instances and approximately 0.32B tokens. Each instance of IEPILE comprises two fields: “instruction” and
“output”, which are formatted for direct use in the instruction tuning.

heading: C Experiments

heading: C.1 Experimental Settings
text: Evaluation Metrics We employ span-based Micro-F1 as the primary metric for measuring model performance. For the NER task, the model is required to accurately identify the boundaries of entities and their corresponding types. For the RE task, the model must precisely determine the subject and object entities within a relation, as well as the type of relation between them. UIE necessitates predefined entity types; given that the FewRel and Wiki-ZSL datasets do not provide such information, we are unable to evaluate UIE’s performance on these datasets. As for the EE task, we match the event triggers, denoted as Trigger, and the arguments, referred to as Argument, independently.
Baseline models To assess the zero-shot generalization capabilities, we select a range of strong models for comparative analysis:
• UIE (Lu et al., 2022a): is a unified textto-structure generation framework that can model various information extraction (IE) tasks generically.
• LLaMA2-13B-Chat (Touvron et al., 2023b): is a series of LLMs ranging from 7 billion to 70 billion parameters.
• Baichuan2-13B-Chat (Yang et al., 2023): is a collection of multilingual LLMs containing 7 billion and 13 billion parameters.
• Qwen1.5-14B-Chat (Bai et al., 2023): is a comprehensive language model series that encompasses distinct models with varying parameter counts.
• Mistral-7B-Instruct-v0.2 (Jiang et al., 2023): is a 7-billion-parameter LLM.
• ChatGPT (Ouyang et al., 2022): also known as GPT-3.5-turbo, represents the most advanced artificial intelligence language model with chat optimization capabilities to date.
• GPT-4 (OpenAI, 2023): Known as the most powerful closed-source chat model to date.
• LLaMA3-8B-Instruct 3: The latest release in the LLaMA model series, achieving significant improvements across various benchmarks.
• InstructUIE (Wang et al., 2023b): a unified IE framework based on multi-task instruction tuning.
• YAYI-UIE (Xiao et al., 2023): is an endto-end, chat-enhanced, universal information extraction framework that supports both Chinese and English, fine-tuned with instructional prompts for generalized information.

heading: C.2 OneKE
text: We leverage IEPILE, InstructIE (Gui et al., 2023), CMRC (Cui et al., 2019), along with certain proprietary business information extraction datasets from Ant Group, to compile a comprehensive training dataset consisting of 2.5 million instances. Subsequently, we undertake full-parameter fine-tuning of the alpaca2-chinese-13b4 model on this training dataset, resulting in the refined model named OneKE.
Zero-shot Dataset To ensure the validity of the zero-shot evaluation and prevent result bias due to data similarity, we select datasets primarily derived from news and biomedical fields as our training sets. This selection is intended to train the model’s capability for instruction following and schemabased extraction. For the evaluation data, we adopt the 13 cross-domain datasets recommended in IEINSTRUCTIONS and YAYI-UIE, which include: for Named Entity Recognition (NER) tasks, we use the CrossNER (Liu et al., 2021), Weibo NER (Peng and Dredze, 2015), and Boson5; in Relation Extraction (RE) tasks, we choose FewRel (Han et al., 2018), Wiki-ZSL (Chen and Li, 2021), COAE20166, IPRE (Wang et al., 2019), and SKE20207; and for Event Extraction (EE), we include RAMS (Ebner et al., 2020), WikiEvents (Li et al., 2021), CrudeOilNews (Lee et al., 2022b), FewFC (Zhou et al., 2021), and CCF law 8. These
3https://ai.meta.com/blog/meta-llama-3/. 4https://huggingface.co/hfl/
chinese-alpaca-2-13b. 5https://github.com/InsaneLife/ 6https://github.com/Sewens/COAE2016 7https://aistudio.baidu.com/datasetdetail/ 177191 8https://aistudio.baidu.com/projectdetail/ 4201483
datasets cover a wide range of fields including literature, music, law, and oil news. It is noteworthy that these evaluation data sets are not used during the training, ensuring that our evaluation accurately reflects the model’s generalization and adaptation capabilities for unseen domains and unseen schema data in zero-shot information extraction.

heading: C.3 Zero-shot performance on Event Extraction
text: As illustrated in Table 3, the model trained with IEPILE exhibits outstanding performance in zeroshot event extraction (EE) tasks, surpassing other baselines. Notably, in the Chinese EE task, the LLaMA2-IEPILE model’s performance is slightly inferior to YAYI-UIE’s, revealing LLaMA2’s limitations in processing Chinese data. However, in the
English EE task, LLaMA2-IEPILE’s performance is significantly superior to that of similar models. This contrast highlights the potential influence of language type on model performance.

heading: C.4 Hyper-parameter
text: In our research, we select four pre-trained models, Baichuan2-13B-Chat and LLaMA2-13B-Chat, Qwen1.5-14B-Chat, and LLaMA3-8B-Instruct, as the base models for our study. Specifically, we employ the LoRA (Hu et al., 2022) technique and utilize 8 NVIDIA A800 GPUs to perform instruction tuning on our IEPILE dataset. Detailed configurations of the hyperparameters during the fine-tuning process are presented in Table 4.

heading: C.5 Supervision Results
text: Due to limited computational resources, I report only the supervised results for the Baichuan2IEPILE, LLaMA2-IEPILE, and OneKE models. Tables 6, 7, and 8 present our experimental results under a supervised learning setting on the training dataset. Specifically, it can be observed that after training on the IEPILE, the model excels in Named Entity Recognition (NER), Relation Extraction (RE), and Event Detection (ED), ranking top 2 across these tasks. The model’s performance is only slightly behind other baselines in the Event Argument Extraction. Additionally, we record the model’s performance in Chinese NER, RE, and EE tasks, where it demonstrates robust results. In a comprehensive assessment, the IEPILE-trained model showcases performance on par with other models in instruction-based information extraction (IE) tasks and significantly improves performance in zero-shot IE tasks compared to other models. This indicates the significant application prospects and potential of IEPILE in the current field of IE.
C.6 Impact of Potential Dataset Bias on Model Performance and Generalization
During the research, we identify that potential biases introduced by the datasets used can affect the model’s performance and generalization capability. Firstly, biases in the definition of schemas within the datasets have a negative impact on model performance (Huang et al., 2024). In the early stages of training, we observe instability in results due to
mutual interference among multiple datasets that contain the same schemas but with differing definitions. For instance, despite wikiann, wikineural, polyglot-NER, and CoNLL2003 all containing common schemas such as people and organization, they each possess distinct scheme definitions. Consequently, in the later stages, only CoNLL2003 is retained. Secondly, the model demonstrates good generalization when dealing with datasets having schemas similar to those in the training set. As shown in Table 5, despite not being included in the training corpus, the People Daily and ACE2004 NER datasets share similar schemas with the MASR and ACE2005 NER dataset in the training set, and the Baichuan2-IEPILE model is still capable of handling them proficiently. Lastly, the use of common, coarse-grained labels (such as “person” and “organization”) within the IEPILE lead the model, after training, to favor these coarse categories over fine-grained ones (such as “scientist” and “company”) when predicting instructions that included both levels of granularity.
shadow represent the top 2 results.

heading: Future Work
text: Additionally, our work is limited to two languages and does not address Open Information Extraction(Open IE), though we plan to extend to more languages and Open IE scenarios in the future. From the model’s perspective, our research evaluates limited models, along with a few baselines due to the computation resources. Theoretically, IEPILE can be applied to any other LLMs, such as ChatGLM (Du et al. , 2022) and Gemma (Mesnard et al. , 2024)