Universal Information Extraction with Meta-Pretrained Self-Retrieval
Year: 2023

Abstract:
Universal Information Extraction (Universal IE) aims to solve different extraction tasks in a uniform text-to-structure generation manner. Such a generation procedure tends to struggle when there exist complex information structures to be extracted. Retrieving knowledge from external knowledge bases may help models to overcome this problem but it is impossible to construct a knowledge base suitable for various IE tasks. Inspired by the fact that large amount of knowledge are stored in the pretrained language models (PLM) and can be retrieved explicitly, in this paper, we propose MetaRetriever to retrieve task-specific knowledge from PLMs to enhance universal IE. As different IE tasks need different knowledge, we further propose a Meta-Pretraining Algorithm which allows MetaRetriever to quicktly achieve maximum task-specific retrieval performance when fine-tuning on downstream IE tasks. Experimental results show that MetaRetriever achieves the new state-of-the-art on 4 IE tasks, 12 datasets under fully-supervised, low-resource and few-shot scenarios.

heading: 1 Introduction
text: Information extraction (IE) is the task of extracting specific information structures of interest from unstructured text (Andersen et al., 1992; Grishman, 2019). The various IE tasks at hand can be highly heterogeneous, as each task may have different extraction targets, such as entities (Lample et al., 2016), relations (Zheng et al., 2017), and events (Lin et al., 2018). Traditional IE methods tend to be task-specific, resulting in specialized and isolated approaches for different tasks.
In recent years, researchers have made efforts towards universal information extraction, or Universal IE (Lu et al., 2022), which aims to develop
∗Corresponding Author
a unified method capable of solving a range of IE tasks. One proposed approach involves the use of a Structure Extraction Language to express different extraction targets in a unified structure form, and pretrains a Transformer model to generate this unified structure through a sequence-to-sequence process. However, this structure generation process can be limited in its ability to fully utilize contextual semantic correlation between the extracted information, as the Transformer generation process is unidirectional. In cases where the information structure to be extracted is complex, the universal IE model may struggle to generate accurate results.
Previous studies on retrieval-augmented generation (Lewis et al., 2020b; Wang et al., 2021; Cai et al., 2022; Geng et al., 2022) have shown that utilizing external knowledge bases to retrieve taskspecific information can improve the ability of models to generate complex sequences. However, this approach is not practical for Universal IE, as it is almost impossible to build a knowledge base suitable for various IE tasks, and it is also the ultimate goal of IE. Recently, research on knowledge probing (Petroni et al., 2019; Roberts et al., 2020; Zhang et al., 2022; Yu et al., 2022) has demonstrated that large amounts of knowledge are stored in pretrained language models (PLMs) and can be retrieved explicitly. Utilizing this knowledge to enhance IE models could avoid the need for extensive resource construction. In light of these findings, the question arises: can PLMs be used as knowledge bases to retrieve knowledge and improve universal IE models? If so, universal IE models would be able to generate more accurate results.
Based on this idea, in this paper, we propose MetaRetriever, a pretrained model for universal IE. Unlike existing method which generates results in a single step, MetaRetriever utilizes a retrieve-then-extract approach: first, it retrieves
4084
MetaRetriever ((PER: Steve (ORG: Apple))
In 1997, Steve became the CEO of Apple.
Text Knowledge
Retrieval
(a) First: Retrieval task-specific knowledge from the model.
Text
task-specific knowledge from itself, and then it uses this knowledge as additional input to extract information. However, different IE tasks require different knowledge, and simply retrieving information from PLM can lead to irrelevant information being included, which can negatively impact performance. To address this, incorporating metalearning techniques (Finn et al., 2017), we develop a Meta-Pretraining Algorithm (MPA) which allows the model to quickly learn the semantics of extraction targets of different tasks to retrieve the relevant knowledge. Specifically, MPA pretrains the model with a bi-level optimization, in which the model are optimized to achieve maximal task-specific retrieval performance given the extraction targets after the parameters have been updated through a small number of gradient steps on a new IE task. Thus, in the fine-tuning phase, MetaRetriever can learn downstream tasks quickly to retrieve relevant knowledge and disregard irrelevant information.
We conduct experiments on 12 datasets of 4 main IE tasks under fully-supervised, few-shot and low-resource scenarios. Experiments show that MetaRetriever significantly surpasses previous works and achieves the new state-of-the-art. On fully-supervised scenario, MetaRetriever achieves 0.7% gains on average over UIE (Lu et al., 2022). On few-shot and low-resource scenarios, our model outperforms UIE with over 2.0% improvement.
Our contributions can be summarized as follows: (1) We introduce a retrieve-then-extract perspective into universal IE. It retrieves task-specific knowledge from the pretrained language model to improve performance. (2) We propose a metapretraining algorithm to make models fast-adapt
into various downstream IE tasks by retrieving taskspecific knowledge. (3) We construct large-scale pretraining corpus for universal IE and it can promote future research1.

heading: 2 Preliminaries

heading: 2.1 Structured Extraction Language
text: The UIE paper (Lu et al., 2022) presents the concept of the Structured Extraction Language (SEL) for expressing the diverse extraction targets of various IE tasks in a uniform representation. As illustrated in Figure 2a, SEL is a hierarchical key-value structure that comprises three components:
(1) SpotName: the targeted class name of the extracted information pieces in a specific IE task, e.g., “PERSON” in named entity recognition;
(2) AssoName: the targeted class name of the relationship between different information, e.g., “Works For” in relation extraction;
(3) InfoSpan: the text span corresponding to the specific SpotName or AssoName.
Additionally, the colon (“:”) symbol denotes the mapping of the text span to its class name or relationship, and the hierarchical structure is indicated by the use of parentheses (“(” and “)”).

heading: 2.2 Structural Schema Instructor
text: To make the model be aware of which extraction targets the IE task focus on, UIE further proposes Structural Schema Instructor (SSI), a schema-based prompt. As shown in Figure 2b, SSI first uses several special symbols ([spot], [asso]) to denote SpotNames and AssoNames. Then these SpotNames and AssoNames are concatenated before each text sequence with a “[text]” symbol as a prompt. UIE takes the prompted text as input to generate the linearized SEL to achieve information extraction.
1Our code and constructed corpus are released at https://github.com/AlibabaResearch/DAMO-ConvAI/ tree/main/metaretriever

heading: 3 Methodology
text: Our proposed MetaRetriever achieves universal IE via a retrieve-then-extract manner. It first retrieves task-specific knowledge from itself according to the input text and then generates the extraction results based on the retrieved knowledge. To make MetaRetriever only retrieve task-specific knowledge and neglect other distracting knowledge, we further proposed a meta-pretraining algorithm based on meta-learning technique. In this section, we first introduce how MetaRetriever works via the retrieve-then-extract manner and then detail the meta-pretraining algorithm.

heading: 3.1 Retrieve-then-Extract
text: Given that providing task-specific knowledge can be beneficial for IE and that pretrained models have acquired a significant amount of knowledge through training on large corpora (Petroni et al., 2019; Roberts et al., 2020; Zhang et al., 2022; Yu et al., 2022), our proposed system, MetaRetriever, utilizes this pretrained knowledge by first retrieving relevant information from the model itself. This retrieved knowledge then serves as an additional input, along with the original input, to produce the final extraction results. The design of MetaRetriever is depicted in Figure 1.

heading: 3.1.1 Retrieval Procedure
text: We use SEL (Lu et al., 2022) to express knowledge for knowledge retrieval and knowledge understanding. To specify which extraction targets MetaRetriever should retrieve, we adopt SSI as a prefix that controls which kinds of knowledge need to be retrieved. Thus, the input is in the form of:
s⊕ x = [s1, s2, ..., s|s|, x1, x2, ..., x|x|] (1)
where s = [s1, s2, ..., s|s|] is SSI and x = [x1, x2, ..., x|x|] is the original text. Defining a unified format of task-specific knowledge for various IE tasks is a challenging issue. In this paper, we adopt a simple strategy: using the ground truth linearized SEL sequence of the corresponding input text as the knowledge we wish to retrieve. Taking SSI (s) and the text sequence (x) as input, MetaRetriever generates the linearized SEL (k) to retrieve task-specific knowledge. The retrieval procedure can represented as follows:
k = MetaRetriever(s⊕ x) (2)
where k = [k1, ..., k|k|] is the retrieved knowledge represented by SEL.
Algorithm 1 Meta-Pretraining Algorithm Require: α, β: inner/outer step size; D: pretraining corpus 1: Initialize parameters θ of the model 2: Construct support-query pair based on D 3: // Outer Loop 4: while not converged do 5: Sample the support set (xS , yS)
and the query set (xQ, yQ) 6: Compute record loss Lrecord(y; θ) 7: Corrupt raw text: x′, x′′ ← Corrupt(x) 8: Compute LM loss LLM(x′, x′′; θ) 9: // Inner Loop
10: for j in number of inner-loop updates J do 11: Compute retrieval loss Lretrv(xS , yS ; θj) 12: Compute extraction loss Lext(xS , kS , yS ; θj) 13: Compute inner loss Linner = Lretrv + Lext 14: Compute adapted parameters θj+1 ← θj − α∇θjLinner 15: end for 16: Compute retrieval loss Lretrv(xQ, yQ; θJ) 17: Compute extraction loss Lext(xQ, kQ, yQ; θJ) 18: Compute outer loss Louter = Lretrv + Lext + Lrecord + LLM 19: Update model parameters θ ← θ − β∇θLouter 20: end while

heading: 3.1.2 Extraction Procedure
text: After retrieval of task-specific knowledge, MetaRetriever incorporates this information as an additional input to generate the extraction results. To ensure that MetaRetriever utilizes the retrieved knowledge in generating the results, we concatenate it with the original input by appending it as a suffix:
s⊕ x⊕ k = [s1, ..., s|s|, x1, ..., x|x|, k1, ..., k|k|] (3)
Then, MetaRetriever will take s⊕x⊕ k as the new input to generate the final extraction results y:
y = MetaRetriever(s⊕ x⊕ k) (4)

heading: 3.2 Meta-Pretraining Algorithm
text: As the knowledge contained in the pretrained model may not always be relevant to the downstream IE tasks, we propose a Meta-Pretraining Algorithm (MPA) that utilizes a meta-learning technique (Finn et al., 2017) to enable MetaRetriever to quickly learn the semantics of the extraction targets of downstream IE tasks, and thus, retrieve task-specific knowledge only. The algorithm is detailed in Algorithm 1.
To make MetaRetriever retrieve task-specific knowledge, MPA meta-pretrain MetaRetriever with a bi-level optimization (Finn et al., 2017) to optimize MetaRetriever to achieve maximum taskspecific retrieval performance after the parameters have been updated through a small number of gradient steps on a new IE task. In the bi-level optimization, models will experience two optimization
trials: the inner loop and the outer loop. The inner loop aims to mimic the downstream fine-tuning procedure to compute a intermediate parameters of the model via SGD (Saad, 1998; Sutskever et al., 2013). The outer loop aims to mimic the downstream evaluation procedure to optimize the model parameters based on the intermediate parameters computed in the inner loop. The inner loop and the outer loop are nested and thus, it calculates highorder gradients to update parameters. Optimized by such a bi-level optimization, MetaRetriever would lean that after finetuning a small number of steps (the inner loop), MetaRetriever can achieve lower retrieval loss (the outer loop), a.k.a. learning a fast-adaptation ability for task-specific retrieval.
To accomplish this, MetaRetriever should be pretrained on a set of simulated “IE tasks” T in the pretraining phase. Each “IE task” contains a support set S and a query set Q. The support set is to mimic the training set used in the inner loop and the query set is to mimic the test set used in the outer loop. To construct various “IE tasks”, a intuitive way is to follow the widely-used episodic training technique in conventional meta-learning methods (Vinyals et al., 2016) to randomly sample instances by category. However, such a random-sampling operation requires random access to all data, which is infeasible when processing large-scale corpora, as such corpora cannot be stored in memory and must be accessed from external storage instead, resulting in intolerable long input/output (IO) time. To address this limitation, we design a support-query pairing method based on the graph maximum-weighted matching algorithm (Galil, 1986), which significantly reduces IO time (see in Section 5.5). Details of our method are introduced in Appendix B.
Inner Loop Since the inner loop simulates the downstream fine-tuning process, MetaRetriever computes the intermediate parameters by considering two inner losses: the retrieval loss Lretrv and the extraction loss Lext. Let MetaRetriever be represented by a parametrized function fθ with parameters θ. When learning a mimic IE task Ti = {Si,Qi}, the retrieval loss is computed based on the support set Si = (xSi , ySi).
Lretrv = CrossEntropy ( fθ(x Si), ySi )
(5)
It takes the text with SSI as the input and calculates the cross-entropy loss between the output sequence and the ground truth retrieved knowledge SEL se-
quence. Then, we compute the extraction loss:
Lext = CrossEntropy ( fθ(x Si , kSi), ySi )
(6)
where MetaRetriever takes the retrieved knowledge kSi with original input and generates the final predicted linearized SEL expression. We compute the cross-entropy loss between the predicted linearized SEL and ground truth as the extraction loss. Thus, the total inner loss is calculated as follows:
Linner = Lretrv + Lext (7)
The inner loop may consist of several update steps, resulting in multiple updates of the intermediate parameters. Let the number of steps in the inner loop be denoted as J , then the model parameters θ are updated iteratively to θJ based on the inner loss Linner:
θj+1 = θj − α∇θjLinner(fθj ) (8)
where α is the inner step size.
Outer Loop In the outer loop, the model parameters are updated by optimizing the performance of fθJ with respect to θ. As the outer loop simulates the downstream evaluation process, given the query set Qi = (xQi , yQi), the retrieval loss is calculated to evaluate the retrieval performance:
Lretrv = CrossEntropy ( fθJ (x Qi), yQi )
(9)
The extraction loss is computed to assess the extraction results based on the retrieved knowledge:
Lext = CrossEntropy ( fθJ (x Qi , kQi), yQi )
(10)
Additionally, to maintain the ability to understand language, we include a T5-style language modeling loss LLM (Raffel et al., 2020a). And to acquire the capability of generating valid linearized SEL structures, we include a record loss Lrecord as proposed in UIE (Lu et al., 2022). The overall objective in the outer loop is calculated using these four losses:
Louter = Lretrv + Lext + LLM + Lrecord (11)
It is worth noting that the language modeling loss and the record loss are not the primary learning objectives in the downstream IE tasks, thus these losses are calculated based on the original parameters θ rather than the intermediate parameters θJ .
Eventually, the model parameters θ are updated as follows:
θ ← θ − β∇θ ∑
Ti∈T Louter(fθJ ) (12)
where β is the outer step size. The bi-level optimization is applied to the model parameters θ, but the objective is computed using the updated model parameters θJ , which calculates high-order gradients to update parameters. Essentially, our proposed method optimizes the model parameters such that one or a small number of gradient steps on a new task will yield optimal performance on that task.

heading: 4 Experiments

heading: 4.1 Pretraining Datasets
text: To pretrain MetaRetriever, we collect a large-scale corpus using CROCODILE(Cabot and Navigli, 2021), an automatic relation extraction dataset construction tool. CROCODILE is based on the distant supervision technique (Mintz et al., 2009) which aligns texts from English Wikipedia 2 and knowledge base from Wikidata 3 to create distant labeled relation extraction data. Since distant supervision inevitably introduces noise (Mintz et al., 2009; Cabot and Navigli, 2021), CROCODILE further uses Natural Language Inference (NLI) to filter data with low confidence. Finally, we construct a 71 million distant labeled pretraining corpus and obtained 6.9 million high-quality pretraining corpus after NLI filtering. Detailed statistics of our pretraining corpus are listed in Appendix A.
Our analysis experiment (Section 5.3) indicates that the pretraining corpus we constructed can yield comparable results in the trained model as the data constructed by UIE. As UIE’s pretraining data is not open-sourced, our data will be the first 10- million-level corpus to be open-sourced, and will be released after the acceptance of the paper.

heading: 4.2 Evaluation Datasets
text: Following previous work (Lu et al., 2022), we conduct experiments across 4 IE tasks: named entity recognization (NER), relational triple extraction (RTE), event extraction (EE), structured sentiment extraction (Senti). Specifically, we use 12 IE benchmarks in total for these 4 IE tasks4: ACE04 (Mitchell et al., 2005), ACE05 (Walker et al., 2006); CoNLL03 (Tjong Kim Sang and De Meulder, 2003), CoNLL04 (Roth and Yih, 2004), SciERC (Luan et al., 2018), CASIE (Satya-
2https://www.wikipedia.org/ 3https://www.wikidata.org/ 4As NYT datasets overlaps with pre-training data, we
didn’t conduct on it for fair comparsion.
panich et al., 2020), SemEval-14 (Pontiki et al., 2014), SemEval-15 (Pontiki et al., 2015), SemEval16 (Pontiki et al., 2016). The detailed statistics of these datasets can be seen in Appendix D.

heading: 4.3 Evaluation Metrics
text: We use span-based offset Micro-F1 as the primary metric to evaluate the model for different IE tasks:
(1) NER: an entity is correct if its offsets and type are correct.
(2) RTE: a relation is correct if its relation type is correct and the offsets and entity types of the related head/tail entity are correct.
(3) EE Trg.: an event trigger is correct if its offsets and event type are correct.
(4) EE Arg.: an event argument is correct if its offsets, role type, and event type are correct.
(5) Senti: a sentiment triple is correct if the offsets boundary of the target, the offsets boundary of the opinion span, and the target sentiment polarity are correct.
After generation, we reconstruct the offset of predicted information pieces by finding the matched utterance in the input sequence one by one.
To validate the effectiveness of our proposed method, we compare MetaRetriever with the stateof-the-art model UIE (Lu et al., 2022) in fullysupervised, few-shot and low-resource scenarios.

heading: 4.4 Main Results
text: Table 1 and Table 2 summarize the results of our proposed MetaRetriever compared to the previous work, UIE, on 4 types of IE tasks. From the experimental results, we have several observations (1) Equipped with task-specific knowledge, MetaRetriever outperforms UIE on all fully-supervised, few-shot, and low-resource scenarios. MetaRetriever surpasses UIE with an average of 0.7%, 2.5%, and 2.3% in F1 score respectively in the fullysupervised, few-shot, and low-resource settings. This strongly demonstrates that MetaRetriever can utilize the retrieved knowledge to improve IE performance. (2) Empowered by meta-pretraining, MetaRetriever can effectively learn downstream task-specific extraction targets. Despite the fact that event extraction and sentiment extraction were not present in the pretraining phase, MetaRetriever still outperforms UIE in these tasks, validating that MetaRetriever can quickly learn downstream IE tasks through the meta-pretraining algorithm to support task-specific extraction. (3) Benefiting from task-specific knowledge and meta-pretraining, MetaRetriever exhibits significant superior performance in data-scarce scenarios. MetaRetriever achieves large gains in the few-shot and low-resource settings. This is attributed to two factors: first, the meta-pretraining algorithm can
quickly learn downstream IE tasks, which reduces the requirement for training data, and second, the retrieved task-specific knowledge provides crucial clues for the model to make correct predictions, thus MetaRetriever can achieve impressive performance in data-scarce scenarios.

heading: 5 Analysis

heading: 5.1 Effect of Meta-Pretraining Algorithm
text: To examine the contribution of our MetaPretraining Algorithm, we remove the inner loop in Algorithm 1 and pretrain our model using only the outer loop. Without the inner loop, the bi-level optimization is reduced to single-level optimization. We name this pretrained model SimpleRetriever and compare its performance with MetaRetriever on CoNLL03 and CoNLL04 datasets in fullysupervised, few-shot(1-shot) and low-resource(1%) settings. The results are presented in Table 3 and it can be observed that: Compared to MetaRetriever, the performance of SimpleRetriever declines on all datasets in all settings. Notably, in data-scarce
scenarios, SimpleRetriever underperforms MetaRetriever by 3.92% on average. This demonstrates that the meta-pretraining algorithm enables the model to quickly learn downstream IE tasks and thus improve performance.

heading: 5.2 Effect of Finetune Epoch
text: In order to demonstrate the efficiency of our MetaRetriever in rapidly learning downstream information extraction tasks, we conducted a series of experiments. Specifically, we compare the performance of MetaRetriever to that of UIE across a range of finetuning epochs, from 1 to 5 on CoNLL03 dataset. The results of these experiments, presented in Figure 3, clearly indicate that MetaRetriever outperforms UIE from the very first finetuning epoch. Furthermore, as the number of finetuning epochs increases, the performance of MetaRetriever consistently surpasses that of UIE, with the greatest difference observed when comparing MetaRetriever’s performance in 2 finetuning epochs to UIE’s performance in 5 finetuning epochs. These results strongly suggest that MetaRetriever’s ability to rapidly learn downstream tasks is significantly enhanced by its use of metapretraining. Additionally, we also conducted experiments on the CoNLL04 datasets, the results of which are provided in Appendix E.

heading: 5.3 Effect of Structure Complexity
text: In order to determine the effectiveness of MetaRetriever in dealing with complex structures when compared to UIE, we conduct an experiment using the test set of CoNLL03, grouping the test instances according to the number of entities present. We report the performance gain of MetaRetriever compared with UIE in different group in Figure 4. We can obverse that, as the number of entities present in a test instance increases, MetaRetriever’s per-
formance advantage over UIE also increases. This demonstrates that MetaRetriever is able to effectively leverage task-specific knowledge to handle complex structures and improve information extraction performance. We also conduct further experiments on the CoNLL04 dataset, the results of which can be found in Appendix F.

heading: 5.4 Effect of Pretraining Corpus Collection
text: To evaluate the quality of the corpus we have collected for pretraining, we use this corpus to pretrain a variant of UIE which we refer to as UIE-ours. We then compared UIE-ours to the original UIE on four representative datasets for each IE task in fully-supervised settings. The results, presented in Table 4, show that UIE-ours, which is pretrained on our collected corpus, is comparable in performance to the original UIE (with an average gain of 0.07%). This indicates that our collected corpus is of similar quality to that used for training the original UIE and we plan to make it publicly available to promote future research. Additionally, this also demonstrates that the improvement of MetaRetriever is due to our proposed method rather than the corpus used for pretraining, further validating the effectiveness of our proposed method.

heading: 5.5 Effect of Support-Query Construction for Meta-Pretraining
text: Conventional meta-learning methods typically employ the episodic training technique to construct "tasks" for meta-training (Vinyals et al., 2016; Finn et al., 2017). This technique creates "tasks" by randomly sampling instances based on their category, which requires random access to all data. However, since large-scale corpora cannot be stored in memory and must be accessed from external storage instead, random-sampling operation will cause long IO time. To address this, we design a support-query pairing method which can significantly reduce IO times. To illustrate the efficiency of our method, we compared the time cost of our method with episodic training on 10K instances. Episodic training required 100.42s to construct "IE tasks" while our method only required 5.57s, demonstrating the significant efficiency of our method.

heading: 5.6 Case Study
text: In this study, we compare the performance of our MetaRetriever model with that of the UIE model using a selection of cases from the CoNLL04 dataset, as illustrated in Table 5.
In the first case, our MetaRetriever, aided by the retrieved task-specific knowledge, “(LOC: El Toro (Located_In: California))”, correctly predict “(LOC: Ontario (Located_In: California))”. In contrast, the UIE model was unable to make a precise prediction without the benefit of this task-specific knowledge. This demonstrates that models can significantly benefit from the incorporation of retrieved task-specific knowledge to generate more accurate linearized SEL expressions.
In the second case, we observe that MetaRetriever makes a redundant prediction, “(LOC: Moscow (Located_In: Soviet))”. Though this relational triple is not labeled in the CoNLL04 dataset, it is a common sense knowledge that Moscow is located in Russia, which is former So-
viet Union. MetaRetriever was able to predict this based on the background knowledge acquired from the pretraining corpus.

heading: 6 Related Work
text: Information Extraction (IE) aims at extracting text spans or a tuple of text spans of interests from plain texts. There exist many specific IE tasks: Named Entity Recognition (Lample et al., 2016; Lin et al., 2019), Relational Triple Extraction (Zheng et al., 2017; Levy et al., 2017), Event Extraction (Lin et al., 2018; Wadden et al., 2019; Du and Cardie, 2020; Li et al., 2022c,b,a), etc. For a long time, researchers are devoted to propose task-customized and isolated methods to achieve these different IE tasks. In the recent years, with the rising of pretraining technique, pretraining a universal model for several NLP tasks has attracted a lot of attention (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020a; Raffel et al., 2020b; Xue et al., 2021). Following this trend, several works make attempts to unify diverse IE tasks. Yan et al. (2021b) first propose a span-offset-based generation to solve various NER tasks in a universal manner. Yan et al. (2021a) use a similar way to solve different aspectbased sentiment analysis tasks. Lu et al. (2022) propose UIE which designs Structured Extraction Language to formulate all IE tasks in a unified form but it will experience poor performance when there exists complex structure to be extracted. Several works have shown that retrieving knowledge from external knowledge bases can empower models to generate complex sequences (Lewis et al., 2020b; Wang et al., 2021; Cai et al., 2022; Geng et al., 2022) and a large number of knowledge stored in pretrained language models can be “retrieved” explicitly (Petroni et al., 2019; Roberts et al., 2020; Zhang et al., 2022; Yu et al., 2022). In this paper, we propose MetaRetriever to retrieve task-specific knowledge from PLMs to enhance universal IE.
Meta-learning aims to learn better learning algorithms from a series of tasks, i.e., learning to learn.
In meta-learning, there exist diverse branches focusing on different aspects of the learning algorithm: learning-to-initialize (Finn et al., 2017, 2018), learning-to-compare (Vinyals et al., 2016; Snell et al., 2017), learning-to-optimize (Ravi and Larochelle, 2017), etc. Existing works are most devoted to solve the few-shot problem, cross-domain problem, etc. We make the first attempt to adapt meta-learning into universal IE.

heading: 7 Conclusion
text: In this work, we present MetaRetriever, a novel approach for enhancing universal information extraction through the retrieval of task-specific knowledge from within the model itself. To ensure that the retrieved knowledge is truly task-specific and to minimize the impact of irrelevant or "noise" knowledge, we also propose a meta-pretraining algorithm that enables MetaRetriever to quickly adapt to downstream information extraction tasks. Empirical results demonstrate the effectiveness of our approach, with MetaRetriever achieving new state-of-the-art performance on a range of information extraction tasks across 12 datasets in fullysupervised, low-resource, and few-shot scenarios. Further analysis experiments also validate the contributions and effectiveness of MetaRetriever.

heading: 8 Limitations
text: While our MetaRetriever has demonstrated its superior performance on 4 IE tasks, 12 datasets, in fullysupervised, few-shot and low-resource scenarios, it still has several limitations. First, since MetaRetriever will first retrieve task-specific knowledge and then make predictions in the inference phase, such a retrieve-then-extract manner will take longer inference time than non-retrieve methods unavoidably. Second, our proposed meta-pretraining algorithm is based on bi-level optimization. In the pretraining phase, it needs to calculate high-order gradients to optimize parameters and calculating high-order gradient requires more time, which will cause longer time to pretrain MetaRetriever. Detailed discussions are shown in Appendix G.

heading: 9 Ethical Considerations
text: This paper constructs a new dataset for universal information extraction, and we discuss some related ethical considerations here. (1) Intellectual property. The Wikipedia corpus is shared under
the CC BY-SA 3.0 license5 and Wikidata is shared under the CC0 1.0 license6. CROCODILE is licensed under the CC BY-SA-NC 4.0 license7. Our data source and the construct tool are all free for research use. (2) Controlling Potential Risks. Since the texts in Wikipedia do not involve private information and annotating entities and relations does not require many judgments about social issues, we believe our collected dataset does not create additional risks. To ensure it, we manually checked some randomly sampled data and did not note risky issues. (3) Worker Treatments. Since the dataset construction procedure is automatic without any manual annotation, it does not create potential risk about the worker treatments.

heading: Acknowledgements
text: This work is supported by the National Key Research and Development Program of China (grant No.2021YFB3100600), the Strategic Priority Research Program of Chinese Academy of Sciences (grant No.XDC02040400) and the Youth Innovation Promotion Association of CAS (Grant No. 2021153).

heading: A Details of Pretraining Corpus
text: We apply CROCODILE8 (Cabot and Navigli, 2021), which is a automatic distant supervised labeling tool developed for relation extraction, to collect our pretraining corpus. CROCODILE utilizes widely-used distant supervision technique (Mintz et al., 2009) to build relation extraction dataset by
8https://github.com/Babelscape/crocodile
aligning English Wikipedia9 texts and Wikidata10 knowledge base. Additionally, CROCODILE applies RoBERTa NLI model 11 to filter distant labeled data with low-confidence. Finally, we construct 71M original pretraining corpus without NLI filtering and obtain 6.9M high-quality pretraining corpus with NLI filtering. The statistics of them are shown in Table 6 and Table 7 respectively. Since our collected corpus contains the 10-million-level sentences covering about 40K entity types and over 1K relation types, we believe the collected corpus is diverse enough in genre and topic, which exhibits a good coverage for general domain . Both of these pretraining corpus will be public after acceptance.

heading: B Details of Meta-Pretraining Dataset Construction Procedure
text: The goal of our meta-pretraining is to train MetaRetriever that can quickly learn the extraction target semantics of various downstream IE tasks for retrieval using only a few training epochs. To accomplish this, MetaRetriever is pretrained on a set of “IE tasks” based on the meta-learning technique (Finn et al., 2017), such that the trained model can quickly learn new IE tasks using only a small number of trials. To construct various “IE tasks”, a intuitive way is to follow the episodic training technique (Vinyals et al., 2016) which are widely used in conventional meta-learning methods. In episodic training, it randomly samples N classes
9https://www.wikipedia.org/ 10https://www.wikidata.org/ 11xlm-roberta-large-xnli
from original class set as a N -classification “task”. For each N -classification task, it will randomly samples K instances as the support set to mimic the training set for this task and then use remain instances as the query set to mimic the test set, i.e., the N -way-K-shot setting. In each training epoch, it will sample different N -way-K-shot tasks to train models. However, when pretraining a model, it is time-consuming and memory-consuming to sample classes from a large-scale corpus. To metapretrain MetaRetriever, we design Support-Query Pairing Algorithm based on the graph maximumweighted matching algorithm (Galil, 1986). Our Support-Query Pairing Algorithm can be divided into three steps: (1) Partition by class; (2) Deduplication; (3) Graph Matching.
Partition by class Different from episodic training which treats each a support set and a query set as a task, we construct a “task” with only two instances. One acts as the support instance and the other as the query instance, i.e., the size of our support set and query are both 1. To make the support instance “support” the query instance, their classes should be as nearly identical as possible so the model can learn the semantics of the target classes from the support instance to make predictions on the query instance. To achieve this goal, we partition all pretraining instances based on the class. Specifically, we denote the raw training dataset D = {(xi, yi)}Ni=1 where xi refers to the raw text, yi refers to the golden linearized SEL expression, and N refers to the number of the ;pretraining instances. Given the training corpus D, we can get the class set C which contains all SpotNames and AssoNames of D. Then, we collect instances for each classes Dc, c ∈ C where Dc refers to all instances which contains c-type target information piece. Therefore, we can get the partitioned dataset DC.
Deduplication Getting the partitioned dataset DC, there exists serious duplication phenomenon because one instance will contain several classes and thus it will be partitioned into multiple subset Dc. Such a duplication will cause a severe data imbalance since if one instance contains m classes, it will be duplicated m times, hurting the model performance. To alleviate this problem, we should deduplicate the partitioned dataset DC. Specifically, we sort the class set C by the number of instances of each class. From less to more, we re-
move these instances from the large subset which have existed in the small subset. After that, we get the deduplicated partitioned dataset D̂C. Graph Matching Since we treat a support-query instance pair as a task, we want to find a optimal pairing solution that the classes of the support instance could cover the classes of the query instance as many as possible globally. To achieve this goal, we first define a pairing score function to evaluate a support-query pair. Given a support instance s and a query instance q, we define the pairing score function as follows,
ρ(s, q) = |FC(s) ∩ FC(q)|
|FC(s)| +
1
|FC(s)| (13)
where FC(·) means the class set contained in the given instance. In this score function, the first term evaluates the ratio how the support instance covers the query instance. The second term evaluates the number of the classes of the support instance. We wish the support instance should cover the query instance exactly so the number of the classes of the support instance should be as small as possible besides the class coverage. Then, we define a matching score function to calculate the pairing degree given any two instances x and y:
φ(x, y) = max {ρ(x, y), ρ(y, x)} (14) Given a subset Dc, c ∈ C, we can construct a weighted undirected complete graph Gc. In Gc, each instance in Dc acts as a node and the arbitrary two nodes x and y have a edge with φ(x, y) weight. Next, we run the maximum-weighted graph matching algorithm (Galil, 1986) to get a graph matching result Mc. In Mc, each node matching pair represents a support-query pair. We run the aforementioned matching procedure for each class c ∈ C to get all matching pairs MC = {Mc}, c ∈ C. We use MC to meta-pretrain our MetaRetriever.
Through the procedure introduced above, the I/O times of support-query pairing are 2. First, it reads all data to get the categories of each instance to run the support-query pairing algorithm. Second, given the pairing results, it takes the second I/O time to re-range all data to construct support-query pair. For the episodic training, every random-sampling will lead to a I/O time while our method can reduce I/O times significantly.

heading: C Details of Model Implementation
text: Since MetaRetriever works in a sequence-tosquence manner, we use the encoder-decoder-style
architecture to achieve this procedure. We choose T5-base architecture (Raffel et al., 2020a) as our model with 220M parameters. We pretrain our model on our constructed pretraining dataset. We use Adam (Kingma and Ba, 2015) as the optimizer with learning rate 1e− 4 with the linear scheduling with a warming up proportion 6%. As previous work (Lu et al., 2022), we randomly sample 10 SpotName and AssoName to preserve the generalization ability of the model when calculating Lretrv and Lgen. For LLM, we set the corruption rate as 15% and the average corrupting span length as 3 as T5 (Raffel et al., 2020b). We truncate the length of the input sequence to 128 during pretraining. In the pretraining phase, we set the step of the inner loop as 1 and the inner step size α and the outer step size β are set as 1e− 4. The inner-loop update step J is set as 1. We adopt the filtered version of our pretraining corpus (6.9M instances) to pretrain MetaRetriever. We pretrain our model for both 4 epochs with batch size 512 on 8 NVIDIA A100 80G GPUs with 56 hours. For the downstream finetuning, we apply the same grid searching strategy for hyper-parameters as UIE (Lu et al., 2022) to find the optimal results. In the inference phase, we use the greedy search strategy to decode the final predicted SEL sequence.

heading: D Details of Downstream Task Datasets
text: We conduct evaluation experiments on 4 IE tasks, 12 datasets, and the detailed statistic of each dataset is shown in Table 8.

heading: E Details of Effect of Finetune Epoch
text: To verify that our MetaRetriever can fast learn the downstream IE tasks, we conduct experiments on both CoNLL03 and CoNLL04. Experimental results are shown in Figure 5. Results of CoNLL04 show the same trend as CoNLL03, which can prove the effect of our proposed Meta-Pretraining Algorithm.

heading: F Details of Effect of Structure Complexity
text: To validate that MetaRetriever own the superiority when there exist complicated structures needed to be extracted. We group the test set of CoNLL03 and CoNLL04 according to the number of entities and the number of relational triples and calculate the performance difference between MetaRetriever and UIE in different group. Experimental results are shown in Figure 6. From it, we can learn that when there exist more entities and relational triples, MetaRetriever performs better.

heading: G Discussion of Efficiency
text: While our MetaRetriever has achieved the new state-of-the-art on 4 IE tasks, 12 datasets, in fullysupervised, few-shot and low-resource scenarios, it still has several limitations on training efficiency and inference efficiency.
First, since MetaRetriever will first retrieve taskspecific knowledge and then make predictions in the inference phase, such a retrieve-then-extract manner will take longer inference time than nonretrieve methods unavoidably. We conduct experiments on the test set of CoNLL04 dataset to compare overall inference time of MetaRetriever with UIE. All hyper-parameters are set to be the same for a fair comparison. Experimental results are shown in Table 9 and we can find that MetaRetriever cost nearly twice time as UIE to make predictions. As MetaRetriever works in a retrieve-than-extract manner, such a time cost is reasonable.
Second, our proposed meta-pretraining algorithm is based on bi-level optimization. In the
pretraining phase, it needs to calculate high-order gradients to optimize parameters and calculating high-order gradient requires more time. Therefore, it takes longer time to pretrain MetaRetriever. To illustrate the time cost, we conduct experiments on 10K instances to compare the pretraining time of MetaRetriever with SimpleRetriever which is pretrained without meta-pretraining algorithm. All hyper-parameters are set to be the same for a fair comparison. Table 10 gives the results. From it, we can obverse that compared with SimpleRetriever, MetaRetriever takes about 1/4 longer time than SimpleRetriever. Finally, We spent 56 hours to pretrain MetaRetriever on filtered pretraining corpus (6.9M instances).
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Section 8

heading: 3 A2. Did you discuss any potential risks of your work?
text: Section 9

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Section 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section 4 and Appendix A

heading: 3 B1. Did you cite the creators of artifacts you used?
text: Section 4 and Appendix A

heading: 3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts?
text: Section 9
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Section 9 and Appendix A

heading: 3 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any
text: information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Section 9

heading: 3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and
text: linguistic phenomena, demographic groups represented, etc.? Appendix A
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Appendix A
C 3 Did you run computational experiments?

heading: Appendix C

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Appendix C
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Appendix C

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix C

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section 4 and Appendix A
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: This indicates that our collected corpus is of similar quality to that used for training the original UIE and we plan to make it publicly available to promote future research. Additionally, this also demonstrates that the improvement of MetaRetriever is due to our proposed method rather than the corpus used for pretraining, further validating the effectiveness of our proposed method