SlowGuess/NAACL_2022_OCR huggingface dataset:
Martin Josifoski, $^{1}$  Nicola De Cao, $^{2,3}$  Maxime Peyrard, $^{1}$  Fabio Petroni, $^{4}$  Robert West $^{1}$
<sup>1</sup>EPFL <sup>2</sup>University of Amsterdam, <sup>3</sup>University of Edinburgh, <sup>4</sup>Meta AI
martin.josifoski@epfl.ch,nicola.decao@gmail.com
maxime.peyrard@epfl.ch, fabiopetroni@fb.com, robert.west@epfl.ch
Structured and grounded representation of text is typically formalized by closed information extraction, the problem of extracting an exhaustive set of (subject, relation, object) triplets that are consistent with a predefined set of entities and relations from a knowledge base schema. Most existing works are pipelines prone to error accumulation, and all approaches are only applicable to unrealistically small numbers of entities and relations. We introduce GenIE (generative information extraction), the first end-to-end autoregressive formulation of closed information extraction. GenIE naturally exploits the language knowledge from the pre-trained transformer by autoregressively generating relations and entities in textual form. Thanks to a new bi-level constrained generation strategy, only triplets consistent with the predefined knowledge base schema are produced. Our experiments show that GenIE is state-of-the-art on closed information extraction, generalizes from fewer training data points than baselines, and scales to a previously unmanageable number of entities and relations. With this work, closed information extraction becomes practical in realistic scenarios, providing new opportunities for downstream tasks. Finally, this work paves the way towards a unified end-to-end approach to the core tasks of information extraction.
The ability to extract structured semantic information from unstructured texts is crucial for many AI tasks such as knowledge discovery (Ji and Grishman, 2011; Trisedya et al., 2019), knowledge maintenance (Tang et al., 2019), symbolic representation, and reasoning (Ji et al., 2021). The interface between free text and structured knowledge is formalized by knowledge base population (KBP; Ji and Grishman, 2011), which proposes to represent the information contained in text using (subject, relation, object) fact triplets. In this work, we focus on closed information extraction (cIE), the problem
Figure 1: Overview of GenIE. We use a transformer encoder-decoder model that takes unstructured text as input and autoregressively generates a structured semantic representation of the information expressed in it, in the form of (subject, relation, object) triplets. GenIE employs constrained beam search with: (i) a high-level constraint which asserts that the output corresponds to a set of triplets; (ii) lower-level constraints which use prefix tries to force the model to only generate valid entity or relation identifiers (from a predefined schema).
of extracting exhaustive sets of fact triplets expressible under the relation and entity constraints defined by a Knowledge Base (KB) schema.
Traditionally, cIE was approached with pipelines that sequentially combine named entity recognition (Tjong Kim Sang, 2002), entity linking (Milne and Witten, 2008), and relation extraction (Miller et al., 1998). Entity linking and relation extraction serve as grounding steps, matching entities and relations to numerical identifiers in a KB, e.g., QIDs and PIDs for Wikidata (Vrandecic, 2012). Recently, Trisedya et al. (2019) pointed out that such pipeline architectures suffer from the accumulation of errors and proposed an end-to-end alternative. Nevertheless, existing methods are still only prac
Human Language Technologies, pages 4626 - 4643
tical for small schemas with unrealistically small numbers of relations and entities.
Alternatively, some works have focused on a simpler syntactic task: open information extraction (oIE), which produces free-form triplets from texts. In this setup, the entities and relations are not grounded in a KB and, usually, do not represent facts (Gashteovski et al., 2020). As oIE triplets contain only surface relations, they have ambiguous semantics, making them hard to use in downstream tasks (Broscheit et al., 2017) if not first aligned with a KB (Gashteovski et al., 2020). Since, in practice, oIE often consists of structured substring selection, it has recently been framed as an end-to-end sequence-to-sequence problem with great success (Huguet Cabot and Navigli, 2021; Dognin et al., 2021). Indeed, such autoregressive formulations can exploit the language knowledge already encoded in pre-trained transformers (Devlin et al., 2019). For example, some tokens can be more easily recognized as possible entities or relations thanks to the pre-training information.
Inspired by recent successes in oIE, we propose the first autoregressive end-to-end formulation of cIE that scales to many entities and relations, making cIE practical for more realistic KB schemas (i.e. schemas with millions of entities).<sup>1</sup> We employ a sequence-to-sequence BART model (Lewis et al., 2020), and exploit a novel bi-level constrained generation strategy operating on the space of possible triplets (from a fixed schema induced by Wikidata) to ensure that only valid triplets are generated. Our resulting model, GenIE, performs Generative Information Extraction and combines the advantages of a known schema with an autoregressive formulation. The high-level overview of GenIE is provided in Fig. 1. The constrained generation encodes the known schema and enables the autoregressive decoder to generate textual tokens but only from the set of allowed entities or relations.
- We present the first end-to-end autoregressive formulation of closed information extraction.
- We describe a constrained decoding strategy that exploits the Wikidata schema to generate only valid fact triplets, demonstrating how constrained beam search can be applied on large, structured, and compositional spaces.
- We propose a model that achieves state-of-the-art performance on the cIE task and scales to previously unmanageable numbers of entities (6M) and relations (more than 800).
- We point out and address weaknesses in the evaluation methodologies of recent previous works stemming from their small scale and the large imbalances in the available data per relation. We demonstrate the importance of reporting performance as a function of the number of relation occurrences in the data.
- We release pre-processed data, pre-trained models, and code within a general template designed to facilitate future research at https://github.com/epfl-dlab/GenIE.
2.1 Closed Information Extraction
In this work, we address the task of closed information extraction (cIE), which aims to extract the exhaustive set of facts from natural language, expressible under the relation and entity constraints defined by a knowledge base (KB).
Most of the existing methods address the problem with a pipeline solution. One line of work starts by first extracting the entity mentions and the relations between them from raw text. This is followed by a disambiguation step in which the entity and relation predicates are mapped to their corresponding items in the KB. The sub-task of extracting the free-form triplets was originally proposed by Banko et al. (2007), and it is commonly referred to as open information extraction (oIE) or text-to-graph in the literature (Guo et al., 2020; Castro Ferreira et al., 2020; Huguet Cabot and Navigli, 2021; Shen et al., 2015). Another line of work employs a pipeline of models for (i) named entity recognition (NER) – detecting the entity mentions; (ii) entity linking (EL) – mapping the mentions to specific entities from the KB; (iii) relation classification (RC) – detecting the relations that are expressed between the entities (Galarraga et al., 2014; Angeli et al., 2015b; Chaganty et al., 2017). Due to their architecture, pipeline methods are plagued by error propagation, which significantly affects their performance (Mesquita et al., 2019; Trisedya et al., 2019).
End-to-end systems that jointly perform the extraction and the disambiguation of entities and relations have been proposed to address the error propagation (Trisedya et al., 2019; Sui et al., 2021;
<sup>1</sup>Note that current methods, due to the atomic classification, have high memory requirements, and suffer from performance deterioration as the number of entities or/and relations grows.
Liu et al., 2018). To mitigate the propagation of errors, these systems are endowed with the ability to leverage entity information in the relation extraction and vice-versa, which has resulted in significant performance gains. Conceptually, for producing the output triplets, existing methods all rely on atomic, multi-class classification-based ranking of relations and entities. Classification methods particularly suffer from imbalances in the data. On the contrary, our model, GenIE, is autoregressive and copes better with imbalances.
While cIE requires the constituent elements of the output triplets to be entities and relations associated with the KB, the output triplets in oIE are free-text. This makes the cIE task fundamentally harder than oIE and renders the majority, if not all, oIE methods inapplicable to the cIE setting. We report an additional discussion on relevant, but not fundamental, related work on oIE in Appendix A.
2.2 Autoregressive Entity Linking
The tasks of entity linking (EL) and entity disambiguation (ED) have been extensively studied in the past (Huang et al., 2015; Wu et al., 2020; Le and Titov, 2018; Kolitsas et al., 2018; Arora et al., 2021). Most existing approaches associate entities with unique atomic labels and cast the retrieval problem as multi-class classification across them. The match between the context and the label can then be represented as the dot product between the dense vector encodings of the input and the entity's meta information (Wu et al., 2020). This general approach has led to large performance gains.
Recently, De Cao et al. (2021a,b, 2022) have suggested that the classification-based paradigm for retrieval comes with several shortcomings such as (i) the failure to capture fine-grained interactions between the context and the entities; (ii) the necessity of tuning an appropriately hard set of negative samples during training. Building on these observations, they propose an alternative solution that casts the entity retrieval problem as one of autoregressive generation in which the entity names are generated token-by-token in an autoregressive fashion. The (freely) generated output will not always be a valid entity name, and to solve this problem De Cao et al. (2021b) propose a constrained decoding strategy that enforces this by employing a prefix trie. Their method scales to millions of entities, achieving state-of-the-art performance on monolingual and multilingual entity linking.
Inspired by the intuition that language models are well suited for predicting entities, we propose a novel approach for cIE by framing the problem in an autoregressive generative formulation.
In this section we formalize GenIE, an autoregressive end-to-end model for closed information extraction. Let us assume a knowledge base (KB) consisting of a collection of entities  $\mathcal{E}$ , a collection of relations  $\mathcal{R}$ , and a set of facts  $(s,r,o)\in \mathcal{E}\times \mathcal{R}\times \mathcal{E}$  stored as (subject, relation, object) triplets. Additionally, we assume that each entity  $e\in \mathcal{E}$  and relation  $r\in \mathcal{R}$  is assigned to a textual label (corresponding to its name). The Wikidata KB (Vrandečić, 2012), with Wikipedia page titles as entity names, and the Wikidata relation labels as relation names satisfy these assumptions.
We cast the task of information extraction as one of autoregressive generation. More concretely, given some text input  $x$ , GenIE strives to generate the linearized sequence representation  $y$  of the exhaustive set of facts expressed in  $x$ . The conditional probability (parameterized by  $\theta$ ) assigned to the output  $y$  is computed in the autoregressive formulation:  $p_{\theta}(y \mid x) = \prod_{i=1}^{|y|} p_{\theta}(y_i \mid y_{<i}, x)$ . This can be seen as translating the unstructured text to a structured, unambiguous representation in a sequence-to-sequence formulation. GenIE employs the BART (Lewis et al., 2020) transformer architecture. It is trained to maximize the target sequence's conditional log-likelihood with teacher forcing (Sutskever et al., 2011, 2014), using the cross-entropy loss. We use dropout (Srivastava et al., 2014) and label smoothing for regularization (Szegedy et al., 2016).
3.2 Output Linearization
To represent the output with a sequence of symbols that is compatible with sequence-to-sequence architectures, we introduce the special tokens  $<\mathrm{sub}>$ ,  $<\mathrm{rel}>$ ,  $<\mathrm{obj}>$  to demarcate the start of the subject entity, the relation type and the object entity for each triplet. The special token  $<\mathrm{et}>$  is introduced to demarcate the end of the object entity, which is also the end of the triplet. We construct the sequence representation by concatenating the textual representations of its constituent triplets. While the sequence representation has an intrinsic notion of
order, the output set of triplets does not. To mitigate the effects of this discrepancy, we enforce a consistent ordering of the target triplets during training. Concretely, whenever the triplets' entities are linked to the entity mentioned in the textual input, we consider first the triplets for which the subject entity appears earlier in the text. Ties are resolved by considering the appearance position of the object entity.
3.3 Inference with Constrained Beam Search
The space of triplets corresponds to  $\mathcal{T} = \mathcal{E}\times$ $\mathcal{R}\times \mathcal{E}$  , and the target space, which consists of triplet sets of arbitrary cardinality, is equivalent to  $\mathcal{S} = \bigcup_{i = 0}^{\infty}[\mathcal{E}\times \mathcal{R}\times \mathcal{E}]^{i}$  . At inference time, GenIE tackles the task of retrieving the linearized representation  $y_{S}\in \mathcal{S}$  of a set of facts  $S = \{t_1,\dots ,t_n\}$  constituted by triplets  $t_i\in \mathcal{T}$  expressed in the input text  $x$  . Ideally, we would consider every element  $y\in \mathcal{S}$  in the target space, assign it a score  $p_{\theta}(y|x)$  and retrieve the most probable  $y$  Unfortunately, this is prohibitively expensive since we are dealing with a compositional target space whose size is gigantic (e.g., if we consider a Wikidata entity catalog of  $|\mathcal{E}|\approx 6\mathrm{M}$  elements and a relation catalog of  $|\mathcal{R}|\approx 1000$  relations, that can express a total of  $|\mathcal{T}|\approx 10^{15}$  triplets; even if we limit ourselves to sentences that express only two facts, this provides us with  $\approx 10^{30}$  different output options).
On the other hand, the output needs to follow a particular structure, and contain only valid entity and relation identifiers. This does not necessarily hold for an arbitrary generation from a sequence-to-sequence model.
GenIE employs constrained beam search (BS; Sutskever et al., 2014; De Cao et al., 2021b) to resolve both of these problems. Instead of explicitly scoring all of the elements in the target space  $\mathcal{S}$ , the idea is to search for the top- $k$  eligible options, using BS with  $k$  beams and a prefix trie. BS considers one step ahead – the next token to be generated – conditioned on the previous ones. The prefix trie restricts the BS to candidate tokens that could lead to valid identifiers. However, for the cIE setting we are interested in, the target space is prohibitively large to pre-compute the necessary trie. Therefore, we enforce a bi-level constraint on the output that allows for compositional, dynamic generation of the valid prefixes. More specifically, GenIE employs: (i) a high-level structural constraint which asserts that the output follows the linearization
schema defined in Sec. 3.2; (ii) lower level validity constraints which use an entity trie and a relation trie to force the model to only generate valid entity or relation identifiers, respectively – depending on the specific element of the structure that is being generated. This outlines a general approach for applying BS to search through large compositional structured spaces.
4.1 Knowledge Base: Wikidata
We use Wikidata² (Vrandecić, 2012) as the target KB to link to, filtering out all entities that do not have an English Wikipedia page associated with them. The filtering guarantees that all entity names are unique. Our final entity set  $\mathcal{E}$  contains 5,891,959 items. We define our relation set  $\mathcal{R}$  as the union of all the relations considered in the datasets described below, resulting in 857 relations. For different datasets, we consider only the subset of annotated relations to better compare with baselines. Although large, the number of entity (and relation) names is not a memory bottleneck as the generated prefix trie occupies  $\approx$ 200MB of storage (e.g., the entity linking system proposed by Wu et al. 2020 needs  $>20$  times more storage).
In this work, we further annotate and adapt REBEL (Huguet Cabot and Navigli, 2021) and Wiki-NRE for training, validation and testing. Additionally, we use Geo-NRE (Trisedya et al., 2019), and FewRel (Han et al., 2018) for testing purposes only. Appendix B contains descriptions of these datasets and their statistics. We measure the performance in terms of micro and macro precision, recall and F1. See Appendix C for a detailed and formal description of these metrics. We also report a 1-standard-deviation confidence interval constructed from 50 bootstrap samples of the data.
We compare GenIE against Set Generation Networks (SetGenNet; Sui et al., 2021) which is, to the best of our knowledge, the strongest model on Wiki-NRE and Geo-NRE. Note that the authors did not release code or the model and there is no other model from the literature trained and evaluated on REBEL for cIE. SetGenNet (Sui et al., 2021) is an
$^{2}$ Dumps from 2019/08/01
end-to-end state-of-the-art model for triplet extraction. It consists of a transformer encoder (Vaswani et al., 2017) that encodes the input followed by a non-autoregressive transformer decoder (Gu et al., 2018). The decoder generates embeddings that are used to predict entities and relations. SetGenNet further uses candidate selection (Ganea and Hofmann, 2017; Kolitsas et al., 2018) to reduce the output space and a bipartite matching loss that handles different prediction orderings (i.e., it generates a set). Note that there are weaker baselines (e.g., Trisedya et al. 2019) we could have used to compare on REBEL, but we were not able to reproduce their code. We report details on the effort made to use these baselines in Appendix D.
We also implement a pipeline baseline, consisting of 4 independent steps, namely: (i) named entity recognition (NER), which selects the spans in the input source likely to be entity mentions; (ii) entity disambiguation (ED), which links mentions to their corresponding identifiers in the KB; (iii) relation classification (RC), which predicts the relation between a given pair of entities, and finally; (iv) triplet classification (TC), which predicts whether a given triplet is actually entailed by the context. TC is necessary because the previous step (RC) predicts a relation for every pair of entities. Each step needs to be trained independently with a specific architecture tailored for the task, and we made an optimal choice for each step. For the NER component we used the state-of-the-art tagger FLAIR $^3$  (Akbik et al., 2019), while for ED we used the GENRE linker $^4$  (De Cao et al., 2021b). These two models were already trained, and we use them for inference only. For RC and TC, we trained a RoBERTa (Liu et al., 2019) model with a linear classification layer on top (as these two sub-tasks are typically cast as classification problems). Trisedya et al. (2019) also proposed many other pipeline baselines but ours outperforms them (see Table 5 in Appendix G for comparison).
Models performing cIE can base their predictions on different schemas. In this section, we distinguish between small and large evaluation schema. The small evaluation schema is consistent with pre
vious approaches where models only have to decide between a small set of relations and entities (the schema induced by Wiki- and Geo-NRE). In the large evaluation schema, models use the schema induced by REBEL. Models also use the large evaluation schema of REBEL when tested on FewRel, as a high-quality and challenging recall-based evaluation. We consider 3 training setups for GenIE and the pipeline baseline comprised of SotA components: (i) the training set of Wiki-NRE (W) only, (ii) the training set of REBEL (R) only, and (iii) pre-training on REBEL and fine-tuning on WikiNRE  $(\mathrm{R} + \mathrm{W})$ . The implementation details are given in Appendix E. We report the macro and micro precision, recall, and F1 in Table 1. Unfortunately, as the code for SetGenNet is not available, we cannot compute its macro performance, thus we report the micro scores only.
First, on Wiki-NRE (W), we observe a large and significant F1 improvement of 8 and 28 absolute points obtained by GenIE over SetGenNet and the pipeline baseline, respectively, when trained on the same dataset. Despite the much bigger schema employed by REBEL, pre-training on it and then finetuning  $(\mathrm{R} + \mathrm{W})$ , improves the performance on WikiNRE and Geo-NRE for  $3\%$  and  $5\%$ , respectively. This highlights that: (i) GenIE can effectively transfer knowledge across datasets/schemas; (ii) GenIE can quickly adapt to new schemas. Due to its rigid, monolithic relation classifier, the pipeline baseline does not possess these qualities. However, the pretraining does improve its macro scores.
Only the newly developed pipeline baseline and GenIE can scale-up to the larger schema<sup>5</sup>, and as expected, this setting is more challenging for both models. However, GenIE still preserves a good F1 score of 68 micro and 34 macro, which is a relative increase of  $60\%$  and  $320\%$ , respectively, over the baseline. While the pipeline has a steeper drop from micro to macro scores, in general, a significant difference between the two is observed in every setting. This suggests that the models perform better for relations associated with many training examples and significantly worse for the rest. These findings call for the fine-grained analysis of performance in Sec. 5.2 that partitions the relations according to their occurrence count in the training data. For completeness, we also provide an analysis of performance as a function of the number of relations considered, in Appendix F.1.
See notes on reproducibility in Appendix D.
3https://github.com/flairNLP/flair 4https://github.com/facebookresearch/ GENRE