SlowGuess/NAACL_2022_OCR huggingface dataset:
Farima Fatahi Bayat
farimaf@umich.edu
Nikita Bhutani
nikita@megagon.ai
H. V. Jagadish
jag@umich.edu
A major drawback of modern neural OpenIE systems and benchmarks is that they prioritize high coverage of information in extractions over compactness of their constituents. This severely limits the usefulness of OpenIE extractions in many downstream tasks. The utility of extractions can be improved if extractions are compact and share constituents. To this end, we study the problem of identifying compact extractions with neural-based methods. We propose COMPACTIE, an OpenIE system that uses a novel pipelined approach to produce compact extractions with overlapping constituents. It first detects constituents of the extractions and then links them to build extractions. We train our system on compact extractions obtained by processing existing benchmarks. Our experiments on CaRB and Wire57 datasets indicate that COMPACTIE finds  $1.5 \times 2 \times 2$  more compact extractions than previous systems, with high precision, establishing a new state-of-the-art performance in OpenIE.
A popular domain-agnostic paradigm to structure the raw text is open information extraction (OpenIE) (Banko et al., 2007). Not relying on any predefined schema, OpenIE systems typically extract information as (subject; relation; object) triples. The extracted information is then used in several downstream applications, including answering questions (Khot et al., 2017), summarizing documents (Hao et al., 2018; Ji et al., 2013), and populating knowledge bases (Fan et al., 2019).
Despite much progress, state-of-the-art neural OpenIE systems focus on covering more information from the input sentence often at the cost of utility and compactness of the extracted triples. The extracted triples have long, over-specific constituents (i.e. the relation and its arguments). Figure 1 illustrates such example triples produced by a popular
Beth is the second child of Henry, born in wedlock.
IMoJIE
Extractions	E1: ( Beth ; is ; the second child of Henry born in wedlock )
Compact
Extractions	E1: ( Beth ; is ; the second child of Henry )
E2: ( the second child of Henry ; born ; in wedlock )
The rest of the group reach a small shop , where the crocodile
breaks through a wall and devours Annabelle.
IMoJIE
Extractions	E1: ( The rest of the group ; reach ; a small shop , where the
crocodile breaks through a wall and devours Annabelle )
E2: ( the crocodile ; devours ; Annabelle a small shop)
Compact
Extractions	E1: ( The rest of the group ; reach ; a small shop )
E2: ( crocodile ; breaks ; through a wall )
E3: ( crocodile ; devours ; Annabelle )
Figure 1: Example sentences with non-compact triples from IMoJIE vs. compact triples from our benchmark. Compact triples can share constituents. Constituents for subjects, relations and objects are indicated in blue, green and orange, respectively.
OpenIE system, IMoJIE (Kolluru et al., 2020b). As shown, the knowledge that the second child of Henry was born in wedlock is embedded in a long argument. This can be problematic for downstream applications, especially knowledge base population (Gashteovski et al., 2020; Stanovsky et al., 2015) that derive power from merging multiple pieces of information extracted about the same entity. In contrast, the compact extractions are more pliable for tasks such as identifying similar facts and merging facts that share constituents. For example, compact extractions in Figure 1 can be merged to derive that Beth is born in wedlock.
Although some prior work (Corro and Gemulla, 2013; Gashteovski et al., 2017; Bhutani et al., 2016) has explored the compactness of OpenIE triples, these systems are rule-based and have been superseded by end-to-end neural OpenIE systems. In this work, we study the problem of identifying compact extractions with neural-based methods. Inspired by (Corro and Gemulla, 2013), we define an extracted triple to be compact if it does not contain information that can be independently represented in another triple. To further improve the suitability of
Human Language Technologies, pages 900 - 910
compact triples for knowledge base population, we require compact triples extracted from a sentence to have overlapping constituents.
Existing neural systems adopt a sequence labeling (Kolluru et al., 2020a; Wang et al., 2021; Ro et al., 2020) or a sequence generation (Kolluru et al., 2020b) approach to identify triples and their constituents, typically all at once, or through a pipeline that first identifies the relations and then their corresponding arguments. None of these methods guarantee that the extracted triples will be compact and share constituents.
We propose a novel pipeline system for finding compact triples that share their constituents. We call our OpenIE system, COMPACTIE. To encourage the constituents to be shared across triples, COMPACTIE first extracts the constituents using a Constituent Extraction model and then links them using a Constituent Linking model to obtain triples.
We adapt a table filling method (Wang et al., 2021) with a new schema for identifying both constituent boundaries and their roles (i.e., subject or object). This allows the constituent extraction model to capture interactions among constituents and minimize ambiguities in boundary detection. For the task of constituent linking, we train a model that builds on contextual representations specific to a given pair of constituents and predicts their relation type. Such a two-step approach enables us to optimize the models for each sub-task with different objectives and also promote the constituent reuse across triples.
Existing neural OpenIE systems are trained on benchmarks that combine extractions from multiple OpenIE systems. However, no such large-scale benchmark exists for compact triples. We develop a new benchmark using a subset of sentences in the OpenIE2016 benchmark (Mausam, 2016). Specifically, we develop a data processing algorithm that targets extraction from individual clauses in a sentence. Given an input sentence, it identifies clauses and then uses OpenIE systems such as IMoJIE over the clauses to find compact triples. We train COMPACTIE on the new benchmark.
Our experiments on a fine-grained benchmark, Wire57, show that COMPACTIE outperforms existing non-neural and neural systems by 5.8 F1 pts and 7.1 F1 pts, respectively. Manual evaluation over a coarse-grained benchmark, CaRB, indicates that COMPACTIE produces  $1.5\mathrm{x} - 2\mathrm{x}$  more compact extractions than existing systems with comparable
precision, establishing a new state-of-the-art for the OpenIE task<sup>1</sup>.
Given a sentence  $s = w_{1}w_{2}\dots w_{n}$ , an OpenIE system generates triples of the form (subject; relation; object), where subject, relation and object are the constituents of a triple.
2.1 Extracting Compact Triples
A recent study (Gashteovski et al., 2020) shows that triples from modern neural OpenIE systems are difficult to align to knowledge bases such as DBpedia. Less than  $77\%$  of triples from neural OpenIE systems had the same arguments as DBpedia facts. In contrast, the corresponding alignment ratio for some of the non-neural OpenIE systems was as high as  $98\%$ . They attribute this behavior to the specificity of the triples. A compact triple, which does not contain complete clauses as part of a constituent or contain additional information, is easier to align to DBpedia. Our goal is to leverage neural-based methods to extract compact triples.
2.2 System Architecture
We focus on extraction from individual clauses within a sentence, where each clause includes a subject, a verb, optionally a direct object, and a compliment. Since extractions from different clauses share information, we split the OpenIE task into two sub-tasks: constituent extraction and constituent linking.
The task of constituent extraction is to find a set of constituents such that each constituent  $c$  is a contiguous span of words  $c.\text{span} = \{(w_i, w_j)\}$  and has a pre-defined type  $c.\text{type} \in Y_c$  where  $Y_c = \{\text{Argument}, \text{Predicate}\}$ . The constituent that takes the relation role in a triple has  $c.\text{type} = \text{Predicate}$ , and subject and object constituents have  $c.\text{type} = \text{Argument}$ . This schema simplifies the task and provides more information to the constituent linking model.
The task of constituent linking is to connect a given set of Predicate constituents  $\{p_1, \ldots, p_m\}$  and Argument constituents  $\{a_1, \ldots, a_n\}$  to obtain triples. We formulate this as a relation classification task where the set of relations is  $Y_r = \{\text{Subject}, \text{Object}\}$ . The model predicts relations  $r$  between each  $p_x$  and  $\{a_1, \ldots, a_n\}$  such that:
<sup>1</sup>Source code, benchmark dataset, and related resources are available at https://github.com/FarimaFatahi/CompactIE
$\exists (i,j):r(a_i,p_x) = Subject, r(p_x,a_j) = Object$
to construct triple  $(a_i; p_x; a_j)$ .
Figure 2: Table filling based on the relation between each pair of words in the sentence. Argument (A) and Predicate (P) are constituent types. Subject (S) and Object (O) declare the relation between two constituents (N stands for no relation).
In this section we describe our pipeline system, COMPACTIE. We first detail the constituent extraction model, its training constraints, and the decoding algorithm in Section 3.1. Then, we describe the constituent linking model in Section 3.2. Figure 3 shows an overview of COMPACTIE architecture.
3.1 Constituent Extraction Model
The constituent extraction model aims to find constituent spans and their types in a sentence. Following recent progress in entity-relation extraction (Wang et al., 2021), we model this as a table filling problem. However, we design a new table schema for the constituent extraction task. Figure 2 shows an example schema. A sentence  $s$  with  $|s|$  tokens corresponds to a table  $T^{|s| \times |s|}$  such that each cell is labeled based on the relation between the pair of words. For each constituent, corresponding cells are labeled with  $y_c \in \{Argument, Predicate\}$ . For relations between different constituents, corresponding cells are labeled with  $y_r \in \{Subject, Object\}$ . The cells with no relations are labeled None. Graphically, constituents are squares on the diagonal, and relations are rectangles off the diagonal.
3.1.1 Table Filling Model
Given the tabular formulation, the constituent extractor performs two tasks: a) fill the table by predicting labels for each word pair, b) extract the constituents given the label probabilities. Following (Wang et al., 2021), we adopt a biaffine attention mechanism, described next, to learn interactions between word pairs when filling the table.
Given the input sentence  $s$ , we first obtain contextual representation  $\mathbf{h_i}$  for each word using a pretrained language model (e.g. BERT (Devlin et al., 2018)). We then employ two MLPs to identify the head and tail role of the word given its vector representation  $h_i$ .
$$
h _ {i} ^ {h e a d} = M L P _ {h e a d} (h _ {i}), h _ {i} ^ {t a i l} = M L P _ {t a i l} (h _ {i})
$$
Next, using the biaffine scoring function, we calculate the scoring vector of each pair of words (e.g.  $w_{i}, w_{j}$ ) as follows:
$$
t _ {i, j} = \left(h _ {i} ^ {\text {h e a d}}\right) ^ {T} U ^ {(1)} h _ {j} ^ {\text {t a i l}} + \left(h _ {i} ^ {\text {h e a d}} \oplus h _ {j} ^ {\text {t a i l}}\right) ^ {T} U ^ {(2)} + b
$$
where  $U^{(1)}, U^{(2)}$  are weight parameters,  $b$  is the bias term and  $\oplus$  denotes concatenation. Then, we feed the score vector  $t_i, j$  into a softmax function to calculate the probability distribution of the corresponding labels  $l \in Y$ , where  $Y = Y_c \cup Y_r \cup None$ .
$$
P \left(y _ {i, j} \mid s\right) = \operatorname {S o f t m a x} \left(t _ {i, j}\right)
$$
Finally, we train the 2D table to minimize the following training objective:
$$
L _ {e n t r y} = - \frac {1}{| s | ^ {2}} \sum_ {i = 1} ^ {| s |} \sum_ {j = 1} ^ {| s |} \log (P (y _ {i, j} = Y _ {i, j} | s))
$$
where  $Y_{i,j}$  is the gold label for cell  $(i,j)$  in the table.
3.1.2 Training Constraints
(Wang et al., 2021) shows that structural constraints imposed on the table during training can significantly enhance the model. We adopt their symmetry and implication constraints. However, we observed that these alone are not sufficient if certain labels are preferred over others. For example, all triples must have a subject, but some may not have an object. We propose a new triple constraint to further enhance our model. In this section, we describe the three constraints in detail. We also introduce  $\mathcal{P} \in R^{|s| \times |s| \times |Y|}$  that denotes the stack of  $P(y_{i,j}|s)$  for all word pairs in sentence  $s$ .
Symmetry: This constraint ensures that the table is symmetric i.e. the squares are symmetric about the diagonal. As shown in Figure 2, this ensures the label assigned to the (second, Henry) cell is the
Figure 3: Overview of system architecture. Given the sentence: "Beth was the second child of Henry, born in wedlock.", the Constituent Extraction model identifies the span and type of constituents (top-right). Next, the Constituent Linking model (b) searches for Arguments of each Predicate constituent independently. Thus, for each of the two extracted Predicates, it modifies the input sentence by inserting typed constituent markers  $(<\mathrm{Arg}>, </\mathrm{Arg}>$  to specify the start and end of arguments and  $<\mathrm{Pr}>, </\mathrm{Pr}>$  for predicates). Finally, the modified sentence is fed into a classifier to find Subject and Object of each Predicate and form triples (c).
same as the cell (Henry, second). Given matrix  $\mathcal{P}$  We formulate this constraint as symmetrical loss:
$$
L _ {s y m} = - \frac {1}{| s | ^ {2}} \sum_ {i = 1} ^ {| s |} \sum_ {j = 1} ^ {| s |} \sum_ {t \in Y _ {r} \cup Y _ {c}} | \mathcal {P} _ {i, j, t} - \mathcal {P} _ {j, i, t} |
$$
Implication: This constraint implies that no relation would appear unless its constituents are present in the table. This is imposed on  $\mathcal{P}$ : for each word in the diagonal, maximum possibility over the constituent type space  $Y_{c} = \{\text{Argument}, \text{Predicate}\}$  is not lower than the maximum possibility for other words in the same row or column over the relation type space  $Y_{r} = \{\text{Subject}, \text{Object}\}$ .
$$
L _ {i m p} = \frac {- 1}{| s |} \sum_ {i = 1} ^ {| s |} \left[ \max _ {t \in Y _ {r}} (\mathcal {P} _ {i,:, t}, \mathcal {P} _ {:, i, t}) - \max _ {t \in Y _ {c}} (\mathcal {P} _ {i, i, t}) \right] _ {*}
$$
Triple Constraint: This constraint enables the model to increase the likelihood of certain roles (e.g. Subject) over the others (e.g. Object) to ensure the triples are valid. We enforce this constraint on  $\mathcal{P}$ : For each column or row corresponding to a Predicate constituent, the maximum possibility of off-diagonal words over Subject type is not lower than the maximum possibility of off-diagonal words over Object type. We formulate this constraint as triple loss.
$$
\begin{array}{l} L _ {t r i p l e} = \frac {- 1}{2 | p s |} \sum_ {i \in p s} \left[ \left\{\max  (\mathcal {P} _ {i,:}, \boldsymbol {O}) - \max  (\mathcal {P} _ {i,:}, \boldsymbol {S} \right\} \right. \\ \left. + \{m a x (\mathcal {P} _ {:, i, \pmb {O}}) - m a x (\mathcal {P} _ {:, i, \pmb {S}}) \} \right] \\ \end{array}
$$
where  $ps$  is union of Predicate spans in sentence.
Finally, we jointly optimize four objectives in training:  $L_{entry} + L_{sym} + L_{imp} + L_{triple}$
3.1.3 Decoding
Given the label probability tensor  $\mathcal{P}$ , we need to decode the constituents in the testing phase. We follow a 2-step decoding procedure that finds spans of constituents first and then assigns a label to each span. The decoder first calculates the distance between adjacent rows and columns of the table to find constituents' boundaries. Next, it assigns a type to each span and filters out any None constituents before passing the output to the linking model. The upper part of Figure 3 shows the output of the decoder, which extracts two constituents ("was", "born") of type Predicate and three constituents ("Beth", "the second child of Henry", "in wedlock") of type Argument. We provide a detailed description of the decoding algorithm in Appendix A.2.
3.2 Constituent Linking Model
The constituent linking model aims to take a Predicate constituent and a set of Argument constituents as input and predict a relation label
$^2 [u]_{*} = max(u,0)$  is the hinge loss.
$Y_{r} = \{Subject, Object, None\}$ . This procedure is repeated for each predicate constituent in the sentence. We formulate this as a relation classification task where the model classifies relation labels of given constituent pairs based on context.
Following prior work (Zhang et al., 2019; Zhong and Chen, 2020), we modify the token sequence of input sentence by adding marker tokens  $\langle \mathrm{Pr} \rangle$ ,  $\langle / \mathrm{Pr} \rangle$ ,  $\langle \mathrm{Arg} \rangle$ ,  $\langle / \mathrm{Arg} \rangle$  to highlight the constituent spans and their types. The markers help the linking model combine context information and constituent information for relation classification. As shown in Figure 3.a, two types of constituents are extracted from the input sentence. For each constituent of type Predicate, we modify the input sentence by highlighting the location of the Predicate and all Argument constituents. Then, we feed this processed sentence to a pre-trained encoder (BERT).
Next, we concatenate the output representation of the start position of predicate  $p$  with the output representation of the start position of argument  $a_{i}$ :
$$
X _ {r} (p, a _ {i}) = h _ {s t a r t (p)} \oplus h _ {s t a r t (a _ {i})}
$$
Finally, we feed the concatenated representation into a multi-layer perceptron (MLP) to predict the probability distribution of the relation type  $r \in Y_r \cup \text{None}$ :
$$
P (r | p, a _ {i}) = M L P (X _ {r})
$$
4 Benchmark Creation
To train the constituent extraction and constituent linking models for extracting compact triples, we need a benchmark of compact triples. Existing OpenIE benchmark  ${}^{3}$  is created by combining extractions from multiple existing OpenIE systems. Although widely adopted, we observed that it includes over-specific and sometimes incorrect extractions from previous systems. This encouraged us to design a data processing algorithm that can extract compact triples from scratch. Inspired by rule-based OpenIE system (Corro and Gemulla, 2013), we find compact triples by extracting the following clauses within a sentence:
Main Clauses are independent clauses that express a complete concept.
Complement Clauses are subordinate clauses that serve to complete the meaning of a verb or noun in the sentence.
Our Benchmark	OIE2016
Total	Train	Valid
# Sentence	54.9k	54.5k	500	92.7k
# Triples	121.8k	120.6k	1155	190.6k
Avg. # triples per sent.	3.165	-	-	2.542
Avg. constituent length	4.587	-	-	7.893
Table 1: Statistics of our benchmark and OpenIE2016 benchmark.
Coordinate Clauses are independent clauses joined to the main clause using coordinating conjunctions such as and, or, but, etc.
We identify clauses within a sentence using its dependency graph. We first build a sentence tree such that the root is the head of the main clauses and the first-level children are clauses modifying the root word. We then perform a postfix traversal of the tree until we find a sub-tree with no clausal children. At this point, we run a standard OpenIE system, IMoJIE (Kolluru et al., 2020b), over the clause corresponding to the sub-tree to obtain triples. We then backtrack and extract triples for other clausal children and lastly the parent. We provide pseudo-code of algorithm in Appendix A. We run our algorithm on each multi-clause sentence in the OpenIE2016 benchmark and obtain a new benchmark tailored for extracting compact triples. Figure 1 shows example sentences and compact triples from this benchmark.
Training Dataset: We train COMPACTIE using the benchmark described in Section 4. Table 1 compares the statistics of our new benchmark and bootstrapped OpenIE2016 benchmark. As shown, our benchmark has 1.25 times more extractions per sentence than OpenIE2016 and its constituents are more compact. We use about  $1\%$  of sentences for validation and the remaining for training.
Comparison Systems: We compare COMPACTIE against state-of-the-art sequence-labeling systems, OpenIE6 (Kolluru et al., 2020a) and Multi2OIE (Ro et al., 2020), and sequence-generation system, IMoJIE (Kolluru et al., 2020b)). We also compare it against traditional non-neural systems designed for extracting compact facts: NestIE (Bhutani et al., 2016) and MinIE (Gashteovski et al., 2017).
Evaluation Datasets and Metrics: We evaluate the OpenIE systems both automatically and manually on standardized benchmarks. For automatic
<sup>3</sup>https://github.com/dair-iitd/imojie/tree/master/benchmark
Dataset	Wire57	CaRB
Proc	Orig	Proc	Orig
# Sentences	56	57	577	641
# Triples	309	325	2101	2715
Table 2: Statistics of evaluation datasets, Wire57 and CaRB, before (Orig) and after processing (Proc).
evaluation, we first assess all systems with  $\mathrm{CaRB^4}$  test and Wire57 datasets. Since these datasets are not targeted for compact triples, for a fair comparison we exclude triples that have at least one clause within a constituent. Table 2 shows the statistics of the original and processed datasets. Each dataset also provides its own scoring function. We report precision (P), recall (R), and F1 computed by these scoring functions. Wire57 contains more fine-grained extractions than the CaRB dataset and its scoring function is more rigorous for compact facts since it penalizes over-specific extractions. However, both CaRB and Wire57 scoring functions are based on token-level matching of system extractions against ground truth facts. Moreover, these benchmarks are incomplete, meaning that the gold extractions do not include all acceptable surface realizations of the same fact. These drawbacks encouraged us to additionally perform a fact-centered evaluation using the BenchIE (Gashteovski et al., 2021) benchmark and scoring paradigm. Finally, we carry out a manual evaluation on 100 sentences to avoid bias towards different scorers.
Implementation Details: Since the schema design of the table filling model does not support conjunctions inside constituents, we follow previous work (Kolluru et al., 2020a) and pre-process the sentences into smaller conjunction-free sentences before passing them to the system.
For a fair comparison to previous work, we use bert-based-uncased (Devlin et al., 2018) as the text encoder for both the constituent extraction model and constituent linking model. Each model contains nearly 110M parameters. For both models, we set the max sequence length to 512, initial learning rate to 5e-5, weight decay to 1e-5, and the batch size to 32. We use AdamW optimizer to fine-tune each model. The batch size is 300 for constituent extraction model and 20 for the constituent linking model, both equipped with early stopping. We use NVIDIA GeForce RTX 2080 Ti GPU to train both models for a cumulative time of 8 hours.
6.1 Automatic Token-level Evaluation
Table 3 summarizes the performance of OpenIE systems across the CaRB and Wire57 datasets and scoring functions. On the fine-grained Wire57 dataset with a strict Wire57 scorer, COMPACTIE outperforms neural OpenIE systems (by 7.2 - 9 F1 pts) and non-neural systems (by 5.8 - 10.8 F1 pts).