SlowGuess/ACL_2020_OCR huggingface dataset:
Zhepei Wei $^{1,2}$ , Jianlin Su $^{4}$ , Yue Wang $^{5}$ , Yuan Tian $^{1,2*}$ , Yi Chang $^{1,2,3*}$
$^{1}$ School of Artificial Intelligence, Jilin University
$^{2}$ Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin University
$^{3}$ International Center of Future Science, Jilin University
4Shenzhen Zhuiyi Technology Co., Ltd.
$^{5}$ School of Information and Library Science, University of North Carolina at Chapel Hill
weizp19@ mails.jlu.edu.cn, bojonesu@ wezhuiyi.com, wangyue  $@$  email.unc.edu,
yuantian@jlu.edu.cn, yichang@jlu.edu.cn
Extracting relational triples from unstructured text is crucial for large-scale knowledge graph construction. However, few existing works excel in solving the overlapping triple problem where multiple relational triples in the same sentence share the same entities. In this work, we introduce a fresh perspective to revisit the relational triple extraction task and propose a novel cascade binary tagging framework (CASREL) derived from a principled problem formulation. Instead of treating relations as discrete labels as in previous works, our new framework models relations as functions that map subjects to objects in a sentence, which naturally handles the overlapping problem. Experiments show that the CAS-REL framework already outperforms state-of-the-art methods even when its encoder module uses a randomly initialized BERT encoder, showing the power of the new tagging framework. It enjoys further performance boost when employing a pre-trained BERT encoder, outperforming the strongest baseline by 17.5 and 30.2 absolute gain in F1-score on two public datasets NYT and WebNLG, respectively. In-depth analysis on different scenarios of overlapping triples shows that the method delivers consistent performance gain across all these scenarios. The source code and data are released online<sup>1</sup>.
The key ingredient of a knowledge graph is relational facts, most of which consist of two entities connected by a semantic relation. These facts are in the form of (subject, relation, object), or  $(s,r,o)$  referred to as relational triples. Extracting relational triples from natural language text is a crucial step towards constructing large-scale knowledge graphs.
Figure 1: Examples of Normal, EntityPairOverlap (EPO) and SingleEntityOverlap (SEO) overlapping patterns.
Early works in relational triple extraction took a pipeline approach (Zelenko et al., 2003; Zhou et al., 2005; Chan and Roth, 2011). It first recognizes all entities in a sentence and then performs relation classification for each entity pair. Such an approach tends to suffer from the error propagation problem since errors in early stages cannot be corrected in later stages. To tackle this problem, subsequent works proposed joint learning of entities and relations, among them are feature-based models (Yu and Lam, 2010; Li and Ji, 2014; Miwa and Sasaki, 2014; Ren et al., 2017) and, more recently, neural network-based models (Gupta et al., 2016; Katiyar and Cardie, 2017; Zheng et al., 2017; Zeng et al., 2018; Fu et al., 2019). By replacing manually constructed features with learned representations, neural network-based models have achieved considerable success in the triple extraction task.
However, most existing approaches cannot efficiently handle scenarios in which a sentence contains multiple relational triples that overlap with each other. Figure 1 illustrates these scenarios, where triples share one or two entities in a sentence. This overlapping triple problem directly challenges conventional sequence tagging schemes that assume each token bears only one tag (Zheng et al., 2017). It also brings significant difficulty to relation classification approaches where an entity
*Joint Corresponding Author
<https://github.com/weizhepei/CasRel>
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1476-1488
pair is assumed to hold at most one relation (Miwa and Bansal, 2016). Zeng et al. (2018) is among the first to consider the overlapping triple problem in relational triple extraction. They introduced the categories for different overlapping patterns as shown in Figure 1 and proposed a sequence-to-sequence (Seq2Seq) model with copy mechanism to extract triples. Based on the Seq2Seq model, they further investigate the impact of extraction order (Zeng et al., 2019) and gain considerable improvement with reinforcement learning. Fu et al. (2019) also studied the overlapping triple problem by modeling text as relational graphs with a graph convolutional networks (GCNs) based model.
Despite their success, previous works on extracting overlapping triples still leave much to be desired. Specifically, they all treat relations as discrete labels to be assigned to entity pairs. This formulation makes relation classification a hard machine learning problem. First, the class distribution is highly imbalanced. Among all pairs of extracted entities, most do not form valid relations, generating too many negative examples. Second, the classifier can be confused when the same entity participates in multiple valid relations (overlapping triples). Without enough training examples, the classifier can hardly tell which relation the entity participates in. As a result, the extracted triples are usually incomplete and inaccurate.
In this work, we start with a principled formulation of relational triple extraction right at the triple level. This gives rise to a general algorithmic framework that handles the overlapping triple problem by design. At the core of the framework is the fresh perspective that instead of treating relations as discrete labels on entity pairs, we can model relations as functions that map subjects to objects. More precisely, instead of learning relation classifiers  $f(s,o) \to r$ , we learn relation-specific taggers  $f_{r}(s) \to o$ , each of which recognizes the possible object(s) of a given subject under a specific relation; or returns no object, indicating that there is no triple with the given subject and relation. Under this framework, triple extraction is a two-step process: first we identify all possible subjects in a sentence; then for each subject, we apply relation-specific taggers to simultaneously identify all possible relations and the corresponding objects.
We implement the above idea in CASREL, an end-to-end cascade binary tagging framework. It consists of a BERT-based encoder module, a sub
ject tagging module, and a relation-specific object tagging module. Empirical experiments show that the proposed framework outperforms state-of-the-art methods by a large margin even when the BERT encoder is not pre-trained, showing the superiority of the new framework itself. The framework enjoys a further large performance gain after adopting a pre-trained BERT encoder, showing the importance of rich prior knowledge in triple extraction task.
This work has the following main contributions:
1. We introduce a fresh perspective to revisit the relational triple extraction task with a principled problem formulation, which implies a general algorithmic framework that addresses the overlapping triple problem by design.
2. We instantiate the above framework as a novel cascade binary tagging model on top of a Transformer encoder. This allows the model to combine the power of the novel tagging framework with the prior knowledge in pretrained large-scale language models.
3. Extensive experiments on two public datasets show that the proposed framework overwhelmingly outperforms state-of-the-art methods, achieving 17.5 and 30.2 absolute gain in F1-score on the two datasets respectively. Detailed analyses show that our model gains consistent improvement in all scenarios.
Extracting relational triples from unstructured natural language texts is a well-studied task in information extraction (IE). It is also an important step for the construction of large scale knowledge graph (KG) such as DBpedia (Auer et al., 2007), Freebase (Bollacker et al., 2008) and Knowledge Vault (Dong et al., 2014).
Early works (Mintz et al., 2009; Gormley et al., 2015) address the task in a pipelined manner. They extract relational triples in two separate steps: 1) first run named entity recognition (NER) on the input sentence to identify all entities and 2) then run relation classification (RC) on pairs of extracted entities. The pipelined methods usually suffer from the error propagation problem and neglect the relevance between the two steps. To ease these issues, many joint models that aim to learn entities and relations jointly have been proposed. Traditional joint models (Yu and Lam, 2010; Li and Ji,
2014; Miwa and Sasaki, 2014; Ren et al., 2017) are feature-based, which heavily rely on feature engineering and require intensive manual efforts. To reduce manual work, recent studies have investigated neural network-based methods, which deliver state-of-the-art performance. However, most existing neural models like (Miwa and Bansal, 2016) achieve joint learning of entities and relations only through parameter sharing but not joint decoding. To obtain relational triples, they still have to pipeline the detected entity pairs to a relation classifier for identifying the relation of entities. The separated decoding setting leads to a separated training objective for entity and relation, which brings a drawback that the triple-level dependencies between predicted entities and relations cannot be fully exploited. Different from those works, Zheng et al. (2017) achieves joint decoding by introducing a unified tagging scheme and convert the task of relational triple extraction to an end-to-end sequence tagging problem without need of NER or RC. The proposed method can directly model relational triples as a whole at the triple level since the information of entities and relations is integrated into the unified tagging scheme.
Though joint models (with or without joint decoding) have been well studied, most previous works ignore the problem of overlapping relational triples. Zeng et al. (2018) introduced three patterns of overlapping triples and try to address the problem via a sequence-to-sequence model with copy mechanism. Recently, Fu et al. (2019) also study the problem and propose a graph convolutional networks (GCNs) based method. Despite their initial success, both methods still treat the relations as discrete labels of entity pairs, making it quite hard for the model to learn overlapping triples.
Our framework is based on a training objective that is carefully designed to directly model the relational triples as a whole like (Zheng et al., 2017), i.e., to learn both entities and relations through joint decoding. Moreover, we model the relations as functions that map subjects to objects, which makes it crucially different from previous works.
3 The CASREL Framework
The goal of relational triple extraction is to identify all possible (subject, relation, object) triples in a sentence, where some triples may share the same entities as subjects or objects. Towards this goal, we directly model the triples and design a training
objective right at the triple level. This is in contrast to previous approaches like (Fu et al., 2019) where the training objective is defined separately for entities and relations without explicitly modeling their integration at the triple level.
Formally, given annotated sentence  $x_{j}$  from the training set  $D$  and a set of potentially overlapping triples  $T_{j} = \{(s,r,o)\}$  in  $x_{j}$ , we aim to maximize the data likelihood of the training set  $D$ :
$$
\begin{array}{l} \prod_ {j = 1} ^ {| D |} \left[ \prod_ {(s, r, o) \in T _ {j}} p ((s, r, o) | x _ {j}) \right] (1) \\ = \prod_ {j = 1} ^ {| D |} \left[ \prod_ {s \in T _ {j}} p (s | x _ {j}) \prod_ {(r, o) \in T _ {j} | s} p ((r, o) | s, x _ {j}) \right] (2) \\ = \prod_ {j = 1} ^ {| D |} \left[ \prod_ {s \in T _ {j}} p (s \mid x _ {j}) \prod_ {r \in T _ {j} \mid s} p _ {r} (o \mid s, x _ {j}) \prod_ {r \in R \backslash T _ {j} \mid s} p _ {r} \left(o _ {\varnothing} \mid s, x _ {j}\right) \right]. (3) \\ \end{array}
$$
Here we slightly abuse the notation  $T_{j}$ .  $s \in T_{j}$  denotes a subject appearing in the triples in  $T_{j}$ .  $T_{j}|s$  is the set of triples led by subject  $s$  in  $T_{j}$ .  $(r,o) \in T_{j}|s$  is a  $(r,o)$  pair in the triples led by subject  $s$  in  $T_{j}$ .  $R$  is the set of all possible relations.  $R\backslash T_j|s$  denotes all relations except those led by  $s$  in  $T_{j}$ .  $o_{\emptyset}$  denotes a "null" object (explained below).
Eq. (2) applies the chain rule of probability. Eq. (3) exploits the crucial fact that for a given subject  $s$ , any relation relevant to  $s$  (those in  $T_{j}|s$ ) would lead to corresponding objects in the sentence, and all other relations would necessarily have no object in the sentence, i.e. a "null" object.
This formulation provides several benefits. First, since the data likelihood starts at the triple level, optimizing this likelihood corresponds to directly optimizing the final evaluation criteria at the triple level. Second, by making no assumption on how multiple triples may share entities in a sentence, it handles the overlapping triple problem by design. Third, the decomposition in Eq. (3) inspires a novel tagging scheme for triple extraction: we learn a subject tagger  $p(s|x_j)$  that recognizes subject entities in a sentence; and for each relation  $r$ , we learn an object tagger  $p_r(o|s,x_j)$  that recognizes relation-specific objects for a given subject. In this way we can model each relation as a function that maps subjects to objects, as opposed to classifying relations for (subject, object) pairs.
Indeed, this novel tagging scheme allows us to extract multiple triples at once: we first run the subject tagger to find all possible subjects in the
sentence, and then for each subject found, apply relation-specific object taggers to find all relevant relations and the corresponding objects.
The key components in the above general framework, i.e., the subject tagger and relation-specific object taggers, can be instantiated in many ways. In this paper, we instantiate them as binary taggers on top of a deep bidirectional Transformer BERT (Devlin et al., 2019). We describe its detail below.
3.1 BERT Encoder
The encoder module extracts feature information  $\mathbf{x}_j$  from sentence  $x_j$ , which will feed into subsequent tagging modules<sup>2</sup>. We employ a pre-trained BERT model (Devlin et al., 2019) to encode the context information.
Here we briefly review BERT, a multi-layer bidirectional Transformer based language representation model. It is designed to learn deep representations by jointly conditioning on both left and right context of each word, and it has recently been proven surprisingly effective in many downstream tasks (Zhong et al., 2019). Specifically, it is composed of a stack of  $N$  identical Transformer blocks. We denote the Transformer block as  $\operatorname{Trans}(\mathbf{x})$ , in which  $\mathbf{x}$  represents the input vector. The detailed operations are as follows:
$$
\mathbf {h} _ {0} = \mathbf {S} \mathbf {W} _ {s} + \mathbf {W} _ {p} \tag {4}
$$
$$
\mathbf {h} _ {\alpha} = \operatorname {T r a n s} \left(\mathbf {h} _ {\alpha - 1}\right), \alpha \in [ 1, N ] \tag {5}
$$
where  $\mathbf{S}$  is the matrix of one-hot vectors of subwords $^3$  indices in the input sentence,  $\mathbf{W}_s$  is the sub-words embedding matrix,  $\mathbf{W}_p$  is the positional embedding matrix where  $p$  represents the position index in the input sequence,  $\mathbf{h}_{\alpha}$  is the hidden state vector, i.e., the context representation of input sentence at  $\alpha$ -th layer and  $N$  is the number of Transformer blocks. Note that in our work the input is a single text sentence instead of sentence pair, hence the segmentation embedding as described in original BERT paper was not taken into account in Eq. (4). For a more comprehensive description of the Transformer structure, we refer readers to (Vaswani et al., 2017).
3.2 Cascade Decoder
Now we describe our instantiation of the novel cascade binary tagging scheme inspired by the previous formulation. The basic idea is to extract triples in two cascade steps. First, we detect subjects from the input sentence. Then for each candidate subject, we check all possible relations to see if a relation can associate objects in the sentence with that subject. Corresponding to the two steps, the cascade decoder consists of two modules as illustrated in Figure 2: a subject tagger; and a set of relation-specific object taggers.
Subject Tagger The low level tagging module is designed to recognize all possible subjects in the input sentence by directly decoding the encoded vector  $\mathbf{h}_N$  produced by the  $N$ -layer BERT encoder. More precisely, it adopts two identical binary classifiers to detect the start and end position of subjects respectively by assigning each token a binary tag (0/1) that indicates whether the current token corresponds to a start or end position of a subject. The detailed operations of the subject tagger on each token are as follows:
$$
p _ {i} ^ {\text {s t a r t - s}} = \sigma \left(\mathbf {W} _ {\text {s t a r t}} \mathbf {x} _ {i} + \mathbf {b} _ {\text {s t a r t}}\right) \tag {6}
$$
$$
p _ {i} ^ {e n d \_ s} = \sigma (\mathbf {W} _ {e n d} \mathbf {x} _ {i} + \mathbf {b} _ {e n d}) \qquad (7)
$$
where  $p_i^{start\_s}$  and  $p_i^{end\_s}$  represent the probability of identifying the  $i$ -th token in the input sequence as the start and end position of a subject, respectively. The corresponding token will be assigned with a tag 1 if the probability exceeds a certain threshold or with a tag 0 otherwise.  $\mathbf{x}_i$  is the encoded representation of the  $i$ -th token in the input sequence, i.e.,  $\mathbf{x}_i = \mathbf{h}_N[i]$ , where  $\mathbf{W}_{(\cdot)}$  represents the trainable weight, and  $\mathbf{b}_{(\cdot)}$  is the bias and  $\sigma$  is the sigmoid activation function.
The subject tagger optimizes the following likelihood function to identify the span of subject  $s$  given a sentence representation  $\mathbf{x}$ :
$$
\begin{array}{l} p _ {\theta} (s | \mathbf {x}) \\ = \prod_ {t \in \{s t a r t _ {-} s, e n d _ {-} s \}} \prod_ {i = 1} ^ {L} \left(p _ {i} ^ {t}\right) ^ {\mathbf {I} \left\{y _ {i} ^ {t} = 1 \right\}} \left(1 - p _ {i} ^ {t}\right) ^ {\mathbf {I} \left\{y _ {i} ^ {t} = 0 \right\}}. \tag {8} \\ \end{array}
$$
where  $L$  is the length of the sentence.  $\mathbf{I}\{z\} = 1$  if  $z$  is true and 0 otherwise.  $y_{i}^{start\_s}$  is the binary tag of subject start position for the  $i$ -th token in  $\mathbf{x}$ , and  $y_{i}^{end\_s}$  indicates the subject end position. The parameters  $\theta = \{\mathbf{W}_{start}, \mathbf{b}_{start}, \mathbf{W}_{end}, \mathbf{b}_{end}\}$ .
This paper uses boldface letters to denote vectors and matrices.
3We use WordPiece embeddings (Wu et al., 2016) to represent words in vector space as in BERT (Devlin et al., 2019). Each word in the input sentence will be tokenized to fine-grained tokens, i.e., sub-words.
Figure 2: An overview of the proposed CASREL framework. In this example, there are three candidate subjects detected at the low level, while the presented  $0/1$  tags at high level are specific to the first subject Jackie R. Brown, i.e., a snapshot of the iteration state when  $k = 1$  is shown as above. For the subsequent iterations ( $k = 2, 3$ ), the results at high level will change, reflecting different triples detected. For instance, when  $k = 2$ , the high-level orange (green) blocks will change to 0 (1), respectively, reflecting the relational triple (Washington, Capital_of, United States Of America) led by the second candidate subject Washington.
For multiple subjects detection, we adopt the nearest start-end pair match principle to decide the span of any subject based on the results of the start and end position taggers. For example, as shown in Figure 2, the nearest end token to the first start token "Jackie" is "Brown", hence the detected result of the first subject span will be "Jackie R. Brown". Notably, to match an end token for a given start token, we don't consider tokens whose position is prior to the position of the given token. Such match strategy is able to maintain the integrity of any entity span if the start and end positions are both correctly detected due to the natural continuity of any entity span in a given sentence.
Relation-specific Object Taggers The high level tagging module simultaneously identifies the objects as well the involved relations with respect to the subjects obtained at lower level. As Figure 2 shows, it consists of a set of relation-specific object taggers with the same structure as subject tagger in low level module for all possible relations. All object taggers will identify the corresponding object(s) for each detected subject at the same time.
Different from subject tagger directly decoding the encoded vector  $\mathbf{h}_N$ , the relation-specific object tagger takes the subject features into account as well. The detailed operations of the relation-specific object tagger on each token are as follows:
$$
\begin{array}{l} p _ {i} ^ {\text {s t a r t - o}} = \sigma \left(\mathbf {W} _ {\text {s t a r t}} ^ {r} \left(\mathbf {x} _ {i} + \mathbf {v} _ {\text {s u b}} ^ {k}\right) + \mathbf {b} _ {\text {s t a r t}} ^ {r}\right) \tag {9} \\ p _ {i} ^ {e n d - o} = \sigma \big (\mathbf {W} _ {e n d} ^ {r} (\mathbf {x} _ {i} + \mathbf {v} _ {s u b} ^ {k}) + \mathbf {b} _ {e n d} ^ {r} \big) (1 0) \\ \end{array}
$$
where  $p_i^{start - o}$  and  $p_i^{end - o}$  represent the probability of identifying the  $i$ -th token in the input sequence as the start and end position of a object respectively, and  $\mathbf{v}_{sub}^k$  represents the encoded representation vector of the  $k$ -th subject detected in low level module.
For each subject, we iteratively apply the same decoding process on it. Note that the subject is usually composed of multiple tokens, to make the additions of  $\mathbf{x}_i$  and  $\mathbf{v}_{sub}^k$  in Eq. (9) and Eq. (10) possible, we need to keep the dimension of two vectors consistent. To do so, we take the averaged vector representation between the start and end tokens of the  $k$ -th subject as  $\mathbf{v}_{sub}^k$ .
The object tagger for relation  $r$  optimizes the following likelihood function to identify the span
of object  $o$  given a sentence representation  $\mathbf{x}$  and a subject  $s$ :
$$
\begin{array}{l} p _ {\phi_ {r}} (o | s, \mathbf {x}) \\ = \prod_ {t \in \{s t a r t _ {-} o, e n d _ {-} o \}} \prod_ {i = 1} ^ {L} \left(p _ {i} ^ {t}\right) ^ {\mathbf {I} \left\{y _ {i} ^ {t} = 1 \right\}} \left(1 - p _ {i} ^ {t}\right) ^ {\mathbf {I} \left\{y _ {i} ^ {t} = 0 \right\}}. \tag {11} \\ \end{array}
$$
where  $y_{i}^{start\_o}$  is the binary tag of object start position for the  $i$ -th token in  $\mathbf{x}$ , and  $y_{i}^{end\_o}$  is the tag of object end position for the  $i$ -th token. For a "null" object  $o_{\emptyset}$ , the tags  $y_{i}^{start\_o\emptyset} = y_{i}^{end\_o\emptyset} = 0$  for all  $i$ . The parameters  $\phi_r = \{\mathbf{W}_{start}^r, \mathbf{b}_{start}^r, \mathbf{W}_{end}^r, \mathbf{b}_{end}^r\}$ .
Note that in the high level tagging module, the relation is also decided by the output of object taggers. For example, the relation "Work_in" does not hold between the detected subject "Jackie R. Brown" and the candidate object "Washington". Therefore, the object tagger for relation "Work_in" will not identify the span of "Washington", i.e., the output of both start and end position are all zeros as shown in Figure 2. In contrast, the relation "Birth_place" holds between "Jackie R. Brown" and "Washington", so the corresponding object tagger outputs the span of the candidate object "Washington". In this setting, the high level module is capable of simultaneously identifying the relations and objects with regard to the subjects detected in low level module.
3.3 Data Log-likelihood Objective
Taking log of Eq. (3), the objective  $J(\Theta)$  is: