SlowGuess/NAACL_2021_OCR huggingface dataset:
Xuelu Chen $^{1*}$ , Michael Boratko $^{2*}$ , Muhao Chen $^{3,4}$
Shib Sankar Dasgupta $^{2}$ , Xiang Lorraine Li $^{2}$ , Andrew McCallum $^{2}$
$^{1}$ Department of Computer Science, UCLA
$^{2}$ College of Information and Computer Sciences, UMass Amherst
$^{3}$ Department of Computer Science, USC
<sup>4</sup>Information Sciences Institute, USC
shirleychen@cs.ucla.edu;mboratko@iesl.cs.umass.edu;
muhaoche@usc.edu; {ssdasgupta, xiangl, mccallum}@cs.umass.edu
Knowledge bases often consist of facts which are harvested from a variety of sources, many of which are noisy and some of which conflict, resulting in a level of uncertainty for each triple. Knowledge bases are also often incomplete, prompting the use of embedding methods to generalize from known facts, however existing embedding methods only model triple-level uncertainty and reasoning results lack global consistency. To address these shortcomings, we propose BEUrRE  $\diamond$ , a novel uncertain knowledge graph embedding method with calibrated probabilistic semantics. BEUrRE models each entity as a box (i.e. axis-aligned hyperrectangle), and relations between two entities as affine transforms on the head and tail entity boxes. The geometry of the boxes allows for efficient calculation of intersections and volumes, endowing the model with calibrated probabilistic semantics and facilitating the incorporation of relational constraints. Extensive experiments on two benchmark datasets show that BEUrRE consistently outperforms baselines on confidence prediction and fact ranking due to its probabilistic calibration and ability to capture high-order dependencies among facts. $^{1}$
Knowledge graphs (KGs) provide structured representations of facts about real-world entities and relations. In addition to deterministic KGs (Bollacker et al., 2008; Lehmann et al., 2015; Mahdisoltani et al., 2015), much recent attention has been paid to uncertain KGs (or UKGs). UKGs, such as ProBase (Wu et al., 2012), NELL (Mitchell et al., 2018), and ConceptNet (Speer et al., 2017), associate each fact (or triple) with a confidence score representing the likelihood of that fact to be true. Such uncertain knowledge representations critically capture
(The Beatles, genre, Rock): confidence?
Figure 1: BEURRE models entities as boxes and relations as two affine transforms.
the uncertain nature of reality, and provide more precise reasoning. For example, while both (Honda, competeswith, Toyota) and (Honda, competeswith, Chrysler) look somewhat correct, the former fact should have a higher confidence than the latter one, since Honda and Toyota are both Japanese car manufacturers and have highly overlapping customer bases. Similarly, while (The Beatles, genre, Rock) and (The Beatles, genre, Pop) are both true, the first one may receive a slightly higher confidence, since the Beatles is generally considered a rock band. Such confidence information is important when answering questions like Who is the main competitor of Honda?, or extracting confident knowledge for drug repurposing (Sosa et al., 2020).
To facilitate automated knowledge acquisition for UKGs, some UKG embedding models (Chen et al., 2019; Kertkeidkachorn et al., 2019) have recently been proposed. Inspired by the works about deterministic KG embeddings (Yang et al., 2015; Bordes et al., 2013), existing approaches model entities and relations as points in low-dimensional vector space, measure triple plausibility with vector similarity (eg. distance, dot-product), and map the plausibility to the confidence range of [0, 1]. For instance, the representative work UKGE (Chen et al., 2019) models the triple plausibility in the
* Indicating equal contribution.
<sup>1</sup>Resources and software are available at https://github.com/stas10217/beurre
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 882-893 June 6-11, 2021. ©2021 Association for Computational Linguistics
form of embedding product (Yang et al., 2015), and trains the embedding model as a regressor to predict the confidence score. One interpretation of existing methods is that they model each triple using a binary random variable, where the latent dependency structure between different binary random variables is captured by vector similarities. Without an explicit dependency structure it is difficult to enforce logical reasoning rules to maintain global consistency.
In order to go beyond triple-level uncertainty modeling, we consider each entity as a binary random variable. However, representing such a probability distribution in an embedding space and reasoning over it is non-trivial. It is difficult to model marginal and joint probabilities for entities using simple geometric objects like vectors. In order to encode probability distributions in the embedding space, recent works (Lai and Hockenmaier, 2017; Vilnis et al., 2018; Li et al., 2019; Dasgupta et al., 2020) represent random variables as more complex geometric objects, such as cones and axis-aligned hyperrectangles (boxes), and use volume as the probability measure. Inspired by such advances of probability measures in embeddings, we present BEUrRE  $\mathbf{\Theta}$  (Box Embedding for Uncertain RLational Data) $^{2}$ . BEUrRE represents entities as boxes. Relations are modeled as two separate affine transforms on the head and tail entity boxes. Confidence of a triple is modeled by the intersection between the two transformed boxes. Fig. 1 shows how a fact about the genre of the Beatles is represented under our framework.
Such representation is not only inline with the human perception that entities or concepts have different levels of granularity, but also allows more powerful domain knowledge representation. UKGE (Chen et al., 2019) has demonstrated that introducing domain knowledge about relation properties (e.g. transitivity) can effectively enhance reasoning on UKGs. While UKGE uses Probabilistic Soft Logic (PSL) (Bach et al., 2017) to reason for unseen facts and adds the extra training samples to training, such a method can lead to error propagation and has limited scope of application when UKG is sparse. In our work, we propose sufficient conditions for these relation properties to be preserved in the embedding space and directly model the relation properties by regularizing relation-specific transforms based on constraints.
This technique is more robust to noise and has wide coverage that is not restricted by the scarcity of the existing triples. Extensive experiments on two benchmark datasets show that BEURRE effectively captures the uncertainty, and consistently outperforms the baseline models on ranking and predicting confidence of unseen facts.
We discuss two lines of related work.
UKG Embeddings. A UKG assigns a confidence score to each fact. The development of relation extraction and crowdsourcing in recent years enabled the construction of many large-scale uncertain knowledge bases. ConceptNet (Speer et al., 2017) is a multilingual KG of commonsense concepts, where triples are assigned with confidence measures reflecting crowdsourcing agreement. NELL (Mitchell et al., 2018) collects facts from web pages with an active-learnable information extraction model, and measures their confidence scores by semi-supervised learning with the Expectation-Maximum (EM) algorithm. Probase (Wu et al., 2012) is a general probabilistic taxonomy obtained from syntactic extraction. Aforementioned UKGs have supported numerous knowledge-driven applications, such as literature-based drug repurposing (Sosa et al., 2020).
Recently, a few UKG embedding methods have been proposed, which seek to facilitate automated knowledge acquisition for UKGs. UKGE (Chen et al., 2019) is the first work of this kind, which models triple plausibility as product of embedding vectors (Yang et al., 2015), and maps the plausibility to the confidence score range of [0, 1]. To further enhance the performance, UKGE incorporates PSL based constraints (Bach et al., 2017) to help enforce the global consistency of predicted knowledge. UOKGE (Boutouhami et al., 2020) jointly encodes the graph structure and the ontology structure to improve the confidence prediction performance, which however requires an additional ontology of entity types that is not always available to all KGs. In addition to the above UKG embeddings models, there is also a matrix-factorization-based approach URGE that seeks to embed uncertain graphs (Hu et al., 2017). However, URGE only considers the node proximity in the networks. URGE cannot handle multi-relational data and only generates node embeddings.
2"Beurre" is French for "butter".
Geometric Embeddings. Developing embedding methods to represent elements using geometric objects with more complex structures than (Euclidean) vectors is an active area of study. Poincaré embeddings (Nickel and Kiela, 2017) represent entities in hyperbolic space, leveraging the inductive bias of negative curvature to fit hierarchies. Order embeddings (Vendrov et al., 2016) take a region-based approach, representing nodes of a graph using infinite cones, and using containment between cones to represent edges. Hyperbolic entailment cones (Ganea et al., 2018) combine order embeddings with hyperbolic geometry. While these methods show various degrees of promise when embedding hierarchies, they do not provide scores between entities that can be interpreted probabilistically, which is particularly useful in our setting.
Lai and Hockenmaier (2017) extend order embeddings with a probabilistic interpretation by integrating the volume of the infinite cones under the negative exponential measure, however the rigid structure imposed by the cone representation limits the representational capacity, and the resulting model cannot model negative correlation or disjointness. Introduced by Vilnis et al. (2018), probabilistic box embeddings represent elements using axis-aligned hyperrectangles (or boxes). Box embeddings not only demonstrate improved performance on modeling hierarchies, such embeddings also capture probabilistic semantics based on box volumes, and are capable of compactly representing conditional probability distributions. A few training improvement methods for box embeddings have been proposed (Li et al., 2019; Dasgupta et al., 2020), and we make use of the latter, which is termed GumbelBox after the distribution used to model endpoints of boxes.
While box embeddings have shown promise in representing hierarchies, our work is the first use of box embeddings to represent entities in multi-relational data. Query2Box (Ren et al., 2020) and BoxE (Abboud et al., 2020) make use of boxes in the loss function of their models, however entities themselves are represented as vectors, and thus these models do not benefit from the probabilistic semantics of box embeddings, which we rely on heavily for modeling UKGs. In (Patel et al., 2020), the authors demonstrate the capability of box embeddings to jointly model two hierarchical relations, which is improved upon using a learned transform in (Dasgupta et al., 2021). Similarly to
Ren et al. (2020) and Dasgupta et al. (2021), we also make use of a learned transform for each relation, however we differ from Ren et al. (2020) in that entities themselves are boxes, and differ from both in the structure of the learned transform.
Before we move on to the presented method in this work, we use this section to introduce the background of box embeddings and the addressed task.
3.1 Uncertain Knowledge Graphs
A UKG consists of a set of weighted triples  $\mathcal{G} = \{(l, s_l)\}$ . For each pair  $(l, s_l)$ ,  $l = (h, r, t)$  is a triple representing a fact where  $h, t \in \mathcal{E}$  (the set of entities) and  $r \in \mathcal{R}$  (the set of relations), and  $s_l \in [0, 1]$  represents the confidence score for this fact to be true. Some examples of weighted triples from NELL are (Honda, competeswith, Toyota): 1.00 and (Honda, competeswith, Chrysler): 0.94.
UKG Reasoning. Given a UKG  $\mathcal{G}$ , the uncertain knowledge graph reasoning task seeks to predict the confidence of an unseen fact  $(h, r, t)$ .
3.2 Probabilistic Box Embeddings
In this section we give a formal definition of probabilistic box embeddings, as introduced by Vilnis et al. (2018). A box is an  $n$ -dimensional hyperrectangle, i.e. a product of intervals
$$
\prod_ {i = 1} ^ {d} \left[ x _ {i} ^ {\mathbf {m}}, x _ {i} ^ {\mathbf {M}} \right], \quad \text {w h e r e} \quad x _ {i} ^ {\mathbf {m}} <   x _ {i} ^ {\mathbf {M}}.
$$
Given a space  $\Omega_{\mathrm{Box}} \subseteq \mathbb{R}^n$ , we define  $\mathcal{B}(\Omega_{\mathrm{Box}})$  to be the set of all boxes in  $\Omega_{\mathrm{Box}}$ . Note that  $\mathcal{B}(\Omega_{\mathrm{Box}})$  is closed under intersection, and the volume of a box is simply the product of side-lengths. Vilnis et al. (2018) note that this allows one to interpret box volumes as unnormalized probabilities. This can be formalized as follows.
Definition 3.1. Let  $(\Omega_{\mathrm{Box}},\mathcal{E},P_{\mathrm{Box}})$  be a probability space, where  $\Omega_{\mathrm{Box}}\subseteq \mathbb{R}^n$  and  $\mathcal{B}(\Omega_{\mathrm{Box}})\subseteq \mathcal{E}$ . Let  $\mathcal{V}$  be the set of binary random variables  $Y$  on  $\Omega_{\mathrm{Box}}$  such that  $Y^{-1}(1)\in \mathcal{B}(\Omega_{\mathrm{Box}})$ . A probabilistic box embedding of a set  $S$  is a function  $f(S)\coloneqq Y_s$  and  $Y_{s}^{-1}(1)\eqqcolon \operatorname {Box}(s)$ .
Essentially, to each element of  $S$  we associate a box which, when taken as the support set of a binary random variable, allows us to interpret each
element of  $S$  as a binary random variable. Using boxes for the support sets allows one to easily calculate marginal and conditional probabilities, for example if we embed the elements  $\{\mathrm{CAT},\mathrm{MAMMAL}\}$  as boxes in  $\Omega_{\mathrm{Box}} = [0,1]^d$  with  $P_{\mathrm{Box}}$  as Lebesgue measure, then
$$
\begin{array}{l} P \left(\text {M A M M A L} \mid \text {C A T}\right) = P _ {\text {B o x}} \left(X _ {\text {M A M M A L}} \mid X _ {\text {C A T}}\right) \\ = \frac {\operatorname {V o l} (\text {B o x (M A M M A L)} \cap \text {B o x (C A T)})}{\operatorname {V o l} (\text {B o x (C A T)})}. \\ \end{array}
$$
3.3 Gumbel Boxes
We further give a brief description of the Gumbel-Box method, which we rely on for training our box embeddings (Dasgupta et al., 2020).
As described thus far, probabilistic box embeddings would struggle to train via gradient descent, as there are many settings of parameters and objectives which have no gradient signal. (For example, if boxes are disjoint but should overlap.) To mitigate this, Dasgupta et al. (2020) propose a latent noise model, where the min and max coordinates of boxes in each dimension are modeled via Gumbel distributions, that is
$$
\begin{array}{l} \operatorname {B o x} (X) = \prod_ {i = 1} ^ {d} \left[ x _ {i} ^ {\mathrm {m}}, x _ {i} ^ {\mathrm {M}} \right] \quad \text {w h e r e} \\ x _ {i} ^ {\mathrm {m}} \sim \operatorname {G u m b e l M a x} \left(\mu_ {i} ^ {\mathrm {m}}, \beta\right), \\ x _ {i} ^ {\mathbf {M}} \sim \operatorname {G u m b e l M i n} (\mu_ {i} ^ {\mathbf {M}}, \beta). \\ \end{array}
$$
$\mu_{i}^{\mathrm{m}}$  thereof is the location parameter, and  $\beta$  is the (global) variance. The Gumbel distribution was chosen due to its min/max stability, which means that the set of all "Gumbel boxes" are closed under intersection. Dasgupta et al. (2020) go on to provide an approximation of the expected volume of a Gumbel box,
$$
\begin{array}{l} \mathbb {E} \left[ \operatorname {V o l} (\operatorname {B o x} (X)) \right] \approx \\ \prod_ {i = 1} ^ {d} \beta \log \left(1 + \exp \left(\frac {\mu_ {i} ^ {\mathrm {M}} - \mu_ {i} ^ {\mathrm {m}}}{\beta} - 2 \gamma\right)\right). \\ \end{array}
$$
A first-order Taylor series approximation yields
$$
\mathbb {E} \left[ P _ {\text {B o x}} \left(X _ {\mathrm {A}} \mid X _ {\mathrm {B}}\right) \right] \approx \frac {\mathbb {E} \left[ \operatorname {V o l} \left(\operatorname {B o x} (A) \cap \operatorname {B o x} (B)\right) \right]}{\mathbb {E} \left[ \operatorname {V o l} (\operatorname {B o x} (B)) \right]},
$$
and Dasgupta et al. (2020) empirically demonstrate that this approach leads to improved learning when targeting a given conditional probability distribution as the latent noise essentially ensembles over a large collection of boxes which allows the model to escape plateaus in the loss function. We therefore use this method when training box embeddings.
Remark 3.1. While we use Gumbel boxes for training, intuition is often gained by interpreting these boxes as standard hyperrectangles, which is valid as the Gumbel boxes can be seen as a distribution over such rectangles, with the Gumbel variance parameter  $\beta$  acting as a global measure of uncertainty. We thus make statements such as  $\operatorname{Box}(X) \subseteq \operatorname{Box}(Y)$ , which, strictly speaking, are not well-defined for Gumbel boxes. However we can interpret this probabilistically as  $P(Y \mid X) = 1$  which coincides with the conventional interpretation when  $\beta = 0$ .
In this section, we present our UKG embedding model BEUrRE. The proposed model encodes entities as probabilistic boxes and relations as affine transforms. We also discuss the method to incorporate logical constraints into learning.
4.1 Modeling UKGs with Box Embeddings
BEURRE represents entities as Gumbel boxes, and a relation  $r$  acting on these boxes by translation and scaling. Specifically, we parametrize a Gumbel box  $\operatorname{Box}(X)$  using a center  $\operatorname{cen}(\operatorname{Box}(X)) \in \mathbb{R}^d$  and offset  $\operatorname{off}(\operatorname{Box}(X)) \in \mathbb{R}_+^d$ , where the location parameters are given by
$$
\mu_ {i} ^ {\mathrm {m}} = \operatorname {c e n} (\operatorname {B o x} (X)) - \operatorname {o f f} (\operatorname {B o x} (X)),
$$
$$
\mu_ {i} ^ {\mathrm {M}} = \operatorname {c e n} (\operatorname {B o x} (X)) + \operatorname {o f f} (\operatorname {B o x} (X)).
$$
We consider transformations on Gumbel boxes parametrized by a translation vector  $\tau \in \mathbb{R}^d$  and a scaling vector  $\Delta \in \mathbb{R}_+^d$  such that
$$
\begin{array}{l} \operatorname {c e n} (f (\operatorname {B o x} (X); \tau , \Delta)) = \operatorname {c e n} (\operatorname {B o x} (X)) + \tau , \\ \operatorname {o f f} (f (\operatorname {B o x} (X); \tau , \Delta)) = \operatorname {o f f} (\operatorname {B o x} (X)) \circ \Delta , \\ \end{array}
$$
where  $\circ$  is the Hadamard product. We use separate actions for the head and tail entities of a relation, which we denote  $f_{r}$  and  $g_{r}$ , and omit the explicit dependence on the learned parameters  $\tau$  and  $\Delta$ .
Remark 4.1. Note that these relations are not an affine transformations of the space,  $\Omega_{\mathrm{Box}}$ , rather they perform a transformation of a box. These functions form an Abelian group under composition, and furthermore define a transitive, faithful group action on the set of (Gumbel) boxes.
Given a triple  $(h,r,t)$ , BEURRE models the confidence score using the (approximate) conditional probability given by
$$
\phi (h, r, t) = \frac {\mathbb {E} [ \operatorname {V o l} (f _ {r} (\operatorname {B o x} (h)) \cap g _ {r} (\operatorname {B o x} (t))) ]}{\mathbb {E} [ \operatorname {V o l} (g _ {r} (\operatorname {B o x} (t))) ]}.
$$
We can think of the box  $f_{r}(\mathrm{Box}(h))$  as the support set of a binary random variable representing the concept  $h$  in the context of the head position of relation  $r$ , for example Box(THEBEATLES) is a latent representation of the concept of The Beatles, and  $f_{\mathrm{GENRE}}(\mathrm{Box}(\mathrm{THeBETLES}))$  represents The Beatles in the context of genre classification as the object to be classified.
4.2 Logical Constraints
The sparsity of real-world UKGs makes learning high quality representations difficult. To address this problem, previous work (Chen et al., 2019) introduces domain knowledge about the properties of relations (e.g., transitivity) and uses PSL over first-order logical rules to reason for unseen facts and create extra training samples. While this technique successfully enhances the performance by incorporating constraints based on relational properties, the coverage of such reasoning is still limited by the density of the graph. In UKGE, the confidence score of a triple can be inferred and benefit training only if all triples in the rule premise are already present in the KG. This leads to a limited scope of application, particularly when the graph is sparse.
In our work, we propose sufficient conditions for these relation properties to be preserved in the embedding space and directly incorporating the relational constraints by regularizing relation-specific transforms. Compared to previous work, our approach is more robust to noise since it does not hardcode inferred confidence for unseen triples, and it has wide coverage that is not restricted by the scarcity of the existing triples.
In the following, we discuss the incorporation of two logical constraints — transitivity and composition — in the learning process. We use capital letters  $A, B, C$  to represent universally quantified entities from UKG and use  $\Phi$  to denote a set of boxes sampled from  $\mathcal{B}(\Omega_{\mathrm{Box}})$ .
Transitivity Constraint. A relation  $r$  is transitive if  $(A, r, B) \wedge (B, r, C) \Rightarrow (A, r, C)$ . An example of a transitive relation is hypernymy.
The objective of imposing a transitivity constraint in learning is to preserve this property of the relation in the embedding space, i.e. to ensure that  $(A,r,C)$  will be predicted true if  $(A,r,B)$  and  $(B,r,C)$  are true. This objective is fulfilled if  $g_{r}(\mathrm{Box}(B))$  contains  $f_{r}(\mathrm{Box}(B))$ . An illustration of the box containment relationships is given in Fig 2. Thus, we constrain  $f_{r}$  and  $g_{r}$  so that  $g_{r}(u)$
Figure 2: Illustration of how the constraint that  $g_{r}(u)$  contains  $f_{r}(u)$  preserves transitivity of relation  $r$  in the embedding space. A triple  $(h, r, t)$  is true if and only if  $f_{r}(\operatorname{Box}(h))$  contains  $g_{r}(\operatorname{Box}(t))$ . By adding this constraint,  $f_{r}(\operatorname{Box}(A))$  is guaranteed to contain  $g_{r}(\operatorname{Box}(C))$  if  $(A, r, B)$  and  $(B, r, C)$  are true.
contains  $f_{r}(u)$  for any  $u \in \Omega_{\mathrm{Box}}$ . We impose the constraint with the following regularization term:
$$
L _ {\mathrm {t r}} (r) = \frac {1}{| \Phi |} \sum_ {u \in \Phi} \| P _ {\mathrm {B o x}} \left(g _ {r} (u) \mid f _ {r} (u)\right) - 1 \| ^ {2}.
$$
Composition Constraint. A relation  $r_3$  is composed of relation  $r_1$  and relation  $r_2$  if  $(A, r_1, B) \wedge (B, r_2, C) \Rightarrow (A, r_3, C)$ . For example, the relationatheletePlaysSports can be composed of relationsatheletePlaysForTeam and teamPlaysSports.
To preserve the relation composition in the embedding space, we constrain that the relation-specific mappings  $f_{r_3}$  and  $g_{r_3}$  are the composite mappings of  $f_{r_1}, f_{r_2}$  and  $g_{r_1}, g_{r_2}$  respectively:
$$
f _ {r _ {3}} = f _ {r _ {2}} \cdot f _ {r _ {1}}; g _ {r _ {3}} = g _ {r _ {2}} \cdot g _ {r _ {1}}.
$$
where  $\cdot$  is the mapping composition operator. Thus, for any  $u \in \Omega_{\mathrm{Box}}$ , we expect that  $f_{r_3}(u)$  is the same as  $f_{r_2}(f_{r_1}(u))$  and  $g_{r_3}(u)$  is the same as  $g_{r_2}(g_{r_1}(u))$ . We accordingly add the following regularization term
$$
\begin{array}{l} L _ {\mathrm {c}} \left(r _ {1}, r _ {2}, r _ {3}\right) = \frac {1}{\left| \Phi \right|} \sum_ {u \in \Phi} f _ {r _ {3}} (u) \oplus f _ {r _ {2}} \left(f _ {r _ {1}} (u)\right) \\ + g _ {r _ {3}} (u) \oplus g _ {r _ {2}} \left(g _ {r _ {1}} (u)\right) \\ \end{array}
$$
where  $\oplus$  is defined as
$$
\begin{array}{l} \operatorname {B o x} _ {1} \oplus \operatorname {B o x} _ {2} = \| 1 - P _ {\operatorname {B o x}} (\operatorname {B o x} _ {1} \mid \operatorname {B o x} _ {2}) \| ^ {2} \\ + \left\| 1 - P _ {\text {B o x}} \left(\operatorname {B o x} _ {2} \mid \operatorname {B o x} _ {1}\right) \right\| ^ {2}. \\ \end{array}
$$