SlowGuess/NAACL_2021_OCR huggingface dataset:
The learning process of BEURRE optimizes two objectives. The main objective optimizes the loss for a regression task and, simultaneously, a constrained regularization loss enforces the aforementioned constraints.
Let  $\mathcal{L}^+$  be the set of observed relation facts in training data. The goal is to minimize the mean squared error (MSE) between the ground truth confidence score  $s_l$  and the prediction  $\phi(l)$  for each relation  $l \in \mathcal{L}^+$ . Following UKGE (Chen et al., 2019), we also penalize the predicted confidence scores of facts that are not observed in UKG. The main learning objective is as follows:
$$
\mathcal {J} _ {1} = \sum_ {l \in \mathcal {L} ^ {+}} | \phi (l) - s _ {l} | ^ {2} + \alpha \sum_ {l \in \mathcal {L} ^ {-}} | \phi (l) | ^ {2}.
$$
where  $\mathcal{L}^{-}$  is a sample set of the facts not observed in UKG, and  $\alpha$  is a hyper-parameter to weigh unobserved fact confidence penalization. Similar to previous works, we sample those facts by corrupting the head and the tail for observed facts to generate  $\mathcal{L}^{-}$  during training.
In terms of constraints, let  $\mathcal{R}_{\mathrm{tr}}$  be the set of transitive relations,  $\mathcal{R}_{\mathrm{c}}$  be the set of composite relation groups, and  $w_{\mathrm{tr}}$  and  $w_{\mathrm{c}}$  be the regularization coefficients. We add the following regularization to impose our constraints on relations:
$$
\mathcal {J} _ {2} = w _ {\mathrm {t r}} \sum_ {r \in \mathcal {R} _ {\mathrm {t r}}} L _ {\mathrm {t r}} (r) + w _ {\mathrm {c}} \sum_ {(r _ {1}, r _ {2}, r _ {3}) \in \mathcal {R} _ {\mathrm {c}}} L _ {\mathrm {c}} (r _ {1}, r _ {2}, r _ {3}).
$$
Combining both learning objectives, the learning process optimizes the joint loss  $J = J_{1} + J_{2}$ .
Once BEUrRE is trained, the model can easily infer the confidence of a new fact  $(h,r,t)$  based on the confidence score function  $\phi (h,r,t)$  defined in Section 4.1. This inference mechanism easily supports other types of reasoning tasks, such as inferring the plausibility of a new fact, and ranking multiple related facts. The experiments presented in the next section will demonstrate the ability of BEUrRE to perform those reasoning tasks.
In this section we present evaluation of our model on two UKG reasoning tasks, i.e. confidence prediction and fact ranking. More experimentation details are in Appendices.
Dataset	#Ent.	#Rel.	#Rel. Facts	Avg(s)	Std(s)
CN15k	15,000	36	241,158	0.629	0.232
NL27k	27,221	404	175,412	0.797	0.242
Table 1: Statistics of the datasets. Ent. and Rel. stand for entities and relations.  $\operatorname{Avg}(s)$  and  $\operatorname{Std}(s)$  are the average and standard deviation of confidence.
Dataset	Transitivity	Composition
CN15k	causes	N/A
NL27k	locationAtLocation	(athetePlaysForTeam, teamPlaysSport) → athetePlaysSport
Table 2: Examples of relations with logical constraints.
Datasets. We follow Chen et al. (2019) and evaluate our models on CN15k and NL27k benchmarks, which are subsets of ConceptNet (Speer et al., 2017) and NELL (Mitchell et al., 2018) respectively. Table 1 gives the statistics of the datasets. We use the same split provided by Chen et al. (2019):  $85\%$  for training,  $7\%$  for validation, and  $8\%$  for testing. We exclude the dataset PPI5k, the subgraph of the protein-protein interaction (PPI) network STRING (Szklarczyk et al., 2016), where the supporting scores of PPI information are indicators based on experimental and literary verification, instead of a probabilistic measure.
Logical constraints. We report results of both versions of our model with and without logical constraints, denoted as BEUrRE (rule+) and BEUrRE respectively. For a fair comparison, we incorporate into BEUrRE (rule+) the same set of logical constraints as UKGE (Chen et al., 2019). Table 2 gives a few examples of the relations on which we impose constraints.
Baselines. We compare our models with UKG embedding models as well as deterministic KG embedding models.
UKG embedding models include UKGE (Chen et al., 2019) and URGE (Hu et al., 2017). While UKGE has multiple versions incorporated with different regression functions, we report the results of the best performing one with the logistic function. We also include results for both settings with and without constraints, marked as UKGE (rule+) and UKGE in result tables respectively.
Dataset	CN15k	NL27k
Metrics	MSE	MAE	MSE	MAE
URGE	10.32	22.72	7.48	11.35
UKGE	9.02	20.05	2.67	7.03
BEUrRE	7.80	20.03	2.37	7.12
UKGE(rule+)	8.61	19.90	2.36	6.90
BEUrRE(rule+)	7.49	19.88	2.01	6.89
URGE was originally designed for probabilistic homogeneous graphs and cannot handle multi-relational graphs, so accordingly we ignore relation information when embedding a UKG. UOKGE (Boutouhami et al., 2020) cannot serve as a baseline because it requires additional ontology information for entities that is not available to these UKGs.
Deterministic KG embedding models TransE (Bordes et al., 2013), DistMult (Yang et al., 2015), ComplEx (Trouillon et al., 2016), RotatE (Sun et al., 2019), and TuckER (Balazevic et al., 2019) have demonstrated high performance on reasoning tasks for deterministic KGs, and we also include them as baselines. These models cannot predict confidence scores for uncertain facts, so we compare our method with them only on the ranking task. Following Chen et al. (2019), we only use facts with confidence above the threshold  $\tau = 0.85$  to train deterministic models.
Model configurations. We use Adam (Kingma and Ba, 2014) as the optimizer and fine-tune the following hyper-parameters by grid search based on the performance on the validation set, i.e. MSE for confidence prediction and normalized Discounted Cumulative Gain (nDCG) for fact ranking. Hyper-parameter search range and the best hyperparameter configurations are given in Appendix A.1. Training terminates with early stopping based on the same metric with a patience of 30 epochs. We repeat each experiment five times and report the average results.
5.2 Confidence Prediction
This task seeks to predict the confidence of new facts that are unseen to training. For each uncertain fact  $(l, s_l)$  in the test set, we predict the confidence of  $l$  and report the mean squared error (MSE) and mean absolute error (MAE).
Results. Results are reported in Table 3. We compare our models with baselines under the uncon
Table 3: Results of fact confidence prediction  $\left( {\times {10}^{-2}}\right)$  .
Variants	uncons. rule+
Metrics	MSE (×10-2)
BEUrRE	7.80	7.49
—w/o Gumbel distribution	8.13	8.14
—Single relation-specific transform	7.81	7.60
Table 4: Ablation study results on CN15k. uncons. represents the unconstrained setting, and rule+ denotes the logically constrained setting.
strained and logically constrained (marked with rule+) settings respectively. Under both settings, BEURRE outperforms the baselines in terms of MSE on both datasets.
Under the unconstrained setting, BEUrRE improves MSE of the best baseline UKGE by 0.012 (ca.  $14\%$  relative improvement) on CN15k and 0.003 (ca.  $11\%$  relative improvement) on NL27k. The enhancement demonstrates that box embeddings can effectively improve reasoning on UKGs. It is worth noting that even without constraints in learning, BEUrRE can still achieve comparable MSE and MAE to the logically constrained UKGE (rule+) on both datasets and even outperforms UKGE (rule+) on CN15k. Considering that constraints of relations in CN15k mainly describe transitivity, the aforementioned observation is consistent with the fact that box embeddings are naturally good at capturing transitive relations, as shown in the recent study (Vilnis et al., 2018).
With logical constraints, BEUrRE (rule+) further enhances the performance of BEUrRE and reduces its MSE by 0.0031 (ca.  $4\%$  relative improvement) on CN15k and 0.0036 (ca.  $15\%$  relative improvement) on NL27k. This is as expected, since logical constraints capture higher-order relations of facts and lead to more globally consistent reasoning. We also observe that BEUrRE (rule+) brings larger gains over BEUrRE on NL27k, where we have both transitivity constraints and composition constraints, than on CN15k with only transitivity constraints incorporated.
In general, with box embeddings, BEURRE effectively improves reasoning on UKGs with better captured fact-wise confidence. Furthermore, the results under the logically constrained setting show the effectiveness of improving reasoning with higher-order relations of uncertain facts.
Ablation Study. To examine the contribution from Gumbel distribution to model box boundaries and the effectiveness of representing relations as two
Dataset	CN15K	NL27k
Metrics	linear exp.	linear exp.
TransE	0.601 0.591	0.730 0.722
DistMult	0.689 0.677	0.911 0.897
ComplEx	0.723 0.712	0.921 0.913
RotatE	0.715 0.703	0.901 0.887
TuckER	0.736 0.724	0.877 0.870
URGE	0.572 0.570	0.593 0.593
UKGE	0.769 0.768	0.933 0.929
BEUrRE	0.796 0.795	0.942 0.942
UKGE(rule+)	0.789 0.788	0.955 0.956
BEUrRE(rule+)	0.801 0.803	0.966 0.970
Table 5: Mean nDCG for fact ranking. linear stands for linear gain, and exp. stands for exponential gain.
separate transforms for head and tail boxes, we conduct an ablation study based on CN15k. The results for comparison are given in Table 4. First, we resort to a new configuration of BEURRE where we use smoothed boundaries for boxes as in (Li et al., 2019) instead of Gumbel boxes. We refer to boxes of this kind as soft boxes. Under the unconstrained setting, using soft boxes increases MSE by 0.0033 on CN15k (ca.  $4\%$  relative degradation), with even worse performance observed when adding logical constraints. This confirms the finding by Dasgupta et al. (2020) that using Gumbel distribution for boundaries greatly improves box embedding training. Next, to analyze the effect of using separate transforms to represent a relation, we set the tail transform  $g_{r}$  as the identity function. For logical constraint incorporation, we accordingly update the constraint on transitive relation  $r$  as  $P_{\mathrm{Box}}(u \mid f_r(u)) = 1, u \in \Omega_{\mathrm{Box}}$ , which requires that  $u$  always contains  $f_{r}(u)$ , i.e. the translation vector of  $f_{r}$  is always zero and elements of the scaling vector are always less than 1. Although there is little difference between using one or two transforms under the unconstrained setting, under the logically constrained setting, the constraint is too stringent to be preserved with only one transform.
Case study. To investigate whether our model can encode meaningful probabilistic semantics, we present a case study about box volumes. We examine the objects of the atLocation predicate on CN15k and check which entity boxes have larger volume and cover more entity boxes after the relation transformation. Ideally, geographic entities with larger areas or more frequent mentions
should be at the top of the list. When using the BEUrRE(rule+) model, the top 10 in all entities are place, town, bed, school, city, home, house, capital, church, camp, which are general concepts. Among the observed objects of the atLocation predicate, the entities that have the least coverage are Tunisia, Morocco, Algeria, Westminster, Veracruz, Buenos Aires, Emilia-Romagna, Tyrrhenian sea, Kuwait, Serbia. Those entities are very specific locations. This observation confirms that the box volume effectively represents probabilistic semantics and captures specificity/granularity of concepts, which we believe to be a reason for the performance improvement.
5.3 Fact Ranking
Multiple facts can be associated with the same entity. However, those relevant facts may appear with very different plausibility. Consider the example about Honda Motor Co. in Section 1, where it was mentioned that (Honda, competeswith, Toyota) should have a higher belief than (Honda, competeswith, Chrysler). Following this intuition, this task focuses on ranking multiple candidate tail entities for a query  $(h,r,\underline{?t})$  in terms of their confidence.
Evaluation protocol. Given a query  $(h, r, \underline{?t})$ , we rank all the entities in the vocabulary as tail entity candidates and evaluate the ranking performance using the normalized Discounted Cumulative Gain (nDCG) (Li et al., 2009). The gain in retrieving a relevant tail  $t_0$  is defined as the ground truth confidence  $s_{(h,r,t_0)}$ . Same as Chen et al. (2019), we report two versions of nDCG that use linear gain and exponential gain respectively. The exponential gain puts stronger emphasis on the most relevant results.
Results. We report the mean nDCG over the test query set in Table 5. Although the deterministic models do not explicitly capture the confidence of facts, those models are trained with high-confidence facts and have a certain ability to differentiate high confidence facts from lesser ones. URGE ignores relation information and yields worse predictions than other models. UKGE explicitly models uncertainty of facts and is the best performing baseline.
The proposed BEUrRE leads to more improvements under both the unconstrained and logically constrained settings. Under the unconstrained setting, BEUrRE offers consistently better per
formance over UKGE. Specifically, on CN15k, BEUrRE leads to 0.027 improvement in both linear nDCG and exponential nDCG. On NL27k, it offers 0.009 higher linear nDCG and 0.013 higher exponential nDCG. Similar to the results on the confidence prediction task, even unconstrained BEUrRE is able to outperform the logically constrained UKGE (rule+) on CN15k without incorporating any constraints of relations. This further confirms the superior expressive power of box embeddings.
In summary, box embeddings improve accuracy and consistency of reasoning and BEURRE delivers better fact ranking performance than baselines.
This paper presents a novel UKG embedding method with calibrated probabilistic semantics. Our model BEUrRE encodes each entity as a Gumble box representation whose volume represents marginal probability. A relation is modeled as two affine transforms on the head and tail entity boxes. We also incorporate logic constraints that capture the high-order dependency of facts and enhance global reasoning consistency. Extensive experiments show the promising capability of BEUrRE on confidence prediction and fact ranking for UKGs. The results are encouraging and suggest various extensions, including deeper transformation architectures as well as alternative geometries to allow for additional rules to be imposed. In this context, we are also interested in extending the use of the proposed technologies into more downstream tasks, such as knowledge association (Sun et al., 2020) and event hierarchy induction (Wang et al., 2020). Another direction is to use BEUrRE for ontology construction and population, since box embeddings are naturally capable of capturing granularities of concepts.
Real-world UKGs often harvest data from open data sources and may include biases. Reasoning over biased UKGs may support or magnify those biases. While not specifically addressed in this work, the ability to inject logical rules could be one way to mitigate bias, and the ability to interpret the learned representation probabilistically allows the investigation of potential learned biases.
All the datasets used in this paper are publicly available and free to download. The model pro
posed in the paper aims to model uncertainty in knowledge graphs more accurately, and the effectiveness of the proposed model is supported by the empirical experiment results.
We appreciate the anonymous reviewers for their insightful comments and suggestions.
This material is based upon work sponsored by the DARPA MCS program under Contract No. N660011924033 with the United States Office Of Naval Research, and by Air Force Research Laboratory under agreement number FA8750-20-2-10002. We also thank our colleagues within IESL at UMass Amherst, for their helpful discussions. Michael, Shib and Xiang were supported in part by the Center for Intelligent Information Retrieval and the Center for Data Science, in part by the IBM Research AI through the AI Horizons Network, in part by the University of Southern California subcontract No. 123875727 under Office of Naval Research prime contract No. N660011924032 and in part by the University of Southern California subcontract no. 89341790 under Defense Advanced Research Projects Agency prime contract No. FA8750-17-C-0106. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Government.
Ralph Abboud, Ismail Ilkan Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. 2020. Boxe: A box embedding model for knowledge base completion. In Proceedings of the 34th Annual Conference on Neural Information Processing Systems, NeurIPS.
Stephen H Bach, Matthias Broecheler, Bert Huang, and Lise Getoor. 2017. Hinge-loss markov random fields and probabilistic soft logic. Journal of Machine Learning Research.
Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. 2019. Tucker: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP, pages 5184-5193. Association for Computational Linguistics.
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of ACM SIGMOD International Conference on Management of Data (SIGMOD).
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems (NIPS), pages 2787-2795.
Khaoula Boutouhami, Jiatao Zhang, Guilin Qi, and Huan Gao. 2020. Uncertain ontology-aware knowledge graph embeddings. In Semantic Technology, pages 129-136. Springer Singapore.
Xuelu Chen, Muhao Chen, Weijia Shi, Yizhou Sun, and Carlo Zaniolo. 2019. Embedding uncertain knowledge graphs. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), pages 3363-3370. AAAI Press.
Shib Sankar Dasgupta, Michael Boratko, Dongxu Zhang, Luke Vilnis, Xiang Lorraine Li, and Andrew McCallum. 2020. Improving local identifiability in probabilistic box embeddings. In Advances in Neural Information Processing Systems.
Shib Sankar Dasgupta, Xiang Li, Michael Boratko, Dongxu Zhang, and Andrew McCallum. 2021. Box-to-box transformation for modeling joint hierarchies.
Octavian Ganea, Gary Becigneul, and Thomas Hofmann. 2018. Hyperbolic entailment cones for learning hierarchical embeddings. In International Conference on Machine Learning, pages 1646-1655.
Jiafeng Hu, Reynold Cheng, Zhipeng Huang, Yixiang Fang, and Siqiang Luo. 2017. On embedding uncertain graphs. In Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM).
Nathawut Kertkeidkachorn, Xin Liu, and Ryutaro Ichise. 2019. Gtranse: Generalizing translation-based model on uncertain knowledge graph embedding. In Annual Conference of the Japanese Society for Artificial Intelligence, pages 170-178. Springer.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR).
Alice Lai and Julia Hockenmaier. 2017. Learning to predict denotational probabilities for modeling entailment. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 721-730. Association for Computational Linguistics.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef,
Soren Auer, et al. 2015. Dbpedia-a large-scale, multilingual knowledge base extracted from wikipedia. Semantic Web, 6(2):167-195.
Hang Li, Tie-Yan Liu, and Chengxiang Zhai. 2009. Learning to rank for information retrieval. Foundations and Trends® in Information Retrieval.
Xiang Li, Luke Vilnis, Dongxu Zhang, Michael Boratko, and Andrew McCallum. 2019. Smoothing the geometry of probabilistic box embeddings. *ICLR*.
Farzaneh Mahdisoltani, Joanna Biega, et al. 2015. Yago3: A knowledge base from multilingual Wikipedias. In Proceedings of the Conference on Innovative Data Systems Research (CIDR).
Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, B Yang, J Betteridge, A Carlson, B Dalvi, M Gardner, B Kisiel, et al. 2018. Never-ending learning. Communications of the ACM.
Maximillian Nickel and Douwe Kiela. 2017. Poincaré embeddings for learning hierarchical representations. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6338-6347. Curran Associates, Inc.
Dhruvesh Patel, Shib Sankar Dasgupta, Michael Boratko, Xiang Li, Luke Vilnis, and Andrew McCallum. 2020. Representing joint hierarchies with box embeddings. Automated Knowledge Base Construction.
Hongyu Ren, Weihua Hu, and Jure Leskovec. 2020. Query2box: Reasoning over knowledge graphs in vector space using box embeddings. In 8th International Conference on Learning Representations. OpenReview.net.
DN Sosa, A Derry, M Guo, E Wei, C Brinton, and RB Altman. 2020. A literature-based knowledge graph embedding method for identifying drug repurposing opportunities in rare diseases. In *Pacific Symposium on Biocomputing*. Pacific Symposium on Biocomputing, volume 25, pages 463-474.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence (AAAI), page 4444-4451. AAAI Press.
Zequn Sun, Muhao Chen, Wei Hu, Chengming Wang, Jian Dai, and Wei Zhang. 2020. Knowledge association with hyperbolic knowledge graph embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5704-5716.
Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. In International Conference on Learning Representations (ICLR).
Damian Szklarczyk, John H Morris, Helen Cook, Michael Kuhn, Stefan Wyder, Milan Simonovic, Alberto Santos, Nadezhda T Doncheva, Alexander Roth, Peer Bork, et al. 2016. The string database in 2017: quality-controlled protein-protein association networks, made broadly accessible. *Nucleic acids research*.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Proceedings of the International Conference on Machine Learning (ICML), volume 48, pages 2071-2080. PMLR.
Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. 2016. Order-embeddings of images and language. In ICLR.
Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. 2018. Probabilistic embedding of knowledge graphs with box lattice measures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 263-272. Association for Computational Linguistics.
Haoyu Wang, Muhao Chen, Hongming Zhang, and Dan Roth. 2020. Joint constrained learning for event-event relation extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguistics.
Wentao Wu, Hongsong Li, Haixun Wang, and Kenny Q Zhu. 2012. Probase: A probabilistic taxonomy for text understanding. In Proceedings of ACM SIGMOD International Conference on Management of Data (SIGMOD).
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. 2015. Embedding entities and relations for learning and inference in knowledge bases. International Conference on Learning Representations (ICLR).
A.1 More Implementation Details
Table 6 lists hyper-parameter search space for obtaining the set of used numbers. We performed grid search to choose the final setting.