Smoothing the Geometry of Probabilistic Box Embeddings

1 INTRODUCTION . Embedding methods have long been a key technique in machine learning , providing a natural way to convert semantic problems into geometric problems . Early examples include the vector space ( Salton et al. , 1975 ) and latent semantic indexing ( Deerwester et al. , 1990 ) models for information retrieval . Embeddings experienced a renaissance after the publication of Word2Vec ( Mikolov et al. , 2013 ) , a neural word embedding method ( Bengio et al. , 2003 ; Mnih & Hinton , 2009 ) that could run at massive scale . Recent years have seen an interest in structured or geometric representations . Instead of representing e.g . images , words , sentences , or knowledge base concepts with points , these methods instead associate them with more complex geometric structures . These objects can be density functions , as in Gaussian embeddings ( Vilnis & McCallum , 2015 ; Athiwaratkun & Wilson , 2017 ; 2018 ) , convex cones , as in order embeddings ( Vendrov et al. , 2016 ; Lai & Hockenmaier , 2017 ) , or axis-aligned hyperrectangles , as in box embeddings ( Vilnis et al. , 2018 ; Subramanian & Chakrabarti , 2018 ) . These geometric objects more naturally express ideas of asymmetry , entailment , ordering , and transitive relations than simple points in a vector space , and provide a strong inductive bias for these tasks . In this work , we focus on the probabilistic Box Lattice model of Vilnis et al . ( 2018 ) , because of its strong empirical performance in modeling transitive relations , probabilistic interpretation ( edges in a relational DAG are replaced with conditional probabilities ) , and ability to model complex joint * Equal contribution . probability distributions including negative correlations . Box embeddings ( BE ) are a generalization of order embeddings ( OE ) ( Vendrov et al. , 2016 ) and probabilistic order embeddings ( POE ) ( Lai & Hockenmaier , 2017 ) that replace the vector lattice ordering ( notions of overlapping and enclosing convex cones ) in OE and POE with a more general notion of overlapping boxes ( products of intervals ) . While intuitively appealing , the “ hard edges ” of boxes and their ability to become easily disjoint , present difficulties for gradient-based optimization : when two boxes are disjoint in the model , but have overlap in the ground truth , no gradient can flow to the model to correct the problem . This is of special concern for ( pseudo- ) sparse data , where many boxes should have nearly zero overlap , while others should have very high overlap . This is especially pronounced in the case of e.g . market basket models for recommendation , where most items should not be recommended , and entailment tasks , most of which are currently artificially resampled into a 1:1 ratio of positive to negative examples . To address the disjoint case , Vilnis et al . ( 2018 ) introduce an ad-hoc surrogate function . In contrast , we look at this problem as inspiration for a new model , based on the intuition of relaxing the hard edges of the boxes into smoothed density functions , using a Gaussian convolution with the original boxes . We demonstrate the superiority of our approach to modeling transitive relations on WordNet , Flickr caption entailment , and a MovieLens-based market basket dataset . We match or beat existing state of the art results , while showing substantial improvements in the pseudosparse regime . 2 RELATED WORK . As mentioned in the introduction , there is much related work on structured or geometric embeddings . Most relevant to this work are the order embeddings of Vendrov et al . ( 2016 ) , which embed a nonprobabilistic DAG or lattice in a vector space with order given by inclusion of embeddings ’ forward cones , the probabilistic extension of that model due to Lai & Hockenmaier ( 2017 ) , and the box lattice or box embedding model of Vilnis et al . ( 2018 ) , which we extend . Concurrently to Vilnis et al . ( 2018 ) , another hyperrectangle-based generalization of order embeddings was proposed by Subramanian & Chakrabarti ( 2018 ) , also called box embeddings . The difference between the two models lies in the interpretation : the former is a probabilistic model that assigns edges conditional probabilities according to degrees of overlap , while the latter is a deterministic model in the style of order embeddings — an edge is considered present only if one box entirely encloses another . Methods based on embedding points in hyperbolic space ( Nickel & Kiela , 2017 ; Ganea et al. , 2018 ) have also recently been proposed for learning hierarchical embeddings . These models , similar to order embeddings and the box embeddings of Subramanian & Chakrabarti ( 2018 ) , are nonprobabilistic and optimize an energy function . Additionally , while the negative curvature of hyperbolic space is attractively biased towards learning tree structures ( since distances between points increase the farther they are from the origin ) , this constant curvature makes the models not as suitable for learning non-treelike DAGs . Our approach to smoothing the energy landscape of the model using Gaussian convolution is common in mollified optimization and continuation methods , and is increasingly making its way into machine learning models such as Mollifying Networks ( Gulcehre et al. , 2016b ) , diffusion-trained networks ( Mobahi , 2016 ) , and noisy activation functions ( Gulcehre et al. , 2016a ) . Our focus on embedding orderings and transitive relations is a subset of knowledge graph embedding . While this field is very large , the main difference of our probabilistic approach is that we seek to learn an embedding model which maps concepts to subsets of event space , giving our model an inductive bias especially suited for transitive relations as well as fuzzy concepts of inclusion and entailment . 3 BACKGROUND . We begin with a brief overview of two methods for representing ontologies as geometric objects . First , we review some definitions from order theory , a useful formalism for describing ontologies , then we introduce the vector and box lattices . Figure 1 shows a simple two-dimensional example of these representations . 3.1 PARTIAL ORDERS AND LATTICES . A non-strict partially ordered set ( poset ) is a pair P , , where P is a set , and is a binary relation . For all a , b , c ∈ P , Reflexivity : a a Antisymmetry : a b a implies a = b Transitivity : a b c implies a c This generalizes the standard concept of a totally ordered set to allow some elements to be incomparable . Posets provide a good formalism for the kind of acyclic directed graph data found in many knowledge bases with transitive relations . A lattice is a poset where any subset of elements has a single unique least upper bound , and greatest lower bound . In a bounded lattice , the set P contains two additional elements , > ( top ) , and ⊥ ( bottom ) , which denote the least upper bound and greatest lower bound of the entire set . A lattice is equipped with two binary operations , ∨ ( join ) , and ∧ ( meet ) . a∨b denotes the least upper bound of a , b ∈ P , and a ∧ b denotes their greatest lower bound . A bounded lattice must satisfy these properties : Idempotency : a ∧ a = a ∨ a = a Commutativity : a ∧ b = b ∧ a and a ∨ b = b ∨ a Associativity : a ∧ b ∧ c = a ∧ ( b ∧ c ) and ( a ∨ b ∨ c ) = a ∨ ( b ∨ c ) Absorption : a ∨ ( a ∧ b ) = a and a ∧ ( a ∨ b ) = a Bounded : ⊥ a > Note that the extended real numbers , R ∪ { −∞ , ∞ } , form a bounded lattice ( and in fact , a totally ordered set ) under the min and max operations as the meet ( ∧ ) and join ( ∨ ) operations . So do sets partially ordered by inclusion , with ∩ and ∪ as ∧ and ∨ . Thinking of these special cases gives the intuition for the fourth property , absorption . The ∧ and ∨ operations can be swapped , along with reversing the poset relation , to give a valid lattice , called the dual lattice . In the real numbers this just corresponds to a sign change . A semilattice has only a meet or join , but not both . Note . In the rest of the paper , when the context is clear , we will also use ∧ and ∨ to denote min and max of real numbers , in order to clarify the intuition behind our model . 3.2 VECTOR LATTICE . A vector lattice , also known as a Riesz space ( Zaanen , 1997 ) , or Hilbert lattice when the accompanying vector space has an inner product , is a vector space endowed with a lattice structure . A standard choice of partial order for the vector lattice Rn is to use the product order from the underlying real numbers , which specifies for all x , y ∈ Rn x y ⇐⇒ ∀i ∈ { 1 .. n } , xi ≤ yi Under this order , meet and join operations are pointwise min and max , which gives a lattice structure . In this formalism , the Order Embeddings of Vendrov et al . ( 2016 ) embed partial orders as vectors using the reverse product order , corresponding to the dual lattice , and restrict the vectors to be positive . The vector of all zeroes represents > , and embedded objects become “ more specific ” as they get farther away from the origin . Figure 1b demonstrates a toy , two-dimensional example of the Order Embedding vector lattice representation of a simple ontology . Shading represents the probability measure assigned to this lattice in the probabilistic extension of Lai & Hockenmaier ( 2017 ) . 3.3 BOX LATTICE . Vilnis et al . ( 2018 ) introduced a box lattice , wherein each concept in a knowledge graph is associated with two vectors , the minimum and maximum coordinates of an axis-aligned hyperrectangle , or box ( product of intervals ) . Using the notion of set inclusion between boxes , there is a natural partial order and lattice structure . To represent a box x , let the pairs ( xm , i , xM , i ) be the maximum and minimum of the interval at each coordinate i . Then the box lattice structure ( least upper bounds and greatest lower bounds ) , with ∨ and ∧ denoting max and min when applied to the scalar coordinates , is x ∧ y = ⊥ if x and y disjoint , else x ∧ y = ∏ i [ xm , i ∨ ym , i , xM , i ∧ yM , i ] x ∨ y = ∏ i [ xm , i ∧ ym , i , xM , i ∨ yM , i ] Here , ∏ denotes a set ( cartesian ) product — the lattice meet is the largest box contained entirely within both x and y , or bottom ( the empty set ) where no intersection exists , and the lattice join is the smallest box containing both x and y . To associate a measure , marginal probabilities of ( collections of ) events are given by the volume of boxes , their complements , and intersections under a suitable probability measure . Under the uniform measure , if event x has an associated box with interval boundaries ( xm , xM ) , the probability p ( x ) is given by ∏n i ( xM , i − xm , i ) . Use of the uniform measure requires the boxes to be constrained to the unit hypercube , so that p ( x ) ≤ 1. p ( ⊥ ) is taken to be zero , since ⊥ is an empty set . As boxes are simply special cases of sets , it is intuitive that this is a valid probability measure , but it can also be shown to be compatible with the meet semilattice structure in a precise sense ( Leader , 1971 ) . Figure 1c demonstrates a toy , two-dimensional example of the Box Embedding lattice representation of a simple ontology .

Summary: The paper proposes a method for learning embedding of hierarchies. Specifically, the paper builds on a a geometrically inspired embedding method using box representations. The key contribution of the paper is facilitating optimization of these models by gradient based methods, which eventually leads to improved accuracy on relevant benchmark data (on par or beyond SOTA). The observation is that when two boxes are disjoint in the model but have overlap in the ground truth, no gradient can flow to the model to correct the problem (which is happens in case of sparse-data.