Box-To-Box Transformation for Modeling Joint Hierarchies

1 INTRODUCTION . Representation learning for hierarchical relations is crucial in natural language processing because of the hierarchical nature of common knowledge , for example , < Bird ISA Animal > ( Athiwaratkun & Wilson , 2018 ; Vendrov et al. , 2016 ; Vilnis et al. , 2018 ; Nickel & Kiela , 2017 ) . The ISA relation represents meaningful hierarchical relationships between concepts and plays an essential role in generalization for other relations , such as the generalization of < organ PARTOF person > based on < eye PARTOF of person > , and < organ ISA eye > . The fundamental nature of the ISA relation means that it is inherently involved in a large amount of compositional human reasoning involving other relations . Modeling hierarchies is essentially the problem of modeling a poset , or partially ordered set . The task of partial order completion , a general term to describe tasks which require learning a transitive relation , was introduced in ( Vendrov et al. , 2016 ) . The authors also introduce a model based on the reverse product order on Rn , which essentially models concepts as infinite cones . Region-based representations have been effective in representing hierarchical data , as containment between regions is naturally transitive . Vilnis et al . ( 2018 ) introduced axis-aligned hyperrectangles ( or boxes ) that are provably more flexible than cones , and demonstrated state-of-the-art performance in multiple tasks . Thus far , not as much effort has been put into modeling joint hierarchies . Patel et al . ( 2020 ) proposed to simultaneously model ISA and HASPART hierarchies from Wordnet ( Miller , 1995 ) . To do so , however , they effectively augmented the graph by duplicating the nodes to create a single massive hierarchy . Their model assigns two boxes BISA and BHASPART for each node n , which are unrelated , and therefore misses out on a large amount of semantic relatedness between ISA and HASPART . In this paper we propose a box-to-box transformation which translates and dilates box representations between hierarchies . Our proposed model shares information between the ISA and HASPART hierarchies via this transformation as well as cross-hierarchy containment training objectives . We compare BOX-TRANSFORM MODEL with multiple strong baselines under different settings . We substantially outperform the prior TWO-BOX MODEL while training with only the transitive reduction of both hierarchies and predicting inferred composition edges . As mentioned above , our model ’ s shared learned features should allow for more generalization , and we test this by training on a subset of the transitive reduction , where we find we are able to outperform strong baselines . Finally , we perform a detailed analysis of the model ’ s capacity to predict compositional edges and transitive closure edges , both from an overfitting and generalization standpoint , identifying subsets where further improvement is needed . 2 RELATED WORK . Recent advances in representing one single hierarchy mainly fall in two categories : 1 ) representing hierarchies in non-Euclidian space ( eg . hyperbolic space , due to the curvature ’ s inductive bias to model tree-like structures ) 2 ) using region-based representations instead of vectors for each node in the hierarchy ( Erk , 2009 ) . Hyperbolic space has been shown to be efficient in representing hierarchical relations , but also encounters difficulties in training ( Nickel & Kiela , 2017 ; Ganea et al. , 2018b ; Chamberlain et al. , 2017 ) . Categorization models in psychology often represent a concept as a region ( Nosofsky , 1986 ; Smith et al. , 1988 ; Hampton , 1991 ) . Vilnis & McCallum ( 2015 ) and Athiwaratkun & Wilson ( 2018 ) use Gaussian distributions to embed each word in the corpus , the latter of which uses thresholded divergences which amount to region representations . Vendrov et al . ( 2016 ) and Lai & Hockenmaier ( 2017 ) make use of the reverse product order on Rn+ , which effectively results in cone representations . Vilnis et al . ( 2018 ) further extend this cone representation to axis-aligned hyperrectangles ( or boxes ) , and demonstrate state-of-the-art performance on modeling hierarchies . Various training improvement methods for box embeddings have been proposed ( Li et al. , 2019 ; Dasgupta et al. , 2020 ) , the most recent of which is termed GumbelBox after it ’ s use of a latent noise model where box parameters are represented via Gumbel distributions . Region representations are also used for tasks which do not require modeling hierarchy . In Vilnis et al . ( 2018 ) , the authors also model conditional probability distributions using box embeddings . Abboud et al . ( 2020 ) and Ren et al . ( 2020 ) take a different approach , using boxes for their capacity to contain many vectors to provide slack in the loss function when modeling knowledge base triples or representing logical queries , respectively . Ren et al . ( 2020 ) also made use of an action on boxes similar to ours , involving translation and dilation , however our work differs in both the task ( i.e . representing logical queries vs. joint hierarchies ) and approach , as their model represents entities using vectors and a loss function based on a box-to-vector distance . The inductive bias of hyperbolic space is also exploited to model multiple relations , Ganea et al . ( 2018a ) learn hyperbolic transformations for multiple relations using Poincare embeddings , and show model improvement in low computational resource settings . Patel et al . ( 2020 ) , which our work is most similar to , represent joint hierarchies using box embeddings . However , they represent each concept with two boxes ignoring the internal semantics of the concepts . Modeling joint hierarchies shares some similarities with knowledge base completion , however the goals of the two settings are different . When modeling joint hierarchies you are attempting to learn simultaneous transitive relations , and potentially learn relevant compositional edges involving these relations . For knowledge base completion , on the other hand , you may be learning many different relations , and primarily seek to recover edges which were removed rather than inferring new compositional edges . Still , the models which perform knowledge base completion can be applied to this task , as the data can be viewed as knowledge base triples with only 2 relations . There have been multiple works that aim to build better knowledge representation ( Bordes et al. , 2013 ; Trouillon et al. , 2016 ; Sun et al. , 2019 ; Balazevic et al. , 2019a ) . Most relevant , Chami et al . ( 2020 ) ; Balazevic et al . ( 2019b ) recently proposed KG embedding methods which embeds entities in the Poincaré ball model of hyperbolic space . These models are intended to capture relational patterns present in multi-relational graphs , with a particular emphasis on hierarchical relations . 3 BACKGROUND . 3.1 BOX LATTICE MODEL . Introduced in Vilnis et al . ( 2018 ) , a box lattice model ( or box model ) is a geometric embedding which captures partial orders and lattice structure using n-dimensional hyperrectangles . Formally , we define the set of boxes B in Rn as B ( Rn ) = { [ x1 , x1 ] × · · · × [ xd , xd ] } , ( 1 ) where xi , xj ∈ R , and we represent all degenerate boxes where xi > xi with ∅ . A box model for a set S is a function Box : S → B ( Rn ) which captures some desirable properties of the set S. As the name implies , the box lattice model is particularly suited to representing partial orders and lattice structures . Definition 1 ( Poset ) . A partially ordered set , or poset , is a set P along with a relation such that , for each a , b , c ∈ P , we have 1. a a ( reflexivity ) 2. if a b and b a then a = b ( antisymmetry ) 3. if a b and b c then a c ( transitivity ) Definition 2 ( Lattice ) . A lattice is a poset where each pair of elements have a unique upper bound called the join , denoted by ∧ , and a unique lower bound called the meet , denoted by ∨ . The authors note that there are natural geometric operations which form a lattice structure on B : Box ( x ) ∧ Box ( y ) : = ∏ i [ max ( xi , yi ) , min ( x i , yi ) ] , ( 2 ) Box ( x ) ∨ Box ( y ) : = ∏ i [ min ( xi , yi ) , max ( x i , yi ) ] , ( 3 ) In other words , the meet of two boxes is the smallest containing box , and the join is the intersection , or ∅ if the boxes are disjoint . These geometric operations map very neatly to hierarchies , where the meet of two nodes is their closest common ancestor and the join is the closest common descendent ( or ∅ if no such node exists ) . The ability of this model to capture lattice structure using geometric operations makes it a natural choice to embed hierarchies . 3.2 PROBABILISTIC BOX MODEL TRAINING . In Vilnis et al . ( 2018 ) , the authors also introduced a probabilistic interpretation of box embeddings and a learning method which was improved upon in Li et al . ( 2019 ) and Dasgupta et al . ( 2020 ) . By using a probability measure µ on Rd ( or by constraining the space to [ 0 , 1 ] d ) , one can calculate box volumes as µ ( Box ( X ) ) . The pullback of this measure yields a probability measure on S , and thus the box model can be imbued with valid probabilistic semantics . In particular , since the box space B is closed under intersection , we can calculate joint probabilities by computing P ( X , Y ) = µ ( Box ( X ) ∧ Box ( Y ) ) and similarly compute conditional probabilities as P ( X | Y ) = µ ( Box ( X ) ∧ Box ( Y ) ) µ ( Box ( Y ) ) . ( 4 ) The conversion from a poset or lattice structure to probabilistic semantics is accomplished by assigning conditional probabilities , namely a b if and only if P ( b | a ) = 1 . We note that the properties required of the relation follow as a natural consequence of the axioms for conditional probability . Apart from providing rigor and interpretability , the calibrated probabilistic semantics also inform and facilitate the training procedure for box embeddings , which is accomplished via gradient descent using KL-divergence with respect to the aforementioned probability distribution as a loss function . As one might expect , care must be taken to handle the case when boxes are disjoint , as there is no gradient . In ( Vilnis et al. , 2018 ) the authors made use of the lattice structure to derive a lower bound on the probability , and ( Li et al. , 2019 ) introduced an approximation to Gaussian convolution over the boxes which similarly handled the case of disjoint boxes . ( Dasgupta et al. , 2020 ) improves this further by taking a random process perspective , ensembling over an entire family of box models . The endpoints of boxes are represented using Gumbel distributions , that is GumbelBox ( X ) = ∏ i [ Xi , X i ] , Xi ∼ MaxGumbel ( µi , β ) , Xi ∼ MinGumbel ( µi , β ) , ( 5 ) where µ , β are the location and scale parameters of the Gumbel distribution respectively . The MaxGumbel distribution is given by f ( x ; µ , β ) = 1 β exp ( −x−µβ − e − x−µβ ) , ( 6 ) and the MinGumbel distribution given by negating x an µ . The Gumbel distribution was chosen due to it ’ s min/max stability , making the set of Gumbel boxes closed under intersection , i.e . the intersection of two Gumbel boxes is another Gumbel box . We denote the space of all such boxes as G. The expected volume of a Gumbel box can be efficiently calculated analytically , and in Dasgupta et al . ( 2020 ) the authors use this expected volume to calculate the conditional probabilities mentioned in equation equation 4 . This training method leads to improved performance on a number of tasks , and is particularly beneficial when embedding trees , thus we will use this Gumbel box approach in our setting .

Summary: The paper focuses on modeling multiple hierarchical relations on a heterogenous graph.  The task “modeling joint hierarchies” is essentially trying to infer whether a given pair of entities has a hierarchical connection especially when there exists multiple hierarchical relations (2 in the paper), and missing links. The paper proposes to embed entities using boxes whose endpoints follow the Gumbel distribution. Given there exists two hierarchical relations, the paper transforms the box of one entity under relation 1 to the box of the entity under relation 2 with a parameterized linear function. This is in contrast to previous work that parameterized the box of two relations using separate independent parameters.