Overlapping Spaces for Compact Graph Representations

Various non-trivial spaces are becoming popular for embedding structured data such as graphs , texts , or images . Following spherical and hyperbolic spaces , more general product spaces have been proposed . However , searching for the best configuration of a product space is a resource-intensive procedure , which reduces the practical applicability of the idea . We generalize the concept of product space and introduce an overlapping space that does not have the configuration search problem . The main idea is to allow subsets of coordinates to be shared between spaces of different types ( Euclidean , hyperbolic , spherical ) . As a result , we often need fewer coordinates to store the objects . Additionally , we propose an optimization algorithm that automatically learns the optimal configuration . Our experiments confirm that overlapping spaces outperform the competitors in graph embedding tasks with different evaluation metrics . We also perform an empirical analysis in a realistic information retrieval setup , where we compare all spaces by incorporating them into DSSM . In this case , the proposed overlapping space consistently achieves nearly optimal results without any configuration tuning . This allows for reducing training time , which can be essential in large-scale applications . 1 Introduction . Building vector representations of various objects is one of the central tasks of machine learning . Word embeddings such as Glove [ 22 ] and Word2Vec [ 19 ] are widely used in natural language processing ; a similar Prod2Vec [ 7 ] approach is used in recommendation systems . There are many algorithms proposed for graph embeddings , e.g. , Node2Vec [ 8 ] and DeepWalk [ 23 ] . Recommendation systems often construct embeddings of a bipartite graph that describes interactions between users and items [ 10 ] . For a long time , embeddings were considered exclusively in Rn . However , the hyperbolic space was shown to be more suitable for graph , word , and image representations due to the underlying hierarchical structure [ 12 , 20 , 21 , 26 ] . Going beyond spaces of constant curvature , a recent study [ 9 ] proposed product spaces , which combine several copies of Euclidean , spherical , and hyperbolic spaces . While these spaces demonstrate promising results , the optimal signature ( types of combined spaces and their dimensions ) has to be chosen via brute force , which may not be acceptable in large-scale applications . In this paper , we propose a more general metric space called overlapping space ( OS ) together with an optimization algorithm that trains signature simultaneously with embedding allowing us to avoid brute-forcing . The main idea is to allow coordinates to be shared between different spaces , which significantly reduces the number of coordinates needed . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Importantly , the proposed overlapping space can further be enhanced by adding non-metric approaches such as weighted inner product [ 13 ] as additional similarity measures complementing metric ones . Thus , we obtain a flexible hybrid measure OS-Mixed that is no longer a metric space . Our experiments show that in some cases , non-metric measures outperform metric ones . The proposed OS-Mixed has advantages of both worlds and thus achieves superior performance . To validate the usefulness of the proposed overlapping space , we provide an extensive empirical evaluation for the task of graph embedding , where we consider both distortion-based ( i.e. , preserving distances ) and ranking-based ( i.e. , preserving neighbors ) objectives . In both cases , the proposed measure outperforms the competitors . We also compare the spaces in information retrieval and recommendation tasks , for which we apply them to train embeddings via DSSM [ 11 ] . Our method works comparable to the best product spaces tested in these cases , while it does not require bruteforcing for the best signature . Thus , using the overlapping space may significantly reduce the training time , which can be crucial in large-scale applications . 2 Background and related work . 2.1 Embeddings and loss functions . For a graph G = ( V , E ) an embedding is a mapping f : V → U , where U is a metric space equipped with a distance dU : U × U → R+.1 On the graph , one can consider a shortest path distance dG : V × V → R+ . In the graph reconstruction task , it is expected that a good embedding preserves the original graph distances : dG ( v , u ) ≈ dU ( f ( v ) , f ( u ) ) . The most commonly used evaluation metric is distortion , which averages relative errors of distance reconstruction over all pairs of nodes : Davg = 2 |V | ( |V | − 1 ) ∑ ( v , u ) ∈V 2 , v 6=u |dU ( f ( v ) , f ( u ) ) − dG ( v , u ) | dG ( v , u ) . ( 1 ) While commonly used in graph reconstruction , distortion is a natural choice for many practical applications . For example , in recommendation tasks , one usually deals with a partially observed graph ( some positive and negative element pairs ) , so a huge graph distance between two nodes in the observed part does not necessarily mean that the nodes are not connected by a short path in the full graph . Also , often only the order of the nearest elements is essential while predicting distances to faraway objects is not critical . In such cases , it is more reasonable to consider a local ranking metric , e.g. , the mean average precision ( mAP ) that measures the relative closeness of the relevant ( adjacent ) nodes compared to the others:2 mAP = 1 |V | ∑ v∈V AP ( v ) = 1 V ∑ v∈V 1 deg ( v ) ∑ u∈Nv |Nv ∩Rv ( u ) | |Rv ( u ) | , Rv ( u ) = { w ∈ V |dU ( f ( v ) , f ( w ) ) ≤ dU ( f ( v ) , f ( u ) ) } , Nv = { w ∈ V | ( v , w ) ∈ E } . ( 2 ) Mean average precision can not be directly optimized since it is not differentiable . In our experiments , we use the following probabilistic loss function as a proxy:3 Lproxy = − ∑ ( v , u ) ∈E log P ( ( v , u ) ∈ E ) = − ∑ ( v , u ) ∈E log exp ( −dU ( f ( v ) , f ( u ) ) ) ∑ w∈V exp ( −dU ( f ( v ) , f ( w ) ) ) . ( 3 ) Note that when substituting dU ( x , y ) = c−f ( x ) T f ( y ) ( assuming that f ( x ) ∈ Rn , so the dot product is defined ) , Lproxy becomes the standard word2vec loss function . 2.2 Spaces , distances , and similarities . In the previous section , we assumed that dU : U × U → R+ is an arbitrary distance . In this section , we discuss particular choices often considered in the literature . 1Note that any discrete metric space corresponds to a weighted graph , so graph terminology is not restrictive . 2For mAP , the relevance labels are assumed to be binary ( unweighted graphs ) . If a graph is weighted , then we say that Nv consists of the closest element to v ( or several closest elements if the distances to them are equal ) . 3See Table 10 in Appendix for the comparison of other ways of converting distance to probability . For many years , Euclidean space was the primary choice for structured data embeddings [ 6 ] . For two points x , y ∈ Rd , Euclidean distance is defined as dE ( x , y ) = ( d∑ i=1 ( xi − yi ) 2 ) 1/2 . Spherical spaces were also found to be suitable for some applications [ 18 , 24 , 30 ] . Indeed , in practice , vector representations are often normalized , so cosine similarity between vectors is a natural way to measure their similarity . This naturally corresponds to a spherical space Sd = { x ∈ Rd+1 : ‖x‖22 = 1 } equipped with a distance dS ( x , y ) = arccos ( xT y ) . In recent years , hyperbolic spaces also started to gain popularity . Hyperbolic embeddings have shown their superiority over Euclidean ones in a number of tasks , such as graph reconstruction and word embedding [ 20 , 21 , 25 , 26 ] . To represent the points , early approaches used the Poincaré model of the hyperbolic space [ 20 ] , but later it has been shown that the hyperboloid ( Lorentz ) model may lead to more stable results [ 21 ] . In this work , we also adopt the hyperboloid model Hd = { x ∈ Rd+1|〈x , x〉h = 1 , x1 > 0 } equipped with a distance dH = arccosh ( 〈x , y〉h ) , where 〈x , y〉h : = x1y1 − d+1∑ i=2 xiyi . Going even further , a recent paper [ 9 ] proposed more complex product spaces that combine several copies of Euclidean , spherical , and hyperbolic spaces . Namely , the overall dimension d is split into k parts ( smaller dimensions ) : d = k∑ i=1 di , di > 0 . Each part is associated with a space Di ∈ { Edi , Sdi , Hdi } and scale coefficient wi ∈ R+ . Varying scale coefficients corresponds to changing curvature of hyperbolic and spherical spaces , while in Euclidean space this coefficient is not used ( wi = 1 ) . Then , the distance in the product space D1 × . . .×Dk is defined as : dP ( x , y ) = √√√√ k∑ i=1 wi dDi ( x [ ti−1 + 1 : ti ] , y [ ti−1 + 1 : ti ] ) 2 , where t0 = 0 , ti = ti−1 + di , and x [ s : e ] is a subvector ( xs , . . . , xe ) ∈ Re−s+1 . If k = 1 , we get a standard Euclidean , spherical , or hyperbolic space . In [ 9 ] , it is proposed to learn an embedding and scale coefficients wi simultaneously . However , choosing the optimal signature ( how to split d into di and which types of spaces to choose ) is challenging . A heuristics proposed in [ 9 ] allows to guess types of spaces if di ’ s are given . If d1 = d2 = 5 , this heuristics agrees well with the experiments on three considered datasets . The generalizability of this idea to other datasets and configurations is unclear . In addition , it can not be applied if a dataset is partially observed and graph distances can not be computed ( e.g. , when there are several known positive and negative pairs ) . Hence , in practice , it is more reliable to choose a signature via the brute-force , which can be infeasible on large datasets . Another way to measure objects ’ similarity , which is rarely compared with metric methods but is frequently used in practical applications , is via the dot product of vectors xT y or its weighted version xTWy with a diagonal matrix W , which is also known as a weighted inner product [ 13 ] . Such measures can not be converted to a metric distance via a monotone transformation . However , they can be used to predict similarity or dissimilarity between objects , which is often sufficient in practice , especially when ranking metrics are used . In this paper , we stress that when comparing different methods , both metric and non-metric variants should be used because different methods are better for different tasks . In particular , the dot product similarity allows one to easily differentiate between more popular and less popular items ( the vector norm can be considered a measure of popularity ) . This feature is also attributed to hyperbolic spaces , where more popular items are placed closer to the origin .

Summary: In this paper, the authors propose to extend product space for embedding structured data. In doing so, the author poses to allow each dimension to possibly belong to multiple signatures t. To optimize, the authors propose to use universal signatures and a regular Adam optimizer. Extensive experiments have been conducted to demonstrate the effectiveness of new embedded space.