Data-Informed Geometric Space Selection
Authors: Shuai Zhang
Wenqi Jiang
Venue: NeurIPS 2023 poster (2023)
Keywords: Geometric representation learning
TL;DR: A data-informed strategy for automating the alignment of geometric representation learning with underlying data structures

Abstract: Geometric representation learning (e.g., hyperbolic and spherical geometry) has proven to be efficacious in solving many intricate machine learning tasks. The fundamental challenge of geometric representation learning lies in aligning the inherent geometric bias with the underlying structure of the data, which is a rarely explored topic in the literature. Existing methods heavily rely on heuristic assumptions on the data structure to decide the type of geometry to be adopted, which often leads to suboptimal performance. This work aims to automate the alignment process via a data-informed strategy such that we optimize model performance with minimal overhead. Specifically, a sparse gating mechanism is employed to enable each input data point $\mathit{p}$ to select $K$ geometric spaces from a given candidate geometric space pool with $N$ ($K<N$) spaces of different geometry. The selected $K$ spaces are then tightly integrated to formulate a Cartesian product space, which is leveraged to process this input data $\mathit{p}$. In doing so, each input data is processed by the spaces it selected with maximum specialization. We empirically show that this method can effectively align data and spaces without human interventions and further boost performance on real-world tasks, demonstrating its potential in eliciting the expressive power of geometric representations and practical usability.

Data-Informed Geometric Space Selection
Shuai Zhang
Wenqi Jiang
ETH Zurich Breungghuai@outloode
ETH Zurich sengijjaangint .ttz..h
Abstract
Geometric representation learning (e.g.. hyperbolic and spherical geometry) has proven be efficacious in solving many intricate machine learning tasks. The fundamental challenge of geometric representation learning lies in aligning the inherent geometric bias with the underlying structure of the data, which is rarely explored topic in the literature. Existing methods heavily rely on heuristic assump- tions on the data structure decide the type of geometry be adopted, which often leads to suboptimal performance. This work aims to automate the alignment process via data-informed strategy such that we optimize model performance with minimal overhead. Specifically, a sparse gating mechanism is employed to enable each input data point p to select K geometric spaces from given candidate geometric space pool with N (K < N) spaces of different geometry. The selected K spaces are then tightly integrated formulate Cartesian product space, which is leveraged to process this input data In doing so, each input data is processed by the spaces it selected with maximum specialization. We empirically show that this method can effectively align data and spaces without human interventions and further boost performance on real-world tasks, demonstrating its potential eliciting the expressive power of geometric representations and practical usability.
Introduction
Representing data in fow-dimensional geometric spaces (c.g., Euclidean, hyperbolic, and spherical) has been invaluable technique in variety of machine learning tasks. Buclidean space has been the most widely adopted and predominant geometric space across many tasks [8] 31 Recent research has shown that -uc-udclid geometry, such as hyperbolic and spherical geometry, can be more effective in representing data with specific/intricate inherent structures. Typically, hyperbolic space, type of manifold with constant negative (sectional) curvature. provides greater ability to model tree-like structures and capture hierarchical patterns 32
Conversely, spherical space, with positive curvature, is well-suited for fitting data with cyclical structures endows models with different inductive bias that prioritizes learning specific patterns in the data.
11 21
30 Different geometry
Geometric representation learning faces significant challenge effectively aligning underlying data structures with appropriate geometric spaces. Earlier approaches tend to map all data to single geometric space, operating under the implicit assumption that the majority, if not all, of the data conforms to the same geometric pattern. This assumption of a uniform geometric pattern across all data points is unrealistic. As such, there has been growing interest hybridizing different geometric spaces 22 37 4 via Cartesian product to form a mixed curvature space. By leveraging the benefits of multiple geometric spaces, this approach can offer improved performance compared methods that rely single geometric space.
Real-world data usually characterized by intricate patterns that single geometric space cannot adequately capture. Thus, employing a one-space-fits-all approach is unlikely to produce optimal solutions. In addition, the Cartesian product space representation learning technique does discriminate between input examples based on their underlying geometric structures. For example,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

if a Cartesian product space is constructed by combining spherical space with hyperbolic space, both spaces will be used to fit all input data examples, irrespective of whether a data point only aligns with one of the geometry. Ideally, each space should calibrate its internal parameters using the input examples that conform to the corresponding manifold to ensure that the modeling capacity of all geometric spaces is maximized.
Accurately aligning input data with appropriate geometric spaces is crucial challenge in geometric representation learning. However, neither the single geometric space method nor the Cartesian product approach fully address this issue. Our primary goal is tackle this challenge by developing an automated approach that seamlessly aligns each data point with its appropriate space. Instead of relying on oversimplified heuristies or manual interventions, we aim to learn the Sata-space alignment component together with the task-specific model using an end-to-end framework.
To this end, we propose data-informed method for geometric space selection. Given N candidate geometric spaces with different geometry, we let each data point p choose K(1 S K < N) spaces from the pool. The chosen K spaces are then tightly combined via Cartesian product to form Cartesian product space, which is used to process p. The selection process is parameterized by sparsely gated neural network that can adaptively activate a subset of the given spaces based on the given input data. This module can be easily plugged into any geometric representation learning task to enable end-to-end optimization. Theoretically, we can construct NCK = (N-K)KK possible Cartesian product spaces in the end, which brings greater degree of freedom and personalization for
N!
representation learning. Moreover, the proposed method will only incur a negligible computational overhead since only the K selected geometric spaces participate in the model inference no matter how large the space pool is.
The contributions of this work are summarized as follows:
We propose a data-informed approach for geometric space selection that offers the advantage of automatically aligning each input data point with its preferred geometric space, thereby increasing specificity while being computationally efficient.
We empirically demonstrate the practical effectiveness of this method with real-world datasets across two downstream tasks. We examine the proposed method's inner workings (e.g., distribution of the selected geometric spaces) via visualization, ablation, and case studies, reaffirming the benefits of the proposed method.
2 Related Work
We briefly review two areas of active research that intersect with the present work.
2.1
Non-Euclidean Geometric Representation Learning
Two popular non-Euclidean geometric spaces with non-zero curvature, including hyperbolic and spherical spaces, have gained increasing attention and can be found in broad spectrum of repre- sentation learning applications. Hyperbolic space has negative curvature and is reminiscent of a continuous version of trees, which makes it excel in capturing hierarchical structures. A bunch of work has demonstrated its effectiveness tasks such as graph modeling and knowledge representa-
[5] 20 26 40). On the other hand, spherical space with positive curvature more suitable for directional similarity modeling and cyclical-structuree data. We can find applications of spherical spaces in text embeddings [30], texture mapping [44), time-warping functions embedding [44]. In these papers, the choice of geometric space is primarily based on expert heuristics about the data characteristics.
tion 32 42
39
5
12
35 (88
[0
To combine the best of different worlds, Gu et al. 22 37
proposed method construct a mixed curvature space via Cartesian product of different spaces. The component spaces in Cartesian product spaces are selected from Euclidean, hyperbolic, and spherical spaces. Each component of Cartesian product space has constant curvature, while the resulting mixed space has a non-constant curvature, which enables it to capture a wide range of patterns with a single model. Empirically, Cartesian product spaces demonstrate its efficacy in graph reconstruction, word embedding with low dimensions [22], node classification in graphs 4 and image reconstruction [37]. Product space designed to handle data with mixed patterns, but the alignment process between data and geometric

---

space is still oversimplified and not customized for each data point. Consequently, the challenge of aligning appropriate spaces with imput data points persists, and solution this problem remains elusive.
2.2 Sparsely-gated Mixture of Experts
Another related research area is sparsely-gated neural networks, which are type of mixture-of-
experts (MOE) architecture 24 25 Over the past few decades, many MOE architectures have been proposed, such as hierarchical structures [47]. sequential experts [2]. deep MOE [18), and sparsely-gated MOE [36]. MOE based on the divide-and-conquer principle, which divides the problem into homogencous regions and assigns expert to each region. The final prediction is made by combining the predictions of all experts via a gating network. The gating network controls the contribution of each expert using probabilistic gating function. If the gating function is sparse, i.e., restricted to assigning non-zer weights to only a subset the experts, unused experts need not be computed. MOE has been widely studied in various tasks, such multi-task learning [29), large language models 36 19), and scaling up vision transformers [34..
We draw inspiration from sparsely gated MOE to automate the alignment between data and its appropriate geometric space. Our approach allows each input data point to control the gating function and select suitable geometric spaces from a candidate geometric space pool, thus enabling customized assignment of data to its suitable spaces. It brings higher specialization, greater degree of freedom, and superior expressiveness.
Preliminaries
3.1 Riemannian Geometric Space
Stereographic Projection model. In general, there are three types of constant curvature spaces with respect the sign of the curvature (Fig. Common realizations are Euclidean space (flat), hypersphere S (positively curved) and hyperboloid HI (negatively curved). For the latter two, we pre- fer their stereographic projection model: projected sphere D and Poincare ball IP. These models are easier optimize and avoid the problem of non- convergence of norm of points with a curvature close 0. and the projection is conformal, i.c.. does not affect the angles between points [32 37
Figure 1: Left: Hyperbolic space IH Center: Euclidean plane E Right: Spherical space S.
An alternative to vector space in non- Eunlddeid geometry is gyrovector space operations such as vector addition and multiplication.
41
which defines
Definition For D and IP (jointly denoted as Me. where denotes curvature), the addition between two points x,y E MM, also known as Möbius addition @e (for both signs of c), defined as:
B. 1--2c(x
c ly
x
2c(x, y)
Where (.) is Euclidean inner product. The distance between points in the gyrovector space defined as:
2 vld
ax,
tan 1V - @ v1l2),
where tan, stands for tan if 0 and tanh ifc < 0. In both spaces, have Euclidean geometry when c + It is easy to prove that:
dM.(x.y) c+0 2
(3)
which means the gyrospace distance converges to Euclidean distance when limiting to zero.

---

Definition 2 Let TxMe be the tangent space to the point x E MM. Mapping between (Euclidean) tangent space and hyperboliccspheriial space performed with exponential map: Tx + Ma and logarithmic map: Me + Tx which are defined as:
(yg vlcA2
tan 1
(Vlll - DD y112)
- -X Bc
B. y
X v
(xp = x Be (Lanal (cl
2
clllv|22
where Xx is a conformal factor, defined as X = 2/(1 + c|xx 12), which is used to transform the metric tensors between (Euclidean) tangent space and non non-nonnnnn -on-Euclidean space.
3.2 Cartesian Product of Spaces
Definition Product space is defined as the Cartesian product of multiple spaces with varying dimensionality and curvature. Let P denote product space composed by N independent component spaces M(), ( (...,MNN) ()) The mixed space P has the form (operator x can be omitted):
P = X M - x M x... M(N)
(5)
Definition 4 The product space P also has distance functions. The squared distance between points x,y E P is defined as:
N
dx((, Σ deuto R
(6)
where X and M(0) denote the corresponding vectors the component space M
Other operations such as exponential map and logarithmic map are element-wise meaning that can decompose the points into component spaces, apply operations on each component space, and then compose them back (e.g., concatenation) (i.e., parametrization) refers to the types of space, the number of spaces, the dimensionality. and the curvature each space. For instance, (P160033 (T)50)2 denotes Cartesian product space of three hyperbolic spaces of hidden dimension 100 and two spherical spaces of hidden dimension 50.
the product space. product space's signature
4
Data-Informed Space Selection
4.1 Geometric Space Pool
Suppose we have space pool with N ge- ometric spaces, !! [E,
Geometric Space Pool
(I...., N.. For simplicity, we assume all spaces have the same dimensionality b. The goal is to select K(1 < K < N) spaces from the given N spaces for each data point p. If K > 1, the selected K spaces will form product space. Theoretically, there will be nCx IN KIKT possible space combinations. Figure 2 illustrates the proposed data-informed space selection pipeline. Here, and are selected by data and the Cartesian product P x E is used to handle p.
N=
P.
D.
00oc 0
Cartesian Product Space
K
PXD
Geometric Space Selector
N!
Embeddings
PX
A(O1
DX E
Figure 2: Data-informed geometric space selection. and are selected to form P x EE for input data p.
4.2 Embeddings
For each geometric space M! we define an embedding dictionary e In the space, data point (6) p is represented by vector ep E 1R, where b is the hidden dimension of this embedding dictionary.

---

We initialize all the embedding vectors in the tangent space, and the exponential map will be applied recover them into hyperbolic or spherical space when necessary. In doing so, standard Euclidean optimization algorithms can be directly applied to enable an end-to-end learning framework to avoid the cumbersome Riemannian optimization 77.
4.3 Geometric Space Selector via Sparsely-gated MOE
We use sparsely-gate MOE network to select a suitable space subset from the space pool. The
selector is differentiable network that takes as input the embedding vectors op (i) and outputs the probability of each space to be selected. Formally, the input is defined as:
Xp = 1e (2) ...... " C
(7)
where Xp is two-dimensional matrix of shape N x b. We use convolutional neural network to transform this matrix into vector of dimension N. Other types of neural networks are also viable, we adopt CNN because it does not introduce a lot of additional parameters. Then, the last hidden layer of the gating network has the following form:
f fxxp) = (xp) + randn() In(1 + exp(f2(x)))),
(8)
where fi and f2 represent CNN layers and f. (x,) E R function "randn()" is used to product Gaussian noise in the training stage to improve loading balancing, i.e., balance the number of samples accepted by each space.
To impose sparsity, we employ TopK function to obtain the largest K elements from f(xp):
((xp) + TopK(
where TopK returns the original value if the element is in the top K list; otherwise, it returns -00. Afterward, N softmax gating functions 91, 92. ... IN are leveraged to control which K space(s) to be selected. For coo, it will output zero.
exp(f(xp)()
(x(xp))
NI.
(10)
ΣSn exp(f(xp)g)
Later, spaces with nonzero gating values will be activated while others remain idle. The selected K space will form a Cartesian product space.
4.4 Task Specifie Prediction Function
The prediction function is specific to the type of task. For instance, in each component space M! we can adopt squared distance to produce the model prediction. Given two points p and g. the prediction function can be defined as $ SM (ep (i) eq ( space not selected, s SM More details will be provided in the experiment section for each task. If K 1, the final prediction score is computed as:
N s (p. q) = log gi (xp) (o (())))
(11)
Here, log-sum-exp technique is leveraged to improve the numerical stability avoid problems such as underflow and overflow. Multiplying the prediction by the gating probability is optional in the framework.
If () is calculated with squared distance and K = N, we can recover the squared distance function defined for product space by removing (x( and the log-sum-exp technique. Our prediction function cannot be viewed as a standard distance metric as it does not satisfy the triangle inequality property. Also, it is flexible to other prediction functions to replace squared distances.
4.5 Load Balancing Regularization
We use load balancing regularization term to improve numerical stability and avoid some spaces from being over-selected. In specific, two additional regularization terms used following (36).
5

---

For batch of X inputs,
( 2 Σ g(xp)))
(12)
PEX
where (v)) warkance(1) mean(())
is the coefficient of variation.
Eg (x((x)))
(13)
Σ Φ f((x)) -(f(xx), PEX
<(X);
(14)
In + exp(f2(Xx))
where ku, is the hthh highest component of excluding component i. + is the cumulative distribution function of standard normal distribution.
The two regularization terms are added to the task-specific loss for model optimization:
i = fask + M1l1 + H262,
(15)
where and #a are scaling factors (set to 0.01 by default).
4.6 Limitation
Compared with Cartesian product space [22]. the proposed method introduces one additional hy- perparameter, K. As we will show later, relatively small K (e.g., K <= 5) can introduce significant performance boost. Just like determining the signature of Cartesian product space, we need determine what geometric spaces be included in the space pool. This overhead is inherited from Cartesian product space and can be potentially mitigated via hyper-parameters optimization algorithms 23
Experiments
In this section, we evaluate the proposed method on real-world datasets demonstrate its capability in dealing with practical tasks, including personalized ranking and link prediction for relational graphs.
5.1 Personalized Ranking
The task of personalized ranking is to provide user with tailored, ranked list of recommended items [33]. Given partially-observed interaction matrix between users and items, where each entry of the matrix represents the rating/preference a user u gives to item if has interacted with the iterm (e.g., watched the movie) and is otherwise missing. We conduct experiments on two well-known datasets, MovieLens 100K and MovieLens 1M. The first dataset consists of 100K ratings between 943 users and 1,682 movies, and the second dataset consists of one million ratings between 6,040 users and 3.706 movies. Over 90% of the entries of these interaction matrices are missing. Completing this matrix enables us to recommend to customers what to watch next.
The personalized ranking model takes user vector u. E R and an item vector v E R input, and approximate the rating with squared distance [42 is estimated by:
That is, the rating of user gives to movie
8 M (u, v) ddycco (exp)(u(")) expo ( (eep)((!"))) ())).
(16)
Alternatively, we can use the dot product, u u to approximate each entry of the matrix (B
The input of the gating network the concatenation of the user and item embeddings. Specifically. vector of size 22 is used as the input, and a linear layer is used as the sparsely-gated network. Let s(u, v) denote the final prediction. We treat the task learning-to- to-rank problem and optimize by minimizing the following contrastive loss:
Pask =
Σ (E
max(0, (ux, + m - -s(u, s(u,)2)),
(17)

---

Vean Prodeee Feditang13 Aacaligoo
Performanc on Mouletens 3M
Performance en MovieL 300K
Preck 2X7 Reculigis
0.250
0.200
0.152
2.00
0:00
Dnt relion
Lorg Long imbord spuce
Det Prssoo
Lary pm lencce
Figure 3: Personalized ranking performance on MovieLens 1M and MovieLens 100K.
where Ω is the collection of training samples, v1 is the movie that watched, and 022 represents unobserved movie for the user; m is hyper-parameter.
We hold 70% entries in each user's interactions as the training set, 10% entries as the validation set for model tuning, and the remaining 20% for model testing. All interactions (e.g., ratings) are binarized following the implicit feedback setting [3 For all methods, the total dimension Σi= b set to 100 for fair comparison ensure the same model size. The curvatures for spherical and hyperbolic models are set to and 1, respectively. N is set to K is tuned among {1, 2, 3, 4). Regularization rate is chosen from {0.1, 0.01, 0.001 m is fixed 0.5. Adam is adopted as the optimizer. We measure the performance based on the widely adopted metrics in personalized ranking: mean average precision (MAP), precision, and recall at the top-ranked ten items (Precision @ 10 and Recall @ 10). For all experiments, we report the average over five runs.
Main Results. Figure 3 summarizes the performances. The proposed method achieves the best scores on all three metrics, outperforming the product space counterpart with the same signature and other single space models. Also, dot-product based approach remains competitive in this task. We estimate the global average curvature with the algorithm described in [22] and obtain a value of 0.695 for MovieLens 1M and 0.19 for MovieLens 100K, which suggests that they lean towards cyclical structures. However, from the performance comparison among 100 D D000' and |000 we find that the model built in Euclidean space offers the best performance, outperforming spherical and hyperbolic counterparts. This observation indicates that the global average curvature estimation algorithms proposed by Gu al. 22 may give misleading information, which also ascertains the difficulty of data-space alignment.
5.2 Link Prediction for Relational Graphs
effective way integrate disparate data sources and model underlying relationships. Encoding the nodes and relations of relational graphs into low-dimensional vector spaces is vital to downstream applications such missing facts completion, question answering. information extraction, and logical reasoning 3
Relational graphs have emerged
Given relational graph G with set of entities E and set of relations R. Each triple, abbreviated as (h, r, t). in G is composed by two entities (i.e., head entity E & and tail entity E E), and the relationship r R between them. In the jth space, entities h, are represented by vectors e oz R and relation is represented by two translation vectors ()) ,,3/ RD and a rotation vector 7 R Also, each head (tail) entity is associated with bias term (b,) R.
Inspired by the method (RotE and RotH) proposed in [1], we propose generic model, RotX, that subsumes the two methods and can be extended to spherical space. In RotX, the head entity is translated twice via Möbius addition and rotated once. Formally, the head entity processed as follows:
Q ( (h,r) = ROTATE (+) @ (a )!?) @ exp6 (B(())
(18)
where exp6 is the exponential map over origin. ROTATE rotation function, and 7 the rotation matrix. The transformed head entity is then compared with the tail entities using squared distance.
7

---

0 V4A
0 mm31
Rasulit en WINLER
Results P515K-237
0.33
0:33
0422
0.03
-0225
0.09
-0033
0.05
-035
Tanssaar
Aroud
Disstoot
every
Coou
Exort
Buttd
capyee go diay
Trmn
Figure 4: Performance on relational graphs link prediction. We use the performance of TransE- AT 46) as the baseline (e.g., AMRR MRR(MMRMM (oodel) MRR(TransE-AT) for the lollipop charts. In our method, for WNISRR, the geometrie space pool consists of 10000 and one E K for FB 13K--237, the geometric space pool consists of 3 x 1p000 and 2 x D000 K = 4.
The final scoring function as follows:
SM (h, -Pucos (Q r), expo(e()) +60 + 6,00
(19)
We make the curvature relation specific and trainable. Each relation has a curvature parameter and is trained simultaneously with the model. We adopt the following cross-entropy loss the objective function:
liask
Σ log(1 + eesp(Y(auu (b,
(20)
(h,r,))
where (I....) E {1, 11 is binary label indicating whether triple is factual (1) not (-1). Ω represents the training collection, including positive and negative triples.
We use two datasets including WNI8RR g 16 and FB15K -237 8 16 for model evaluation.
WN18RR taken from WordNet, a lexical database of semantic relations between words. It has 40, 943 entities, 1 relations, and 86, 835/3, 034/3, 134 training/validationnn a subset of the Freebase knowledge graph. global resource of common and general information. It has 14, 541 entities, 237 relations, and 272, 115/17, 11511,,35//2 535/20, 466 The performance is evaluated using two standard metrics, including mean reciprocal rank (MRR) and hit rate (HR) with a given cut-off value (1,3, 10}.
triples. FB15K-237
triples.
The total dimension fixed 500 for fair comparison. Learning rate is tuned amongst (0.01, 0.005, 0.001}. For all experiments, we report the average over 5 runs. We set the kernel size and stride 3 for convolution operation in the gating network. N is set to and K tuned among {1, 2, 3, 4}. The number of negative samples (uniformly sampled) per factual triple is set to 50. Optimizer Adam is used for model learning. We perform early stopping if the validation MRR stops increasing after 10 epochs.
We compare method with several baselines, including Euclidean methods TransE 8 Dist- Mult [45). ConvE [6). TuckER (6), RotE [1]], BoxE I complex number based methods ComplEx-
N3, [27 and RotatE [38; quaternion model QuatE [ spherical models MuRS [3]; hyper-
bolic methods MurP (S RotH [I: Cartesian product space model MCGNN 43 and Trans-
former/pretrained model based models HiuER [44 and CSProm-KG 1K3 CSProm-KG uses the
pretrained large language model (LLM), BERT 17 as the backbone.
TransE-AT [46] as basclind" and report the differences. We make the following observations. Firstly, our method outperforms all compared baselines without pretrained large language model external sources. Specifically, it outperforms Euclidean, hyperbolic, spherical, complex-valuee and Cartesian product spaces methods and obtains clear performance gain over the second-best model. Secondly, the
Main Results. We report the performance comparison in Figure where
concurrent work. TransE-AA on WNISRR: MRR=0.479, HR@ PRR1=00.45. 100,434, HR@3-0.495, HR @ 10=0.571; on FB 237: MRR=0.351, HR @ 1=0.257, HR @ 3-0.386. HR @ 10=0.538.

---

9033
2030
PO Fuh Pun 7ut PUN Pv's
AARA APAA
PA0.0 ANDA
APA
Spaces
Spaces
PoPsPs PiPSh PuP PSPODDD Prhtout
Figure 5: Left: distribution of the selected geometric space on WN18RR; Center: distribution of the selected geometric space on FB15K 237; Right: geometric space distribution of four relations from FB15K-237.
AAXI
MAM
ANE
DM lemile
Figure 6: Distribution of the selected spaces shown in a sub-graph of the FB15K-237.
improvement over the second-best model on FB15K-237 is larger than that on WN18RR. One possible explanation that FB15K-237 has a more diverse set of relations and a higher structure heterogeneity [5]. so better geometric space alignment could bring more value. It is worth noting that WN18RR sampled from the WordNet lexical database and BERT is trained on a large corpus of lexical data, it is unsurprising that CSProm-KC can obtain very high score on WN18RR. However, its relatively modest scores on FB15K-237 suggest limited practicality for non-lexical knowledge graphs. As such, it is not fair to compare an LLM based approach with our approach using LLM may lead to data leakage.
Distribution of the selected Geometric Space. Figure 5 (left and center) presents the distribution of the selected spaces. We observe that certain product spaces are preferred depending on the dataset. For example, the product space 13 is the most selected space on WN18RR. While on more preferable. To show more fine-grained examples, we randomly select four relations from FB15K-233 and visualize the distribution in Figure 5 (Right). We find that each relation has their own desirable product spaces. For example, relation "locationn location/time seeeee__oes prefers space P, P2 but relation "people/person/proffee P,PaD,,D 222 This reconfirms the specification capability our method.
FB15K-237, the product space P2 ,11,,""333 1
prefers space
To show how the space distribu- tion coincides with data structures, we randomly extract three connected graph components from the test set of FB15K-237 and visualize them in Figure 6 Each triple can be repre- sented with an edge and its two end nodes. We render edges with different colors based on the type of product spaces used to model the correspond- ing triples. From the left two graphs, we see some clear clusters with a dom- inant color, which suggests that neigh- borhood triples are more likely to be processed with the same selected prod- uct space. From the right-most graph, we find that different relations usually prefer different spaces, which indicates that the type of relations plays a critical role in the space selection process.
M
Signature
WNISKK
FHI5K-237
FFR
Single
D5000
0,492
0.514
0.295
0.322
(2100)3 (p:1000
0,484
0.498
0,311
0,344
(g100)
Product
(=100)4g:.00
0.468
0.479
0-488
0.321
0.356
0.344
0.479
0.196
0.308
0.342
3 x p400 x 200
0.500
0.521
0.530
0.545
Ours
x p40o p2000
p1oo" gı00
0.904
0.526
0.536
0.515
0.551
0.527
2 p0oo 100 100
Table 1: Comparison with Cartesian product space mod- els [22] (a few signatures are reported due to length con- straints). K 2 for WNISRR and K 4 for FB15K-237
Comparison with Cartesian Product Space with Varying Signatures. Compared with GNNN (its signature is 200000)00002000 (in Figure 4 Our model achieves the best performance even with

---

fewer trainable parameters. Then, we compare our method with Cartesian product space models with varying signatures in [able[] We observe that our method constantly outperforms pure product spaces with the same prior signatures. It is worth noting that our method requires merely two/four active spaces to outperform product space models with five active spaces. These observations reconfirm the effectiveness of the data-informed space alignment method.
Impact of N and K.
The left figure of 7 shows the impact of N by fixing K. We observe that (a) the model performance does not ben- efit much from increasing N this dataser; (b) the inference time remains nearly constant when increase N, confirming its efficiency. The effect of K is shown in Figure 7 (right). We find that K has higher impact on the model performance and inference time.
Infferenc S Time b9
0.559
0.50
0.525
Inference Time
0.525
0.500
0.500
0.455
0.453
Figure 7: Effect of N (left) and K (right) on WN18RR.
Increasing K will generally degrade the performance on this datasel and yield additional computational overhead.
6 Conclusion
In this paper, propose data-informed geometric space selection method. Our method enables an automatic alignment between data and geometric spaces by allowing each data point to automatically select its preferred geometric spaces via sparsely-gan -agttdd MOE network. It brings a greater extent of specification for geometric representations while remaining efficient. We show that the proposed approach can boost representation performance in real-world applications with noticeable margin.
Broader Impact. This approach exhibits considerable potential facilitating enhanced and versatile geometric representation learning, thereby engendering broader and more comprehensive applications across various domains.
References
[1] Ralph Abboud, Ismail Ceylan, Thomas Lukasiewicz, and Tommaso Salvatori. Boxe: A box embedding model for knowledge base completion. Advances in Neural Information Processing Systems, 33, 2020.
[2] Rahaf Aljundi, Punarjay Chakravarty, and Tinne Tuytelaars. Expert gate: Lifelong learning with a network of experts. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3366-3375, 2017.
[3] Alfonso Amayuelas, Shuai Zhang, Xi Susie Rao, and Ce Zhang. Neural methods for logical reasoning over knowledge graphs. International Conference on Learning Representations, 2022.
[4] Gregor Bachmann, Gary Bécigneul, and Octavian Ganea. Constant curvature graph convolu- tional networks. In International Conference on Machine Learning, pages 486-496. PMLR. 2020.
[5] Ivana Balazevic, Carl Allen, and Timothy Hospedales. Multi-relational poincaré graph embed- dings. In Advances in Neural Information Processing Systems, pages 4463-4473, 2019.
[6] Ivana Balazevic, Carl Allen, and Timothy Hospedales. Tucker: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNV) IJCNLP,, pages 3188-5197, 2019.
[7] Silvere Bonnabel. Stochastic gradient descent on riemannian manifolds. IFFE Transactions on Automatic Control, 58(9):2217-2229) 2013.
10

---

[8] Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26:2787-2795. 2013.
[9] Joey Bose, Ariella Smofsky, Renjie Liao, Prakash Panangaden, and Will Hamilton. Latent variable modelling with hyperbolic normalizing flows. International Conference on Machine Learning, pages 1045-1055. PMLR, 2020.
[10] Ines Chami, Albert Gu, Vaggos Chatziafratis, and Christopher Ré. From trees to continuous embeddings and back: Hyperbolic hierarchical clustering. Advances in Neural Information Processing Systems, 2020.
[11] Ines Chami, Adva Wolf, Da- Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Low- dimensional hyperbolic knowledge graph embeddings. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6901-6914, Online, July 2020. Association for Computational Linguistics.
[12] Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. In Advances in neural information processing systems, pages 4868 -8689,879, 2019.
[13] Chen Chen, Yufei Wang, Aixin Sun, Bing Li, and Kwok- Yan Lam. Dipping PLMs sauce: Bridging structure and text for effective knowledge graph completion via conditional soft prompting. In Findings of the Association for Computational Linguistics: ACI. 2023, pages 11489-11503, Toronto, Canada, July 2023. Association for Computational Linguistics.
[14] Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, and Yangfeng Ji. HittER: Hierarchical transformers for knowledge graph embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10395-10407, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[15] Calin Cruceru, Gary Becigneul, and Octavian-Eugen Ganea. Computationally tractable rieman- nian manifolds for graph embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 7133-7141, 2021.
[16] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[17] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:18/0.04805. 2018.
[18] David Eigen, Marc' Aurelio Ranzato, and llya Sutskever. Learning factored representations in deep mixture of experts. arXiv preprint 131iv:/312.433: 2013.
[19] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961, 2021.
[20] Shanshan Feng, Lucas Vinh Tran, Gao Cong, Lisi Chen, Jing Li, and Fan Li. Hme: A
hyperbolic metric embedding approach for next-poi recommendation. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1429-1438, 2020.
[21] Octavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. arXiv preprint 1805vv1805.091 12, 2018
[22] Albert Gu, Frederic Sala, Beliz. Gunel, and Christopher Ré. Learning mixed-curvature rep- resentations in product spaces. In International Conference on Learning Representations, 2019.
[23] Xin He, Kaiyong Zhao, and Xiaowen Chu. Automl: survey of the state-of-the-art, Knowledge- Based Systems, 212:106622. 2021.

---

[24] Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. Adaptive mixtures of local experts, Neural computation. 3(1):79-87) 87, 1991.
[25] Michael Jordan and Robert A Jacobs. Hierarchical mixtures of experts and the em algorithm. Neural compulation, 6(2): 1994.
[26] Max Kochurov, Rasul Karimov, and Serge Kozlukov. Geoopt: Riemannian optimization in pytorch. arXiv preprint arXiv:2005. 02819, 2020.
[27] Timothee Lacroix, Nicolas Usunier, and Guillaume Obozinski. Canonical tensor decomposition for knowledge base completion. In International Conference on Machine Learning, pages 2863-2872, 2018.
[28] Qi Liu, Maximilian Nickel, and Douwe Kiela. Hyperbolic graph neural networks. In Advances Neural Information Processing Systems, pages 1230-8241, 2019.
[29] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. Modeling task relationships in multi-task learning with multi-gate cexxuure-of-xperte In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1930-1939, 2018.
[30] Yu Meng, Jiaxin Huang, Guangyuan Wang, Chao Zhang, Honglei Zhuang. Lance Kaplan, and Jiawei Han. Spherical text embedding. In Advances in Neural Information Processing Systems, pages 2208-8217, 2019.
[31] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint
2013.
[32] Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical represen- tations. In Advances in neural information processing systems, pages 6338-6344 6347, 2017.
[33] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thiems Bpr: Bayesian personalized ranking from implicit feedback. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, pages 452-461, 2009.
[34] Carlos Riquelme, Joan Puigeerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems, 34:8583 8595, 2021.
[35] Ramit Sawhney, Shivam Agarwal, Megh Thakkar, Arna Wadhwa, and Rajiv Rain Shah. Hyperbolic online time stream modeling. In Proceedings of the 44th International ACM SIGIR Conference Research and Development in Information Retrieval, pages 1682-1686, 2021.
[36] Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-exxerrs layer. arXiv preprint arXiv: h00l::10000000 /701106538.... 2017.
[37] Ondrej Skopek, Octavian- Eugen Ganea, and Gary Bécigneul. Mixed-curvature variational autoencoders. In International Conference on Learning Representations, 2020.
[38] Zhiqing Sun, Zhi-Hong Deng, Jian- Yun Nie, and Jian Tang. Rotate: Knowledge graph em- bedding by relational rotation in complex space. In International Conference on Learning Representations, 2019.
[39] Yi Tay, Luu Anh Tuan, and Siu Cheung Hui. Hyperbolic representation learning for fast and efficient neural question answering. In Proceedings of the Eleventh ACM International Conference Web Search and Data Mining, pages 583-591, 2018.
[40] Alexandru Tifrea, Gary Bécigneul, and Octavian- Eugen Ganea. Poincaré glove: Hyperbolic word embeddings. In Proceedings of the International Conference on Learning Representations (ICLR 2019). OpenReview, 2018
[41] Abraham A Ungar. Thomas precession and its associated grouplike structure. American Journal of Physics, 59(9))824-834 1991.
12

---

[42] Lucas Vinh Tran, Yi Tay, Shuai Zhang, Gao Cong, and Xiaoli Li. Hyperml: A boosting metric learning approach in hyperbolic space for recommender systems. In Proceedings of the 13th International Conference on Web Search and Data Mining, pages 609-617, 2020.
[43] Shen Wang, Xiaokai Wei, Cícero Nogueira dos Santos, Zhiguo Wang, Ramesh Nallapati, Andrew Aroll, Bing Xiang, Yu Philip, and Isabel Cruz. Mixed-curvature multi-relational graph neural network for knowledge graph completion. In The Web Conference, 2021.
[44] Richard C Wilson, Edwin R Hancock, Elžbieta Pekalska, and Robert PW Duin. Spherical and hyperbolic embeddings of data. IEEE transactions on pattern analysis and machine intelligence, 36(1 1::2:)):2222055 2014.
[45] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. ICLR, 2015.
[46] Jinfa Yang, Yongjie Shi, Xin Tong, Robin Wang, Taiyan Chen, and Xianghua Ying. Improving knowledge graph embedding using affine transformations of entities corresponding to each relation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 508-517, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
[47] Bangpeng Yao, Dirk Walther, Diane Beck, and Li Fei-Fei. Hierarchical mixture of classification experts uncovers interactions between brain regions. Advances in Neural Information Processing Systems, 22:2178 21861 2009.
[48] Shuai Zhang, Yi Tay, Lina Yao, and Qi Liu. Quaternion knowledge graph embeddings. In Advances in Neural Information Processing Systems, pages 2735-2745, 2019.
13


Reviews:
[{"review_id": "OjH0cQud1e", "reviewer": "Reviewer_fz8R", "summary": "In many applications (especially those involving discrete data structures), choosing the right geometry for the embedding space, matching the structure of the data, can lead to significant performance gains. Extant approaches often make an ad hoc choice or use heuristics for the type of geometry applicable globally for the entire data. The paper proposes a novel strategy for the local selection of the product space with appropriate geometry for each data point, using a sparse gating mechanism. The approach is validated on the matrix completion for the movie ratings prediction problem, and link prediction on relational graphs (WordNet, Freebase KG) demonstrating significant performance gains.", "questions": "Kindly address the weaknesses pointed in the critique above. I summarize them below:\n- appropriateness of algebra involving embeddings in different spaces.\n- quantitative comparison with SOTA on the movie-ratings problem\n- local selection of K.\n- Clarify with references (a) Rheinmain geometric space, (b) how the regularizers achieve load balancing.\n- Discuss limitations of the work (see below)", "limitations": "Authors don’t address limitations of the work. \n\nI don’t think there are any direct negative societal implications. Other limitations and opportunities for improvement are addressed in my responses to previous questions.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "Strengths of the paper are listed below: \n\n**Relevance** The paper addresses a problem – automatic, and local selection of a product of subspaces with appropriate geometry – of relevance to a wide audience including those working on knowledge graphs, and recommendation systems. \n\n**Originality** While the components of the proposed approach, embedding in different geometric spaces, and sparsely gated MOEs, are not novel, I have not seen the utilization of the latter to make a local (per data-point) selection of the optimal product space. While I’m not deeply familiar with the literature in this area, I believe the proposed approach is novel. I rate the core technical novelty incremental.   \n\n**Technical Quality** The technical approach mostly appears sound apart from some doubts that I point to in the weaknesses section. \n\n**Evaluation** The evaluation using the tasks of (a) matrix completion in the movie-ratings setting, and (b) link prediction in relational graphs (WordNet, and Freebase KG), and the demonstrated significant gains in performance (especially on link prediction in the Freebase KG over the SOTA) validate the utility of the proposed approach. This strength is weakened by points listed in the weaknesses section.\n\n**Significance** The approach has the potential for significant impact over wide areas of research once the weaknesses have been addressed.", "weaknesses": "Weaknesses of the paper are listed below: \n\n**Technical Quality** I'm not convinced regarding the appropriateness of using the CNN layers $f_1(.)$ and $f_2(.)$. Since the vectors $e_p^{(i)}$ lie in different spaces, even if lifted to the corresponding tangent spaces, $f_j(.)$ layer would perform algebra on them and take weighted combinations of entities (embeddings) in entirely different spaces. To me this doesn’t seem proper. \n\n**Evaluation** \n- Matrix completion for movie-ratings can be considered a classical problem with a lot of algorithms benchmarked on MovieLens 1M. However, the paper doesn’t directly provide comparisons with the state of the art (bar chart in Figure 3). Secondly, it doesn’t provide a table with quantitative results, enabling a better understanding of the exact performance gains achieved. \n- Figure 6 shows that while the performance may be more robust to the choice of N, K has a significant impact. In the spirit of local selection of the relevant product spaces, it stands to reason that optimal K may also vary from point to point. Was this aspect investigated?  \n\n**Clarity** \n- (l. 111) What is a ‘Rheinmain Geometric Space’? If a typo, kindly fix, else provide a definition with a reference to the literature.\n- (l. 201-208; (12)-(15)) Kindly add explanations for how the two regularizers, $l_1$ and $l_2$,  achieve load balancing. Point to the relevant literature where they were introduced or mention that they are novel introductions.  \n- (Fig. 6) The value of K for the left panel, and N for the right panel are not identified.\n- some typos need to be fixed (not considered in evaluating the paper).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for confirming the novelty as well as the potential of the proposed approach. Also, the suggestions can certainly help us improve the manuscript. \n\nBelow we answer the questions and make some clarifications.\n\n##### **Why do we use CNN layers (appropriateness of algebra involving embeddings in different spaces)?**\nAll the vectors are initialized in the Euclidean space then mapped to corresponding geometric spaces, the CNN layers are applied over the initialized euclidean vectors regardless of which geometric space these vectors will be mapped to for simplicity. We agree that it might be possible to use hyperbolic/spherical neural networks in the process. However, in this case, it requires each geometric space to have its own selection network, and we still need to combine the outputs of all the selection networks to arrive at a single decision. We will leave it as a future work to explore more advanced selection mechanisms.\n\n##### **Comparison with state of the art on Movielens**\nWe provide comparisons with other stoa methods on ML-1M below. We can see that, compared with the most recent models, the performance gain is still significant.\n\nOur method:  Recall@10=0.116; \n\nMultiVAE (WWW 2018 [a]), an variational autoencoder based model; Recall@10=0.101;\n\nLightGCN (SIGIR 2020 [b]), a graph convolutional network based model: Recall@10=0.0987;\n\nDiffRec (SIGIR 2023, first available on April 2023 [c[), a diffusion recommender model; Recall@10=0.106;\n\n##### **local selection of K**\nIn our current design, K is selected for the whole dataset instead of each data point. It is reasonable that each data point may require a different K, but it is quite challenging to achieve this goal. One potential solution is to build an MOE network with different K for each data point, but this can be unrealistic as the model size increases linearly to the data size.\n\nMoreover, K is usually set to a small value in our experiments (select from 2, 3, 4), as such, manual hyper-parameter search is good in this case.  \n\n##### **Clarification**\n- (1) Rheinmain geometric space ⇒ it is a typo, and it should be Riemannian geometric space.\n- (2) The loading balance approach is not a novel contribution, but from reference ***. We will add these references [d][e] to the paper.\n\nThe two regularization losses can be viewed as a soft constraints on the space selector.\n- In the first regularization loss (eq-12), the batch-wise sum of the gate values are considered as the importance of an expert. Minimizing the coefficient of variation the importance distribution encourages all experts to have equal importance.\n- The second loss (eq-13) is used to encourage all experts to receive roughly equal numbers of training examples. Although one can ensure equal expert importance with the first loss, the number of training examples received by each expert can be different (e.g., one expert receives a few examples with large weights, and the other receives many examples with small weights). Here, $\\kappa$ can be viewed as a soft estimator of the number of examples assigned to each expert for a batch of inputs. Minimizing the coefficient of variation this distribution will help encourage an equal number of training examples for each expert.\n\n\n##### **Discuss limitations of the work**\nWe have a short subsection in 3.4 to discuss the limitations of the proposed method. Based on the review, we also identified one more limitations: the proposed approach can not assign different K for each data point.\n\n##### **Reference**\n- [a] Liang, Dawen, et al. \"Variational autoencoders for collaborative filtering.\" Proceedings of the 2018 world wide web conference. 2018.\n- [b] He, Xiangnan, et al. \"Lightgcn: Simplifying and powering graph convolution network for recommendation.\" Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval. 2020.\n- [c] Wang, Wenjie, et al. \"Diffusion Recommender Model.\" SIGIR (2023).\n- [d] Shazeer, Noam, et al. \"Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.\" arXiv preprint arXiv:1701.06538 (2017).\n- [e] Bengio, Emmanuel, et al. \"Conditional computation in neural networks for faster models.\" arXiv preprint arXiv:1511.06297 (2015).\n\n\n\n"}, {"review_id": "8RI5Yau08A", "reviewer": "Reviewer_Wewy", "summary": "Data representation is an important problem in today's deep learning world. Representation beyond Euclidean geometry, such as spherical or hyperbolic spaces can provide additional flexibility and benefits in capturing underlying properties of data. For example, hyperbolic space can better capture data that has inherent hierarchical structure, and spherical space can better model cyclical structure. In this paper, the authors provide a method to automatically map the individual data points to different geometric spaces using a mixture of experts network. By mapping the data points to different geometric spaces automatically, the authors show advantages in many real world tasks such as matrix completion and link prediction for relational graphs. ", "questions": "Overall, this is a well written paper addressing an important problem. Please see concerns above. ", "limitations": "No concerns. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The paper addresses an important problem of automatically mapping the data points to different geometric spaces and motivates the problem well in the paper. \n\n2) This is a challenging problem and many prior methods typically focus on mapping all the data points to a single geometric space.  The formalism is also elegant and the proposed solution shows improvement on matrix completion and graph link prediction over other baselines. ", "weaknesses": "1) While the matrix completion and link prediction seems like good applications, it is not clear whether the proposed techniques can be extended to some of the other mainstream learning applications in vision and language domains. Furthermore, the early layers in many deep neural networks can already exploit the underlying intrinsic data properties to extract features to show benefits. It is not completely clear whether explicit assignment to individual spaces would provide any additional benefits in some of these newer applications.  \n\n2) There has been many prior methods addressing this problem, and the novelty is not explicitly discussed. It would be good if the authors could better clarify this w.r.t MOE and other methods that exploit hybrid geometrical spaces. \n\n3) In many problem settings the underlying data distribution may be predominantly hierarchical or cyclical, and not sure whether it provides strong advantages with the hybrid approaches, and furthermore, the number of geometrical spaces and identification of these geometrical spaces still seem manual and the modern deep learning machinery may be implicitly learning and mapping them as they find them useful in reducing the training loss. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the positive feedback and reconfirming the importance of the explored problem. \n\n##### **Q1**: While the matrix completion and link prediction seems like good applications, it is not clear whether the proposed techniques can be extended to some of the other mainstream learning applications in vision and language domains. Furthermore, the early layers in many deep neural networks can already exploit the underlying intrinsic data properties to extract features to show benefits. It is not completely clear whether explicit assignment to individual spaces would provide any additional benefits in some of these newer applications.\n\n**Answer**: Due to a limited rebuttal time frame, we are not able to conduct experiments on the mentioned applications. But we can provide some thoughts on how to apply this idea on these applications. For example, in current mainstream NLP tasks  (e.g., classification, recognizing textual entailment), hyperbolic neural networks [a] (it’s been shown that hyperbolic neural networks can outperform euclidean neural networks[a]), spherical neural works, and normal euclidean neural networks can be built for specific NLP tasks. Then, we can use the sentence representations obtained from pretrained language models (e.g., bert) and the selection network to choose which type of geometric networks to be employed. In doing so, we can also realize the goal of data-informed geometric space selection. \n\n\n##### **Q2**: There has been many prior methods addressing this problem, and the novelty is not explicitly discussed. It would be good if the authors could better clarify this w.r.t MOE and other methods that exploit hybrid geometrical spaces.\n\nThank you very much for the constructive suggestion. We will explicitly clarify the relationship of the proposed approach with MOE and  existing hybrid geometrical spaces.\n\nThe proposed approach is orthogonal to the existing MOE works. Existing works usually apply the MOE paradigm to euclidean neural networks for better performance or model size scaling up. However, to the best of knowledge, this work is the first attempt to seamlessly integrate the sparsely gated MOE learning paradigm into geometric representation learning. The selected spaces are tightly coupled to form a product space that has strict mathematical definition and meanings.  Compared with existing hybrid geometric spaces (e.g., product space representation learning), in our approach, each data point can inform the model on which geometric space to use, while existing methods treat every data point equally without any customization. In doing so, the proposed method can fully elicit the expressive power of geometric representation learning.\n\n\n##### **Q3**: In many problem settings the underlying data distribution may be predominantly hierarchical or cyclical, and not sure whether it provides strong advantages with the hybrid approaches, and furthermore, the number of geometrical spaces and identification of these geometrical spaces still seem manual and the modern deep learning machinery may be implicitly learning and mapping them as they find them useful in reducing the training loss.\n\n**Answer**: It is true that in some problem settings, the underlying data distribution can be predominantly hierarchical or cyclical. However, we would like to clarify that we did not make any assumption on the underlying data structure. And in most real-world applications, the underlying data structure is difficult to know. Moreover, our approach subsumes non-hybrid approaches. If we know that the data is predominantly hierarchical, we can set all the candidate space to be hyperbolic. \n\nPertaining to modern deep learning methods, although it is not in the scope of this paper, we can see from existing literature that non-Euclidean neural networks usually performs better than euclidean neural networks on many tasks [a], indicating that it is not trivial for modern deep learning methods to implicitly inferring the methods. Our method introduces an inductive bias that ensures an effective capture of the data-geometric space relationships, while existing deep learning methods can hardly achieve this.\n\n\n\n##### Reference\n- [a] Ganea, Octavian, Gary Bécigneul, and Thomas Hofmann. \"Hyperbolic neural networks.\" Advances in neural information processing systems 31 (2018).\n"}, {"review_id": "OfA6ghPTre", "reviewer": "Reviewer_TAMD", "summary": "The goal of this paper is to learn the geometry (manifold) underlying given data points. Rather than learning an arbitrary Riemannian manifold from the data, the paper models this manifold as a Cartesian product of manifolds with constant curvature (three prototyprs are use: Euclidean, spherical, hyperbolic). It imposes a certain vector space structure (‘gyrovector’) on these prototypes, where the operations are functions of the curvature. This imposition is a key step as it allows the definition of log and exp maps from the manifolds to tangent spaces, and reach vector representations of points on manifolds.\n\nWith this geometry, the paper derives a framework to fit a manifold to the training data. Each data point has components in some number of these manifolds. The paper first estimates the probability of each data point having a component in a specific component manifold. They then add a task specific objective function seeks to optimization data representation in terms of maximizing task performance. With these cost functions, the manifold optimization becomes an optimization problem. \n\nThe conceptual applications of this framework include matrix completion, and link prediction for relational graphs. There are a number of experimental results demonstrating the ideas and their performance over the past ideas.\n", "questions": "\nMinor point: \n1.\tI was just checking the Mobius sum for elements of a unit sphere in R^n – the result was (Y - <X,Y> X)/(1 - <X,Y>), which is not on the unit sphere. Does it mean that the sum leaves the set {\\cal M}_c? Please clarify. This is probably my calculation mistake. \n\n2. Also this expression does not seem to be symmetric in X and Y. Is that correct ? ", "limitations": "\nThe paper discusses one limitation -- addition of a hyperparameter (K) - -with respect to the previous work on Euclidean product space. However they don't discuss how general their formulation is with respect to an arbitrary data manifold. That is, a manifold with varying curvature. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "\n-- Improvement over the previous data-driven manifold learning by including curved manifolds as basic components. \n\n-- Interesting use of the gyrovector machinery to derive Euclidean representations for feeding into optimization tools. \n\n-- The experiments provide evidence of success in learning some geometry from the data. \n\n", "weaknesses": "\n-- Can one represent arbitrary manifolds using a direct product of these prototypes? Perhaps not. Then what is lost in posing the problem in this way as opposed to a completely nonparametric approach of manifold learning? \n\n-- The paper can provide some intuition on the overall objective function and some more details on how the optimization is performed. \n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the positive rating and constructive suggestions!\n\n##### **Can one represent arbitrary manifolds using a direct product of these prototypes?** \nWe cannot represent arbitrary manifolds in this way. We mainly focus on three popular manifolds (spherical, hyperbolic, and euclidean) which have well-defined projections forms and metrics. Other manifolds such as symplectic manifold are out of the scope in this paper.  \n\nIt is nontrivial to represent manifold with varying curvature in a uniform form. However, the curvature can be learnable in our model. That is, it can be treated as a hyper-parameter or being a learnable parameter that can be optimized together with the model.\n\n##### **Then what is lost in posing the problem in this way as opposed to a completely nonparametric approach of manifold learning?** \nThis study centers around parametric learning algorithms, and the highlighted applications also exhibit a preference for parametric approaches over non-parametric ones. To illustrate, numerous works [a, b] in the literature have shown that matrix factorization yields better results compared to non-parametric techniques like KNN-based models. In the future, we plan to carry forward this concept into more sophisticated deep neural networks, such as hyperbolic neural networks or spherical neural networks.\n\n#### **The paper can provide some intuition on the overall objective function and some more details on how the optimization is performed.**\nThank you very much for the suggestion, we will add more intuition/explanations on the objective functions to the paper.\n\nFirst, the two regularization losses can be viewed as a soft constraints on the space selector. \n\n- In the first regularization loss (eq-12), the batch-wise sum of the gate values are considered as the importance of an expert. Minimizing the coefficient of variation the importance distribution encourages all experts to have equal importance. \n\n- The second loss (eq-13) is used to encourage all experts to receive roughly equal numbers of training examples. Although one can ensure equal expert importance with the first loss, the number of training examples received by each expert can be different (e.g., one expert receives a few examples with large weights, and the other receives many examples with small weights). Here, $\\kappa$ can be viewed as a soft estimator of the number of examples assigned to each expert for a batch of inputs. Minimizing the coefficient of variation this distribution will help encourage an equal number of training examples for each expert.\n\nSecond, the two task specific losses are contrastive losses which encourage positive pairs to be closer and make the distance between negative entities larger.\n\nFor the optimization, since all the tensors are projected via the stereographic projection, we use Adam as the optimizer.\n\n##### **I was just checking the Mobius sum for elements of a unit sphere in R^n – the result was (Y - <X,Y> X)/(1 - <X,Y>), which is not on the unit sphere. Does it mean that the sum leaves the set {\\cal M}_c? Please clarify. This is probably my calculation mistake.**\n\nThe calculation is correct. However, the mobius sum is defined for the stereographic projection model instead of the original manifold. We provided some examples in Figure 1 and Figure 2 on what the projected model looks like for both hyperbolic and spherical models. We also provided two examples in Figure 3 on the mobius sum calculation results when c=1 and c=-1. See [pdf](https://openreview.net/attachment?id=VzCnW8Uuls&name=pdf).\n\n##### **Also this expression does not seem to be symmetric in X and Y. Is that correct ?**\nYes, the expression is not symmetric. But in some special cases, they are symmetric: (1) zero vector case; e.g., of the vector is zero. (2) zero curvature case that is same as Euclidean addition.\n\n\n\n##### Reference\n\n- [a] Hu, Yifan, Yehuda Koren, and Chris Volinsky. \"Collaborative filtering for implicit feedback datasets.\" 2008 Eighth IEEE international conference on data mining. Ieee, 2008.\n- [b] Koren, Yehuda. \"Factorization meets the neighborhood: a multifaceted collaborative filtering model.\" Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining. 2008."}, {"review_id": "2UpghAc0Pl", "reviewer": "Reviewer_c7Xk", "summary": "The paper proposes a new end-to-end distance learning method to facilitate solving downstream prediction tasks. Their core idea is to select a subset of geometric spaces from a candidate space sets that include Euclidean, projected sphere and Poincare ball types of spaces, and computes the final distance in the Cartesian product space of the selected spaces.  The selector is built by applying the sparsely-gated MOE technique. To improve the stability of the training, some balancing regularization terms are added to the training loss. Performance of the proposed method is assessed by one matrix completion task and one link prediction task using benchmark datasets. Reported results show performance improvement.", "questions": "(1)\tExplain how the two candidate hyperbolic and two spherical spaces were constructed?\n\n(2)\tIn line 236, the authors said that the training examples are formed based on watched (v1) and unobserved (v2) movies. This is odd. In MovieLens data, the matrix elements are ratings like 1,2,3,4,5.  Isn’t the matrix completion supposed to predict the ratings of unobserved movies for a user?  Why is it meaningful to push s(u,v1) -s(u,v2) +m < = 0?\n\n(3)\tCan the authors perform a brief investigate on what experiment settings some state-of-the-art works on matrix completion and link prediction have used for the same three benchmark datasets, and what performance they have obtained, and compare the published results with the authors' results?\n\n(4)\tPerform some ablation studies to examine the role of l_1 and l_2.", "limitations": "Discussion on geometric representation learning is quite basic. It would be interesting to discuss why it is important and useful to research geometric representation learning, and whether the proposed approach has other potentials, in addition to boosting prediction performance.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed idea has some originality (a kind of creative combination of existing ideas) and the proposed algorithm design is reasonable.", "weaknesses": "A major weakness is the writing and clarity. First of all, Sections 3.1 and 3.2 are not proposed method, but a short repeat of Section 2 of [34], but in poorer quality. The subsection of “Geometric Space Selector via Sparsely-gated MOE” from line 172 to line 188 is almost a repeat of Section 2.1 of [33] published in ICLR 2017. The design motivations of the two balancing regularisation terms are not clearly explained, especially for l_2 that is not as straightforward as l_1. It is not clear that how to construct different candidate geometric spaces. The authors only mention that there are three types of spaces. How exactly to compute distance using tangent vectors in a geometric space is not explained. More such information in the Embedding section in line 166 would be helpful. It is a pity that in the proposed method section the authors use quite some space for a poor summary of existing knowledge rather than putting good effort to explain well their own proposed method.\n\nIt seems that the goal of the research is to boost prediction performance by improving geometric representation learning. Under such goal, experiment section has weakness. It only compares between different spaces, but lacks comparison with published results by mainstream existing works and state of the art results on the same benchmark datasets for both matrix completion and link prediction. Without such comparison, it is hard to recognise the value/importance of  geometric representation learning for solving downstream prediction tasks. Ablation studies on the effect of the regularization terms are missing.\n\nThe contribution is a little limited given that the only objective seems to boost performance. Under such goal, the direction of geometric representation learning may not even be the best direction to pursue for each prediction task. It would be more interesting if the work could explore other potential, in addition to prediction performance improvement, of geometric representation learning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the review, especially the constructive suggestion on the writing/clarification. We will revise the manuscript as suggested and clarify some of the confusing points. \n\n##### **For Section 3.1 and section 3.2**\nWe will move Section 3.1 and Section 3.2  to a separate preliminary section to ensure the paper to be self-contained.\n\n##### **The design motivations of the two balancing regularisation terms are not clearly explained**\n- In the first regularization loss (eq-12), the batch-wise sum of the gate values are considered as the importance of an expert. Minimizing the coefficient of variation the importance distribution encourages all experts to have equal importance.\n- The second loss (eq-13) is used to encourage all experts to receive roughly equal numbers of training examples. Although one can ensure equal expert importance with the first loss, the number of training examples received by each expert can be different (e.g., one expert receives a few examples with large weights, and the other receives many examples with small weights). Here, $\\kappa$ can be viewed as a soft estimator of the number of examples assigned to each expert for a batch of inputs. Minimizing the coefficient of variation this distribution will help encourage an equal number of training examples for each expert.\n\n##### **Explain how the two candidate hyperbolic and two spherical spaces were constructed?**\nWe will provide a formal description in the paper (e.g., after line 166).\n\nTaking the recommendation task as an example, suppose that we have two candidate hyperbolic spaces ($P_1$, $P_2$)  and two spherical spaces ($D_1$, $D_2$) in the candidate space pool. \nIn the first step, we will initialize four embedding vectors for each item (say $ i_1, i_2, i_3, i_4$) and four embedding vectors for each user ($u_1, u_2, u_3, u_4$), and all vectors are in the tangent space. \nSecondly, we assume that $i_1$ and $u_1$ are in the hyperbolic space $P_1$; $i_2$ and $u_2$ are in the hyperbolic space $P_2$; $i_3$ and $u_3$ are in the spherical space $D_1$; $i_4$ and $u_4$ are in the same spherical space $D_2$. The distance of $i_*$ and $u_*$ is computed via eq(16). \nThirdly, if we set $K=2$, and the model selected $P_1$ and $D_2$ for the final prediction. We will combine the distance in $P_1$ (computed by $i_1$ and $u_1$) and distance in $D_2$ (computed by $i_4$ and $u_4$) via equation (11). \n\n\n##### **How exactly to compute distance using tangent vectors*\nBecause we have two separate applications, we the distance for each application is defined in different sections. One can get the final distance calculation from eq 2, 11, 16 (for matrix factorization) and 19 (relational link prediction)\n\n\n##### **Loss for Movielens-1m**\nFor MovieLens 1M, as we mentioned in line 240, we have binarized the ratings into 0/1, where ratings greater than or equal to 4 are treated as 1, otherwise, it is treated as unknown (0). This is a common setting in matrix factorization/collaborative filtering. As in most real-world applications, feedback is in the form of likes/dislikes, viewed/not viewed, etc. As such, we adopted this contrastive loss function.\nTo avoid confusion, we will rename it to **matrix factorization** instead of matrix completion.\n\n##### **Compare with STOA**\n\nWe provide comparisons with other stoa methods on ML-1M below. The performance boost is still significant.\n\nOur method:  Recall@10=0.116; \n\nMultiVAE [a]: Recall@10=0.101;\n\nLightGCN [b]: Recall@10=0.0987;\n\nDiffRec  [c]: Recall@10=0.106;\n\nOn the relational link prediction task, we have already included 13 baselines. We add one more model here.\nHittER (EMNLP 2021, [d]): FB15K-237 (MRR=.373; HR@1=.279; HR@3= .409; HR@10=.558); WN18RR (MRR=.503; HR@1=.462; HR@3=.516; HR@10=.584);  We can see that our model still outperforms them. \n\n##### **Apart from performance boosts**\nIt is worth noting that performance boost is not trivial given the recent advance in these fields. Here, we'd like highlight the two advantages of the proposed method:\n- **Enhanced expressiveness**: the proposed approach can enhance the representation power of current geometric representation learning, and performance boost is the best indicator.\n- **Flexibility**: the model flexibility is shown in two aspects: (a) the proposed approach is applicable to different domains and applications; (b) each data point has the flexibility to choose from a number of candidate product spaces to represent itself (e.g,. Figure 4 and 5). \n\n##### **Perform some ablation studies to examine the role of l_1 and l_2**\nWe conduct ablation studies on the WN18RR dataset.\n- w/o $\\ell_1$ and $\\ell_2$, we have MRR: 0.515 | H@1: 0.463| H@3: 0.536| H@10: 0.595\n- w/o $\\ell_2$ but with $\\ell_1$ : MRR: 0.519 | H@1: 0.471 | H@3: 0.540 | H@10: 0.602\n- w/o $\\ell_1$ but with $\\ell_2$: MRR: 0.520 | H@1: 0.473 | H@3: 0.543| H@10: 0.605\n\n##### **why geometric representation learning and the potential of the proposed approach**\nGeometric representation learning is fundamental problem and useful in main domains (as reaffirmed by other reviewers). It can introduce a strong inductive bias to the model, making the learning process easier and boosting the expressiveness of the representation.  In the future, \nwe plan to carry forward this concept into more sophisticated deep neural networks, such as hyperbolic neural networks or spherical neural networks. In doing so, more complex tasks can benefit from the proposed data-informed geometric space selection idea.\n\n##### Reference\n- [a] Liang, Dawen, et al. \"Variational autoencoders for collaborative filtering.\"  \n- [b] He, Xiangnan, et al. \"Lightgcn: Simplifying and powering graph convolution network for recommendation.\" \n- [c] Wang, Wenjie, et al. \"Diffusion Recommender Model.\" SIGIR (2023).\n- [d] Chen, Sanxing, et al. \"HittER: Hierarchical Transformers for Knowledge Graph Embeddings.\" Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2021.\n"}]

Author rebuttal:
We thank all the reviewers for their suggestions and we answer corresponding questions under each review's rebuttal. The pdf contains some figures to show the stereographic projection models for hyperbolic & spherical spaces. Mobius sum is also demonstrated in this pdf.

Meta-review: The authors propose a method to embed data in constant curvature manifolds (i.e., Euclidean, hyperbolic, and spherical spaces) and their product manifolds. In doing so, they propose a mechanism to adaptively and sparsely select product manifolds, using a gating mechanism. The efficacy of the proposed method is verified on several problems, including matrix completion.  This paper was well received by reviewers, acknowledging the clarity of the manuscript and the value of the core idea. During the authors and reviewers' discussion period, the authors provide detailed responses to all raised concerns. As such, the AC agrees with the evaluation by the reviewers and recommends the acceptance of the work; congratulations. Please revise your work based on comments and incorporate the new results provided during the rebuttal phase.
Decision: Accept (poster)