Representing Hyperbolic Space Accurately using Multi-Component Floats
Year: 2022

Abstract:
Hyperbolic space is particularly useful for embedding data with hierarchical structure; however, representing hyperbolic space with ordinary floating-point numbers greatly affects the performance due to its ineluctable numerical errors. Simply increasing the precision of floats fails to solve the problem and incurs a high computation cost for simulating greater-than-double-precision floats on hardware such as GPUs, which does not support them. In this paper, we propose a simple, feasibleon-GPUs, and easy-to-understand solution for numerically accurate learning on hyperbolic space. We do this with a new approach to represent hyperbolic space using multi-component floating-point (MCF) in the Poincaré upper-half space model. Theoretically and experimentally we show our model has small numerical error, and on embedding tasks across various datasets, models represented by multi-component floating-points gain more capacity and run significantly faster on GPUs than prior work.

heading: 1 Introduction
text: Representation learning is of particular interest nowadays in machine learning. Real-world data is often discrete—such as natural language sentences in the form of syntax trees and graphs [7, 22]— which makes it critical to do a good job of representing and storing them as numbers which we can better understand and deal with. Perhaps the most common way of doing this representation is with embedding, in which discrete objects are assigned points in some continuous (typically high-dimensional Euclidean) space. Standard embeddings in NLP include Word2Vec [24] and GloVe [30]. However, many kinds of real-world data are hierarchical or graph-like [25], in which case embedding into non-Euclidean space can outperform the standard Euclidean embedding. Hyperbolic space is such an embedding target of particular interest, showing impressive performance on many tasks such as taxonomies, link prediction and inferring concept hierarchies [27, 28, 15].
Hyperbolic space [9] is helpful for many types of embeddings because of its constant negative curvature, as distinguished from Euclidean space with zero curvature. This negative-curvature geometry endows hyperbolic space with many useful properties not present in Euclidean space. For example, the volume of a ball in even two-dimensional hyperbolic space increases exponentially with respect to the radius, as shown in [9], as a contrast to the polynomial rate at which balls grow in Euclidean space. This makes it possible to embed more instances in the hyperbolic ball than Euclidean ball with the same radius, making it ideal for structures like trees where the number of objects at a distance d from any given object can grow exponentially with d. Reasoning from this intuition, the hyperbolic space “looks like” a finite-degree tree and can be regarded as a continuous version of trees [6]. The combination of low dimension (which facilitates learning) with high volume (which increases representational capacity) makes hyperbolic space inherently well-suited for embedding hierarchical structures data like trees and graphs.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
However, the advantages of hyperbolic space do not come for free. The large volumes that are so beneficial from a embedding-capacity perspective create unavoidable numerical problems. Informally called “the NaN problem" [37, 34], representing hyperbolic space with ordinary floating-point numbers can cause significant errors and lead to severe numerical instability that compounds throughout training, causing NaNs to pop out everywhere in the computations. Common models of hyperbolic space used to learn hyperbolic embeddings using floating-point arithmetic, such as the Poincaré ball model [27, 15] and the Lorentz hyperboloid model [28], all suffer from significant numerical errors. These errors originate in the ordinary floating-point representation, and are further amplified in subsequent computations by the very ill-conditioned Riemannian metrics involved in their construction. Yu and De Sa [37] show that in the Lorentz hyperboloid model, the error of representing an exact point with ordinary floating-point numbers becomes unbounded as the point moves away from the origin, which is what will happen in most embedding tasks: points unavoidably move far away from the origin as more and more data instances crowd in.
Many efforts and compromises have been made so as to solve or alleviate the ‘NaN’ problem. One commonly adopted technical solution firstly exploited in [35] is using an appropriate scaling factor, i.e., scale down all the distances by the scaling factor before embedding, then recover the original distances by dividing to the scaling factor. However, this scaling will increase the distortion of the embedding, and the distortion gets worse as the scale factor increases. Sala et al. [34] show that this sort of tradeoff is unavoidable when embedding in hyperbolic space: one can either choose to increase the number of bits used for the floating-point numbers or increase the dimension of the space, and this is independent of the underlying models. However, increasing the dimension is not a panacea, as the ‘NaN’ problem still exists in high dimensional space (albeit to a lesser degree).
A straightforward solution to the ‘NaN’ problem would be to compute with high precision floatingpoint numbers, such as BigFloats (specially designed with a large quantity of bits), as adopted in the combinatorial construction of hyperbolic embedings in [34], where floating-point numbers with thousands of bits are used with the support of Julia [4]. However, there are two evident problems with representing hyperbolic space using BigFloats. Firstly, BigFloats are not supported on ML accelerator hardware like GPUs: computations using BigFloats on CPUs will be slow and hence not realistic for many meaningful tasks. Secondly, BigFloats are currently not supported in most deep learning frameworks even on CPUs, so one would need to develop the algorithms from scratch so as to learn over hyperbolic space, which is tedious and fraught with the danger of numerical mistakes.
A hybrid solution to the ‘NaN’ problem proposed by Yu and De Sa [37] is to avoid floating-point arithmetic as much as possible by using integer arithmetic instead—as integer arithmetic can be done exactly. Their tiling-based model works by decomposing hyperbolic space into a tiling: the product of a compact fundamental region (which can be represented in floating-point with bounded error) and a discrete group (which supports integer computation). We can also interpret their approach as decomposing a large floating-point number into a multiply of a small floating-point number with a large integer number using some group actions [3, 33]. This allows for a guaranteed bound on the numerical error of learning in hyperbolic space. However, their approach is limited somewhat in that its guarantees apply to loss functions that depend only the hyperbolic distance between points, and their best-performing model (their L-tiling model) is limited to 2-dimensional space. A bigger issue with this type of approach is computational: the big-integer matrix multiplication involved in the model is not supported on GPUs and is not yet implemented in many deep learning frameworks such as PyTorch [29] and TensorFlow [1]. This effectively prevents the method from being used in training of complex models that depend on GPU acceleration, such as hyperbolic neural networks [14, 18, 10, 23].
In this paper, we propose a more direct way to represent and learn in hyperbolic space without sacrificing either accuracy guarantees or GPU acceleration. To achieve high precision computations and controllable numerical errors, we choose to represent hyperbolic space with multiple-component floating-point numbers (MCF) [31, 32, 19, 36], an alternative approach different from BigFloats where a number is expressed as an unevaluated sum of multiple ordinary floating-point numbers. We make the following contributions:
• We propose representing hyperbolic space with the Poincaré upper-half space model for optimization in section 3, and show how to represent it with MCF in section 4 to eliminate the ‘NaN’ problem.
• We provide algorithms for computing with these MCF that are adapted to hyperbolic space in section 5 and section 6, where we prove numerical errors can be reduced to any degree by simply increasing the number of components of the MCF. • We experimentally measure our model on embedding tasks across several datasets, and show that one can gain more learning capacity (compared to models that use ordinary floatingpoint numbers) with only a mild computational slowdown on GPUs in section 7, meanwhile significantly faster than prior high precision hyperbolic models.

heading: 2 Related Work
text: The non-Euclidean geometry of hyperbolic space offers many distinctive properties useful in machine learning. It is well-suited to model hierarchical structures such as trees and more complex graphs: for instance, Sala et al. [34] show that a tree can be embedded into a 2-dimensional hyperbolic space with arbitrarily low distortion, and Gromov [16] shows that any finite tree can be embedded into a finite hyperbolic space such that distances are preserved approximately.
Nickel and Kiela [27] propose to embed symbolic data into the Poincaré ball model of hyperbolic space for learning the hierarchical representations, where the Poincaré ball embeddings outperform Euclidean embeddings significantly on hierarchical data in terms of both representation capacity and generalization ability. Subsequently, Ganea et al. [15] further improve the empirical results of Poincaré ball embeddings on directed acyclic graphs by theoretically grounding entailment using the hierarchical relations. Later work uses other models of hyperbolic space [28, 17] besides the Poincaré ball model for various purposes. For example, the Lorentz hyperboloid model used in Nickel and Kiela [28] to learn continuous hierarchies from large-scale unstructured data, was observed to be substantially more efficient and numerically stable. Along a different direction, hyperbolic embeddings can be used in many down-streaming tasks, by simply treating the hyperbolic embedding as a layer to form neural networks, such as the hyperbolic neural networks [14] for textual entailment and noisy-prefix recognition tasks, hyperbolic attention networks [18] for neural machine translation and visual question answering tasks, and hyperbolic graph networks [10, 23] for link prediction and node classification tasks.
Although widely adopted in many applications, the ‘NaN’ problem in hyperbolic representations can be observed easily in practice and generally remains unsolved. Sala et al. [34] suggests that hyperbolic embeddings can have high quality on hierarchies but require large dimensions or high precision for graphs with long chains: they suggest alleviating the ‘NaN’ problem by using scaling factors together with BigFloats. Yu and De Sa [37] introduced tiling-based models, the first theoretical solution to the ‘NaN’ problem with bounded-everywhere numerical error. The tiling-based models enable deriving high-precision embeddings without using high precision floating-point numbers.
In the literature of arbitrary precision algorithms, there are two different ways to achieve high precision computations, and BigFloat is not the only option. The first one is straightforward: the multiple-digit format, which consists of a sequence of digits coupled with a single exponent. Examples in this category include the computation package Mathematica [20], the BigFloats format in Julia [4], and the Fortran multiple-precision arithmetic packages [8, 2]. The other option is the multiple-component format, in which an exact number is written as an unevaluated sum of ordinary floating-point numbers, each with its own significand and exponent [11, 31, 32, 13].

heading: 3 The ‘NaN’ Problem
text: Hyperbolic space can be modelled with several well-known isomorphic models. Previous works have proposed to optimize over the hyperbolic space with the Poincaré ball model [27] and Lorentz hyperboloid model [28]. In this paper, we propose to optimize within the Poincaré upper-half space model instead, because its properties allow for localization of potential sources of numerical error.
The Poincaré upper-half space model is the manifold (Un, gu), where Un = {x ∈ Rn : xn > 0} is the upper half space of n-dimensional Euclidean space. The metric and corresponding distance function are
gu(x) = ge x2n , du(x,y) = arcosh
( 1 + ‖x− y‖2
2xnyn
)
where ge is the Euclidean metric. Notably, the tangent space of the upper-half space model is the same as the ambient vector space, which makes it easier and numerically stable to derive Riemannian gradients without projection to the tangent space. In the two dimensional case, the Poincaré upper-half space model can be interpreted as a mapping on the complex plane (the “half-plane model”).
Numerical error analysis. Yu and De Sa [37] shows that in the Lorentz hyperboloid model of the hyperbolic space, the error of representing an exact point x with ordinary floating-point fl(x) is unbounded and proportional to the distance to the origin d(x,O). Here, as a baseline to compare our MCF methods to, we show that the same sort of representation error d(x,fl(x)) happens when representing points in the Poincaré upper-half space model with ordinary floating-point numbers.
Theorem 1 The representation error of storing a particular point x ∈ Un using floating-points fl is δfl(x) = du(x,fl(x)), and the worst case representation error defined as a function of the distance-to-origin d in the Poincaré upper-half space model is
δd := max x∈Un,du(x,O)≤d
δfl(x) = arcosh(1 + 2 machine cosh 2(d)).
where machine is the machine epsilon of the underlying floating-point arithmetic. This becomes δd = 2 machine + o( machine) if d < log(1/ machine) and δd = 2d + 2 log( machine) + o( −1machine exp(−2d)) if d ≥ log(1/ machine).
We plot the averaged representation error and the ‘NaN’ problem practically in Figure 1, the distance error d(x,fl(x)) when x is represented as fl(x) using floating-point. The error explodes at at distance of around 55 from the origin, which will then potentially cause NaNs in subsequent computations.
Identify potential numerical errors. Optimizing x in the Poincaré upper-half space model will typically push x away from the origin, with xi, i 6= n being large floating-point numbers and xn being floating-point numbers close to 0. Hence, the subtraction of two big floating-point numbers in ‖x−y‖2 of the distance calculation leads to catastrophic cancellation and accounts for a large part of the numerical errors. We can localize this part of the numerical errors easily in simple subtractions in the Poincaré upper-half space model. (For comparison, in the Lorentz model [28], the multiplication in Lorentzian scalar product accounts for the major numerical errors in distance computations.)
RSGD in the Poincaré upper-half space model. We are interested in gradient-based optimizations within the Poincaré upper-half space model, hence we provide Riemannian stochastic gradient descent (RSGD) algorithm for different learning algorithms in the hyperbolic space. For each z ∈ Un in the Poincaré upper-half space model, let f : Un → R be a smooth real-valued loss function over the manifold, denote ∇zf to be the Euclidean gradient of f w.r.t. z. In order to do RSGD in the Poincare upper-half space model, we firstly derive the Riemannian gradient in the tangent space from the Euclidean gradient using the pull-back metric computed as:
gradz f = zn∇zf,
then we take the learning rate η into consideration, denote v = −η · gradzf . Next, we will need the exponential map Exp in the Poincaré upper-half space model, which takes a tangent vector v in the tangent space to z′ on the manifold Un via the following arithmetic:
z′i =zi + zn
s tanh s − vn
· vi, i < n (1)
z′n = zn
cosh s− sinh ss vn (2)
where s = √ vTv. We also provide a more numerically stable form of this exponential map in the Appendix. As an example, assume an exact point x is given, we do RSGD following the above arithmetic and get the accurate y as:
y = Expx(−η · gradxf). We can derive a y′ using ordinary floating-point arithmetic and compute the distance error between the exact y and y′. In Figure 2, we show how this error du(y,y′) varies as the starting point x gradually moves away from the origin. At a distance of around 60, the halfspace RSGD algorithm will be numerically unstable with numerical errors large enough to cause NaNs.
In the Poincaré upper-half space model, the major part of the numerical errors in the exponential map happens in Equation 1, when a big floating-point number is added to a small floating-point gradient number. This lets us localize the numerical errors within a simple addition operation in the Poincaré upper-half space model; in comparison, for the exponential map in the Lorentz model [28], multiple operations cause numerical errors. Hence, we propose to use multiple-component floating-point numbers in the Poincaré upper-half space model to avoid the numerical errors in addition and subtraction, rather than base our methodology in other models like the Lorentz model.

heading: 4 Multiple-Component Floating-point (MCF)
text: Currently, most high precision algorithms are developed with one of two approaches: (1) the standard multiple-digit “BigFloat” approach, involving a sequence of digits coupled with a single exponent and (2) the multiple-component format (MCF), which uses unevaluated sums of multiple ordinary floating-point numbers. The multiple-digit approach can represent compactly a much larger range of numbers, whereas the multiple-component approach has an advantage in speed, as it better takes advantage of existing floating-point accelerators. Since speed and scalability is critical for ML applications, MCF is to be preferred here. In this section, we will detail how MCF works.
For a machine which supports p-bit floating-point number arithmetic, exact arithmetic often produces values that require more than p-bits to express and store. In the multi-component floating-point format, each arbitrary precision value x is expressed as an expansion [36]:
x = xn + · · ·+ x2 + x1, where each xi is called a component of x and is represented by a floating-point number with a p-bit significand. As the structure requirements of this expansion definition, all components are ordered by magnitude (from largest xn to smallest x1). What’s more, every two different components xi, xj are required to be non-overlapping: the least significant nonzero bit of xi is more significant than the most significant nonzero bit of xj or vice versa.
Multiple-component floating-point format allows roundoff error to occur when performing exact arithmetic, then accounts for it afterwards by adding more components. To measure roundoff quickly and correctly, the algorithms presented herein rely on the assumption that ordinary floating-point ops are performed with standard to-nearest rounding. MCF allows the boundaries between components to wander freely, and boundaries are still enforced but can fall at any bit position. For example, the sum of 2300 + 2−300, will be represented by (2300, 2−300) in the multiple-component floating-point format. In this sense, the multiple-component approach may store numbers with fewer bits, stored as two words of memory, whereas the multiple-digit approach will use at least 601 bits to store the sum [36].
From Figure 1, we see that the ‘NaN’ problem occurs at a distance around 50 to the origin, thus for the representation of hyperbolic space, the range of distances is not the main concern, rather, the issue is that when points are far from the origin, a small error in magnitude causes a high numerical
Algorithm 1: Two-Sum Input: p-bit floats a, b, where p ≥ 3
x← fl(a+ b) bvirtual ← fl(x− a) avirtual ← fl(x− bvirtual) broundoff ← fl(b− bvirtual) aroundoff ← fl(a− avirtual) y ← fl(aroundoff + broundoff)
Return: (x, y)
Algorithm 2: Grow-Expansion Input: m p-bits expansion e, p-bit float b initialize Q0 ← b for i = 1 to m do
(Qi, hi)← Two-Sum(Qi−1, ei) end for hm+1 ← Qm Return: h
error in subsequent computations. Multiple-component floats, which can have very small numerical errors, will thus be better-suited to represent hyperbolic space compared to the multiple-digit format.
Multiple-component floating-point arithmetic can be faster than multiple-digit arithmetic because the latter requires expensive normalization of results to fixed digit positions. MCF can also be implemented to run efficiently on GPUs because their unevaluated-sum structure is amenable to parallelism. This allows learning applications that use MCF to enjoy not only accurate results but also benefit from the hardware acceleration of GPUs.

heading: 5 Computing with MCF
text: We provide some basic numerical operations for computing with multiple-component floating-point numbers. To start with, we consider the sum of two p-bit floating-point numbers to form a nonoverlapping expansion.
Theorem 2 [21, 36]. For floating point numbers a, b, Alg. 1 produces a non-overlapping expansion (x, y) such that a + b = x + y, where x is an approximation to a + b and y is the roundoff error. Particularly, when exact rounding is adopted, the roundoff error y is bounded as
|y| ≤ 1 2 ulp(fl(a+ b)), |y| ≤ min(|a|, |b|).
where ulp(x) is the unit of the least precision of x.
This theoretical bound on the roundoff error y serves an important role in computations involving Alg. 1 (most MCF algorithms). Now we can derive an expansion by adding two p-bit floating-point values with this algorithm, next, we need to add a single floating-point number to an expansion. Alg. 2 shows how to add a single p-bit value to any arbitrary precision value expressed as an expansion.
Theorem 3 [36] Let e = ∑m
i=1 ei be a non-overlapping m p(≥ 3)-bit components expansion, sorted in increasing order, except that any of the ei may be zero. Let b be a p(≥3)-bit value, then Alg. 2 produces a non-overlapping m+ 1 cmponents expansion h = ∑m+1 i=1 hi = e+ b, where the components are also in increasing order, except that any of them may be zero. Notably, Qi in Alg. 2 can serve as an approximate sum of b and the first i components of e.
As a result of Alg. 2, addition of a value to an m-components expansion will grow the expansion and outputs an expansion with m+ 1 components. However, if there are no constraints on the number of components, then different values will be highly irregular and vary a lot, which will slow down the computations and make the algorithms much more complicated. Hence, in our implementation, rather to grow the number of components without any constraints, we would like to fix the number of components for all expansions we may use. With Alg. 3, we can renormalize an m+ 1 expansion back to an m components expansion.
Priest [32] proves that if the input expansion of Alg. 3 does not overlap by more than 51 bits, then the algorithm works correctly. This condition holds when ordinary floating-point numbers are used. Table 1 summarizes MCF algorithms used in this paper, some of them detailed in the Appendix. Note that firstly these MCF algorithms can be easily vectorized to apply on high dimensional vectors, secondly, they are linear in the number of components (quadratic for Scale-Expansion) and dimension, hence enjoy a great speedup.
6 Learning using MCF
Algorithm 3: Renormalize [32] Input: (m+ 1)-components expansion (a0, a1, · · · , am) in decreasing order. initialize s← am, k ← 0 for i = m to 1 do
(s, ti)← Two-Sum(ai−1, s) end for for i = 1 to m do
(s, e)← Two-Sum(s, ti) if e 6= 0 then bk ← s s← e k ← k + 1
end if end for Return: (b0, b1, · · · , bm−1)
We solve the ‘NaN’ problem in the Poincaré upper-half space model using multiple-component floating-point arithmetic in the addition & subtraction computations. As analyzed in Sec. 3, we only need to use m-multiplecomponent floating-point numbers for the first n−1 coordinates of the model, denoted as m-xMC-Halfspace. We will mostly compute with MCF arithmetic (in Table 1) for the addition & subtraction involved in the computations, and leave the rest of computations to ordinary floating-point arithmetic.
For example, we compute the distance between x,y ∈ Un as:
du(x,y) = arcosh
( 1 +
Add-Expansion(x,−y)21 2xnyn ) where we add two expansions x,−y to get the summed expansion and then take the first, largest component as an approximation of the scalar ‖x − y‖. Further more, in Appendix we show how to do RSGD using the multi-component floating-point numbers in the mxMC-Halfspace model.
When we compute with the introduced multi-components floating-point numbers in the model, we show theoretically the worse case numerical representation error in Theorem 4.
Theorem 4 The worst case representation error of storing an exact point x ∈ Un with m-multicomponent floating-point expansion (x(m), · · · ,x(2),x(1)) defined as a function of the distance-toorigin d is
δd = arcosh ( 1 + 2machine + 2mmachine(1 + machine)
22m cosh2(d)
) ,
where machine is the machine epsilon. This becomes δd = 2 machine + o( machine) if d < m log(1/ machine) and δd = 2d+ 2m log( machine/2) + o( −2mmachine exp(−2d)) if d ≥ m log(1/ machine).
Table 2: Datasets Information
Datasets Nodes Edges
WordNet[12] 74374 75834
Nouns 82115 769130 Verbs 13542 35079
Mammals 1181 6541 Gr-QC[22] 4158 13422
Hence, there’s a region around the origin whose radius is proportional to the number of components m, where we can be confident there are no numerical issues. This is the scaling we’d expect, and is in line with the results from [34]. In Figure 1, we plot the averaged representation error d(x,MCF(x)) when x is represented using 2-component floating-points. The error will not explode until a distance of 125 from the origin simply by using 2 components, compared to the distance of 55 for vanilla halfspace.
Particularly, we also show the numerical errors of the adapted RSGD algorithm in the 2-xMCHalfspace model in Figure 2, with way smaller error compared to the increasing numerical errors in Poincaré upper-half space model using ordinary floating numbers. Practitioners can choose the number of components according to the desired precision requirements, yet many applications would get full benefits from using merely a small multiple of (such as two or three) components.
As a matter of fact, MCF can be more generally adopted in many operations, in all hyperbolic models and different generic loss functions for better performance. Hence we also implement the model with all coordinates represented with MCF, denoted as m-MC-Halfspace. We defer the discussion of this model to Appendix since it is more sophisticated by making use of the Scale-Expansion algorithm.

heading: 7 Experiments
text: Firstly proposed in Nickel and Kiela [27], hyperbolic embedding was used to embed data with hierarchical structure such as taxonomies, producing results that outperform Euclidean embeddings. Motivated by this, we conduct embedding experiments on datasets including the Gr-Qc and commonly used WordNet Nouns, Verbs and Mammals dataset whose statistics are shown in Table 2. We are interested in two aspects of the proposed MCF-based models in practice: firstly the capacity of the models, and secondly the running speed of model training and optimization.
Reconstruction. We focus on the tasks which can show the capacity of the models. Given an observed transitive closure D = {(u, v)} of all objects, we embed all objects into hyperbolic space and then reconstruct it from the embedding. Hence, the error from the reconstructed embedding to the ground truth will be a reasonable measure for the capacity of the model and reflect how the underlying floating-point arithmetic may affect the performance. Specifically, we learn embeddings of the transitive closure D such that related objects are close in the embedding space. To do this, we aim to minimize the loss introduced in [27]:
L(Θ) = ∑
(u,v)∈D
log e−d(u,v)∑
v′∈N (u) e −d(u,v′) ,
where N (u) = {v | (u, v) 6∈ D} ∪ {u} is the set of negative examples for u (including u). We do negative sampling randomly per positive example during training for different datasets.
We implemented the upper-half space model and xMC-Halfspace models trained with RSGD in our experiments, as for baselines, we consider the current state-of-the-art L-tiling models, H-tiling models [37] trained with RSGD and the Lorentz model [28]. Models were trained using ordinary
float64 with the same hyper-parameters starting from same initializations of parameters, see Appendix for more implementation details. To evaluate the quality of the embeddings, we make use of the standard graph embedding metrics in [5, 26]. For an observed relationship (u, v), we rank the distance d(u,v) among the set {d(u, v′)|(u, v′) ∈ D)}, then we evaluate the ranking on all objects in the dataset and record the mean rank (MR, smaller⇒better) as well as the mean average precision (MAP, larger⇒better) of the ranking. We firstly evaluate different 2-dimensional embeddings on the light-scale Mammals dataset. As shown in Table 3, we choose 2 components for the MCF-based models. The proposed upper-half space model already outperforms all previously proposed baseline models. In particular, the xMC-Halfspace model further improves the embedding performance and achieves a 2.25% MAP improvement on Mammals compared to the best baseline models.
We present the large scale experiments on other three datasets in Table 4. These results show that MCF-based models generally perform better than baseline models in various dimensions with respect to MAP and MR, particularly in 2 dimensional hyperbolic space, since higher dimension can alleviate the ‘NaN’ problem to some degree. On 2 dimensional embeddings of the largest WordNet Nouns dataset, MCF-based models gain as large as a 1.03% MAP improvement compared to the L-tiling model and a 1.11% MAP improvement compared to the upper-half space model.
The limitation of the work is that in some cases of embedding problems that stay close to the origin, MCF will give no improvement over plain half space because it was already numerically accurate enough, also our method will eventually break down at very large distances due to overflow or underflow of the floating point exponent.
Training time cost. Running the model on GPUs confers important training advantage for modern machine learning tasks. In order to provide a comprehensive comparison result, we train most models in our experiments on GPUs including MCF-based half space models, upper-half space model and the Lorentz model. As for the L-tiling and H-tiling model, since big-integer matrix multiplications are not supported in GPUs, we train both models with CPUs. For all timing results within this section, we use GPU: GeForce RTX 2080 Ti and CPU: Intel(R) Xeon(R) Gold 6240 @2.60GHZ.
Note that negative sampling size and batch size can make a great difference to the timing performance over CPUs and GPUs. Herein, we fix the negative sampling size to be 50 and only vary the batchsize in 32, 64, 128. Readers may question the performances of the model as the batchsize changes, we note that with a simple scale of the learning rate, one can achieve comparable results for different batchsize, detailed in the Appendix. We report the training time (seconds per epoch) for different models on three different datasets in Table 5. We can conclude that:
• On large-scale datasets or large batchsize (e.g. 128) training, MCF-based models (high precision) can be trained efficiently, with timing close to that of models using ordinary floating-point.
• MCF-based models (high precision) can run on GPUs, enjoying a great training speedup compared to other high precision models such as the L-tiling and H-tiling models which can only run over CPUs. As a matter of fact, on WordNet Nouns, relatively it takes MCF-based models around 2/3 less time and saves 30.27 hours to finish 1000 epochs training (batchsize = 128), compared to the H-Tiling model with best MAP performance.
More experiment details are in the Appendix. We will release our code in PyTorch publicly for reproducibility, and in hopes that practitioners can use MCF to reliably compute in hyperbolic space.

heading: 8 Conclusion
text: In this work, we proposed upper-half space model of hyperbolic space and presented new multicomponent floating-point representations. Our approach avoids the numerical errors inherent to popular models of hyperbolic space, such as the Lorentz model, while supporting efficient computations on GPUs. This allows our MCF approach to effectively trade off computational cost and numerical accuracy for learning over hyperbolic space.
Acknowledgments and Disclosure of Funding
This work is supported by NSF IIS-2008102. This work was supported in part by a gift from SambaNova Systems, Inc. The authors would like to thank Aaron Lou and anonymous reviewers from NeurIPS 2021 for providing valuable feedbacks on earlier versions of this paper.

Review summaries:
To solve the 'NaN' problem in hyperbolic embeddings, the paper proposes to learn the hyperbolic embeddings in the Poincare upper-half space model using multi-component floating-point (MCF). Theoretically, the paper proves that numerical errors can be reduced to any degree by simply increasing the number of components of MCF. In experiments, it shows the proposed MCF can have better results while not compromising much on efficiency.