Representing Hyperbolic Space Accurately using Multi-Component Floats

1 Introduction . Representation learning is of particular interest nowadays in machine learning . Real-world data is often discrete—such as natural language sentences in the form of syntax trees and graphs [ 7 , 22 ] — which makes it critical to do a good job of representing and storing them as numbers which we can better understand and deal with . Perhaps the most common way of doing this representation is with embedding , in which discrete objects are assigned points in some continuous ( typically high-dimensional Euclidean ) space . Standard embeddings in NLP include Word2Vec [ 24 ] and GloVe [ 30 ] . However , many kinds of real-world data are hierarchical or graph-like [ 25 ] , in which case embedding into non-Euclidean space can outperform the standard Euclidean embedding . Hyperbolic space is such an embedding target of particular interest , showing impressive performance on many tasks such as taxonomies , link prediction and inferring concept hierarchies [ 27 , 28 , 15 ] . Hyperbolic space [ 9 ] is helpful for many types of embeddings because of its constant negative curvature , as distinguished from Euclidean space with zero curvature . This negative-curvature geometry endows hyperbolic space with many useful properties not present in Euclidean space . For example , the volume of a ball in even two-dimensional hyperbolic space increases exponentially with respect to the radius , as shown in [ 9 ] , as a contrast to the polynomial rate at which balls grow in Euclidean space . This makes it possible to embed more instances in the hyperbolic ball than Euclidean ball with the same radius , making it ideal for structures like trees where the number of objects at a distance d from any given object can grow exponentially with d. Reasoning from this intuition , the hyperbolic space “ looks like ” a finite-degree tree and can be regarded as a continuous version of trees [ 6 ] . The combination of low dimension ( which facilitates learning ) with high volume ( which increases representational capacity ) makes hyperbolic space inherently well-suited for embedding hierarchical structures data like trees and graphs . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . However , the advantages of hyperbolic space do not come for free . The large volumes that are so beneficial from a embedding-capacity perspective create unavoidable numerical problems . Informally called “ the NaN problem '' [ 37 , 34 ] , representing hyperbolic space with ordinary floating-point numbers can cause significant errors and lead to severe numerical instability that compounds throughout training , causing NaNs to pop out everywhere in the computations . Common models of hyperbolic space used to learn hyperbolic embeddings using floating-point arithmetic , such as the Poincaré ball model [ 27 , 15 ] and the Lorentz hyperboloid model [ 28 ] , all suffer from significant numerical errors . These errors originate in the ordinary floating-point representation , and are further amplified in subsequent computations by the very ill-conditioned Riemannian metrics involved in their construction . Yu and De Sa [ 37 ] show that in the Lorentz hyperboloid model , the error of representing an exact point with ordinary floating-point numbers becomes unbounded as the point moves away from the origin , which is what will happen in most embedding tasks : points unavoidably move far away from the origin as more and more data instances crowd in . Many efforts and compromises have been made so as to solve or alleviate the ‘ NaN ’ problem . One commonly adopted technical solution firstly exploited in [ 35 ] is using an appropriate scaling factor , i.e. , scale down all the distances by the scaling factor before embedding , then recover the original distances by dividing to the scaling factor . However , this scaling will increase the distortion of the embedding , and the distortion gets worse as the scale factor increases . Sala et al . [ 34 ] show that this sort of tradeoff is unavoidable when embedding in hyperbolic space : one can either choose to increase the number of bits used for the floating-point numbers or increase the dimension of the space , and this is independent of the underlying models . However , increasing the dimension is not a panacea , as the ‘ NaN ’ problem still exists in high dimensional space ( albeit to a lesser degree ) . A straightforward solution to the ‘ NaN ’ problem would be to compute with high precision floatingpoint numbers , such as BigFloats ( specially designed with a large quantity of bits ) , as adopted in the combinatorial construction of hyperbolic embedings in [ 34 ] , where floating-point numbers with thousands of bits are used with the support of Julia [ 4 ] . However , there are two evident problems with representing hyperbolic space using BigFloats . Firstly , BigFloats are not supported on ML accelerator hardware like GPUs : computations using BigFloats on CPUs will be slow and hence not realistic for many meaningful tasks . Secondly , BigFloats are currently not supported in most deep learning frameworks even on CPUs , so one would need to develop the algorithms from scratch so as to learn over hyperbolic space , which is tedious and fraught with the danger of numerical mistakes . A hybrid solution to the ‘ NaN ’ problem proposed by Yu and De Sa [ 37 ] is to avoid floating-point arithmetic as much as possible by using integer arithmetic instead—as integer arithmetic can be done exactly . Their tiling-based model works by decomposing hyperbolic space into a tiling : the product of a compact fundamental region ( which can be represented in floating-point with bounded error ) and a discrete group ( which supports integer computation ) . We can also interpret their approach as decomposing a large floating-point number into a multiply of a small floating-point number with a large integer number using some group actions [ 3 , 33 ] . This allows for a guaranteed bound on the numerical error of learning in hyperbolic space . However , their approach is limited somewhat in that its guarantees apply to loss functions that depend only the hyperbolic distance between points , and their best-performing model ( their L-tiling model ) is limited to 2-dimensional space . A bigger issue with this type of approach is computational : the big-integer matrix multiplication involved in the model is not supported on GPUs and is not yet implemented in many deep learning frameworks such as PyTorch [ 29 ] and TensorFlow [ 1 ] . This effectively prevents the method from being used in training of complex models that depend on GPU acceleration , such as hyperbolic neural networks [ 14 , 18 , 10 , 23 ] . In this paper , we propose a more direct way to represent and learn in hyperbolic space without sacrificing either accuracy guarantees or GPU acceleration . To achieve high precision computations and controllable numerical errors , we choose to represent hyperbolic space with multiple-component floating-point numbers ( MCF ) [ 31 , 32 , 19 , 36 ] , an alternative approach different from BigFloats where a number is expressed as an unevaluated sum of multiple ordinary floating-point numbers . We make the following contributions : • We propose representing hyperbolic space with the Poincaré upper-half space model for optimization in section 3 , and show how to represent it with MCF in section 4 to eliminate the ‘ NaN ’ problem . • We provide algorithms for computing with these MCF that are adapted to hyperbolic space in section 5 and section 6 , where we prove numerical errors can be reduced to any degree by simply increasing the number of components of the MCF . • We experimentally measure our model on embedding tasks across several datasets , and show that one can gain more learning capacity ( compared to models that use ordinary floatingpoint numbers ) with only a mild computational slowdown on GPUs in section 7 , meanwhile significantly faster than prior high precision hyperbolic models . 2 Related Work . The non-Euclidean geometry of hyperbolic space offers many distinctive properties useful in machine learning . It is well-suited to model hierarchical structures such as trees and more complex graphs : for instance , Sala et al . [ 34 ] show that a tree can be embedded into a 2-dimensional hyperbolic space with arbitrarily low distortion , and Gromov [ 16 ] shows that any finite tree can be embedded into a finite hyperbolic space such that distances are preserved approximately . Nickel and Kiela [ 27 ] propose to embed symbolic data into the Poincaré ball model of hyperbolic space for learning the hierarchical representations , where the Poincaré ball embeddings outperform Euclidean embeddings significantly on hierarchical data in terms of both representation capacity and generalization ability . Subsequently , Ganea et al . [ 15 ] further improve the empirical results of Poincaré ball embeddings on directed acyclic graphs by theoretically grounding entailment using the hierarchical relations . Later work uses other models of hyperbolic space [ 28 , 17 ] besides the Poincaré ball model for various purposes . For example , the Lorentz hyperboloid model used in Nickel and Kiela [ 28 ] to learn continuous hierarchies from large-scale unstructured data , was observed to be substantially more efficient and numerically stable . Along a different direction , hyperbolic embeddings can be used in many down-streaming tasks , by simply treating the hyperbolic embedding as a layer to form neural networks , such as the hyperbolic neural networks [ 14 ] for textual entailment and noisy-prefix recognition tasks , hyperbolic attention networks [ 18 ] for neural machine translation and visual question answering tasks , and hyperbolic graph networks [ 10 , 23 ] for link prediction and node classification tasks . Although widely adopted in many applications , the ‘ NaN ’ problem in hyperbolic representations can be observed easily in practice and generally remains unsolved . Sala et al . [ 34 ] suggests that hyperbolic embeddings can have high quality on hierarchies but require large dimensions or high precision for graphs with long chains : they suggest alleviating the ‘ NaN ’ problem by using scaling factors together with BigFloats . Yu and De Sa [ 37 ] introduced tiling-based models , the first theoretical solution to the ‘ NaN ’ problem with bounded-everywhere numerical error . The tiling-based models enable deriving high-precision embeddings without using high precision floating-point numbers . In the literature of arbitrary precision algorithms , there are two different ways to achieve high precision computations , and BigFloat is not the only option . The first one is straightforward : the multiple-digit format , which consists of a sequence of digits coupled with a single exponent . Examples in this category include the computation package Mathematica [ 20 ] , the BigFloats format in Julia [ 4 ] , and the Fortran multiple-precision arithmetic packages [ 8 , 2 ] . The other option is the multiple-component format , in which an exact number is written as an unevaluated sum of ordinary floating-point numbers , each with its own significand and exponent [ 11 , 31 , 32 , 13 ] . 3 The ‘ NaN ’ Problem . Hyperbolic space can be modelled with several well-known isomorphic models . Previous works have proposed to optimize over the hyperbolic space with the Poincaré ball model [ 27 ] and Lorentz hyperboloid model [ 28 ] . In this paper , we propose to optimize within the Poincaré upper-half space model instead , because its properties allow for localization of potential sources of numerical error . The Poincaré upper-half space model is the manifold ( Un , gu ) , where Un = { x ∈ Rn : xn > 0 } is the upper half space of n-dimensional Euclidean space . The metric and corresponding distance function are gu ( x ) = ge x2n , du ( x , y ) = arcosh ( 1 + ‖x− y‖2 2xnyn ) where ge is the Euclidean metric . Notably , the tangent space of the upper-half space model is the same as the ambient vector space , which makes it easier and numerically stable to derive Riemannian gradients without projection to the tangent space . In the two dimensional case , the Poincaré upper-half space model can be interpreted as a mapping on the complex plane ( the “ half-plane model ” ) . Numerical error analysis . Yu and De Sa [ 37 ] shows that in the Lorentz hyperboloid model of the hyperbolic space , the error of representing an exact point x with ordinary floating-point fl ( x ) is unbounded and proportional to the distance to the origin d ( x , O ) . Here , as a baseline to compare our MCF methods to , we show that the same sort of representation error d ( x , fl ( x ) ) happens when representing points in the Poincaré upper-half space model with ordinary floating-point numbers . Theorem 1 The representation error of storing a particular point x ∈ Un using floating-points fl is δfl ( x ) = du ( x , fl ( x ) ) , and the worst case representation error defined as a function of the distance-to-origin d in the Poincaré upper-half space model is δd : = max x∈Un , du ( x , O ) ≤d δfl ( x ) = arcosh ( 1 + 2 machine cosh 2 ( d ) ) . where machine is the machine epsilon of the underlying floating-point arithmetic . This becomes δd = 2 machine + o ( machine ) if d < log ( 1/ machine ) and δd = 2d + 2 log ( machine ) + o ( −1machine exp ( −2d ) ) if d ≥ log ( 1/ machine ) . We plot the averaged representation error and the ‘ NaN ’ problem practically in Figure 1 , the distance error d ( x , fl ( x ) ) when x is represented as fl ( x ) using floating-point . The error explodes at at distance of around 55 from the origin , which will then potentially cause NaNs in subsequent computations . Identify potential numerical errors . Optimizing x in the Poincaré upper-half space model will typically push x away from the origin , with xi , i 6= n being large floating-point numbers and xn being floating-point numbers close to 0 . Hence , the subtraction of two big floating-point numbers in ‖x−y‖2 of the distance calculation leads to catastrophic cancellation and accounts for a large part of the numerical errors . We can localize this part of the numerical errors easily in simple subtractions in the Poincaré upper-half space model . ( For comparison , in the Lorentz model [ 28 ] , the multiplication in Lorentzian scalar product accounts for the major numerical errors in distance computations . ) RSGD in the Poincaré upper-half space model . We are interested in gradient-based optimizations within the Poincaré upper-half space model , hence we provide Riemannian stochastic gradient descent ( RSGD ) algorithm for different learning algorithms in the hyperbolic space . For each z ∈ Un in the Poincaré upper-half space model , let f : Un → R be a smooth real-valued loss function over the manifold , denote ∇zf to be the Euclidean gradient of f w.r.t . z . In order to do RSGD in the Poincare upper-half space model , we firstly derive the Riemannian gradient in the tangent space from the Euclidean gradient using the pull-back metric computed as : gradz f = zn∇zf , then we take the learning rate η into consideration , denote v = −η · gradzf . Next , we will need the exponential map Exp in the Poincaré upper-half space model , which takes a tangent vector v in the tangent space to z′ on the manifold Un via the following arithmetic : z′i =zi + zn s tanh s − vn · vi , i < n ( 1 ) z′n = zn cosh s− sinh ss vn ( 2 ) where s = √ vTv . We also provide a more numerically stable form of this exponential map in the Appendix . As an example , assume an exact point x is given , we do RSGD following the above arithmetic and get the accurate y as : y = Expx ( −η · gradxf ) . We can derive a y′ using ordinary floating-point arithmetic and compute the distance error between the exact y and y′ . In Figure 2 , we show how this error du ( y , y′ ) varies as the starting point x gradually moves away from the origin . At a distance of around 60 , the halfspace RSGD algorithm will be numerically unstable with numerical errors large enough to cause NaNs . In the Poincaré upper-half space model , the major part of the numerical errors in the exponential map happens in Equation 1 , when a big floating-point number is added to a small floating-point gradient number . This lets us localize the numerical errors within a simple addition operation in the Poincaré upper-half space model ; in comparison , for the exponential map in the Lorentz model [ 28 ] , multiple operations cause numerical errors . Hence , we propose to use multiple-component floating-point numbers in the Poincaré upper-half space model to avoid the numerical errors in addition and subtraction , rather than base our methodology in other models like the Lorentz model .

Summary: To solve the 'NaN' problem in hyperbolic embeddings, the paper proposes to learn the hyperbolic embeddings in the Poincare upper-half space model using multi-component floating-point (MCF). Theoretically, the paper proves that numerical errors can be reduced to any degree by simply increasing the number of components of MCF. In experiments, it shows the proposed MCF can have better results while not compromising much on efficiency.