Hyperbolic Busemann Learning with Ideal Prototypes

1 Introduction . Classification by prototypes has a long tradition in machine learning . Foundational solutions such as the Nearest Mean Classifier [ 46 ] represent classes as the mean prototypes in a fixed feature space . Following approaches that learning a metric space for class prototypes [ 28 ] , deep learning with prototypes as points in network output spaces has gained traction [ 14 , 17 , 26 , 32 , 36 , 41 , 49 , 51 ] . In line with foundational approaches , the prototypes are positioned by computing the mean over all training examples for each class . Deep learning by mean prototypes has shown to be effective for tasks such as few-shot learning [ 14 , 32 , 36 , 41 ] and zero-shot recognition [ 41 , 49 , 51 ] . To avoid the need to re-learn prototypes or to enable the use of prior label knowledge , several recent works have proposed deep networks with prototypes in non-Euclidean output spaces . Hyperspherical prototype approaches initialize prototypes on the sphere or its higher-dimensional generalization , based on pair-wise angular separation [ 29 , 39 ] or with the help of word embeddings [ 4 , 43 ] using a cosine similarity loss between example outputs and class prototypes . Hyperspherical prototypes alleviate the need to re-position prototypes continuously and allow to optionally include prior semantic knowledge about the classes . Recently , a few works have extended prototype-based learning to the hyperbolic domain , where prototypes are obtained by embedding label hierarchies [ 23 , 24 ] . While hyperbolic prototype approaches provide more hierarchically coherent classification results , prior hierarchical knowledge is required to embed prototypes . This paper strives to combine the best of both non-Euclidean worlds , namely efficient low-dimensional embeddings from hyperbolic prototypes and the knowledge-free positioning from hyperspherical prototypes . We make three contributions in this work . First , we introduce a hyperbolic prototype network with class prototypes given as points on the ideal boundary of the Poincaré ball model of hyperbolic geometry . Second , we propose the penalized Busemann loss , which enables us to compute proximities between example outputs in hyperbolic space and prototypes at the ideal boundary , an impossible task for existing distance metrics , which put the ideal boundary at infinite distance from all other 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) points in hyperbolic space . Third , we provide a theoretical link between our hyperbolic prototype approach and logistic regression . We show that our choices for output manifold , for prototypes on the ideal boundary , and for the proposed loss provide a direct generalization of the logistic regression model . Experiments on three datasets show that our approach outperforms both hyperspherical and hyperbolic prototype approaches . Moreover , our hyperbolic output space provides a natural interpretation of closeness to the ideal boundary as classification confidence , see Figure 1 . The code is available at https : //github.com/MinaGhadimiAtigh/Hyperbolic-Busemann-Learning . 2 Related work . As an early adaptation of prototypes in deep networks , the center loss [ 47 ] served as an additional loss for softmax cross-entropy optimization , where all outputs of a class were pulled to one point to help minimize intra-class variation . Closer to the original Nearest Mean Classifier objective , Prototypical Networks introduce a prototype-only approach where each class is represented as the mean output of its corresponding training examples [ 41 ] . Learning with prototypes from class means has shown to be effective for few-shot learning [ 14 , 32 , 36 , 41 ] , zero-shot recognition [ 41 , 49 , 51 ] , and domain adaptation [ 33 ] . Defining prototypes as class means is feasible in few-shot learning by sampling all samples of a class in a mini-batch . For general supervised learning , DeepNCM [ 17 ] and DeepNNO [ 26 ] propose alternating optimization schemes , where the prototypes are fixed to update the network backbone and vice versa . To avoid the need for constant prototype updating , several works have proposed to use the hypersphere as output space instead of the Euclidean space . On the hypersphere , prototypes can be positioned a priori based on angular separation [ 29 , 39 ] , using an orthonormal basis [ 4 ] , or using prior knowledge from word embeddings [ 4 , 29 , 43 ] . Similar to hyperspherical prototype approaches , we seek to position prototypes at unit distance from the origin . In contrast , we operate on a hyperbolic manifold , which provides a theoretical link to logistic regression and obtains better empirical results than a hypershperical manifold and loss . In this work , we strive for a prototype-based network on hyperbolic manifolds , which have become a popular choice for embedding data and deep network layers . Hyperbolic embeddings have shown to represent tree-like structures [ 38 , 37 ] , text [ 1 , 44 , 55 ] , cells [ 20 ] , networks [ 18 ] , multi-relational knowledge graphs [ 3 ] , and general taxonomies [ 16 , 21 , 30 , 31 , 50 ] with minimal distortion . In deep networks , hyperbolic alternatives have been proposed for fully-connected layers [ 15 , 40 ] , convolutional layers [ 40 ] , recurrent layers [ 15 ] , classification layers [ 13 , 15 , 40 ] , and optimizers [ 5 , 6 , 52 ] . Hyperbolic generalizations have also been introduced for graph networks [ 2 , 12 , 22 , 25 , 53 , 54 ] and generative models [ 7 , 27 ] . Where hyperbolic-based deep learning approaches commonly operate within the Poincaré model represented by the open unit ball , we propose to position class prototypes on its ideal boundary – the shell of the unit ball – allowing us to benefit from the representational power of hyperbolic geometry without requiring prior knowledge to operate . A few works have recently proposed prototype-based approaches with hyperbolic output spaces . Hyperbolic Image Embeddings by Khrulkov et al . [ 19 ] include a prototype-based setting as a direct hyperbolic generalization of Prototypical Networks [ 41 ] , allowing for few-shot learning through hyperbolic averaging . Beyond few-shot learning , Long et al . [ 24 ] introduce a hyperbolic action network , where action prototypes are projected to a hyperbole based on prior knowledge about hierarchical relations . Videos are in turn projected to the same space and compared to the prototypes using the hyperbolic distance , enabling hierarchically consistent action recognition and zero-shot recognition . Similarly , Liu et al . [ 23 ] perform zero-shot recognition using hyperbolic prototypes positioned based on known hierarchical relations . In this work , we also employ prototypes in hyperbolic space , but do not require prior knowledge about hierarchies . Recent approaches with hyperbolic output spaces have shown to be effective in low dimensionalities [ 19 , 24 , 23 ] . Low dimensional embeddings have several practical use cases and benefits . We list two important use cases . Compactness : Hyperbolic Busemann Learning enables us to embed high-dimensional inputs in low-dimensional output spaces . This ability has potential benefits in applications such as data transmission , compression , and storage . Interpretability : Obtaining high performance with few output dimensions ( i.e. , two or three output dimensions ) provides the potential to interpret the model performance and visualize the output space , see Figure 1 . 3 Learning with hyperbolic ideal prototypes . 3.1 Problem formulation . We are given a training set { ( xi , yi ) } Ni=1 with N examples , where xi ∈ RI denotes an input with dimensionality I and yi ∈ { 1 , .. , C } denotes a label from a set of sizeC . We seek to learn a projection of inputs to a hyperbolic manifold in which can compute proximities to class prototypes . We do so through a transformation F in Euclidean space , followed by an exponential map from the tangent space TxM to a hyperbolic manifoldM : z = expv ( F ( x ; θ ) ) , ( 1 ) where θ denotes the parameters to learn the transformation . The transformation can be any function , e.g. , a deep network or a linear layer akin to logistic regression . The hyperbolic manifold in which z operates is described throughout this work by the Poincaré ball model ( Bd , gBz ) , with open ball Bd = { z ∈ Rd | ||z||2 < 1 } and the Riemannian metric tensor gBz = 4 ( 1 − ||z||2 ) −2Id [ 11 , 34 ] . Setting its center to v = 0 the exponential map in the Poincaré model is given by : exp0 ( x ) = tanh ( ||x||/2 ) x ||x|| . ( 2 ) Furthermore , the geodesic distance between two points z1 , z2 ∈ Bd is given by : dB ( z1 , z2 ) = arcosh ( 1 + 2 ||z1 − z2||2 ( 1− ||z1||2 ) ( 1− ||z2||2 ) ) . ( 3 ) To train our model , we compare the projected inputs to their class labels , represented by the prototypes P = { p1 , .. , pC } . That is , the overall training loss is given as the aggregation of distance-based individual losses ` between projected inputs and the prototype of their respective class labels : L ( θ ) = 1 N N∑ i=1 ` ( exp0 ( F ( x ; θ ) ) , pyi ) . ( 4 ) 3.2 Ideal prototypes and the penalized Busemann loss . The central idea of our approach is to position class prototypes at ideal points , which represent points at infinity in hyperbolic geometry . In the Poincaré model , the ideal points form the boundary of the ball : Id = { z ∈ Rd : z21 + · · ·+ z2d = 1 } . ( 5 ) Thus , the set of ideal points of hyperbolic space Bd is homeomorphic to the hypersphere Sd . As a consequence , any of the methods used for prototype embedding into Sd can be used to embed prototypes into Id . In particular , for d = 2 , prototypes can be placed uniformly on the unit sphere S2 , while class-agnostic prototype embedding based on separation can be used for d ≥ 3 [ 29 ] . A key issue that needs to be solved , is that ideal points are at infinite geodesic distance from all other points in Bd and hence the geodesic distance can not be used directly for prototype-based learning . Therefore , we propose the penalized Busemann loss for hyperbolic learning with ideal prototypes . The Busemann function , originally introduced in [ 9 ] ( see also Def . II.8.17 in [ 8 ] ) , can be considered a distance to infinity and may be defined in any metric space . Let p be an ideal point and γp a geodesic ray , parametrized by arc length , tending to p. Then the Busemann function with respect to p is defined for z ∈ Bd as : bp ( z ) = lim t→∞ ( dB ( γp ( t ) , z ) − t ) . ( 6 ) In the Poincaré model , the limit can be explicitly calculated and the Busemann function is given as : bp ( z ) = log ||p− z||2 ( 1− ||z||2 ) . ( 7 ) The full derivation of ( 7 ) is provided in Appendix A . Our proposed loss combines the Busemann function with a penalty term : ` ( z , p ) = bp ( z ) − φ ( d ) · log ( 1− ||z||2 ) , ( 8 ) with φ ( d ) a scaling factor for the penalty term which is a function of the dimension of the hyperbolic space . The first term in the penalized Busemann loss steers a projected input z towards class prototype p , while the second term penalized overconfidence , i.e. , values close to the ideal boundary of Bd . The prototypes are positioned before training and remain fixed , hence the backpropagation only needs to be done with respect to the inputs . The gradient of ` ( exp0 ( x ) , p ) with respect to x is given as : ∇x ` ( exp0 ( x ) , p ) = ( x−p ) tanh ( ||x|| ) ||x|| − tanh ( ||x|| ) p · x +1d p·x tanh ( ||x|| ) /||x|| − 1 ||x|| − tanh ( ||x|| ) p · x +tanh ( ||x||/2 ) , ( 9 ) with 1d a d-dimensional vector of ones . Figure 2 shows a color map of the penalized Busemann loss itself and its radial gradient for a prototype located at ideal point p = ( 1 , 0 ) . In the penalized Busemann loss , φ ( d ) governs the amount of regularization . Below , we show that this function should be linear in the number of dimensions . Theorem 1 . For any d ≥ 4 and ideal point p the penalized Busemann loss ` ( z , p ) is the log-likelihood of a density f ( z , p ) = 1C ( d ) exp ( − ` ( z , p ) ) on the Poincaré ball Bd if and only if φ ( d ) > d− 2 . Proof . To show that f ( z , p ) is a density , we have to show that the normalization constant C ( d ) = ∫ Bd exp ( − ` ( z , p ) ) dV is finite . Here , dV is the hyperbolic volume element on Bd , which is equal to 2d ( 1 − ||z||2 ) −ddz1 · · · dzd , see [ 34 ] . Furthermore , using invariance of the integrand under rotation , we can assume without loss of generality that p is equal to the unit vector e1 = ( 1 , 0 , . . . , 0 ) , which leads to C ( d ) = 2d ∫ Bd ( ( 1− ||z||2 ) φ ( d ) +1−d 1− 2z1 + ||z||2 ) dz1 · · · dzd . Switching to ( Euclidean ) hyperspherical coordinates ( r , ϕ1 , . . . , ϕd ) , the integral factorizes into C ( d ) = 2d · C1 ( d ) · C2 ( d ) , where C1 ( d ) = ∫ 1 0 ∫ π 0 ( ( 1− r2 ) φ ( d ) +1−d 1− 2r cos ( ϕ1 ) + r2 ) rd−1 sin ( ϕ1 ) d−2dr dϕ1 C2 ( d ) = ∫ π 0 sin ( ϕ2 ) d−3dϕ2 . . . ∫ π 0 sin ( ϕd−2 ) 2dϕd−2 · ∫ 2π 0 sin ( ϕd−1 ) dϕd−1 . Clearly , C2 ( d ) is finite , independently of d , such that it remains to consider C1 ( d ) . Instead of solving the integral exactly , we use the elementary estimate sin ( ϕ1 ) 2 ≤ 1− 2r cos ( ϕ1 ) + r2 ≤ 4 , valid for all r ∈ [ 0 , 1 ] . This estimate allows us to obtain both a lower and an upper bound on C1 ( d ) . The upper bound becomes C1 ( d ) ≤ ∫ 1 0 ( 1− r2 ) φ ( d ) +1−d rd−1dr ∫ π 0 sin ( ϕ1 ) d−4dϕ1 , which ( given that d ≥ 4 ) is finite if and only if φ ( d ) > d− 2 . The lower bound becomes C1 ( d ) ≥ 1 4 ∫ 1 0 ( 1− r2 ) φ ( d ) +1−d rd−1dr ∫ π 0 sin ( ϕ1 ) d−2dϕ1 , which ( given that d ≥ 2 ) is also finite if and only if φ ( d ) > d− 2 , which completes the proof . Given a trained model , inference is performed by projecting the test example x and selecting the class with the smallest prototype distance : y ? = argmin p∈P ` ( exp0 ( F ( x ; θ ) ) , p ) . ( 10 ) The Poincaré ball model of hyperbolic geometry has the conformal property , i.e. , angles in the Poincaré ball model coincide with angles in Euclidean space [ 35 ] . Thus , the cosine similarity between two ideal points remains a meaningful measure of their similarity , even in hyperbolic geometry . As the penalty term is the same for all prototypes , we can equivalently minimize the Busemann function of the prototypes directly . Moreover , as the Busemann function is a decreasing function of the cosine similarity , prediction is equivalent to hyperspherical prototype inference : y ? = argmax p∈P z ||z|| · p , z = exp0 ( F ( x ; θ ) ) . ( 11 )

Summary: This paper introduces a novel model which uses prototypes placed "at infinity" in hyperbolic space for classification. Specifically, ideal prototypes are placed on the surface of the sphere, and the distance from a point inside the sphere the this idealized point is calculated using the Busemann function, which defines a distance to points at infinity using a limit. Thus, the embedded points representing instances can be interpreted as points in hyperbolic space, and prototypes as points at infinity. The authors perform a number of rigorous evaluations and ablation analysis, demonstrating their model performs well in a number of classification settings.