Hyperbolic Busemann Learning with Ideal Prototypes

1 Introduction . Classification by prototypes has a long tradition in machine learning . Foundational solutions such as the Nearest Mean Classifier [ 46 ] represent classes as the mean prototypes in a fixed feature space . Following approaches that learning a metric space for class prototypes [ 28 ] , deep learning with prototypes as points in network output spaces has gained traction [ 14 , 17 , 26 , 32 , 36 , 41 , 49 , 51 ] . In line with foundational approaches , the prototypes are positioned by computing the mean over all training examples for each class . Deep learning by mean prototypes has shown to be effective for tasks such as few-shot learning [ 14 , 32 , 36 , 41 ] and zero-shot recognition [ 41 , 49 , 51 ] . To avoid the need to re-learn prototypes or to enable the use of prior label knowledge , several recent works have proposed deep networks with prototypes in non-Euclidean output spaces . Hyperspherical prototype approaches initialize prototypes on the sphere or its higher-dimensional generalization , based on pair-wise angular separation [ 29 , 39 ] or with the help of word embeddings [ 4 , 43 ] using a cosine similarity loss between example outputs and class prototypes . Hyperspherical prototypes alleviate the need to re-position prototypes continuously and allow to optionally include prior semantic knowledge about the classes . Recently , a few works have extended prototype-based learning to the hyperbolic domain , where prototypes are obtained by embedding label hierarchies [ 23 , 24 ] . While hyperbolic prototype approaches provide more hierarchically coherent classification results , prior hierarchical knowledge is required to embed prototypes . This paper strives to combine the best of both non-Euclidean worlds , namely efficient low-dimensional embeddings from hyperbolic prototypes and the knowledge-free positioning from hyperspherical prototypes . We make three contributions in this work . First , we introduce a hyperbolic prototype network with class prototypes given as points on the ideal boundary of the Poincaré ball model of hyperbolic geometry . Second , we propose the penalized Busemann loss , which enables us to compute proximities between example outputs in hyperbolic space and prototypes at the ideal boundary , an impossible task for existing distance metrics , which put the ideal boundary at infinite distance from all other 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) points in hyperbolic space . Third , we provide a theoretical link between our hyperbolic prototype approach and logistic regression . We show that our choices for output manifold , for prototypes on the ideal boundary , and for the proposed loss provide a direct generalization of the logistic regression model . Experiments on three datasets show that our approach outperforms both hyperspherical and hyperbolic prototype approaches . Moreover , our hyperbolic output space provides a natural interpretation of closeness to the ideal boundary as classification confidence , see Figure 1 . The code is available at https : //github.com/MinaGhadimiAtigh/Hyperbolic-Busemann-Learning . 2 Related work . As an early adaptation of prototypes in deep networks , the center loss [ 47 ] served as an additional loss for softmax cross-entropy optimization , where all outputs of a class were pulled to one point to help minimize intra-class variation . Closer to the original Nearest Mean Classifier objective , Prototypical Networks introduce a prototype-only approach where each class is represented as the mean output of its corresponding training examples [ 41 ] . Learning with prototypes from class means has shown to be effective for few-shot learning [ 14 , 32 , 36 , 41 ] , zero-shot recognition [ 41 , 49 , 51 ] , and domain adaptation [ 33 ] . Defining prototypes as class means is feasible in few-shot learning by sampling all samples of a class in a mini-batch . For general supervised learning , DeepNCM [ 17 ] and DeepNNO [ 26 ] propose alternating optimization schemes , where the prototypes are fixed to update the network backbone and vice versa . To avoid the need for constant prototype updating , several works have proposed to use the hypersphere as output space instead of the Euclidean space . On the hypersphere , prototypes can be positioned a priori based on angular separation [ 29 , 39 ] , using an orthonormal basis [ 4 ] , or using prior knowledge from word embeddings [ 4 , 29 , 43 ] . Similar to hyperspherical prototype approaches , we seek to position prototypes at unit distance from the origin . In contrast , we operate on a hyperbolic manifold , which provides a theoretical link to logistic regression and obtains better empirical results than a hypershperical manifold and loss . In this work , we strive for a prototype-based network on hyperbolic manifolds , which have become a popular choice for embedding data and deep network layers . Hyperbolic embeddings have shown to represent tree-like structures [ 38 , 37 ] , text [ 1 , 44 , 55 ] , cells [ 20 ] , networks [ 18 ] , multi-relational knowledge graphs [ 3 ] , and general taxonomies [ 16 , 21 , 30 , 31 , 50 ] with minimal distortion . In deep networks , hyperbolic alternatives have been proposed for fully-connected layers [ 15 , 40 ] , convolutional layers [ 40 ] , recurrent layers [ 15 ] , classification layers [ 13 , 15 , 40 ] , and optimizers [ 5 , 6 , 52 ] . Hyperbolic generalizations have also been introduced for graph networks [ 2 , 12 , 22 , 25 , 53 , 54 ] and generative models [ 7 , 27 ] . Where hyperbolic-based deep learning approaches commonly operate within the Poincaré model represented by the open unit ball , we propose to position class prototypes on its ideal boundary – the shell of the unit ball – allowing us to benefit from the representational power of hyperbolic geometry without requiring prior knowledge to operate . A few works have recently proposed prototype-based approaches with hyperbolic output spaces . Hyperbolic Image Embeddings by Khrulkov et al . [ 19 ] include a prototype-based setting as a direct hyperbolic generalization of Prototypical Networks [ 41 ] , allowing for few-shot learning through hyperbolic averaging . Beyond few-shot learning , Long et al . [ 24 ] introduce a hyperbolic action network , where action prototypes are projected to a hyperbole based on prior knowledge about hierarchical relations . Videos are in turn projected to the same space and compared to the prototypes using the hyperbolic distance , enabling hierarchically consistent action recognition and zero-shot recognition . Similarly , Liu et al . [ 23 ] perform zero-shot recognition using hyperbolic prototypes positioned based on known hierarchical relations . In this work , we also employ prototypes in hyperbolic space , but do not require prior knowledge about hierarchies . Recent approaches with hyperbolic output spaces have shown to be effective in low dimensionalities [ 19 , 24 , 23 ] . Low dimensional embeddings have several practical use cases and benefits . We list two important use cases . Compactness : Hyperbolic Busemann Learning enables us to embed high-dimensional inputs in low-dimensional output spaces . This ability has potential benefits in applications such as data transmission , compression , and storage . Interpretability : Obtaining high performance with few output dimensions ( i.e. , two or three output dimensions ) provides the potential to interpret the model performance and visualize the output space , see Figure 1 . 3 Learning with hyperbolic ideal prototypes . 3.1 Problem formulation . We are given a training set { ( xi , yi ) } Ni=1 with N examples , where xi ∈ RI denotes an input with dimensionality I and yi ∈ { 1 , .. , C } denotes a label from a set of sizeC . We seek to learn a projection of inputs to a hyperbolic manifold in which can compute proximities to class prototypes . We do so through a transformation F in Euclidean space , followed by an exponential map from the tangent space TxM to a hyperbolic manifoldM : z = expv ( F ( x ; θ ) ) , ( 1 ) where θ denotes the parameters to learn the transformation . The transformation can be any function , e.g. , a deep network or a linear layer akin to logistic regression . The hyperbolic manifold in which z operates is described throughout this work by the Poincaré ball model ( Bd , gBz ) , with open ball Bd = { z ∈ Rd | ||z||2 < 1 } and the Riemannian metric tensor gBz = 4 ( 1 − ||z||2 ) −2Id [ 11 , 34 ] . Setting its center to v = 0 the exponential map in the Poincaré model is given by : exp0 ( x ) = tanh ( ||x||/2 ) x ||x|| . ( 2 ) Furthermore , the geodesic distance between two points z1 , z2 ∈ Bd is given by : dB ( z1 , z2 ) = arcosh ( 1 + 2 ||z1 − z2||2 ( 1− ||z1||2 ) ( 1− ||z2||2 ) ) . ( 3 ) To train our model , we compare the projected inputs to their class labels , represented by the prototypes P = { p1 , .. , pC } . That is , the overall training loss is given as the aggregation of distance-based individual losses ` between projected inputs and the prototype of their respective class labels : L ( θ ) = 1 N N∑ i=1 ` ( exp0 ( F ( x ; θ ) ) , pyi ) . ( 4 ) 3.2 Ideal prototypes and the penalized Busemann loss . The central idea of our approach is to position class prototypes at ideal points , which represent points at infinity in hyperbolic geometry . In the Poincaré model , the ideal points form the boundary of the ball : Id = { z ∈ Rd : z21 + · · ·+ z2d = 1 } . ( 5 ) Thus , the set of ideal points of hyperbolic space Bd is homeomorphic to the hypersphere Sd . As a consequence , any of the methods used for prototype embedding into Sd can be used to embed prototypes into Id . In particular , for d = 2 , prototypes can be placed uniformly on the unit sphere S2 , while class-agnostic prototype embedding based on separation can be used for d ≥ 3 [ 29 ] . A key issue that needs to be solved , is that ideal points are at infinite geodesic distance from all other points in Bd and hence the geodesic distance can not be used directly for prototype-based learning . Therefore , we propose the penalized Busemann loss for hyperbolic learning with ideal prototypes . The Busemann function , originally introduced in [ 9 ] ( see also Def . II.8.17 in [ 8 ] ) , can be considered a distance to infinity and may be defined in any metric space . Let p be an ideal point and γp a geodesic ray , parametrized by arc length , tending to p. Then the Busemann function with respect to p is defined for z ∈ Bd as : bp ( z ) = lim t→∞ ( dB ( γp ( t ) , z ) − t ) . ( 6 ) In the Poincaré model , the limit can be explicitly calculated and the Busemann function is given as : bp ( z ) = log ||p− z||2 ( 1− ||z||2 ) . ( 7 ) The full derivation of ( 7 ) is provided in Appendix A . Our proposed loss combines the Busemann function with a penalty term : ` ( z , p ) = bp ( z ) − φ ( d ) · log ( 1− ||z||2 ) , ( 8 ) with φ ( d ) a scaling factor for the penalty term which is a function of the dimension of the hyperbolic space . The first term in the penalized Busemann loss steers a projected input z towards class prototype p , while the second term penalized overconfidence , i.e. , values close to the ideal boundary of Bd . The prototypes are positioned before training and remain fixed , hence the backpropagation only needs to be done with respect to the inputs . The gradient of ` ( exp0 ( x ) , p ) with respect to x is given as : ∇x ` ( exp0 ( x ) , p ) = ( x−p ) tanh ( ||x|| ) ||x|| − tanh ( ||x|| ) p · x +1d p·x tanh ( ||x|| ) /||x|| − 1 ||x|| − tanh ( ||x|| ) p · x +tanh ( ||x||/2 ) , ( 9 ) with 1d a d-dimensional vector of ones . Figure 2 shows a color map of the penalized Busemann loss itself and its radial gradient for a prototype located at ideal point p = ( 1 , 0 ) . In the penalized Busemann loss , φ ( d ) governs the amount of regularization . Below , we show that this function should be linear in the number of dimensions . Theorem 1 . For any d ≥ 4 and ideal point p the penalized Busemann loss ` ( z , p ) is the log-likelihood of a density f ( z , p ) = 1C ( d ) exp ( − ` ( z , p ) ) on the Poincaré ball Bd if and only if φ ( d ) > d− 2 . Proof . To show that f ( z , p ) is a density , we have to show that the normalization constant C ( d ) = ∫ Bd exp ( − ` ( z , p ) ) dV is finite . Here , dV is the hyperbolic volume element on Bd , which is equal to 2d ( 1 − ||z||2 ) −ddz1 · · · dzd , see [ 34 ] . Furthermore , using invariance of the integrand under rotation , we can assume without loss of generality that p is equal to the unit vector e1 = ( 1 , 0 , . . . , 0 ) , which leads to C ( d ) = 2d ∫ Bd ( ( 1− ||z||2 ) φ ( d ) +1−d 1− 2z1 + ||z||2 ) dz1 · · · dzd . Switching to ( Euclidean ) hyperspherical coordinates ( r , ϕ1 , . . . , ϕd ) , the integral factorizes into C ( d ) = 2d · C1 ( d ) · C2 ( d ) , where C1 ( d ) = ∫ 1 0 ∫ π 0 ( ( 1− r2 ) φ ( d ) +1−d 1− 2r cos ( ϕ1 ) + r2 ) rd−1 sin ( ϕ1 ) d−2dr dϕ1 C2 ( d ) = ∫ π 0 sin ( ϕ2 ) d−3dϕ2 . . . ∫ π 0 sin ( ϕd−2 ) 2dϕd−2 · ∫ 2π 0 sin ( ϕd−1 ) dϕd−1 . Clearly , C2 ( d ) is finite , independently of d , such that it remains to consider C1 ( d ) . Instead of solving the integral exactly , we use the elementary estimate sin ( ϕ1 ) 2 ≤ 1− 2r cos ( ϕ1 ) + r2 ≤ 4 , valid for all r ∈ [ 0 , 1 ] . This estimate allows us to obtain both a lower and an upper bound on C1 ( d ) . The upper bound becomes C1 ( d ) ≤ ∫ 1 0 ( 1− r2 ) φ ( d ) +1−d rd−1dr ∫ π 0 sin ( ϕ1 ) d−4dϕ1 , which ( given that d ≥ 4 ) is finite if and only if φ ( d ) > d− 2 . The lower bound becomes C1 ( d ) ≥ 1 4 ∫ 1 0 ( 1− r2 ) φ ( d ) +1−d rd−1dr ∫ π 0 sin ( ϕ1 ) d−2dϕ1 , which ( given that d ≥ 2 ) is also finite if and only if φ ( d ) > d− 2 , which completes the proof . Given a trained model , inference is performed by projecting the test example x and selecting the class with the smallest prototype distance : y ? = argmin p∈P ` ( exp0 ( F ( x ; θ ) ) , p ) . ( 10 ) The Poincaré ball model of hyperbolic geometry has the conformal property , i.e. , angles in the Poincaré ball model coincide with angles in Euclidean space [ 35 ] . Thus , the cosine similarity between two ideal points remains a meaningful measure of their similarity , even in hyperbolic geometry . As the penalty term is the same for all prototypes , we can equivalently minimize the Busemann function of the prototypes directly . Moreover , as the Busemann function is a decreasing function of the cosine similarity , prediction is equivalent to hyperspherical prototype inference : y ? = argmax p∈P z ||z|| · p , z = exp0 ( F ( x ; θ ) ) . ( 11 )

Summary: Paper proposes an hyperbolic learning method based on ideal prototypes. The originality of the paper lies in the fact that there is no need to include prior information on the hierarchy to position the prototypes. There are located at the boundary of the Poincaré ball, and their position is computed by using algorithm defined in ref [28]. As points located close to the boundary have an infinite distance to the origin, authors propose to use a penalized busemann loss that can be defined in the hyperbolic space. The penalty term allows defining how far the embeddings can be from the origin. Paper also shows that, in 1d, it corresponds to logistic regression. Experiments are provided, considering CIFAR10 and 100 datasets, on classification tasks. Results show that the proposed method provides better results than the baselines when the dimension is low. When no information on the structure of the prototypes is given, the mAP indicator is better than when the prototypes are inferred from the data, while hierarchical metrics such as s-mAP and c-mAP show that the learnt prototypes perform better when privileged information is provided.