Title: Horospherical Decision Boundaries for Large Margin Classification in Hyperbolic Space
Venue: NeurIPS 2023
Accepted: yes
Keywords: Large-margin clssifier

Hyperbolic space

Horosphere

SVM

Geodesically convex

Global optimility

Busemann function
Abstract: Hyperbolic spaces have been quite popular in the recent past for representing hierarchically organized data. Further, several classification algorithms for data in these spaces have been proposed in the literature. These algorithms mainly use either hyperplanes or geodesics for decision boundaries in a large margin classifiers setting leading to a non-convex optimization problem. In this paper, we propose a novel large margin classifier based on horospherical decision boundaries that leads to a geodesically convex optimization problem that can be optimized using any Riemannian gradient descent technique guaranteeing a globally optimal solution. We present several experiments depicting the competitive performance of our classifier in comparison to SOTA.

Horospherical Decision Boundaries for Large Margin Classification in Hyperbolic Space

Abstract

Hyperbolic spaces have been quite popular in the recent past for representing hierarchically organized data. Further, several classification algorithms for data in these spaces have been proposed in the literature. These algorithms mainly use either hyperplanes or geodesics for decision boundaries in a large margin classifiers setting leading to a non-convex optimization problem. In this paper, we propose a novel large margin classifier based on horospherical decision boundaries that leads to a geodesically convex optimization problem that can be optimized using any Riemannian gradient descent technique guaranteeing a globally optimal solution. We present several experiments depicting the competitive performance of our classifier in comparison to SOTA.

1 Introduction

Hyperbolic space, a non-Euclidean space with constant negative curvature, has been shown [25, 23, 29, 24] to be effective for representing hierarchically organized data. For example, authors in [25] showed that a tree can be embedded in a hyperbolic space with arbitrarily small distortion. The main reason for this is that a hyperbolic space can be regarded as a continuous version of trees - the volume of the space grows exponentially as one moves away from the center in hyperbolic space. This matches the growth pattern of the number of nodes in a tree which grows exponentially as the depth of the tree increases. Hyperbolic space embedding has been shown to be a promising approach for representing data with a (latent) hierarchical structure [23, 24, 29, 15].

Recently, representation of data in hyperbolic space for the fundamental tasks of unsupervised and supervised learning has been popularized in various contexts, e.g., dimensionality reduction [10, 13], clustering [22], large-margin classifier [12, 32, 11], regression [20], etc. Existing 'linear' classifiers in hyperbolic spaces are predominantly based on geodesics i.e., using geodesics as decision boundaries. In [12], the decision boundary is chosen to be the intersection of the hyperboloid model and a hyperplane in the ambient space, which in this case is the Minkowski space. Then, the support vector machine (SVM) in hyperbolic space is formulated as a nonconvex optimization problem. In [32], authors followed the same parameterization of the hyperbolic geodesic decision plane and provided a series of algorithms to provably learn large margin classifiers in hyperbolic space. However, as pointed out by [11], the algorithm in [32] fails to converge in practice. Authors in [11] used the Poincaré ball model and parameterized the geodesic decision plane as a hyperplane mapped using the exponential map from the tangent space at some reference point. They first constructed convex hulls for each data cluster in the hyperbolic space and the reference point is then chosen to be the midpoint

between different convex hulls. Then they apply a Euclidean perceptron/SVM algorithm to data lifted into the tangent space at the aforementioned reference point. Although the optimization problem in the tangent space is convex, the procedure of the tangent space approximation introduces inaccuracies and distortions. Moreover, convex hull learning is highly unstable and their implementation is only applicable to the 2-dimensional hyperbolic space.

Finally, it is worth mentioning that linear classification within hyperbolic space, which can be considered as the last layer, referred to as the hyperbolic logistic regression (LR), is a fundamental component of hyperbolic neural networks (HNNs) [16, 27]. The calculation of logits in this layer is based on the distances between samples and the geodesic decision boundary. Notably, this hyperbolic LR employs a geodesic decision boundary but is not a large-margin classifier.

1.1 Horospherical Decision Boundaries for Classification in Hyperbolic Sapce

Horospheres , which are the level sets of the Busemann function in hyperbolic spaces, are the analogs of Euclidean hyperplanes [4]. Horospheres (horocycles) are contained in the Poincaré ball (disk) and are tangential to the ball (disk) at an ideal point as shown in Figure 1. A collection of horospheres centered at the same ideal point are parallel to each other and the lengths of geodesic segments between two horospheres are all equal, just as the lengths of line segments between parallel hyperplanes in Euclidean space are all equal. This property of horospheres was explored by [9] to develop a dimensionality reduction method for data in hyperbolic space. By using horospherical projection, they are able to preserve the distance information in the original data. However, there is no literature on constructing a 'linear' classifier in hyperbolic space using horospheres as the decision boundaries although the horospheres are the hyperbolic equivalent of Euclidean hyperplanes. Therefore, it is natural to consider the use of horospheres as decision boundaries for classifi-

cation in hyperbolic spaces. In this work, we propose a novel hyperbolic large-margin classifier using horospheres as decision boundaries in the Poincaré model. We term this classifier as a HoroSVM . The horospheres are well-defined in the Poincaré ball model. A toy example as shown in Figure 2 demonstrates the advantage of horospherical decision boundaries over geodesic decision boundaries. As the tree-structured data grows in depth, leaf nodes are embedded closer to each other within a subtree and among different subtrees. One of the classification problems in hyperbolic space is to determine whether a node belongs to a chosen subtree given the embedding. For comparison purposes, the decision boundaries of HoroSVM (Figure 2(a)) and hyperboloid SVM [12] (Figure 2(b)) are shown in the figure. As evident, the horospherical decision boundary perfectly separates (the root node is excluded in training) the data while the geodesic decision boundary makes several mistakes on both positive and negative samples. We present a novel formulation of the classification problem in the hyperbolic space as a geodesically convex optimization problem on a Riemannian manifold. This optimization problem can be easily solved using any Riemannian gradient descent technique guaranteeing global optimality. Gradient-based optimizations for geodesically convex problems guaranteeing global optimal solutions are the topic of investigation in optimization literature and we refer the reader to [35] for detailed convergence analysis of several such optimization methods. Further, we empirically validate our method on several real and synthetic data sets.

It should be noted that a horosphere decision boundary has been used in some recent works [31, 28] in constructing HNNs. For example, authors in [31] proposed hyperbolic neuron models using the Busemann function as a generalization of the Euclidean inner product to extract horosphere features from data. Authors in [28] proposed a shallow fully-connected continuous network spanned by (hyperbolic) neurons, on noncompact symmetric space (including hyperbolic space) using the Helgason-Fourier transform. Authors in [33] produced Euclidean features from hyperbolic embeddings via the eigenfunctions of the Laplace operator in the hyperbolic space where the eigenfunctions involve horosphere features. Note that none of the above works developed a large margin classifier using the horosphere as a decision boundary. To the best of our knowledge, our work is the first in the

(a) Horospherical decision boundary

(b) Geodesic decision boundary

Horocycle decision boundary

Geodesic decision boundary

+

subtree root

+

subtree root

negative samples

negative samples

positive samples

positive samples

literature to present a convex optimization formulation of a large-margin classifier using a horosphere decision boundary in a hyperbolic space.

The rest of this paper is organized as follows. In Section 2, we present some background on hyperbolic geometry pertinent to the work presented here. In Section 3, we present our horospherical boundarybased classification methods. Experimental results are presented in Section 4 to demonstrate the advantage of our HoroSVM over competing hyperbolic classifiers. Finally, we conclude in Section 5.

2 Background

In this section, we review some basic concepts of hyperbolic geometry including the generalization of the Euclidean hyperplane to the hyperbolic space namely, the horosphere.

2.1 Hyperbolic Space and the Poincaré Ball Model

There are five isometric models of the hyperbolic space: the Poincaré ball model, the Lorentz model, the Klein model, the upper-half space model, and the Hemisphere model [7]. We choose the Poincaré Ball model in this paper as it is easy to visualize and the Busemann function has a nice closed-form expression in this model. Note that the decision boundary of choice in our work is the level-set of the Busemann function namely, the horosphere.

An n -dimensional Poincaré Ball model, denoted by ( B n , g B ) , consists of all points in an open ball of radius 1, i.e., B n = { x ∈ R n : ‖ x ‖ < 1 } , and equipped with the Riemannian metric g B ( x ) = 4(1 -‖ x ‖ 2 ) -2 g R , where ‖·‖ is the Euclidean L 2 norm, and g R is the Euclidean metric.

2.2 Horospheres

Geodesics, geodesic rays, and ideal points The shortest path that connects two points in the Poincaré Ball model is called a geodesic segment . A geodesic ray is a geodesic segment that can be infinitely extended in one direction. We call the endpoint at infinity of a geodesic ray an ideal point . For B n , ideal points form the boundary of the ball: ∂ B n = S n -1 = { x ∈ R n : ‖ x ‖ = 1 } , where S n -1 is the ( n -1) -dimensional hypersphere. The hypersphere ( S n -1 , g S ) is a Riemannian manifold equipped with the Riemannian metric g S = 4(1 + ‖ x ‖ 2 ) -2 g R .

Busemann function [6] Let ω ∈ ∂ B n be an ideal point and γ ω : [0 , ∞ ) → B n a geodesic ray pointing ω . The Busemann function is defined as

In the Poincaré Ball model, Eq. (1) has a closed form : b ω ( x ) = -log 1 -‖ x ‖ 2 ‖ ω -x ‖ 2 .

Horospheres [6] In B n , a horosphere is a ( n -1) -dimensional sphere that is internally tangent to ∂ B n at an ideal point. For a given ω ∈ ∂ B n , the level sets of Busemann function b ω ( x ) in the Poincaré ball model is a series of horospheres tangent at ω . Horospheres are hyperbolic hyperplanes in the sense that the corresponding construction in Euclidean space gives a hyperplane.

Given an ideal point ω ∈ ∂ B n , the function defined by 〈 ω, x 〉 B : x ↦→ -b ω ( x ) is constant over horosphere tangent at ω . 1 Hence any horosphere can be parameterized with an ideal point ω and an offset value b of the level set.

Let Π denote the set of horospheres of B n . Ahorosphere π ∈ Π can be parameterized by 0 < µ ∈ R + 2 , ω ∈ S n -1 , and b ∈ R as

We will use π or π · to represent a horosphere in different parameterizations hereafter.

3 Horospherical Boundary-based Classification

In this section, we present the key theoretical contributions of our work namely, a horosphere-based SVM classifier that involves formulating and solving a geodesically convex optimization problem. First, we present some preliminary facts and results about the horospheres. Then, we present the optimization problems for the horospherical perceptron and SVM along with analysis.

3.1 Point to Horosphere Distance

While the distance from the origin o to a horosphere has been known for decades [18] (Introduction 4.1, p.31), we present a natural generalization of previous results by providing a closed-form expression for measuring the hyperbolic distance from any arbitrary point x ∈ B n to a given horosphere π µ, ω ,b . The following remark provides this result.

Proposition 3.1. Let π µ, ω ,b be a horosphere. The hyperbolic distance of a point x ∈ B n to a horosphere π µ, ω ,b is given by

Notice that it shares a similarity to the Euclidean distance of a point to a hyperplane. Before presenting the proof for Proposition 3.1, we recall the following Fact 3.2 and Lemma 3.3 from [31].

Fact 3.2. Given an ideal point ω and a point x ∈ B n , there is a unique horosphere passing through x and tangent at ω .

Lemma 3.3. [31] Let Π ω be the set of horocycles of B n tangent at ω . Given λ ∈ R , let π λ, ω be the unique horosphere that passes through tanh( λ/ 2) · ω and tangent at ω . Note that Π ω = ∪ λ ∈ R { π λ, ω } . We have the following two results: (i) the hyperbolic lengths of geodesic (that pass through ω ) segments between π λ 1 , ω and π λ 2 , ω are equal to | λ 1 -λ 2 | ; (ii) 〈 ω , x 〉 B = λ for any x ∈ π λ, ω .

Figure 3 shows a 2D Poincaré disk model B 2 and its boundary S 1 . The point o is the origin of the disk, x ∈ B 2 is a point, and ω ∈ S 1 is a point at infinity (an ideal point). Two geodesics ending at the same ω from x and o respectively are shown in the figure (black solid line/curve). The circle π µ, ω ,b is a given horocycle tangent (red solid circle) at ω . The hyperbolic distance d B ( x , π µ, ω ,b ) between x and π µ, ω ,b is identified as the distance between x and y x where y x is the projection of x to π µ, ω ,b along the geodesic ending at ω . Let π x (red dashed circle) be the unique horocycle that passes through x and is tangent at ω . Note that the lengths of all geodesic segments between two horocycles are the same. That is, d B ( x , π µ, ω ,b ) = d B ( x , y x ) = d B ( x 0 , y 0 ) , where x 0 , y 0 are horocyclic projections [9] of x , y x along π x and π µ, ω ,b respectively.

Proof of Proposition 3.1. Given a point x ∈ B n and an ideal point ω ∈ S n -1 , let π ∈ Π ω be a horosphere tangent at ω and π x ∈ Π ω be the unique horosphere that passes through x . Write 〈 ω , x 〉 B = λ x and then x 0 = tanh( λ x / 2) · ω is the horospherical projection of x along π x . For consistency in notation, we write π x as π λ x , ω . Since the horosphere π µ, ω ,b can be reparameterized as π λ, ω , where λ = b µ , the hyperbolic distance from x to π µ, ω ,b is the hyperbolic distance between π λ x , ω and π λ, ω which is | λ x -λ | = | µ 〈 ω , x 〉 B -b | µ (by Lemma 3.3). This completes the proof. glyph[squaresolid]

3.2 Horospherical Decision Boundaries

We consider classification problems in hyperbolic space of the following form: X ⊂ B n denotes the feature space and Y = {± 1 } denotes the binary label space. In the following, we denote the training set by S ⊂ X × Y . The decision rule using a horosphere as its decision boundary can be written as the following function f : X ↦→ Y where

The positive samples are expected to lie inside a horosphere while the negative samples are expected to lie outside a horosphere. This is analogous to the linear decision boundary in Euclidean space and we will build a horospherical perceptron and a horospherical SVM based on this decision boundary.

It should be noted that in R n the hyperplane ξ a, w ,b = { z ∈ R n | a 〈 w , z 〉 -b = 0 } where a ∈ R + , b ∈ R , w ∈ S n -1 is the hyperplane ξ a, -w , -b . However, the horospheres π µ, ω ,b and π µ, -ω , -b respectively represent two distinct horospheres, centered at ω and -ω respectively. Let Π + = { π µ, ω ,b ∈ Π | b > 0 } and Π -= { π µ, ω ,b ∈ Π | b < 0 } . Thus, Π + , Π -⊂ Π and the radius of π ∈ Π + is less than 1 / 2 and the radius of π ∈ Π -is greater than 1 / 2 . In most cases of classification in hyperbolic space, the positive samples are clustered near the boundaries. Hence, we restrict ourselves to finding a horosphere π ∈ Π + that separates data, instead of searching over Π . Intuitively, we are looking for a 'small' horosphere that captures the positive samples. We are now ready to present the Horospherical Perceptron followed by the Horospherical SVM.

3.3 Horospherical Perceptron

The loss function for the proposed horospherical perceptron is given by

which is zero when the instance is classified correctly and is proportional to the signed distance of the instance from the horosphere when it is misclassified. The empirical loss for a given data set S is

Hence, the optimal horosphere is learned by solving the above optimization problem on the manifold R + × S n -1 × R + , i.e.,

To further analyze this optimization problem, we first recall some facts about geodesic convexity.

Definition 3.4. (Geodesically convex sets [30]). Let ( M , g ) be a Riemannian manifold. A set A ⊆ M is said to be a geodesically convex set if, for any two points p , q ∈ A , the geodesic γ pq that connects them is contained in A .

Definition 3.5. (Geodesically convex/concave functions [30]) Let A ⊆ M be a geodesically convex set. A function f : A → R is said to be a geodesically convex function if, for any p , q ∈ A the composition f ◦ γ pq : [0 , 1] → R is a convex function, where γ pq : [0 , 1] →M is a geodesic that connects p , q . f is said to be a geodesically concave function if -f is a geodesically convex function.

Theorem 3.6. [30] Let A ⊆ M be a geodesically convex set. A function f : A → R is geodesically convex if and only if its epigraph epi ( f ) = { ( p , c ) | f ( p ) ≤ c } ⊂ A × R is a convex set.

Now we present the main theoretical result of this paper.

Theorem 3.7. For a given training data sample { x , y } ∈ S, 0 < ‖ x ‖ < R < 1 (the hyperbolic feature x neither lie on the center nor lie on the boundary), l ( µ, ω , b ; x , y ) is a geodesically convex function on R + × A × R + and is a geodesically concave function on R + × B × R + , where

Note that both A and B are geodesically convex sets.

Proof. Let l ( µ, ω , b ; x , y ) = max(0 , -g ( µ, ω , b ; x , y )) where g ( µ, ω , b ; x , y ) = y · ( µ 〈 ω , x 〉 B -b ) . Since max(0 , -a ) is a convex function in a ∈ R and g ( · ) is linear in µ and b , we only need to show that g ( · ) , as a function of ω ∈ S n -1 , is geodesically convex (concave). Without loss of generality, let µ = 1 and b = 0 . Also note that y is the label of data that takes values from {-1 , +1 } which may flip the inequality. It suffices to validate results for the positive sample, i.e, y = 1 .

With a slight abuse of notation, let g ( ω ; x ) = g (1 , ω , 0; x , 1) = 〈 ω , x 〉 B = ln 1 -‖ x ‖ 2 ‖ ω -x ‖ 2 defined on

A = { ν ∈ S n -1 ∣ ∣ x T ν ‖ x ‖ > 0 } ⊂ S n -1 . Since -ln( · ) is decreasing and convex, we only need to check h ( ω ; x ) = ‖ ω -x ‖ 2 is geodesically convex on A , i.e, check that the epigraph of h is a convex set. Note that x T ω ‖ x ‖ = cos( θ ω ) for ω ∈ S n -1 where θ ω = ∠ ( ω , x ) .

where d ( c ) is a real number depending on c and ‖ x ‖ and A d = { ( ω , c ) | cos( θ ω ) ≥ d ( c ) } is the collection of unit vectors where the angle between the vectors given the data x is small i.e., restricted to a small region on the sphere. The last equality follows from the definition of A and A d : if d ( c ) ≤ 0 , then { ω : cos( θ ω ) ≥ d ( c ) } ∩ A = A . Similarly, if d ( c ) > 0 , { ω : cos( θ ω ) ≥ d ( c ) } ∩ A = { ω ∈ S n -1 | cos( θ ω ) ≥ d ( c ) } := A d ( c ) . Both A and A d are geodesically convex sets and this completes the proof. glyph[squaresolid]

The convex sets A , B are the hemispheres of S n -1 separated by the hyperplane { z ∈ R n | x T z = 0 } (the hyperplane has x as its normal vector) in its ambient space R n . Theorem 3.7 tells us that given one data sample ( x , y ) , the optimal value described in Eq. (7) exists in R + × A × R + , and it is globally optimal. For a collection of training samples S = { ( x i , y i ) } N i =1 , let A i = { ν ∈ S n -1 ∣ ∣ y i · x T i ν ‖ x i ‖ > 0 } . If the data are separable by a horosphere, it follows that ∩ N i =1 A i is nonempty and convex. Then the loss function given S is geodesically convex on R + ×∩ N i =1 A i × R + and the global optimum can be obtained using any gradient-based optimization. Numerically, we apply a Riemannian gradient descent method on the entire space R + × S n -1 × R + since g ( µ, ω , b ; x , y ) is continuous.

3.4 Horospherical SVM

Given a horospherical decision boundary π µ, ω ,b parameterized by ω ∈ S n -1 , µ ∈ R + , and b ∈ R , the margin γ is the minimal distance from training samples S to the decision boundary:

The maximum margin classifier can be obtained by solving the following optimization problem:

Theorem 3.8. The maximum margin classification problem in hyperbolic space with horosphere as its decision boundary described in Eq. (11) is equivalent to the following optimization problem:

The proof is analogous to that in the Euclidean case [3]. Note that the margin is unchanged if we apply the following scale transformation: µ → µ/γ and b → b/γ . We can also build a soft-margin horospherical SVM, dubbed HoroSVM, by minimizing the following loss function:

where C is a hyperparameter that controls the tradeoff between minimizing misclassification and maximizing margin.

It is easy to see that the same result as in Theorem 3.7 holds for the loss function in Eq. (13). That is, the loss function is a geodesically convex function on one geodesically convex subset of the parameter space and a geodesically concave function on the other geodesically convex subset of the parameter space. Recall that the idea behind proving that the loss function in the horospherical perceptron, max(0 , -g ( · )) , where g ( · ) = y · ( µ 〈 ω , x 〉 B -b ) , is geodesically convex is based on the important fact that max(0 , -a ) is a convex function in a ∈ R . Similarly, the same idea applies to HoroSVM, where the hinge loss is used, and the loss function becomes 1 2 µ 2 + max(0 , 1 -g ( · )) . Note that max(0 , 1 -a ) is also a convex function in a ∈ R and 1 2 µ 2 is a convex function in µ , these facts complete the proof of the desired property for the HoroSVM, which is analogous to Theorem 3.7 for horospherical perceptron.

Wecan then apply any Riemannian gradient descent optimization methods for updating the parameters in HoroSVM since the problem is an optimization problem over a product space of Riemannian manifolds, R + × S n -1 × R + . We refer the readers to [5] and [1] for more details about optimization techniques on Riemannian manifolds.

4 Experiments

In this section, we present several experimental results obtained from an application of our HoroSVM to synthetic data as well as real data sets used in published literature. Our implementation is based on Pymanopt [19] using the Riemannian conjugate gradient method [26] on Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz.

4.1 Network Data Set

Here, we follow the experimental setup in [12], and evaluate our HoroSVM over four real-world network data sets used by [8]: karate [34] (2 classes, 34 nodes ), polblogs [2] (2 classes, 1224 nodes ), polbooks 3 (3 classes, 105 nodes ), and football [17] (12 classes, 115 nodes ).

The network data is embedded in a 2D hyperbolic space using the method of [8]. Given the hyperbolic embeddings, we compare our HoroSVM with three other competing large margin classifiers: Euclidean SVM (even though it violates the hyperbolic geometry), hyperboloid SVM [12], and Poincaré SVM [11]. The absence of comparison with [32] in this experiment is due to two reasons. First, [32] aims to provide a theoretical understanding of hyperbolic spaces in classification, focusing on 'linearly' separable data (data that can be separated by a geodesic). They do not address extensions to nonlinearly separable data, which limits their applicability to many practical datasets. Second, for datasets that are linearly separable in hyperbolic space, the approach in [32] aligns with the hyperboloid SVM when adversarial training is not applied in [32].

For multiclass classification, a one-verses-rest strategy is applied. We conducted a five-fold crossvalidation on each data set, where we chose the hyperparameter C from { 1 , 5 , 10 } during the cross-validation procedure. Note that a more extensive search space for C may lead to potential performance improvements. The mean of the F1 score followed by the standard deviation over five

trials are summarized in Table 1. As evident from the table, our method yields the best results on all the data sets.

HoroSVM outperformed other methods on all four data sets. The data in karate are well-separated and thus both Euclidean SVM and hyperboloid SVM performed equally well. Our method outperforms the others since the horospheres have several nice properties, the most important of which is that the Busemann function whose level sets are the horospheres is a convex function that guarantees global optimality in the optimization. Notice that the performance of Poincaré SVM on karate is inferior to others by a significant amount. The reason is that the performance of Poincaré SVM is sensitive to the choice of the reference point. We demonstrate our performance gain on the remaining data sets, and our method is more consistent, compared to Euclidean SVM and hyperboloid SVM, in terms of lower standard deviation, on football data set where data exhibit a larger variance/spread.

4.2 Subtree Classification in WordNet

A task of considerable interest in hyperbolic space classification problems is to determine whether a node belongs to a given subtree in the hyperbolic embedding. We obtained hyperbolic embeddings in various dimensions using the approach in [15] for WordNet 4 noun hierarchy (82,115 nodes). We consider four subtrees whose roots are the following synsets: ANIMAL.N.01 , GROUP.N.01 , WORKER.N.01 , and MAMMAL.N.01 .

We split all nodes in a subtree into positive training (80%) and test (20%) nodes and applied the same process to the remaining WordNet nodes to create negative training and test sets. The average F1 scores and the standard deviations over 3 trials are shown in Table 2. The number of positive training/test samples of each data set are listed as well. We exclude Poincaré SVM from the comparisons in this task. The reason being, data are highly imbalanced in this task and the positive samples are clustered near the boundary. The reference point learned in Poincaré SVM will be close to the boundary where the tangent approximation of data at this reference point is highly distorted, as opposed to the original hyperbolic embeddings. It is therefore hard to locate a hyperplane in the

tangent space that separates the lifted (mapped) data. In addition, the learning of the reference point is only applicable to 2D hyperbolic space.

As well known in the Euclidean SVM literature, a vanilla (unweighted) implementation of SVM performs poorly on extremely imbalanced data, we observed the same behavior in training HoroSVM on this task. We preprocessed the data by downsampling the majority class of samples (the negative samples) to train a robust model. Note that our HoroSVM can be naturally extended to a class/instanceweighted version by assigning a class/instance weight to the penalty term C for each sample, allowing us to address more general imbalanced data. Since there is no protocol for dealing with unbalanced data in training a hyperboloid SVM, we presented the Euclidean SVM results using the same preprocessed data for reference. In addition, we presented results of hyperbolic logistic regression (LR) [15], which is not a large-margin classifier on this task, where the imbalanced data is handled by sampling the equal number of negative and positive nodes in each mini-batch of size 16 during training.

Now, we highlight several results in Table 2. The superior performance of hyperboloid SVM over hyperbolic LR is expected, as both methods use geodesic decision boundaries but hyperboloid SVM aims to maximize the margin. However, the training of hyperboloid SVM is highly unstable as we mentioned earlier due to the non-convex optimization process. Euclidean SVM under-performs as it does not take into account the hyperbolic geometry. Our HoroSVM exhibits a significant improvement in predicting words in a subtree, as evidenced by higher F1 scores across all the subtrees. The small number of nodes within a subtree, compared to the whole WordNet, causes the nodes to cluster near the boundary in their hyperbolic embeddings. Thus, horosphere is an ideally suited decision boundary (in comparison to the geodesic boundary in [12]) to isolate the subtree.

4.3 Synthetic Data with Noisy Labels

To demonstrate the robustness of our HoroSVM, we apply it to synthetic data with noisy labels at varying levels/amounts of noise. Specifically, we generated 100 synthetic datasets by sampling from a Gaussian mixture model defined on the Poincaré disk model as in [12]. The isotropic Gaussian distribution in hyperbolic space is referred to as the Riemannian normal distribution, and we used the sampling method presented in [21]. For each dataset, we

1.00

1.00

0.95

0.95

0.90

0.90

0.85

0.85

score

0.80

0.75

score

0.80

0.75

Training F1 s

0.70

0.70

0.65

0.65

0.60

0.60

0.55

+

Hyperboloid SVM

0.55

Hyperboloid SVM

0.50

0.45

HNN

HoroSVM

0.50

0.45

HNN

HoroSVM

0.40

0

10

20

30

40

50

0.40

0

10

20

30

40

50

Noise level (%)

Noise level (%)

sampled two centroids from a zero-mean Riemannian normal distribution with a variance of 1.5. We then sampled 200 data points from a unit-variate Riemannian normal distribution centered at each centroid, resulting in a dataset of 400 points classified into positive and negative classes. We split the dataset into training and test sets with 100 positive/negative samples in the training set and 100 positive/negative samples in the test set. We then generated datasets with noisy labels at noise levels: η ∈ { 0 , 0 . 05 , 0 . 1 , . . . , 0 . 5 } by flipping the labels of a proportion η of the training (not test) samples, with an equal number of positive and negative samples flipped. The train/test average F1 scores of each method across all datasets at varying noise levels are shown in Fig 4. We compared our HoroSVM with hyperboloid SVM, hyperbolic LR, and a two-layer HNN [15] (with a hidden dimension of 5). While all methods depict decreasing training F1 scores as the noise level increases, HoroSVM outperforms the others consistently throughout the training process. Hyperbolic LR and HNN exhibit the least resistance to label noise, with test F1 scores dropping (faster) with increasing noise level. Both hyperboloid SVM and HoroSVM demonstrate consistent performance across different noise levels, owing to the inherent robustness of large-margin classifiers. However, the training of hyperboloid SVM is highly unstable resulting in its inferior performance. HoroSVM demonstrates its superiority in accuracy and robustness as evidenced in the results. In addition, we present the average training times for each method on one dataset (200 samples) as follows: 6.57 seconds (Hyperboloid SVM), 3.98 seconds (Hyperbolic LR), 9.06 seconds (HNN), and 3.73 seconds (HoroSVM). Notably, HoroSVM stands out as the fastest.

5 Discussion and Conclusions

In this paper, we presented a novel large margin classifier, dubbed HoroSVM, whose decision boundaries are horospheres that are the level sets of a Busemann function. We presented a novel formulation leading to the optimization of a geodesically convex loss performed using a Riemannian gradient-based method and guaranteeing a globally optimal solution. We demonstrated superior to competitive performance of the HoroSVM over SOTA large margin classifiers.

In Euclidean space, a kernel SVM is usually favored over the linear SVM due to its ability to cope with non-linearly separable data. In Hyperbolic space, the challenge lies in developing valid positive definite kernels (see [14] for details on validity of kernels on Riemannian manifolds). The only reported work on KSVM in hyperbolic space that we are aware of is [12], which uses a kernel that violates the positive definiteness property of RKHS kernels. Thus the problem of interest is primarily defining a valid family of kernels in hyperbolic space. We will address this in our future work.

References

P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. In Optimization Algorithms on Matrix Manifolds . Princeton University Press, 2009.

Lada A Adamic and Natalie Glance. The political blogosphere and the 2004 us election: divided they blog. In Proceedings of the 3rd international workshop on Link discovery , pages 36-43, 2005.

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

Roberto Bonola. Non-Euclidean geometry: A critical and historical study of its development . Courier Corporation, 1955.

Nicolas Boumal. An introduction to optimization on smooth manifolds. Available online, May , 3, 2020.

Martin R Bridson and André Haefliger. Metric spaces of non-positive curvature , volume 319. Springer Science & Business Media, 2013.

James W. Cannon, William J. Floyd, Richard Kenyon, and Walter R. Parry. Hyperbolic Geometry , volume 31. MSRI Publications, 1997.

Benjamin Paul Chamberlain, James Clough, and Marc Peter Deisenroth. Neural embeddings of graphs in hyperbolic space. arXiv preprint arXiv:1705.10359 , 2017.

Ines Chami, Albert Gu, Dat P Nguyen, and Christopher Ré. Horopca: Hyperbolic dimensionality reduction via horospherical projections. In International Conference on Machine Learning , pages 1419-1429. PMLR, 2021.

Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. Advances in Neural Information Processing Systems , 32:4868-4879, 2019.

Eli Chien, Chao Pan, Puoya Tabaghi, and Olgica Milenkovic. Highly scalable and provably accurate classification in poincaré balls. In 2021 IEEE International Conference on Data Mining (ICDM) , pages 61-70. IEEE, 2021.

Hyunghoon Cho, Benjamin DeMeo, Jian Peng, and Bonnie Berger. Large-margin classification in hyperbolic space. In The 22nd international conference on artificial intelligence and statistics , pages 1832-1840. PMLR, 2019.

Xiran Fan, Chun-Hao Yang, and Baba C Vemuri. Nested hyperbolic spaces for dimensionality reduction and hyperbolic nn design. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 356-365, 2022.

Aasa Feragen, Francois Lauze, and Soren Hauberg. Geodesic exponential kernels: When curvature and linearity conflict. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3032-3042, 2015.

Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In International Conference on Machine Learning , pages 1646-1655. PMLR, 2018.

Octavian-Eugen Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in Neural Information Processing Systems 31 , pages 5345-5355, 2019.

Michelle Girvan and Mark EJ Newman. Community structure in social and biological networks. Proceedings of the national academy of sciences , 99(12):7821-7826, 2002.

Sigurdur Helgason. Groups and geometric analysis: integral geometry, invariant differential operators, and spherical functions , volume 83. American Mathematical Society, 2022.

Niklas Koep and Sebastian Weichwald. Pymanopt: A python toolbox for optimization on manifolds using automatic differentiation. Journal of Machine Learning Research , 17:1-5, 2016.

Gian Marconi, Carlo Ciliberto, and Lorenzo Rosasco. Hyperbolic manifold regression. In International Conference on Artificial Intelligence and Statistics , pages 2570-2580. PMLR, 2020.

Emile Mathieu. Charline le lan, chris j maddison, ryota tomioka, and yee whye teh. continuous hierarchical representations with poincaré variational auto-encoders. Advances in Neural Information Processing Systems , pages 12544-12555, 2019.

Nicholas Monath, Manzil Zaheer, Daniel Silva, Andrew McCallum, and Amr Ahmed. Gradient-based hierarchical clustering using continuous representations of trees in hyperbolic space. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 714-722, 2019.

Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. Advances in neural information processing systems , 30:6338-6347, 2017.

Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation tradeoffs for hyperbolic embeddings. In International conference on machine learning , pages 4460-4469. PMLR, 2018.

Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. In International Symposium on Graph Drawing , pages 355-366. Springer, 2011.

Hiroyuki Sato. Riemannian conjugate gradient methods: General framework and specific algorithms with convergence analyses. SIAM Journal on Optimization , 32(4):2690-2717, 2022.

Ryohei Shimizu, YUSUKE Mukuta, and Tatsuya Harada. Hyperbolic neural networks++. In International Conference on Learning Representations , 2020.

Sho Sonoda, Isao Ishikawa, and Masahiro Ikeda. Fully-connected network on noncompact symmetric space and ridgelet transform based on helgason-fourier analysis. In International Conference on Machine Learning , pages 20405-20422. PMLR, 2022.

Alexandru Tifrea, Gary Becigneul, and Octavian-Eugen Ganea. Poincare glove: Hyperbolic word embeddings. In International Conference on Learning Representations , 2018.

Constantin Udriste. Convex functions and optimization methods on Riemannian manifolds , volume 297. Springer Science & Business Media, 2013.

Ming-Xi Wang. Laplacian eigenspaces, horocycles and neuron models on hyperbolic spaces. 2021.

Melanie Weber, Manzil Zaheer, Ankit Singh Rawat, Aditya K Menon, and Sanjiv Kumar. Robust largemargin learning in hyperbolic space. Advances in Neural Information Processing Systems , 33:17863-17873, 2020.

Tao Yu and Christopher De Sa. Hyla: Hyperbolic laplacian features for graph learning. arXiv preprint arXiv:2202.06854 , 2022.

Wayne W Zachary. An information flow model for conflict and fission in small groups. Journal of anthropological research , 33(4):452-473, 1977.

Hongyi Zhang and Suvrit Sra. First-order methods for geodesically convex optimization. In Conference on Learning Theory , pages 1617-1638. PMLR, 2016.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the support vector machines for data with latent hierarchical relationships, which are represented in hyperbolic spaces. Similar to the linear SVM in Euclidean spaces, the SVM in hyperbolic spaces have been seen in the literature but they are challenging due to some issues such as non-convexity or algorithmic convergence. In this work, the authors explored the analogs between horospheres and Euclidean hyperplanes, and proposed to use geodesic segment (counterpart of the Euclidean distances) to define the margin for the SVM. A nice theory shows the resulting problem is convex so a stable algorithm is developed. Numerical studies demonstrate the superior performance over some competitors.\n\nSTRENGTHS:\nThis paper is working on an integral question for classifying network data. It introduces an interesting idea of defining the SVM on horospheres and shows good performance on benchmark data. In general, the paper appears to be technically correct with sound theory for showing convexity.\n\nWEAKNESSES:\n1. The authors may comment on the computational speed of the proposed algorithm, as it may outperform Hyperboloid SVM due to its convexity. \n2. From Table 2, it seems HoroSVM well outperforms the other classifiers when D = 2, and the advantage gets smaller when D is large. What is the performance of HoroSVM when D is larger?  Furthermore, it would be useful to discuss how D is chosen in real-world scenarios. \n3. The authors should also compare with the kernel SVM to see if the latent data structure can be implicitly handled by the kernel SVM.\n4. Most datasets chosen for the study display a significant imbalance. It would be interesting to see if it is possible to incorporate cost-sensitive weights into these observations. Doing so could mirror the approach taken by weighted SVM to tackle the imbalance issue.\n\nQUESTIONS:\nIn Section 4.1, the hyperparameter C is selected from {1, 5, 10}. It might be helpful to comment on how sensitive this parameter is. The performance of a linear SVM is typically highly sensitive to this parameter, and the paper could potentially mislead readers by implying that a non-rigorous selection from only three candidate values would suffice for optimal performance.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper presented a novel large margin classifier, dubbed HoroSVM, whose decision boundaries are horospheres in hyperbolic space, and proved it’s a convex optimization problem.
This paper presented several experiments depicting the competitive performance of the classifier in comparison to SOTA.

STRENGTHS:
1. This paper presented a novel large margin classifier, dubbed HoroSVM, whose decision boundaries are horospheres in hyperbolic space. It’s innovative compared to the predecessors.
2. This paper systematically and clearly explain the proof, which is easy to follow.

WEAKNESSES:
1. The experiments “Synthetic Data with noisy Labels” are over synthetic data, which is not convincing enough compared to apply random perturbations over real-world dataset
2. The motivation of classification in hyperbolic space is explained inexplicit.

QUESTIONS:
The method in this paper and the compared methods are basically based on SVM. Wouldn’t the result better if neural networks are applied to classify in hyperbolic space?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a large margin classifier in hyperbolic space, Poincare ball models. To this end, horospherical decision boundaries (which are based on the Buseman function and are different level sets of the Busmann function at the ideal point) are used for the large-margin classifier. They also show that the formed classifier is convex and optimization can be performed using Riemannian gradient descent. They perform experiments on four datasets and compare the results with Euclidean Hyperboloid, Poincare, and Horo SVM.\n\nSTRENGTHS:\nThe paper proposes a large margin classifier based on the horospheres. The research question is valid and the paper is well motivated, and well written. The paper also provides the proofs for the theoretical claims as well. Figures are also beneficial to understand, for example figure 2 is beneficial to compare the horocycle vs geodesic decision boundary. \n\nThe experiments also show the efficacy of the proposed approach specially on 2dimensional embedding space. Synthetic data with noisy labels is also an interesting experiment showing the robustness of the proposed approach and generally hyperbolic version to the noisy data.\n\nThe analysis on the performance on the experiments section is beneficial as well.\n\nWEAKNESSES:\nThere is one main question about comparison with [33]. The paper generally is well written , there a few questions which I will add to the questions section. \n\nI would suggest the authors to do a proofreading. The paper discusses horosphere decision boundary in several different ways like horospheres decision boundary, horocycle decision boundary, and horospherical decision boundary. Using a fixed terminology throughout the paper can increase the comprehensiveness of the paper.\n\nQUESTIONS:\nWhy does not the paper compare the method with [33] in the experiments? \nLines 176- 184 are not very clear. How are the positive and negative $\\pi$ subsets of the same one? shouldn't they be on the opposite sides of the Poincare ball model? Why do the authors claim that the positive samples are clustered near the boundaries? \n\nWhat is the reference point in Poincare SVM? and why is it sensitive to the reference point?\n\nLIMITATIONS:\nThe paper discusses future work shortly in the conclusions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBased on recent successes with hyperbolic embeddings of data with a (latent) hierarchical structure, this paper proposes a new type of SVM in hyperbolic space. Their SVM, named HoroSVM, uses horospheres as decision boundaries and the authors derive a way to compute the distance of a point to such a horosphere. This makes the method different from other recently proposed hyperbolic SVMs, which use geodesic hyperplanes as decision boundaries. The authors argue that these horospheres are a more suitable generalization of hyperplanes than their geodesic-based counterparts and that, therefore, HoroSVM is likely to outperform the other methods.\n\nAfter introducing their method, the authors show that their loss function is geodesically convex with respect to the learnable parameters. As a result, they state that a global optimum to the optimization problem, that arises when using HoroSVM, can be found using gradient-based optimization methods. They further state that, while their method has this nice property, the other methods to which they compare do not. Therefore, HoroSVM should again yield better results.\n\nWith a series of three experiments, the authors attempt to empirically show the superiority of their method over geodesic hyperplane methods and Euclidean SVM. The first experiment aims to show the superior performance on the classification of hyperbolic embeddings of network data. The second experiment shows that HoroSVM is better than its competitors at predicting the subtree to which nodes belong within WordNet. The third and final experiment shows that their method is more robust than their competitors to noisy labels.\n\nLastly, based on their theoretical analysis and empirical results, the authors conclude that their method is more performant and robust than its competitors, while also alluding to future work with hyperbolic kernel SVMs.\n\nSTRENGTHS:\nThe paper contains several, mostly theoretical strengths in my opinion:\n\n1. The paper derives a way to compute distances from points on the Poincaré ball to horospheres. While directly relevant to their HoroSVM method, this formulation is likely useful to any other method that implements these horospheres on the Poincaré ball, making it a nice contribution.\n2. The authors prove the geodesic convexity of their newly proposed loss function, resulting in a very strong statement about the possibility of finding a global optimum to their optimization problem. If it is indeed the case that the competitive hyperbolic methods do not have this property, then this is a very clear theoretical argument for the superiority of their method w.r.t. these other hyperbolic methods.\n3. In my opinion the paper introduces an interesting theoretical analysis of their method, which could by itself be useful as a tool for studying other methods in hyperbolic space or even on other Riemannian manifolds.\n4. The experiments show the superiority of their method with respect to their competitors both w.r.t. classification performance and noisy label robustness.\n\nWEAKNESSES:\nThere are a few concerns that I have with this paper, which mostly have to do with the motivation for using horocycles over geodesic hyperplanes and with the setup of the experiments. Firstly, my concerns with the motivation for the use of horocycles are:\n\n1. There are multiple geometric definitions of hyperplanes possible and it is not clear which one is chosen here. However, it is stated that horocycles are the hyperbolic equivalent of Euclidean hyperplanes and that, therefore, horocycles are a natural choice for decision boundaries. This does not provide a clear motivation for choosing horocycles over geodesic hyperplanes. What is the geometric motivation for this choice and why should it lead to better results in practice?\n2. Figure 2 is supposed to provide a clear motivation for why horocycles are better as decision planes than geodesic hyperplanes, but the geodesic hyperplane example (on the right) seems poorly chosen to me. In fact, it seems very easy to find a hyperplane by hand that would lead to significantly better results than the hyperplane in the figure. What is going on here? Is this really a problem with geodesic hyperplanes or is it a problem with the optimization process. If the latter is the case, then this is still not a direct argument for using horocycles over geodesic hyperplanes.\n\n\nMy concerns regarding the experiments (mostly experiment 1) are:\n\n1. In the experiment from Subsection 4.1, it is unclear how the hyperbolic embeddings of the network data were generated. As the method for generating these embeddings is likely to affect the outcome of the experiment, it is difficult to judge its validity without a description hereof.\n2. Moreover, after the 5-fold cross-validation, the differences between the results obtained with the different methods are quite small for at least two of the datasets, especially compared to the standard deviation. This makes the experiment seem a bit weak as a motivation for using HoroSVM.\n3. Also, given that solving the optimization problem for HoroSVM leads to a global optimum, why do you not simply choose the optimal C? And in that case, would the standard deviation not simply be 0?\n4. What happens when you use a Euclidean embedding of the data and then Euclidean SVM? Even though the Euclidean embeddings would be distorted, it seems plausible to me that Euclidean SVM could obtain better results in such a setting. The current comparison does not seem completely fair to me.\n5. In the second experiment the embeddings are obtained through the application of hyperbolic entailment cones. However, there are some issues with this method and there is a method from the paper \"Representation Tradeoffs for Hyperbolic Embeddings\" by de Sa et al. which boasts greater performance than hyperbolic entailment cones. How do the results change if this method had been used to generate the embeddings?\n6. Lastly, in Table 2 the focus is put on the case of low dimension D = 2, where HoroSVM is significantly better. However, for this low dimension the performance of every method is rather low and for higher dimensions the difference in performance seems to fade. Is there a reason to still focus on low dimensions? What happens if the dimension is chosen even greater? Does the advantage of HoroSVM disappear completely?\n\nQUESTIONS:\nAlongside the questions posed in the weaknesses above, I have a few additional questions:\n\n1. Why are there both a mu and a b parameter in equation (2)? You only need 1 parameter here. Is it for notational simplicity later on? If so, a small note might help to clarify this for the reader. If not, removing this over-parameterization would also make the different versions of pi with subscripts a bit easier to follow.\n2. Theorem 3.7 tells us that, for a single sample, the loss function is geodesically convex w.r.t. the learnable parameters. Is this result strong enough to guarantee convergence to a global optimum in case of a collection of samples? Based on the discussion in the introduction I presume it is, but I think it would be nice to mention this for the reader.\n3. Why is the Poincaré inner product named an inner product? This name seems confusing to me as it suggests that it actually has the properties of an inner product, which it clearly cannot have.\n\nLIMITATIONS:\nYes, the authors point out that kernel SVMs are usually preferred over linear SVMs and indicate that they will work on this in the future. I think that the current work is still an important step in this direction.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-13"}