SlowGuess/NeurIPS_2023_OCR huggingface dataset:
With a slight abuse of notation, let  $g(\boldsymbol{\omega}; \boldsymbol{x}) = g(1, \boldsymbol{\omega}, 0; \boldsymbol{x}, 1) = \langle \boldsymbol{\omega}, \boldsymbol{x} \rangle_{\mathbb{B}} = \ln \frac{1 - \| \boldsymbol{x} \|^2}{\|\boldsymbol{\omega} - \boldsymbol{x}\|^2}$  defined on  $A = \left\{ \nu \in \mathbb{S}^{n-1} \mid \frac{\boldsymbol{x}^T \boldsymbol{\nu}}{\|\boldsymbol{x}\|} > 0 \right\} \subset \mathbb{S}^{n-1}$ . Since  $-\ln(\cdot)$  is decreasing and convex, we only need to check  $h(\boldsymbol{\omega}; \boldsymbol{x}) = \| \boldsymbol{\omega} - \boldsymbol{x} \|^2$  is geodesically convex on  $A$ , i.e., check that the epigraph of  $h$  is a convex set. Note that  $\frac{\boldsymbol{x}^T \boldsymbol{\omega}}{\|\boldsymbol{x}\|} = \cos(\theta_\omega)$  for  $\boldsymbol{\omega} \in \mathbb{S}^{n-1}$  where  $\theta_\omega = \angle(\boldsymbol{\omega}, \boldsymbol{x})$ .
$$
\begin{array}{l} \operatorname {e p i} (h) = \left\{(\boldsymbol {\omega}, c) \in \boldsymbol {A} \times \mathbb {R} | \| \boldsymbol {\omega} - \boldsymbol {x} \| ^ {2} \leq c \right\} \\ = \left\{(\boldsymbol {\omega}, c) \left| \frac {\boldsymbol {x} ^ {T} \boldsymbol {\omega}}{\| \boldsymbol {x} \|} \geq \frac {1}{2 \| \boldsymbol {x} \|} (1 + \| \boldsymbol {x} \| ^ {2} - c) \right. \right\} \tag {9} \\ = \{(\boldsymbol {\omega}, c) | \cos (\theta_ {\boldsymbol {\omega}}) \geq d (c) \} = \left\{ \begin{array}{l l} \boldsymbol {A} \times [ d (c), \infty) & \text {i f} d (c) \leq 0 \\ \boldsymbol {A} _ {d} \times [ d (c), \infty) & \text {i f} d (c) > 0 \end{array} \right. \\ \end{array}
$$
where  $d(c)$  is a real number depending on  $c$  and  $\| \pmb{x}\|$  and  $\pmb{A}_{d} = \{(\omega ,c)|\cos (\theta_{\omega})\geq d(c)\}$  is the collection of unit vectors where the angle between the vectors given the data  $\pmb{x}$  is small i.e., restricted to a small region on the sphere. The last equality follows from the definition of  $\pmb{A}$  and  $\pmb{A}_{d}$ : if  $d(c)\leq 0$ , then  $\{\omega :\cos (\theta_{\omega})\geq d(c)\} \cap \pmb {A} = \pmb{A}$ . Similarly, if  $d(c) > 0$ ,  $\{\omega :\cos (\theta_{\omega})\geq d(c)\} \cap \pmb {A} = \{\omega \in \mathbb{S}^{n - 1}|\cos (\theta_{\omega})\geq d(c)\} := A_{d(c)}$ . Both  $\pmb{A}$  and  $\pmb{A}_{d}$  are geodesically convex sets and this completes the proof.
The convex sets  $A, B$  are the hemispheres of  $\mathbb{S}^{n-1}$  separated by the hyperplane  $\{z \in \mathbb{R}^n | x^T z = 0\}$  (the hyperplane has  $x$  as its normal vector) in its ambient space  $\mathbb{R}^n$ . Theorem 3.7 tells us that given one data sample  $(x, y)$ , the optimal value described in Eq. (7) exists in  $\mathbb{R}^+ \times A \times \mathbb{R}^+$ , and it is globally optimal. For a collection of training samples  $S = \{(x_i, y_i)\}_{i=1}^N$ , let  $A_i = \{\nu \in \mathbb{S}^{n-1} | y_i \cdot \frac{x_i^T \nu}{\|x_i\|} > 0\}$ . If the data are separable by a horosphere, it follows that  $\cap_{i=1}^N A_i$  is nonempty and convex. Then the loss function given  $S$  is geodesically convex on  $\mathbb{R}^+ \times \cap_{i=1}^N A_i \times \mathbb{R}^+$  and the global optimum can be obtained using any gradient-based optimization. Numerically, we apply a Riemannian gradient descent method on the entire space  $\mathbb{R}^+ \times \mathbb{S}^{n-1} \times \mathbb{R}^+$  since  $g(\mu, \omega, b; x, y)$  is continuous.
3.4 Horospherical SVM
Given a horospherical decision boundary  $\pi_{\mu, \omega, b}$  parameterized by  $\omega \in \mathbb{S}^{n-1}, \mu \in \mathbb{R}^+$ , and  $b \in \mathbb{R}$ , the margin  $\gamma$  is the minimal distance from training samples  $S$  to the decision boundary:
$$
\gamma (\mu , \boldsymbol {\omega}, b) = \inf  _ {\{\boldsymbol {x}, y \} \in S} y \cdot f (\boldsymbol {x}; \mu , \boldsymbol {\omega}, b) \cdot d _ {\mathbb {B}} (\boldsymbol {x}, \pi_ {\mu , \boldsymbol {\omega}, b}) = \inf  _ {\{\boldsymbol {x}, y \} \in S} y \cdot \frac {(\mu \langle \boldsymbol {\omega} , \boldsymbol {x} \rangle_ {\mathbb {B}} - b)}{\mu}. \tag {10}
$$
The maximum margin classifier can be obtained by solving the following optimization problem:
$$
\max  _ {\mu , \boldsymbol {\omega}, b} \gamma (\mu , \boldsymbol {\omega}, b) \quad s. t. \quad y \cdot \frac {(\mu \langle \boldsymbol {\omega} , \boldsymbol {x} \rangle_ {\mathbb {B}} - b)}{\mu} \geq \gamma \quad \text {f o r a l l} (\boldsymbol {x}, y) \in S. \tag {11}
$$
Theorem 3.8. The maximum margin classification problem in hyperbolic space with horosphere as its decision boundary described in Eq. (11) is equivalent to the following optimization problem:
$$
\min  _ {\mu , \boldsymbol {\omega}, b} \quad \frac {1}{2} \mu^ {2} \quad s. t. \quad y \cdot (\mu \langle \boldsymbol {\omega}, \boldsymbol {x} \rangle_ {\mathbb {B}} - b) \geq 1 \quad f o r a l l (\boldsymbol {x}, y) \in S. \tag {12}
$$
The proof is analogous to that in the Euclidean case [3]. Note that the margin is unchanged if we apply the following scale transformation:  $\mu \rightarrow \mu /\gamma$  and  $b\rightarrow b / \gamma$  . We can also build a soft-margin horospherical SVM, dubbed HoroSVM, by minimizing the following loss function:
$$
l (\mu , \omega , b; \boldsymbol {x}, y) = \frac {1}{2} \mu^ {2} + C \sum_ {i = 1} ^ {| S |} \max  (0, 1 - y _ {i} \cdot (\mu \langle \omega , \boldsymbol {x} _ {i} \rangle_ {\mathbb {B}} - b)). \tag {13}
$$
where  $C$  is a hyperparameter that controls the tradeoff between minimizing misclassification and maximizing margin.
It is easy to see that the same result as in Theorem 3.7 holds for the loss function in Eq. (13). That is, the loss function is a geodesically convex function on one geodesically convex subset of the parameter space and a geodesically concave function on the other geodesically convex subset of the parameter space. Recall that the idea behind proving that the loss function in the horospherical perceptron,  $\max(0, -g(\cdot))$ , where  $g(\cdot) = y \cdot (\mu \langle \pmb{\omega}, \pmb{x} \rangle_{\mathbb{B}} - b)$ , is geodesically convex is based on the important fact that  $\max(0, -a)$  is a convex function in  $a \in \mathbb{R}$ . Similarly, the same idea applies to HoroSVM, where the hinge loss is used, and the loss function becomes  $\frac{1}{2} \mu^2 + \max(0, 1 - g(\cdot))$ . Note that  $\max(0, 1 - a)$  is also a convex function in  $a \in \mathbb{R}$  and  $\frac{1}{2} \mu^2$  is a convex function in  $\mu$ , these facts complete the proof of the desired property for the HoroSVM, which is analogous to Theorem 3.7 for horospherical perceptron.
We can then apply any Riemannian gradient descent optimization methods for updating the parameters in HoroSVM since the problem is an optimization problem over a product space of Riemannian manifolds,  $\mathbb{R}^{+}\times \mathbb{S}^{n - 1}\times \mathbb{R}^{+}$ . We refer the readers to [5] and [1] for more details about optimization techniques on Riemannian manifolds.
In this section, we present several experimental results obtained from an application of our HoroSVM to synthetic data as well as real data sets used in published literature. Our implementation is based on Pymanopt [19] using the Riemannian conjugate gradient method [26] on Intel(R) Xeon(R) CPU E5-2683 v3 @ 2.00GHz.
4.1 Network Data Set
Here, we follow the experimental setup in [12], and evaluate our HoroSVM over four real-world network data sets used by [8]: karate [34] (2 classes, 34 nodes), polblogs [2] (2 classes, 1224 nodes), polbooks  $^{3}$  (3 classes, 105 nodes), and football [17] (12 classes, 115 nodes).
The network data is embedded in a 2D hyperbolic space using the method of [8]. Given the hyperbolic embeddings, we compare our HoroSVM with three other competing large margin classifiers: Euclidean SVM (even though it violates the hyperbolic geometry), hyperboloid SVM [12], and Poincaré SVM [11]. The absence of comparison with [32] in this experiment is due to two reasons. First, [32] aims to provide a theoretical understanding of hyperbolic spaces in classification, focusing on 'linearly' separable data (data that can be separated by a geodesic). They do not address extensions to nonlinearly separable data, which limits their applicability to many practical datasets. Second, for datasets that are linearly separable in hyperbolic space, the approach in [32] aligns with the hyperboloid SVM when adversarial training is not applied in [32].
For multiclass classification, a one-verses-rest strategy is applied. We conducted a five-fold cross-validation on each data set, where we chose the hyperparameter  $C$  from  $\{1, 5, 10\}$  during the cross-validation procedure. Note that a more extensive search space for  $C$  may lead to potential performance improvements. The mean of the F1 score followed by the standard deviation over five
<sup>3</sup>http://www-personal.umich.edu/~mejn/netdata/
Table 1: F1 scores for node classification on network datasets. Boldface indicates best performance.
Methods	Karate	Polblogs	Polbooks	Football
Euclidean SVM	0.95 ± 0.06	0.92 ± 0.02	0.83 ± 0.03	0.29 ± 0.12
Hyperboloid SVM	0.95 ± 0.06	0.92 ± 0.01	0.83 ± 0.03	0.30 ± 0.14
Poincaré SVM	0.78 ± 0.16	0.92 ± 0.02	0.84 ± 0.03	0.32 ± 0.04
HoroSVM (Ours)	0.98 ± 0.04	0.93 ± 0.01	0.85 ± 0.04	0.34 ± 0.06
Table 2: F1 scores for subtree classification on four subtrees of WordNet. Boldface indicates the best performance on 2D embeddings of each dataset.
Methods	animal.n.013218/798	group.n.016649/1727	worker.n.01861/254	mammal.n.01953/228
Hyperboloid SVM (D = 2)	0.53 ± 0.07	0.52 ± 0.01	0.54 ± 0.04	0.39 ± 0.03
Hyperbolic LR (D = 2)	0.46 ± 0.08	0.52 ± 0.04	0.54 ± 0.07	0.32 ± 0.10
Hyperbolic LR (D = 5)	0.95 ± 0.03	0.76 ± 0.07	0.80 ± 0.08	0.78 ± 0.04
Hyperbolic LR (D = 10)	0.96 ± 0.01	0.86 ± 0.05	0.84 ± 0.04	0.94 ± 0.04
Euclidean SVM (D = 2)	0.39 ± 0.01	0.39 ± 0.00	0.32 ± 0.02	0.20 ± 0.01
Euclidean SVM (D = 5)	0.95 ± 0.00	0.79 ± 0.01	0.38 ± 0.02	0.44 ± 0.01
Euclidean SVM (D = 10)	0.97 ± 0.00	0.91 ± 0.00	0.46 ± 0.04	0.72 ± 0.05
HoroSVM (D = 2)	0.57 ± 0.07	0.65 ± 0.01	0.62 ± 0.01	0.42 ± 0.01
HoroSVM (D = 5)	0.93 ± 0.01	0.88 ± 0.00	0.82 ± 0.04	0.88 ± 0.01
HoroSVM (D = 10)	0.95 ± 0.02	0.91 ± 0.01	0.86 ± 0.01	0.93 ± 0.02
trials are summarized in Table 1. As evident from the table, our method yields the best results on all the data sets.
HoroSVM outperformed other methods on all four data sets. The data in karate are well-separated and thus both Euclidean SVM and hyperboloid SVM performed equally well. Our method outperforms the others since the horospheres have several nice properties, the most important of which is that the Busemann function whose level sets are the horospheres is a convex function that guarantees global optimality in the optimization. Notice that the performance of Poincaré SVM on karate is inferior to others by a significant amount. The reason is that the performance of Poincaré SVM is sensitive to the choice of the reference point. We demonstrate our performance gain on the remaining data sets, and our method is more consistent, compared to Euclidean SVM and hyperboloid SVM, in terms of lower standard deviation, on football data set where data exhibit a larger variance/spread.
4.2 Subtree Classification in WordNet
A task of considerable interest in hyperbolic space classification problems is to determine whether a node belongs to a given subtree in the hyperbolic embedding. We obtained hyperbolic embeddings in various dimensions using the approach in [15] for WordNet  $^4$  noun hierarchy (82,115 nodes). We consider four subtrees whose roots are the following synsets: ANIMAL.N.01, GROUP.N.01, WORKER.N.01, and MAMMAL.N.01.
We split all nodes in a subtree into positive training  $(80\%)$  and test  $(20\%)$  nodes and applied the same process to the remaining WordNet nodes to create negative training and test sets. The average F1 scores and the standard deviations over 3 trials are shown in Table 2. The number of positive training/test samples of each data set are listed as well. We exclude Poincaré SVM from the comparisons in this task. The reason being, data are highly imbalanced in this task and the positive samples are clustered near the boundary. The reference point learned in Poincaré SVM will be close to the boundary where the tangent approximation of data at this reference point is highly distorted, as opposed to the original hyperbolic embeddings. It is therefore hard to locate a hyperplane in the
<sup>4</sup>https://wordnet.princeton.edu/
tangent space that separates the lifted (mapped) data. In addition, the learning of the reference point is only applicable to 2D hyperbolic space.
As well known in the Euclidean SVM literature, a vanilla (unweighted) implementation of SVM performs poorly on extremely imbalanced data, we observed the same behavior in training HoroSVM on this task. We preprocessed the data by downsampling the majority class of samples (the negative samples) to train a robust model. Note that our HoroSVM can be naturally extended to a class/instance-weighted version by assigning a class/instance weight to the penalty term  $C$  for each sample, allowing us to address more general imbalanced data. Since there is no protocol for dealing with unbalanced data in training a hyperboloid SVM, we presented the Euclidean SVM results using the same preprocessed data for reference. In addition, we presented results of hyperbolic logistic regression (LR) [15], which is not a large-margin classifier on this task, where the imbalanced data is handled by sampling the equal number of negative and positive nodes in each mini-batch of size 16 during training.
Now, we highlight several results in Table 2. The superior performance of hyperboloid SVM over hyperbolic LR is expected, as both methods use geodesic decision boundaries but hyperboloid SVM aims to maximize the margin. However, the training of hyperboloid SVM is highly unstable as we mentioned earlier due to the non-convex optimization process. Euclidean SVM under-performs as it does not take into account the hyperbolic geometry. Our HoroSVM exhibits a significant improvement in predicting words in a subtree, as evidenced by higher F1 scores across all the subtrees. The small number of nodes within a subtree, compared to the whole WordNet, causes the nodes to cluster near the boundary in their hyperbolic embeddings. Thus, horosphere is an ideally suited decision boundary (in comparison to the geodesic boundary in [12]) to isolate the subtree.
4.3 Synthetic Data with Noisy Labels
To demonstrate the robustness of our HoroSVM, we apply it to synthetic data with noisy labels at varying levels/amounts of noise. Specifically, we generated 100 synthetic datasets by sampling from a Gaussian mixture model defined on the Poincaré disk model as in [12]. The isotropic Gaussian distribution in hyperbolic space is referred to as the Riemannian normal distribution, and we used the sampling method presented in [21]. For each dataset, we
Figure 4: Training (left) and test (right) F1 scores of several methods on synthetic data with noisy labels at different noise levels.
sampled two centroids from a zero-mean Riemannian normal distribution with a variance of 1.5. We then sampled 200 data points from a unit-variate Riemannian normal distribution centered at each centroid, resulting in a dataset of 400 points classified into positive and negative classes. We split the dataset into training and test sets with 100 positive/negative samples in the training set and 100 positive/negative samples in the test set. We then generated datasets with noisy labels at noise levels:  $\eta \in \{0,0.05,0.1,\dots ,0.5\}$  by flipping the labels of a proportion  $\eta$  of the training (not test) samples, with an equal number of positive and negative samples flipped. The train/test average F1 scores of each method across all datasets at varying noise levels are shown in Fig 4. We compared our HoroSVM with hyperboloid SVM, hyperbolic LR, and a two-layer HNN [15] (with a hidden dimension of 5). While all methods depict decreasing training F1 scores as the noise level increases, HoroSVM outperforms the others consistently throughout the training process. Hyperbolic LR and HNN exhibit the least resistance to label noise, with test F1 scores dropping (faster) with increasing noise level. Both hyperboloid SVM and HoroSVM demonstrate consistent performance across different noise levels, owing to the inherent robustness of large-margin classifiers. However, the training of hyperboloid SVM is highly unstable resulting in its inferior performance. HoroSVM demonstrates its superiority in accuracy and robustness as evidenced in the results. In addition, we present the average training times for each method on one dataset (200 samples) as follows: 6.57 seconds (Hyperboloid SVM), 3.98 seconds (Hyperbolic LR), 9.06 seconds (HNN), and 3.73 seconds (HoroSVM). Notably, HoroSVM stands out as the fastest.
5 Discussion and Conclusions
In this paper, we presented a novel large margin classifier, dubbed HoroSVM, whose decision boundaries are horospheres that are the level sets of a Busemann function. We presented a novel formulation leading to the optimization of a geodesically convex loss performed using a Riemannian gradient-based method and guaranteeing a globally optimal solution. We demonstrated superior to competitive performance of the HoroSVM over SOTA large margin classifiers.
In Euclidean space, a kernel SVM is usually favored over the linear SVM due to its ability to cope with non-linearly separable data. In Hyperbolic space, the challenge lies in developing valid positive definite kernels (see [14] for details on validity of kernels on Riemannian manifolds). The only reported work on KSVM in hyperbolic space that we are aware of is [12], which uses a kernel that violates the positive definiteness property of RKHS kernels. Thus the problem of interest is primarily defining a valid family of kernels in hyperbolic space. We will address this in our future work.
This research was in part funded by the NSF grant IIS 1724174 and the NIH NINDS and NIA grant RF1NS121099 to Vemuri.
[1] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds. In Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2009.
[2] Lada A Adamic and Natalie Glance. The political blogosphere and the 2004 us election: divided they blog. In Proceedings of the 3rd international workshop on Link discovery, pages 36-43, 2005.
[3] Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning, volume 4. Springer, 2006.
[4] Roberto Bonola. Non-Euclidean geometry: A critical and historical study of its development. Courier Corporation, 1955.
[5] Nicolas Boumal. An introduction to optimization on smooth manifolds. Available online, May, 3, 2020.
[6] Martin R Bridson and André Haefliger. Metric spaces of non-positive curvature, volume 319. Springer Science & Business Media, 2013.
[7] James W. Cannon, William J. Floyd, Richard Kenyon, and Walter R. Parry. Hyperbolic Geometry, volume 31. MSRI Publications, 1997.
[8] Benjamin Paul Chamberlain, James Clough, and Marc Peter Deisenroth. Neural embeddings of graphs in hyperbolic space. arXiv preprint arXiv:1705.10359, 2017.
[9] Ines Chami, Albert Gu, Dat P Nguyen, and Christopher Ré. Horopca: Hyperbolic dimensionality reduction via horospherical projections. In International Conference on Machine Learning, pages 1419-1429. PMLR, 2021.
[10] Ines Chami, Zhitao Ying, Christopher Ré, and Jure Leskovec. Hyperbolic graph convolutional neural networks. Advances in Neural Information Processing Systems, 32:4868-4879, 2019.
[11] Eli Chien, Chao Pan, Puoya Tabaghi, and Olgica Milenkovic. Highly scalable and provably accurate classification in poincaré balls. In 2021 IEEE International Conference on Data Mining (ICDM), pages 61-70. IEEE, 2021.
[12] Hyunghoon Cho, Benjamin DeMeo, Jian Peng, and Bonnie Berger. Large-margin classification in hyperbolic space. In The 22nd international conference on artificial intelligence and statistics, pages 1832-1840. PMLR, 2019.
[13] Xiran Fan, Chun-Hao Yang, and Baba C Vemuri. Nested hyperbolic spaces for dimensionality reduction and hyperbolic nn design. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 356-365, 2022.