Learning Structured Representations with Hyperbolic Embeddings
Authors: Aditya Sinha
Siqi Zeng
Makoto Yamada
Han Zhao
Venue: NeurIPS 2024 poster (2024)
Keywords: hierarchical representation
representation learning
hyperbolic geometry
Primary area: other
TL;DR: We propose a structured regularization method for learning label-hierarchy informed representations using hyperbolic geometry.

Abstract: Most real-world datasets consist of a natural hierarchy between classes or an inherent label structure that is either already available or can be constructed cheaply. However, most existing representation learning methods ignore this hierarchy, treating labels as permutation invariant. Recent work [Zeng et al., 2022] proposes using this structured information explicitly, but the use of Euclidean distance may distort the underlying semantic context [Chen et al., 2013]. In this work, motivated by the advantage of hyperbolic spaces in modeling hierarchical relationships, we propose a novel approach HypStructure: a Hyperbolic Structured regularization approach to accurately embed the label hierarchy into the learned representations. HypStructure is a simple-yet-effective regularizer that consists of a hyperbolic tree-based representation loss along with a centering loss, and can be combined with any standard task loss to learn hierarchy-informed features. Extensive experiments on several large-scale vision benchmarks demonstrate the efficacy of HypStructure in reducing distortion and boosting generalization performance especially under low dimensional scenarios. For a better understanding of structured representation, we perform eigenvalue analysis that links the representation geometry to improved Out-of-Distribution (OOD) detection performance seen empirically.

Learning Structured Representations with Hyperbolic Embeddings
Aditya Sinha*t
Siqi Zeng'
University of Illinois, Arbana-Champala as1460illinois.ed
University of Illinois, Urbana- Champaign siqi60i1linois edu
Makoto Yamada
Han Zhao
Okinawa Institute of Science and Technology makoto. makoto. yamddaosstam
University of Illinois, Urbana- trtana-Champaige hanzhao@i iilao0iiliiiiii edu
Abstract
Most real-world datasets consist of natural hierarchy between classes in- herent label structure that either already available or can be constructed cheaply. However, most existing representation learning methods ignore this hierarchy, treating labels as permutation invariant. Recent work [T04) proposes using this structured information explicitly, but the of Euclidean distance may distort the underlying semantic context [8]. In this work, motivated by the advantage of hyper- bolic spaces in modeling hicrarchical relationships, we propose novel approach HypStructure: Hyperbolic Structured regularization approach to accurately embed the label hierarchy into the learned representations. HypStructure is a regularizer that consists of hyperbolic tree-based representa- tion loss along with a centering loss. can be combined with any standard task loss to learn hierurchy-informel features. Extensive experiments on several large-scale vision benehmarks demonstrate the efficacy HypStructure in reducing distor- tion and boosting generalization performance, especially under low-dimensional scenarios. For a better understanding of structured representation, we perform eigenvalue analysis that links the representation geometry improved Out-of- Distribution (OOD) detection performance seen empirically. The code available at huttps://github http:///tttuuu.cmmmuuueeeee http:///iiiccc///l//sttuuuuuuuuu
Introduction
Real-worid datasets, such as ImageNet [72] and CIFAR 45), often exhibit natural hicrarchy or an inherent label structure that deseribes structured relationship between different classes in the data. In the absence of an existing hierarchy, is often possible to cheaply construct or infer this hierarchy from the label space directly
64 However, the majority of existing representation
treat the labels as permutation invariant, ignoring this semantically-rien hierarchical label information. Recently, Zeng et [104] offer promising approach to embed the tree-hierarchy explicitly in representation learning using tree-metric-based regularizer, leading improvements generalization performance. The approach uses a computation of shortest paths between two classes in the tree hierarchy to enforce the same structure in the feature space, by means of Cophenetic Correlation Coefficient (CPCC) [79 based regularizer. However, their approach uses the & distance the Euclidean space, distorting the parent-child representations in the hierarchy 770 50 owing to the bounded dimensionality the Euclidean space (8).
learning methods 43 7 95 29 87
33
39
Authors contributed equally.
Now at Netlix Inc.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Hyperbolic geometry has recently gained growing interest in the field of representation learning Hyperbolic spaces can be viewed as the continuous analog of tree, allowing for embedding
666 67
tree-like data in finite dimensions with minimal distortion [4 73
75 24 Unlike Fuclidean spaces with zero curvature and spherical spaces with positive curvature, the hyperbolic spaces have negative curvature enabling the length to grow exponentially with its radius. Owing to these advantages, hyperbolic geometry has been used for various applications such as natural language processing [52] [7 [16], image classification 440 [03] [8], object detection [46 (21). action retrieval [55), and hierarchical clustering [TOO]. However, the aim of using hyperbolic geometry these approaches is often to implicitly leverage the hierarchical nature of the data.
In this work, given a label hierarchy, we argue that accurately and explicitly embedding the hierarchical information into the representation space has several benefits. and for this purpose, we propose HypStructure, a hyperbolic label-structure based regularization approach that extends the proposed methodology in [104] for semantically structured learning in the hyperbolic space. HypStructure can be easily combined with any standard task loss for optimization, and enables the learning of discriminative and Sherarch-iiffomee features. In summary, our contributions are follows:
We propose HypStructure and demonstrate its effectiveness in the supervised hierarchical classification tasks on three real-world vision benchmark datasets, and show that our pro- posed approach is effective in both training from scratch, fine-tuning if there are resource constraints.
We qualitatively and quantitatively assess the nature of the learned representations and demonstrate that along with the performance gains, using HypStructure as a regularizer leads to more interpretable as well as tree-like representations as side benefit. The low-dimensional representative capacity of hyperbolic geometry is well-known [ō] and interestingly, we observe that training with HypStructure allows for learning extremely low-dimensional representations with distortion values lower than even their corresponding high-dimensional Fuclidean counterparts.
We argue that representations learned with underlying hierarchical structure beneficial not only for the in-distribution (ID) classification tasks but also for -uit---distiibution (OOD) detection tasks. We empirically demonstrate that learning ID representations with HypStructure leads to improved OOD detection on 9 real-world OOD datasets without sacrificing ID accuracy [106].
Inspired by the improvements in OOD detection, we provide a formal analysis of the eigenspectrum of the in-distribution hierarchy-informed features learned with CPCC-style structured regularization methods, thus leading to a better understanding of the behavior of structured representations in general.
Preliminaries
In this section, we first provide a background of structured representation learning and then discuss the limited representation capacity of the Euclidean space for hierarchical information, which serves as the primary motivation for our work.
2.1 Background
Structured representation learning [104] breaks the permutation invariance of flat representation learning by incorporating a hierarchical regularization term with standard classification loss. The regularization term is specifically designed to enforce class-oonditione grouping or partitioning in the feature space, based given hierarchy.
More specifically, given weighted tree T= (V, e) with vertices V, edges E and edge weights e, let us compute tree metric dr for any pair of nodes u. v' E V, as the weighted length of the shortest path in T between v and v'. For a real world dataset = {( y: ] we can specify a label tree T where a node 0; E V, corresponds to a subset of classes, and D, C D denote the subset of data points with class label us- We denote dataset distance between D; and D, as plv;, 23) = (D., D,), where d(:, -) is any distance metric in the feature space, varied by design.

---

With collection of tree metric dr and dataset distances p. we can use the Cophenetic Correlation Coefficient (CPCC) [79], inherently a Pearson's correlation coefficient, to evaluate the correspondence between the nodes of the tree, and the features in the representation space, Let dr, P denote the mean of the collection of distances, then CPCC is defined as
Ede,(dy (vi, U2) (p U1) PP)
CPCC(dT = Eiee(d 13) dr) ())/))())( i/, (p(ur) U2) 2)2)1/22
For the supervised classification task, we consider the training set D (1x y1 }i and we aim to learn the network parameter 0 for feature encoder Se X Z, where S Rd denotes the representation/feeauuee space. For structured representation learning, the feature encoder is usually followed by a classifier fur. and the parameters flat (non-hierarcel -io-hiiearrhhcaa) classification loss, for instance, Cross-Entrope (CE) or Supervised Contrastive (SupCon) [3 loss, with the structured regularization term as:
are learnt by minimizing L along with standard
((D) = Σ
(x y, 0, w) CPCC(dT,
(x.9))D
Using a composite objective as defined in Equation (2,, we can enforce the distance relationship between pair of representations in the feature space, to behave similarly to the tree metric between the same vertices. For instance, consider a simple label tree with a root node, coarse level, and a fine level, where subsets of fine classes share the same coarse parent. For this hierarchy, we would expect the fine classes of the same parents (e-g., apple and banana are fruits) to have closer representations in the feature space, whereas fine classes with different coarse parents (e.g.. an apple is fruit and tulip is a flower) should be further apart. The learned structure-ifoommed representations reflect these hierarchical relationships and lead to interpretable features with better generalization [104]
2.2 CPCC
Equation ① offers the flexibility of designing a metric to measure the similarity between two data subsets, and [104] define the Euclidean dataset distance as Pt, (D;, Dÿ) == D.. EeeD, f(x)
Pí ExEED, l2- The distance between datasets is thus the Eg distance between two Fuclidean centroids of their class-conditioned representations, which is unsuitable for modeling tree-like data 8 Additionally, this regularization approach in [T04] is applied only to the leaf nodes of TT for efficiency. However, this leaf-only formulation of the CPCC offers an approximation of the structured information, since the distance between non-leaf nodes is not restricted explicitly by the regularization. This approximation, therefore, leads to loss of information contained in the original hierarchy T. Actually, impossible embed dy into & exactly. Or more formally, there exists no bijection ço such that dr (f((z)) (p(z)) = || 2y |lz irrespective of how large the feature dimension d is. We provide two such examples for toy label tree in Figure [ below.
Example 1. We intend to embed all nodes in T, including purple internal nodes. Notice that G, C, D, E is star graph centered at G. Since CG DG = CD = 2, by triangle inequality C, D, G must be the same line where G is the center of CD. Similarly, G must be at the center of DE. Hence, the location of E must be at C, which contradicts the uniqueness of all nodes in T.
H
F
B
Figure I: (left) An unweighted label tree with two coarse nodes: F, G. F contains two fine classes A, B and G contains three fine classes C, D, E. We cannot embed this in & exactly (right).
Example 2. As easier problem, let us only embed leaf nodes into the Euclidean space as shown in Figure [] Since CD DE CE = they must be on plane with an equilateral triangle ACDE in Euclidean geometry. Then all the green classes have the same distance 4 to each yellow class. Therefore, A, B must be on the line perpendieular ÂCDE and intersecting the plane with O which is the barycenter of ACDE. Due to the uniqueness and symmetry of A, B, we must have AO = BO = to satisfy AB = 2. 2 AE 4 which contradicts the Pythagorean Theorem.
AO OF 2v
Since we cannot embed an arbitrary tree T into & without distortion, it would also affect the optimization of the CPCC in classification problem, where the tree weights encode knowledge
3

---

zoot
traneportation
animaz
dr
fall CPCC 0.971
Weaf, CPCC 0.963
3
airplane
botombbie
elpp
OBF
364
Exog
horea
Euclidean Dataset Distance
Figure 2: Using CPCC for structured representation CIFARIO. CIFARIO hierarchy (left) has three level structure with 13 vertices. For a 512-dimensional embedding, we apply f2-CPCC either for the full tree (middle) or the leaf nodes only (right) and plot the ground truth tree metric against pairwise Euclidean centroid distances of the learnt representation. The optimal train CPCC is 1.
of class similarity. To verify our claims, we consider the optimization of 51 2-dimennionnal C2CCPC structured representations for CIFAR10 [5. The CIFAR10 dataset consists of small label hierarchy as shown in Figure 2 (left). The optimal CPCC is achieved when each tree metric value corresponds single Pēz. However, in Figure 2 (right), even with an optimization of the CPCCC loss for the entire tree, we observe a sub-optimal train CPCC less than 1, where the distance between two coarse nodes, transportation and animal, is far away from the desired solution. Furthermore, optimization of the CPCC loss for only the leaf nodes, leads to an even larger distortion of the tree metrics.
3 Methodology
Hyperbolic spaces more suitable for embedding hierarchical relationships with low distortion 75 and low dimensions. Hence, motivated by the aforementioned challenges, we propose a Hyperbolic Structured regularizer for hierarchy-informeed representation learning. We begin with the basics of hyperbolic geometry, followed by the detailed methodology of our proposed approach.
3.1 Hyperbolic Geometry
Hyperbolic spaces are Euclidean spaces with negative curvature where given fixed point and a line, there exist infinitely many parallel lines that can pass through this point. There are several commonly used isometric hyperbolic models 4. For this work, we mainly use the Poincaré Ball model.
Hyperbolold
Kieh
Definition 3.1 (Manifold). A manifold M is a set of points z that are locally Euclidean. Every point of the manifold M is attached to a tangent space M,, which is a vector space over the reals of the same dimensionality M that contain all the possible directions that can tangentially pass through x.
Figure 3: Lines on different models for 2-dimensional hy- perbolic space.
Definition 3.2 (Poincaré Ball Model). Given Poincaré ball model (B gæ) is defined by a manifold of an open ball IBd = {= Rdd : cl| 12 < 1} and metric tensor @ that defines an inner product of TB.. The model equipped with the distance
constant, the
88 as
da, (2), = tanh - (val)
- (1 + 2c (- z1, 22) + c
2 (1 c 21 12)
(3)
2c (2), 22)
=
12
For e - 0, we can recover the properties of the Euclidean geometry since lim ds, (21, x2) = 2||z| - 22l1- Since T is isomorphic to gd, we can connect vectors in Euclidean space and hyperbolic space with the bijection between TB and Bd [8]] For x O, the exponential map exp6 Tg Bd - Bd and logarithm map logå Bd T Bd have the closed form of
1
exp6 (v) tanh (Vel)
logā (u)
Ve tanh (Vellul)
(4)
Valuil
Alternatively, to guarantee the correctness of Poincaré distance computation, we can also process any Euclidean vector with a clipping module [66]
(0:
v. -E)
if ||ll < 1/Ve otherwise
c
(ttet
(5)
T
so the clipped vector is within the Poincare disk. We set small positive number in practice.

---

Definition 3.3 (Klein Model). Klein model (KK 9x) consists of an 1/Ve-radius open ball K {= E Rd c||=||² < }} and a metric tensor 9K different from 8B- Similar to the mean computation in Euclidean space, let 7/2 = 1 1/v1 c||zi||? 2 the Einstein midpoint of group of Klein vectors 21.... Zn E k has simple expression of a weighted average
.n)
72
%
(6)
We illustrate the relationship between the different hyperbolic models in Figure The hyperboloid space models d-dimensional hyperbolic geometry on a d + 1-dimensional space. When d = 2, the Klein model is the tangent plane of the hyperboloid model at (0, O, 1), and the Poincaré disk shares the same support as the Klein disk, although shifted downwards and centered at the origin. Given a triangle the hyperboloid model, its projection the projection of a line on the Poincaré model is a part of a circular arc or the diameter of the disk. Let 23, zg be coordinates of z under Poincaré and Klein model respectively, the prototype operations on B require transformations between B and I as
the Klein model preserves the straight sides, but
KK
2z
V - 23 I+
2
+ c| 2B ||
For example, in Figure 3 O' is the HypAvex of A', B', C", and can be mapped back to the Poincaré disk to get the Poincare prototype (HypAveg) O of points A, B, by a change of coordinates.
3.2 HypStructure: Hyperbolic Structured Regularization
At a high level, HypStructure uses a combination of two losses: a Hyperbolic Cophenetic Correla- tion Coefficient Loss (HypCPCC)), and Hyperbolic centering loss (HypCenter) for embedding the hierarchy in the representation space. Below we describe the two components of HypStructure. The pseudocode of HypStructure is shown in Algorithm in Appendix B.2]
HypCPCC (Hyperbolic Cophenetic Correlation Coefficient): We extend the C2CCPCC method- ology in [104 to the hyperbolic space in HypCPCC. Three major steps of HypCPCC are (i) map Euclidean vectors to Poincaré space (ii) compute class prototypes (iii) use Poincaré distance for CPCC. Specifically,
first project each E Edd to Id and compute the Poincaré centroid for each vertex of T using hyperbolic averaging as shown in Equation (6) and Equation (7). Alternatively, we also compute Fuclidean centroids Σ aad EXED, for each vertex, and project each E TR.
to B either by expo or clip". After the computation of hyperbolic centroids, we use the pairwise distances between all vertex pairs in T in the Poincaré ball, to compute the HypCPCC loss using Equation (T) by setting p ddd
HypCenter (Hyperbolic Centering): Inspired by Sarkar's low-distortion construction 75 that places the root node of a tree at the origin, we propose a centering loss for this positioning, that enforces the representation of the root node to be close the center of the Poincaré disk, and the representations of its children to be closer to the border of Poincaré disk. We enforce this constraint by minimizing the norm of the hyperbolic representation of the root node as feenter ||HypAve a (exp) (21).... expo (2x) Alternatively, for centroids computed in the Euclidean space and mapped to the Poincaré disk, we minimize feenter = II Sx())) directly due to the monotonicity of tanh(-) the exponential map.
Finally, for a, B > 0, we can learn the hierarchy-informeee represenaattios by minimizing
((D) = Σ (x y, 8) - a HypCPCC((d, da.) + B Eeemee (xc f),
(8)
(D
where lAa is standard classification loss, such as the CE loss or the SupCon loss.
Time Complexity: In a batch computation setting with a batch size and the number of classes (leaf nodes) as k, the computational complexity for a HypStructure computation to embed the full tree will still be O(d min (b2, 1²2)). which is the same as the complexity of a Euclidean leaf-only CPCC. The additional knowledge gained from internal nodes allows us to reason about the relationship between higher-level concepts, and the hyperbolic representations help in achieving a low distortion of hierarchical information for better performance in downstream tasks.
5

---

4 Experiments
We conduct extensive experiments on several large-scale image benchmark datasets evaluate the performance of HypStructure as compared to the Flat and &2CCPCC baselines for hierarchy embedding, classification, and OOD detection tasks.
Datasets and Setup Following the common benchmarks in the literature, we consider three real- world vision datasets, namely CIFARIO, CIFAR 100 [45 and ImageNet1 (0) 59 for training. which
vary in scale, number of classes, and number of images per class. We construct the ImageNet1 (00 dataset by sampling 100 classes from the ImageNet-Ik Ik [72 dataset following [59]. For CIFAR 100,
three-level hierarchy is available with the dataset release 45 Since no hierarchy is available for the
CIFARIO and ImageNet100 datasets, we construct a hierarchy for CIFARIO manually in Figure 2 For Im0geNer100, we create a subtree from the WordNet [19] given 100 classes as leaves. More
details regarding the datasets, network, training and setup are provided the Appendix B.4
4.1 Quality of Hierarchical Information
Table I: Evaluation of hierarchical information distortion and classification accuracy using SupCon 39] as Fim All metrics are reported as mean (standard deviation) over 3 seeds.
Gremev
Dataset (Backbone)
Method
Distortion of Hierarchy
Classification Accuracy
5..1 (.)
CPCC
Fine (1)
Course (1)
Fiat (512)
CIFARIO
0.232
94.64
(0.04)
(ResNet-18)
Gc-CPC
0.174 (0.002)
0.966 (0.001)
91.47 (0.13)
98.91 (0.92)
HypStructure (Ours)
HypScructure
0.094 (0.003)
0.992 (0.001)
94.79 (0.14)
99.18 (0.04)
57
(ResNctt344
CIFARIO0
HypScracture
CPCC
Flat
0.209 (0.002)
0.127 (0.016)
0.213 (0.006)
0.766 (0.007)
0.534 (0.11)
(0.002)
74.96 (0.14)
76.68 (0.22)
(0.19)
86.01 (0.13)
85.15
(0.32)
Figure 4: Evaluation of distor- tion vs feature dimensions for HypStructure.
Embedding Dimension
100
(ResNet-34)
CPCC
Flat
0.168 (0.003)
0.213 (0.000
0.429 (0.002)
90.01 (0.07)
90.77 (0.11)
(0.2%)
HypScracture
(0.001)
0.841 (0.001)
90.12 (0.01)
90.84 (0.02)
First, to assess the tree-likeness of the learnt representations, we measure the Gromov's 's hyperbolicity dret 2 3 40) of the features in Table I Lower 6yet indicates higher tree-likeness and a perfect tree metric space has 8rat = O (more details in Appendix (B.5). To also evaluate the correspondence of the feature distances with ground truth tree metrics, compute CPCC on test sets. We observe that HypStructure reduces distortion of hierarchical information over Flat by upto 59.4 % and over C2CCPCC by upto 45.4%, while also consistently improving the test CPCC for most datasets.
We also perform qualitative analysis of the learnt representations from HypStructure on the CIFAR10 dataset, and visualize them in a Poincaré disk using UMAP (57] in Figure Sa We can observe clearly that the samples for fine classes arrange themselves in the Poincaré disk based on the hierarchy tree seen in Figure 2 being closer to the classes which share a coarse class parent.
To examine the impact of feature dimension on the representative capacity of the hyperbolic space, we vary the feature dimension for HypStructure and compute the 5rel for each learnt feature. Comparing the distortion of features with the Flat and C2-CPCC settings in Figure we observe that 5yet decreases consistently with increasing dimensions, implying that high dimension features using HypStructure are more tree-like, and better than Flat and (2-CPCCs' 512-dimension baselines.
4.2 Classification
Following [104), we treat leaf nodes in the hierarchy as fine classes and their parent nodes as coarse classes. To evaluate the quality of the learnt representations, we perform classification task on the fine and coarse classes using KNN-classifier following (27 95. 5 [10] and report the performance on the three datasets in Table We observe that HypStructure leads upto 2.2% improvements over Flat and upto 0.8% % improvements over CPCC on both fine and coarse accuracy. We also visualize the learnt test features from Flat vs HypStructure on the CIFAR 100 dataset using Fuclidean t-SNE 89 and show the visualizations in Figure 5b and Figure respectivell. We observe that HypStructure leads to sharper and more discriminative representations Fuclidean space. Additionally, we that the fine classes belonging to coarse class (the same shades of colors)

---

100
(a) Hyperbolic UMAP: HypStructure
(b) Euclidean t-SNE: Flat
(c) Euclidean t-SNE:
HypStructure
Figure 6: Left: Hyperbolic UMAP visualization of CIFARIO's HypStructure representation on Poincaré disk. Middle and Right: t-SNE visualization of learnt representations on CIFARI00.
which are semantically closer in the label hierarchy, are grouped closer and more compactly in the feature space as well, as compared to Flat. We also perform evaluations using the linear evaluation protocol 339 and observe identical trend in the accuracy, report these results in Appendix C.I
4.3 OOD Detection
In addition to leveraging the hierarchy explicitly for the purpose of learning tree-like IID represen- tations, we argue that a structured separation of features in the hyperbolic space as enforced by HypStructure helpful for the OOD detection task well. To verify our claim, we perform an exhaustive evaluation on 9 real-world OOD datasets and demonstrate that HypStructury leads to improvements in the OOD detection AUROC. We share more details below.
OOD Dataset AUROC (f)
Overall AUROC
Method
SVHN
Textures
Places365
LSUN
iSUN
Avg.(1)
B.C.(f)
PrchyAnchor 41
82.43
84.99
79.84
91.68
84.96
84.78
51.42
CE + SinCLR 9
94.45
82.01
71.48
89.00
83.82
84.15
31.42
CSI [85
92.65
86.47
76.27
83.78
84.98
84.83
40.00
CIDER 61
95.16
90.42
73.43
96.33
82.98
87.67
60.00
SND+ (SupCon) 76
94.19
86.18
79.90
85.18
84.08
85.90
54.28
KNN+ (SupCon) 83
88.35
82.69
4s-CPCC [104]
93.08
90.45
77.21
82.77
82.79
85.26
40.00
HypStructure
95.97
88.43
78.12
97.01
84.51
88.81
82.85
(b) CIFARI00 (ID) vs.
SVHN (OOD).
(a) OOD detection performance with CIFAR100 100 as IID dataset.
Figure 7: Left: OOD detection score across various datasets on the CIEAR100 ID dataset. Right: Hy- perbolic UMAP of the CIFARI00 (ID) test SVHN (OOD) test features learnt from HypStructure with clear separation in the Poincaré disk.
4.3.1 Problem Setting
Duttof-distribution (OOD) data refers to samples that do not belong to the in-distribution (ID) and whose label set is disjoint from yd and therefore should not be predicted by the model. Therefore the goal of the OOD detection task is to design a methodology that can solve a binary problem of whether an incoming sample æ X is from Px i... y gin (ID) & y (OOD).
OOD datasets We evaluate on OOD image datasets when CIFARIO and CIFARI00 are used as the ID datasets, namely SVHN 665 Places365 [109), Textures 99. LSUN [102], and iSUN [99]. and 4 large scale OOD test datasets, specifically SUN 102]. Places365 [109], Textures [9] and iNaturalist [90] when ImageNet100 is used as the ID dataset. This subset of datasets is prepared by [59] and is created with overlapping classes from ImageNet- 1k removed from these datasets to ensure there is overlap in the distributions.
OOD detection scores While several scores have been proposed for the of OOD detection, we evaluate our proposed method using the Mahalanobis score (76), computed by estimating the mean

---

Table 2: OOD detection AUROC with CIFARIO and ImageNet100 ID.
Method
AUROC
Method
AUROC
CIFAR10
ImageNet100
SSD+
97.38
SSD+
92.46
KNN+
97.22
KNN+
92.74
b2-CPCC
76.67
C2CCPC
91.33
HypStructure
97.75
HypStructure
93.83
and covariance of the in-distribution training features. The Mahalanobis score defined as s(x) (f(x) - (1) - 1s),
where μt, are the mean and covariance of in-distribution training features. 76 present the Maha- lanobis score (eq. (9) in generalized version for multiple feature clusters. However, since they empirically observe that the single-cluster version achieves the highest performance [76), we will focus on this version. After computing the OOD detection scores, we measure the area under the receiver operating characteristic curve (AUROC) as the primary evaluation metric following [47 76..
4.3.2 Main Results and Discussion
We report the AUROC averaged over all the OOD datasets (5 datasets for CIFARIO and CIFARION 100. 4 datasets for ImageNet100) Figure 7a and Table 2 In addition the Flat (SupCon) and & CPCC baselines, we also compare our method with other state-of-the-art methods (see Appendix C.3. for more details about existing OOD detection methods). We observe that HypStructure leads consistent improvement in the OOD detection score, with upto 2% in average AUROC. We also report the dataset-wist -iise OOD detection results for the CIFAR100 ID dataset Table allnn with Average AUROC. To remove the bias in the Average AUROC metric towards any single dataset, we also evaluate Borda Count (B.C.) [58] and report the same, along with a detailed comparison with more OOD detection methods in Table 7a and 6 and 7 in the Appendix C.3
and
We observe that HypStructure ranks in the highest performing methods consistently, thereby demonstrating higher Borda Count well. We additionally visualize the CIFAR100 (ID) vs SVHN (OOD) features learnt from HypStructure, using a hyperbolic UMAP visualization in Figure 7b We observe that
training with HypStructure leads to improvement in the separation of ID vs OOD features in the Poincaré disk.
Additional Experiments, Ablations and Visualizations: More experiments using hyperbolic contrastive losses and hyperbolic networks, ablation studies on each component of HypStructure and additional visualizations can be found in Appendix C
Eigenspectrum Analysis of Structured Representations
Flat
0.04
CPCCCC
HypStructure
Elgenvalue
0.03
AUROC
0.5
0.4
0.02
0.3
Flat
0.01
4-CPPCC
HypStructure
0.00
60
80
100
40
100
120
140
Index
Index
Figure 8: CIFAR100 as in-distribution dataset. Left (a): Hierarchical block pattern of K. Middle (b): Top 100 eigenvalues of K for different representation. Right (c): OOD detection for CIFAR100 vs.
SVHN with the top k-th principal component.

---

As seen in Figure 7a we observe a significant improvement in the OOD detection performance using HypStructure with the Mahalanobis score eq. (9. Afte training a composite loss with CPCC till convergence, let denote the matrix of the normalized in-distribution trained feature as Z E R Naturally, we inspect the eigenvalue properties of Σ (i.e, 2), and observe that K Z2 E Rax exhibits hierarchical block structure (Figure (aa) where the diagonal blocks have significantly higher correlation than other off-diagonal values, leading us to the following theorem.
Theorem 5.1 (Eigenspectrum of Structured Representation with Balanced Label Tree). Let T be a balanced tree with height H, such that each level has C₁, nodes, h E [O, H). Let us denote each entry of K as ph where h is the height of the lowest common ancestor of the row and the column sample. If r 2 0, vh, then: (i) For h = O, we have Co - C1 eigenvalues X = 1-1 72. (ii) For 0 < h H- 1, we have Cn Ch+11 eigenvalues Xn An-1 + (rth phtty (iii) The last
eigenvalue is λи- AH-1 + Cor"!
We defer the eigenspectrum analysis for an arbitrary label tree to Appendix A Theorem 5.1 implies phase transition pattern in the cigenspectrum. There always exists a significant gap in the eigenvalues representing each level of nodes in the hierarchy, and the eigenvalues corresponding to the coarsest level are the highest in magnitude. CIFAR 100 has a balanced three-level label hierarchy where each coarse label has five fine labels as its children. In Figure [bb] we visualize the eigenspectrum of CIFAR 100 for HypStructure, CPCC and the Flat objective. We observe a significant drop in the eigenvalues for features learnt from two hierarchical regularization approaches, C2-CPCC and HypStructure, at approximately the 20th largest eigenvector (which corresponds to the number of coarse classes), whereas these phase transitions do not appear for standard flat features. We also observe that the magnitude of coarse eigenvalues are approximately at the same scale.
In summary. Theorem 5.1 helps us to formally characterize the difference between flat and structured representations. CPCC style (eq. (T) regularization methods can also treated as dimensionality reduction techniques, where the structured features can be explained mostly by the coarser level features. For the OOD detection setting, this property differentiates the ID and OOD samples the coarse level itself using a lower number of dimensions, and makes the OOD detection task easier. We visualize the OOD detection AUROC on SVHN (OOD) corresponding to the CIFAR 100 (ID) features with the top-k k principal component for different methods, in Figure 8c We observe that for features learnt using HypStructure, accurately embedding the hierarchical information leads to the top 20 eigenvectors (corresponding to the coarse classes) being the most informative for OOD detection. Recall that CIDER [61] is a statc-of-the- the-art method proposed specifically for improving OOD detection by increasing inter-class dispersion and intra-class compactness. We note that CPCC style (eq. ① methods can be seen as generalization of CIDER on higher-level concepts, where the same rules are applied for coarse labels well, along with the fine classes. When the IDD and OOD distributions are far enough, using coarse level feature might be sufficient for OOD detection.
Related Work
Learning with Label Hierarchy. Several recent works have explored how to leverage hierarchical information between classes for various purposes such as relational consistency (14, designing
specific hierarchical classification architectures [10T] 25 68 hierarchical conditioning of the logits
[13, learning order preserving embeddings [15], and improving classification accuracy [91 86 48)
49 108, 34). The proposed structural regularization framework in [04]]] offers an interesting approach to embed a tree metric to learn structured representations through an explicit objective term, although they rely on the f distance, which is less than ideal for learning hierarchies.
Hyperbolic Geometry. [66] first proposed using the hyperbolic space to learn hierarchical represen- tations of symbolic data such as text and graphs by embedding them into a Poincaré ball. Since then, the use of hyperbolic geometry has been explored in several different applications.
440 proposed
hyperbolic image embedding for few-shot learning and person re- re-identification.
22 proposed
hyperbolic neural network layers, enabling the development of hybrid architectures such as hyperbolic convolutional neural networks (78), graph convolutional networks [12], hyperbolic variational au- toencoders (56) and hyperbolic attention networks (44. Additionally, these hybrid architectures have also been explored for different tasks such as deep metric learning [18] [00], object detection [46]
and natural language processing [16]. There have also been several investigations into the properties
of hyperbolic spaces and models such as low distortion (75), small generalization error (84] and

---

representation capacity [62]. However, none of these works have leveraged hyperbolic geometry for explicitly embedding hierarchy the representation space via structured regularization, and usually attempt to leverage the underlying hierarchy implicitly using hyperbolic models.
7 Discussion and Future Work
In this work, we introduce HypStructure, a for incorporating the label hierarchy into the representation space using hyperbolic geometry. In particular, we demonstrate that accurately embedding the hierarchical relationships leads to empirical improvements in both classification well as the OOD detection tasks, while also learning hierarchy- informed features that are more interpretable and exhibit less distortion with respect to the label hierarchy. We are also the first to formally characterize properties of hierarchy-infone an eigenvalue analysis, and also relate it to the OOD detection task, to the best of our knowledge. We acknowledge that our proposed method depends hierarchy for computing the HypCPCC objective. If the hierarchy is unavailable or contains noise, this could present challenges. Therefore. it important evaluate how injecting noisy hicrarchies into CPCC-based based methods impacts downstream tasks. While the current work uses the Poincaré ball model, exploring the representation trade-off's and empirical performances using other models of hyperbolic geometry in HypStructure, such as the Lorentz model [67] is an interesting future direction. Further theoretical investigation into establishing the error bounds of CPCC style structured regularization objectives is of interest as well.
structural regularization framework
features via
the availability or construction of external
Acknowledgement
SZ and HZ are partially supported by an NSF IIS grant No. 2416897. HZ would like to thank the support from a Google Research Scholar Award. MY was supported by MEXT KAKENHI Grant Number 24K03004. We would also like to thank the reviewers for their constructive feedback during the review process. The views and conclusions expressed in this paper solely those of the authors and do not necessarily reflect the official policies or positions of the supporting companies and government agencies.
10

---

References
[1] Aaron Adcock, Blair D Sullivan, and Michael W Mahoney. Tree-like structure large social and information networks. In 2013 IEEE 13th international conference data mining, pages 1-10. IEEE, 2013.
[2] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563-1572, 2016.
[3] Jorge Cadima, Francisco Lage Calheiros, and Isabel P. Preto. The eigenstructure of block- structured correlation matrices and its implications for principal component analysis. Journal of Applied Statistics, 37(4):577-589, April 2010.
[4] James W Cannon, William J Floyd, Richard Kenyon, Walter R Parry, et al. Hyperbolic geometry. Flavors of geometry, 31(59-115):2, 1997.
[5] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 13:9912-9924, 2020.
[6] lnes Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. oowddimenioonal hyperbolic knowledge graph embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6901-6914, 2020.
[7] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[8] Wei Chen, Wenjie Fang, Guangda Hu, and Michael W Mahoney. On the hyperbolicity of small-world and treelike random graphs. Internet Mathematics, 944:4434-41,, 2013.
[9] Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos. Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEÉ Conference on Computer Vision and Pattern Recognition, pages 3606-3613 2014.
[10] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc VL. Autoaugment: Learning augmentation strategies from data. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 113-123, 2019.
[11] Ekin Dogus Cubuk, Barret Zoph, Jon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with a reduced search space. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 06613-18624 Curran Associates, Inc., 2020.
[12] Jindou Dai, Yuwei Wu, Zhi Gao, and Yunde Jia. A hyperbolic-tor torpbplio-tt-yyyrboolii graph convolu- tional network. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 154-163, 2021.
[13] Jim Davis, Tong Liang, James Enouen, and Roman Ilin. Hierarchical classification with confi- dence using generalized logits. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 1874-1881. IEEE, 2021.
[14] Jia Deng, Nan Ding, Yangqing Jia, Andrea Frome, Kevin Murphy, Samy Bengio, Yuan Li, Hartmut Neven, and Hartwig Adam. Large-scale object classification using label relation graphs. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision - ECCV 2014, pages 48 48-64, Cham, 2014. Springer International Publishing.
[15] Ankit Dhall, Anastasia Makarova, Octavian Ganea, Dario Pavllo, Michael Greeff, and Andreas Krause. Hicrarchical image classification using entailment cone embeddings. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 836 837, 2020.
[16] Bhuwan Dhingra, Christopher Shallue, Mohammad Norouzi, Andrew Dai, and George Dahl. Embedding text in hyperbolic spaces. Proceedings of the Twelfih Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-12) pages 59 2018

---

[17] Gamaleldin Elsayed. Dilip Krishnan, Hossein Mobahi, Kevin Regan, and Samy Bengio. Large margin deep networks for classification. Advances in neural information processing systems,
31, 2018.
[18] Aleksandr Ermolov, Leyla Mirvakhabova, Valentin Khrulkov, Nicu Sebe, and Ivan Oseledets. Hyperbolic vision transformers: Combining improvements in metric learning. Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 7409-7419, 2022.
[19] Christiane Fellbaum. WordNet: An Electronic Lexical Database. Bradford Books, 1998.
[20] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. In Advances in Neural Information Processing Systems, pages 5345-5355, 2018.
[21] Songwei Ge, Shlok Mishra, Simon Komblith, Chun-Liang Li, and David Jacobs. Hy- perbolic contrastive learning for visual representations beyond objects. arXiv preprint 1rXi::2222.00533 2022.
[22] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix Wichmann, and Wieland Brendel. Imagenet-trainee trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019.
[23] Mikhael Gromov. Hyperbolic groups. In Essays in group theory, pages 75-263. Springer, 1987.
[24] Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter Battaglia, Victor Bapst, David Raposo, Adam Santoro, and Nando de Freitas. Hyperbolic attention networks. In International Conference on Learning Representations, 2019.
[25] Yanming Guo, Yu Liu, Erwin M Bakker, Yuanhao Guo, and Michael S Lew. Con-rnn: large-scale hierarchical image classification framework. Multimedia tools and applications, 77(8):10251-1027) 2018.
[26] Yunhui Guo, Xudong Wang, Yubei Chen, and Stella X Yu. Clipped hyperbolic classifiers are super-hyperbolic classifiers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11-20, 2022.
[27] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-97739 9938, 2020.
[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference computer vixion and pattern recognition, pages 770-778, 2016.
[29] Olivier Henaff. Dfta-efficient image recognition with contrastive predictive coding. In International conference machine learning, pages 4182-4192 PMLR, 2020.
[30] Dan Hendryeks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distribution examples in neural networks. In International Conference on Learning Rep- resentations, 2017.
[31] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019.
[32] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representations, 2020.
[33] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildonn Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations, 2019.
12

---

[34] David T Hoffmann, Nadine Behrmann, Juergen Gall, Thomas Brox, and Mehdi Noroozi. Ranking info noise contrastive estimation: Boosting contrastive learning via ranked positives. In AAAI Conference on Artificial Intelligence, 2022.
[35] Roger A. Horn and Charles R. Johnson. Matrix analysis. Cambridge University Press, Cambridge; New York, 2nd ed edition, 2012.
[36] Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out- of-distribution image without learning from out-of-distributio data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10951-10960, 2020.
[37] Rui Huang and Yixuan Li. Towards scaling out-of-distribution detection for large semantic space. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[38] Edmond Jonekheere, Poonsuk Lohsoonthorn and Francis Bonahon. Scaled gromov hyperbolic graphs. Journal of Graph Theory, 57(2):157-180, 2008.
[39] Prannay Khosla, Piotr Teterwak. Chen Wang, Aaron Sama, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In Advances in Neural Information Processing Systems, volume 33, pages 18661-18673, 2020.
[40] Valentin Khrulkov, Leyla Mirvakhabova, Evgeniya Ustinova, Ivan Oseledets, and Victor Lempitsky. Hyperbolic image embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6418-6428, 2020.
[41] Sungyeon Kim, Dongwon Kim, Minsu Cho. and Suha Kwak. Proxy anchor loss for deep metric learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 3238-3247, 2020.
[42] Polina Kirichenko, Pavel Izmailov, and Andrew G Wilson. Why normalizing flows fail to detect dutto--distt data. Advances in Neural Information Processing Systems, 33, 2020.
[43] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Large scale learning of general visual representations for transfer. arXiv preprint 1rXiv:1912.11332 13370 2(8), 2019.
[44] Dmitri Krioukov, Fragkiskos Papadopoulos, Maksim Kitsak, Amin Vahdat, and Marián Boguná, Hyperbolic geometry of complex networks. Physical Review E, 82(3):036106, 2010.
[45] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[46] Christopher Lang, Alexander Braun, Lars Schillingmann, and Abhinav Valada. On hyperbolic embeddings in object detection. In Pattern Recognition: 44th DAGM German Conference, DAGM GCPR 2022, Konstanz, Germany, September 27-30, 2022, Proceedings, pages 462-476. Springer, 2022.
[47] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for de- tecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Processing Systems, pages 7167-7177, 2018
[48] Jie Lei, Zhenyu Guo, and Yang Wang. Weakly supervised image classification with coarse and fine labels. In 2017 14th Conference on Computer and Robot Vision (CRV), pages 240-247, IEEE, 2017.
[49] Li-Jia Li, Chong Wang, Yongwhan Lim. David M. Blei, and Li Fei-Fei. Building and using a semantivisual image hierarchy. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3336-3343, 2010.
[50] Lingxiao Li, Yi Zhang, and Shuhui Wang. The euclidean space is evil: hyperbolic attribute edit- ing for few-shot image generation. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 22714-22724, 2023.
13

---

[51] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distributie image detection in neural networks. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
[52] Shaoteng Liu, Jingjing Chen, Liangming Pan, Chong-Wah Ngo, Tat-Seng Chua, and Yu-Gang Jiang. Hyperbolic visual embedding learning for zero-shot recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9273-9288 2020.
[53] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distributiin detection. Advances in Neural Information Processing Systems, 2020.
[54] Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolutional neural networks. In International Conference on Machine Learning, pages 507-516. PMLR, 2016.
[55] Teng Long, Pascal Mettes, Heng Tao Shen, and Cees G. M. Snoek. Searching for actions on the hyperbole. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
[56] Emile Mathieu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Teh. Continuous hierarchical representations with poincaré variational auto-encoders. Advances in neural information processing systems, 32, 2019.
[57] Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 1802.03426 2018.
[58] I. McLean, A.B. Urken, and F. Hewitt. Classics of social choice. University of Michigan Press, 1995.
[59] Yifei Ming, Ziyang Cai. Jiuxiang Gu, Yiyou Sun, Wei Li, and Yixuan Li. Delving into out-of-distribution detection with vision-language representations. In Advances in Neural Information Processing Systems, 2022.
[60] Yifei Ming, Ying Fan, and Yixuan Li. POEM: Out-of-distributioo detection with posterior sampling. In Proceedings of the 39th International Conference on Machine Learning, pages 15650-15665, 2022.
[61] Yifei Ming, Yiyou Sun, Ousmane Dia, and Yixuan Li. How to exploit hyperspherical embed- dings for dfitt--ddit detection? In The Eleventh International Conference on Learning Representations, 2022.
[62] Gal Mishne, Zhengchao Wan, Yusu Wang, and Sheng Yang. The numerical stability of hyperbolic representation learning. In International Conference on Machine Learning, pages 24925-24949, PMLR, 2023.
[63] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? International Conference on Learning Representations, 2019.
[64] Nelson Nauata, Hexiang Hu, Guang-Tone Tong Zhou, Zhiwei Deng, Zicheng Liao, and Greg Mori. Structured label inference for visual understanding. IEEE transactions on pattern analysis and machine intelligence, 42(5))
2019.
[65] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop deep learning and sead learning, volume 2011, page 4. Granada, 2011.
[66] Maximillian Nickel and Douwe Kiela. Poincaré embeddings for Icarning hierarchical rep- resentations. In Advances in Neural Information Processing Systems, pages 6338-6344 2017.
[67] Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the Lorentz model of hyperbolic geometry. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 3779-3788. PMLR, 10-15 Jul 2018.
14

---

[68] Khondaker Tasrif Noor, Antonio Robles-Kelly and Brano Kusy. A capsule network for hierarchical multi-label image classification. In Joint IAPR International Workshops on Statistical Techniques in Puttern Recognition (SPR) and Structural and Syntactic Puttern Recognition (SSPR), pages 163-172. Springer, 2022.
[69] Francesco Pinto, Harry Yang. Ser-Nam Lim, Philip Torr, and Puneet K. Dokania. Using mixup as a regularizer can surprisingly improve accuracy & out-of-distribution robustness. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[70] Erzsébet Ravasz and Alsert-László Barabási. Hierarchical organization in complex networks. Physical review E, 67(2):0261 12, 2003.
[71] Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. In Advances in Neural Information Processing Systems, pages 14680-14691, 2019.
[72] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei- Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):111-52 2015.
[73] Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation tradeoffs for hyperbolic embeddings. In Proceedings of the International Conference on Machine Learning, pages 4460-4469, 2018.
[74] Shibani Santurkar, Dimitris Tsipras, and Aleksander Madry. Breeds: Benchmarks for subpop- ulation shift. In International Conference on Learning Representations, 2021.
[75] Rik Sarkar. Low Distortion Delaunay Embedding of Trees in Hyperbolic Plane, volume 7034 of Lecture Notes in Computer Science, page 355-366. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012.
[76] Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self- supervised outlier detection. In International Conference Learning Representations, 2021.
[77] Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood- generative models. In International Conference Learning Representations, 2020.
[78] Ryohei Shimizu, Yusuke Mukuta, and Tatsuya Harada. Hyperbolic neural networks+++ In International Conference on Learning Representations, 2020.
[79] Robert R. Sokal and F, James Rohlf. The comparison of dendrograms by objective methods. Taxon, 1962.
[80] Yue Song, Nieu Sebe, and Wei Wang. Rankfeat: Rank-1 feature removal for autoo-ddistributinn detection. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[81] Sainbayar Sukhbaatar, Joan Bruna, Manohar Paluri, Lubomir Bourdev, and Rob Fergus. Training convolutional networks with noisy labels. arXiv preprint 1 1406.0000, 2014.
[82] Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In Advances Neural Information Processing Systems, 2021.
[83] Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. ICMI., 2022.
[84] Atsushi Suzuki, Atsushi Nitanda, Jing Wang. Linchuan Xu, Kenji Yamanishi, and Marc Cavazza. Generalization error bound for hyperbolic ordinal embedding. In International Conference on Machine Learning, pages 10011-10021 PMLR, 2021.
15

---

[85] Jihoon Tack, Sangwoo Mo. Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In Advances in Neural Information Processing Systems, 2020.
[86] Fariborz Taherkhani, Hadi Kazemi, Ali Dabouci, Jeremy Dawson, and Nasser M Nasrabadi. A weakly supervised fine label classifier enhanced by coarse supervision. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6459-6468, 2019.
[87] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XI 16, pages 776-794. Springer, 2020.
[88] Abraham A Ungar. Hyperbolic trigonometry and its application in the poincaré ball model of hyperbolic geometry. Computers & Mathematics with Applications, 41(1-2):135-144 147, 2001.
[89] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. Journal of machine learning research, 9), 2008
[90] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[91] Panqu Wang and Garrison W. Cottrell. Basic level categorization facilitates visual object recognition, 2015.
[92] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML, 2022.
[93] Hermann Weyl. Das asymptotische verteilungsgesetz der eigenwerte linearer partieller differen- tialgleichungen (mit einer anwendung auf die theorie der hohlraumstrahlung), Mathematische Annalen, 4144):441-49, December 1912.
[94] Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Led- sam, Patricia WilWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Con- trastive training for improved out-of-distribution detection. arXiv preprint 2007::0007.05566 2020.
[95] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 3733-3742, 2018
[96] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference computer vision and pattern recognition, pages 3485-3492. 3492. IEEE, 2010.
[97] Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An out-o--distribution detection score for variational auto-encoder. Advances in Neural Information Processing Systems, 33, 2020.
[98] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10687-10698, 2020.
[99] Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint / /544066555 2015.
[100] Jiexi Yan, Lei Luo, Cheng Deng, and Heng Huang. Unsupervised hyperbolic metric learning. In Proceedings the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12465-12474, 2021.
16

---

[101] Zhicheng Yan, Hao Zhang, Robinson Piramuthu, Vignesh Jagadeesh, Dennis DeCoste. Wei Di, and Yizhou Yu. Hd-cnn: hierarchical deep convolutional neural networks for large scale visual recognition. In Proceedings of the IEEE international conference on computer vision, pages 2740-2748, 2015.
[102] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song. Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of scage image dataset using deep learning with humans in the loop. arXiv preprint 1rxivv1506.03366 2015.
[103] Yun Yue, Fangzhou Lin, Kazunori D Yamada, and Ziming Zhang. Hyperbolic contrastive learning. arXiv preprint wrXiv:2302.01409, 2023.
[104] Siqi Zeng, Remi Tachet des Combes, and Han Zhao. Learning structured representations by embedding class hierarchy. In The Eleventh International Conference on Learning Represen- tations, 2022.
[105] Jingyang Zhang, Nathan Inkawhich, Randolph Linderman, Yiran Chen, and Hai Li. Mixture outlier exposure: Towards out-of-distribution detection in fine-grained environments. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV),
pages 5531 January 2023.
[106] Jingyang Zhang, Jingkang Yang, Pengyun Wang, Haoqi Wang, Yueqian Lin, Haoran Zhang, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, et al. Openood v1. 5: Enhanced benchmark for nut-of-distribution detection. arXiv preprint arXiv:2306.09950 093001 2023.
[107] Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems, 31, 2018.
[108] Yu Zheng, Jianping Fan, Ji Zhang, and Xinbo Gao. Hierarchical learning of multi-task sparse metries for large-scale image classification. Puttern Recognition, 67:97-109, 2017.
[109] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places:
10 million image database for scene recognition. IEEE transactions pattemn analysis and machine intelligence, 1456) 1464, 2017.
[110] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins. Local aggregation for unsupervised learning of visual embeddings. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6002-6012, 2019.
17

---

Appendix
This appendix segmented into the following key parts.
1. Section A continues the analysis of the eigenspectrum of CPCC-optimized representation matrix and generalizes it for an arbitrary label tree.
2. Section discusses additional details about our proposed method HypStructure, its implementation and broader impact of our work. In particular, an overview of the method is first provided, and then we describe hyperparameter settings of our method and the main baselines, followed by extra dataset details and explanation of evaluation metrics.
3. Section Creports ablation studies, detailed results on OOD detection and provides additional experimental results and visualizations not included in the main paper due to lack of space.
A Details of Eigenspectrum Analysis
In this section, we first introduce some notations, discuss the setup for our analysis, followed by pre- liminary lemmas, and then characterize the cigenspectrum of CPCC-based structured representations in Theorem A.2 an arbitrary label tree and Theorem 5. for a balanced tree presented in the main body.
Proof Sketch The proof of Theorem A.2 and Theorem 5.1 relies on the important observation of a hierarchical block structure of the covariance matrix of CPCC-regularized features, as shown in Figure Sa which will also supported by Lemma A. and Corollary A. Theorem A and Lemma A.2 characterize the eigenvalues of block correlation matrix induced from a basic tree where the matrix only has three types of values: diagonal values of 1s, one for within group entry, and another for across group entry. Larger within group entries lead to the larger cigenvalues. Theorem A. ( and Lemma A.2 are then used as the base case for the induction proof of Theorem 5.1 For an arbitrary tree, in Theorem A.2 we use Weyl's Theorem [93] to bound the gap between within group entries and across group entries that leads to the phase transition of eigenvalues.
Setup details After training with the HypStructure loss till convergence, let denote the feature matrix as Z E Raxd, where each row of a d-dimensional vector of an in distribution training sample, and the CPCC is maximized to 1. We let Co n. C2,,C,,.... Cи of class labels at height of the tree T. Following the standard pre-processing steps in OOD detection [76].
be the number
assume that the features |2 = 1, Besides, we assume that in T, the distance from root node to each leaf node the same. Otherwise, following (74, we can insert dummy parents or children into the tree make sure vertices at the same level have similar visual granularity. We then apply CPCC to each node in the extended tree, where each leaf node is one sample. We note that although this is slightly different from the implementation where the leaf nodes are fine class nodes, the distance for samples within fine classes are automatically minimized by classification loss like cross-entropy and supervised contrastive loss.
standardized and normalized that BJ = 0 and
want to analyze the eigenspectrum of the inverse sample covariance matrix n-1 Z Z, which is the same as investigating the eigenvalues of K = ZZ where Z is ordered by classes at all levels, i.e., samples having the same ane-grained labels and coarse labels should be placed together. This is because the matrix scaling and permutation will not change the order of singular values.
Given these assumptions, 1
Since CPCC (eq. (1) is a correlation coefficient, when it is maximized, the by n pairwise Poincaré distance matrix is perfectly correlated with the ground truth pairwise tree-metric matrix, where each entry is the tree distance between two samples on the tree, no matter we apply CPCC to leaves or all vertices. This implies that in the similarity matrix K, the relative order of entries are the opposite of tree matrix, and it is trivial to show it as follows
Lemma A.I. The relative order of entries in K will be the reverse of the order in tree distance.
2(u,, u) = 2(u, v). Now considering the CPCC computation, if the CPCC is maximized, the pairwise Euclidean matrix is of
Proof. When |||||| = ||a||| = 1, (u, v)² = ||u m1/ = ||||| + ||||||
18

---

the scalar factor of the tree distance matrix. Since each entry of K is the dot product of two samples.
the relative order in K is the opposite.
Corollary A.I. If we use the Poincaré distance (eq. B in CPCC and let the curvature constant 1, the statement of cosine distance in Lemma still holds.
Proof. Since the Poincaré distance (eq. (3) is only defined for vectors with magnitude less than 1, let us consider the case where before the clipping operation, both and are outside the unit ball. After
applying clip'. ||u/| = ||
c, where e small constant (10-5). Then ||wl] (1 - c)²
1- 2E + e². Define 2 e² as ç. making ||u]] == 1 - & where & is also small constant such that Os²²) is negligible.
|| - vll + V |/ull2 ||ll12 - 2u w + 1
Poincaré(u, u) = 21n
v (1 |||| (1 11/013)
| - 1/ + v 2 - 2u v 25 + E²
21n
||u - vll + /2
2u 10- 2
23 2
5
||u wll + /uull + 110:11 2u
|u
ull
2 In
We can see that the Poincaré distance monotonically increases with Euclidean distances ||u e|]- This property ensures the relative order of any two entries for Euclidean CPCC and Poincare CPCC matrices in K to be the same. Then, Poincare, to have the hierarchical diagonalized structure as in Figure Sal So any statement applied for Poincaré version of CPCC will also hold for the Euclidean CPCC counterpart.
argue about the structure of K, either Euclidean or
For each level of the tree, due to the optimization of CPCC loss, the corresponding off diagonal entries of K, which represent the intra-level-class similarities, much smaller than inter-level-clas values. We thus have a symmetric similarity matrix that takes on the following structure, where the red regions are greater than orange regions, which are further greater than the blue regions.
12
11
12
12
13
14
[
I2
T12
122
112
K
13
T23
33
r
23
33
T
7IA
24
Each non non-diagonal entry is called r where i.j are the index of the diagonal block, or the finest label id of one sample, and h is the height of the lowest common ancestor of the two samples in the row and the column. Since every two leaves sharing the lowest common ancestor of the same height have the same tree distance, each entry of K with the same superscript will be the same so we can drop the j subscript in the notation. The size of each block is defined by the number of samples within one label. Then, the shown submatrix K corresponds to the following tree in Figure 9 Next, we present several useful lemmas and theorems.
19

---

Figure 9: Subtree corresponds to the shown submatrix of K.
Theorem A.1 (3)) Let R be a x p full-rank correlation matrix that has k-group block structure, with groups of size Pæ (f = k, Σ P: = p). Let ri be the correlation for any pair of different variables within group Let rty be the common correlation between any variable in group and j. Denote the mean of the i-th diagonal block of by R3 (1/p:) (pa-1)rs)) 1)r;))- Then:
k
has cigenvalues 1 - Ta (i = 1: k).
2. The rest the eigenvalues are those from & symmetric matrix whose diagonal elements are au = p,R, and whose off-diagonal elements are dzy vP: Py
Lemma A.2. Given d by d matrix M where Mt. 1, Vi [d]. and My p otherwise, i.c.
p
M
1
p p.
Lp
p Aa
has eigenvalues A + p(d 1) and X2
Proof. Using the definition of cigenvalues, we want to compute the determinant of matrix
p
P
1-X
P
A
Adding the second till the last row to the first row, we have
A (d 1)p 1 1- 1 1)p
X (d 1)p
p
1-X
P
1-X
Dividing the first row by + (d 1)p, we now have
1-A X
p
p
20

---

Subtracting the second till the last row by times the first row results in an upper triangular matrix
1
x P
1 --X-P
0
0
p
Thus, det(M - AT) = A + (d 11
p)d-1
Notice that Lemma A.2 is a special case Theorem A. where the label tree is two level basic tree with the root node and leaves in the second label all being the direct children of the root node. Now we can leverage Theorem A. and Lemma A.2 to investigate eigenspectrum of by proving the following theorem:
Theorem A.2 (Eigenspectrum of CPCC-based Structured Representation). If T a tree whose root node has height H where each level has Cn nodes, h E [0, H]. K ZZ" is a block structured correlation matrix as result CPCC optimization, where each off-diagonal entry E [-1, can be written as h and h the height of the lowest common ancestor of the i-th row and the 00j-th column sample. Let A = r² r". Pi, E [Ch,] be the number of children for nodes at height h, and Pmax be the maximum. For any h 2 1, if
M 2 0, ,++ < m, then
(i) We have Co Cn eigenvalues, that come from the eigenvalues of a Co x Co matrix sharing the same C₁ of p; p; diagonal blocks from K subtracting n and off diagonal values are all zero.
iii) The of C₁ eigenvalues come from a C₁ C₁ matrix, whose diagonal entries are the mean of each p; p; diagonal block from K, and the off diagonal entries are /PiP5Taj where ri is the correlation between node and node j of height h.
M2 1) Co Cn eigenvalues are all smaller than C₁ eigenvalues. Paus (Ch 1I
(iii) If m
Proof. Part (i) and (ii) can be extended from the proof of Theorem A.1 Let G' be the C₁ matrix where G.y = if the i-th sample is in group j. otherwise Gi = 0. For any n C₁, eigenvectors in the orthogonal complement of the column space of G, the eigenvector of K is also the eigenvector of
K1
0
0
K2
K
where due to the hierarchical structure of block matrix, K₁ has the format of
r
0
0
p
p
The rest of O₁ eigenvectors come from the symmetric C₁ x Cs. matrix A (CT (G) G)21/2 -1) by some basica algebra, we know Oij
/P
1 For more details, we refer the reader to Theorem (sum of all rij entries in x Py block) "P5"
3.1 in 3 where C₁ k using their notation.
Since the largest absolute value of K's eigenvalues, is bounded above by the largest row sum of the absolute values of K [35] first n C₁ eigenvectors bounded above by U max, (1 rd)
(p. - 1) 7²) (1 nh + (Pmax 1)4.
21

---

On the other hand, for the rest of Cj eigenvalues, we analyze matrix A:
A < A + A2
1 + (p1 1)r
0 1 (p2 1)
0
+ (px 1)ra
(Pmax
1) m
Pmax m
Pmmx M
(pmax 1)(r" ,ht
Pmaxm1
Pmax T
(Pmax 1)(rt r(t)
Prmx
Pmnx 7
The inequality comes from the effect of the maximization of CPCC that r and n < m.. The eigenvalues of A1, A2 have the analytical form, where A eigenvalues have the form of 1 (p= 1)r and Ag's eigenvalues can be derived by Lemma A.2 By Weyl's inequality
....ph 2 p 2 .....
193] the minimum of these C cigenvalues is at least L (1 (Pmin 1)r*) [(maxx 1)A
Pmax + kpmax 2 (1 + - 1)r*) [(Pmax 1) Pmax + m1
To guarantee eigenvalues from Part (ii) are larger, want U. We solve this inequality with m. and we will get the desired range of m.
When 1h pd1 Theorem A.2 we have
Therefore, for a three level basic tree with
1)), Co C1 eigenvalues are all smaller than C1 eigenvalues. In general, we have shown that when m, i.e., the across group similarity is sufficiently small, the eigenvalue gap always exists. When the label tree T is balanced, we further specify the expression of each eigenvalue and the amount of eigenvalue gaps.
only rl , if < MMMPpmuu C1
We now formally restate the Theorem 4.1 from the main paper and give its proof.
Theorem 5.1 (Figenspectrum
Structured Representation with Balanced Label Tree). Let T be a balanced tree with height H, such that each level has Cn nodes, h E [0. H). Let us denote each entry of K as h where h is the height of the lowest common ancestor of the row and the column sample. T h 0, Vh, then: (i) For h O, we have Co Ci eigenvalues An r22 (ii) For Co. 0 < h < H - 1, we have Cn Ch+1 eigenvalues An = λn-1 (r p ( C. (iii The last eigenvalue is λн- A + Cor
Proof. From Corollary A.1 know that K CoxCC has block-wise structure.
Since all statements are presented recursively, we prove the theorem by structural induction on the height of the tree.
The base case is Lemma A.2 with a two level hierarchy tree where only (i) and (iii) are applicable, and p 22, Co d, C1 = I. By Lemma A.2 K has Co - eigenvalues as A - r and one eigenvalue as A = 1 + (Co 1)r
) +
Let us now assume that the theorem is true for the balanced tree whose root node is at height H 1. Then if have tree with height H. We call the resulting matrix Kg-
By the first bullet point of Theorem A.I we directly get Ao from (i). Then by the second bullet point of Theorem A.I the rest of the eigenvalues are from the symmetric matrix A#-1 R2 xC; whose diagonal elements are + (Co/ - 1)r" and whose off diagonal elements are Co/C, r for j 1
The key is to observe that A is still block structured matrix. After A is scaled by y, the resulting matrix be also seen result of maximizing CPCC where the off diagonal blocks have smaller values.
Applying the hypothesis induction, then know the expression of eigenvalues for AH-1
22

---

ii we have C1 C2 eigenvalues of the form
d = 7(1 1 Co/oCC
Co
Cl p
Co
+
1)r²
Ci ,
C1
Co
r + 9
(r' -P2)
Co (1² p2)
λo +
C₁
( For 0 < h H 2, we have Ch+1 Ch+2 eigenvalues of the form
Co
ph+2
Ci
Ah+1 - An = -
7
Ch+1
An+1 = An + (phttl n,,t Co
Ch+1
(iii) The last eigenvalue is Therefore, we proved the theorem by showing the induction step from KH-1 to K holds.
Cp
C1
Cor
Note that the true symmetric covariance matrix K' might not be having the exact format as K, but can be seen as a perturbation of K where | - K|| S e, is small constant. By Weyl's inequality 933 the approximation error of each eigenvalue bounded by [}, e,
+ +
B Additional Algorithm and Experimental Details
In this section, we first provide an overview of our algorithm, followed by a discussion on the choice of the flat loss and additional experimental details about the training and evaluation metrics.
B.1 Broader Impact Statement
Our work proposes HypStructure. structured hyperbolic regularization approach to embed hi- crarchical information into the learnt representations. This provides significant advancements in understanding and utilizing hicrarchical real-world data, particularly for tasks such representation learning, classification and OOD detection, and we recognize both positive societal impacts and po- tential risks of this work. The ability better model hierarchical data in structured and interpretable fashion particularly helpful for domains such as Al for science and healthcare, where the learnt rep- resentations will be more reflective of the underlying relationships in the domain space. Additionally, the low-dimensional capabilities of hyperbolic geometry can lead to gains in computational efficiency and reduce the carbon footprint in large scale machine learning. However, real-world hierarchical data often incorporates existing biases which may be amplified by structured representation learning, and hence it is important incorporate fairness constraints mitigate this risk.
B.2 Pseudocode for HypStructure
The training scheme for our HypStructure based structured regularization framework is provided in Algorithm [ At high level, in HypStructure, we optimize a combination of the following hyperbolic space to correspond with the label hierarchy, (2) hyperbolic centering loss to position the representation corresponding the root of the node the centre of the Poincaré ball and the children nodes around it.
two losses: (1) hyperbolic CPCC loss to encourage the representations
23

---

Algorithm/ I HypStructure: Hyperbolic Structured Representation Learning
Input: Batch size B, Label tree T=
= (V, E, e), Number of epochs K, Task Loss formulation Flat Encoder fo. Classifier Head fw. Learning Rate 7) Hyperparameters a.
1: Initialize model parameters: 8, w
2: for epoch = 2.. K do
3: for batch = 2.... B do
4:
Get image-label pairs: {(.,, y: }=
5:
Forward pass to compute the representations: (Z) (z1...zn) + (Jo(x))).. ... ((((ai))..((((zn)))
6:
Compute the Task loss: (z,) y/)
7:
Get hyperbolic representations using exp. map (eq. (4); 2, t -epp((z))
8:
Calculate class prototypes using hyp. Averaging (eq. 66); wi HHppAve (#f ( . 29)
9:
Compute pairwise hyp. distances (eq. (3) Veg, By E V. p(v;, 02) + d3. W))
10:
Get hyp. CPCC loss (eq. ( HypCPCC(dr
11:
Compute hyp. centering loss using (Equation (6:: feenter
||HppAee ( ............ mB
12:
Get total loss using Equation (8): (DDB)
13:
Compute Gradients for learnable parameters at time t: un (0, w)) & Vo
14:
Refresh the parameters: (0, w)++1 <- (0,w); D ue (0,
Output:
(Z1....2N);66 w
B.3
Choice of Flat loss
We use the Supervised Contrastive [39] (SupCon) loss as the first choice for flat loss in our experi- mentation. Let dy be the one-hot vector with the y-th index as 1. The Cross Entropy (CE) loss, defined between the predictions g o fo(x)) and the labels y, lcu(g o f(z), v) == Eiels 9 log fx))(())))))
has been used quite extensively in large-scale classification problems in the literature [0
43 98
However, several shortcoming of the CE loss, such as lack of robustness [8]]
107 and poor gener- have been discovered in recent research. Contrastive learning has emerged
alization 17 54
viable alternative to the CE loss, to address these shortcomings 95. 29
87
33 27 The underlying
principle for these methods is pull together embeddings for positive pairs and push apart the embeddings for negative samples, in the feature space. In the absence of labels, positive samples are created by data augmentations of images and negative samples are randomly chosen from the minibatch. However, when the labels are available, the class information can be leveraged to extend this methodology as Supervised Contrastive loss (SupCon) by pulling together embeddings from the same class, and pushing apart the embeddings from different classes. This offers a more stable solution for a variety of tasks [39 76 83
Definition B.1 (SupCon Loss). Given a training sample æ;, feature encoder fa(-) and projection head (()). we denote the normalized feature representations from the projection head
h (J)()))
(10)
(M(f(()))))
For the N training samples {(,, (()),y))) ) we denote the training batch of 2N (augmented) pairs as {@ ū:) and define the SupCon loss as:
2N
A==1 1kk i = D.) mu/t
1
2N,
CappCn
logg
(11)
2N
2N Σx=1 1 + i)c"? uLTT
where Ny. refers to the number of images with label in the batch, 7 is the temperature param- eter, - refers the inner product, and u, and un are the normalized feature representations using Equation (10) for z; and Ts respectively.
While the numerator in the formulation in Equation only considers the samples (and its augmen- tations) belonging to the same class, the denominator sums over all the negatives as well. Overall, this encourages the network to closely align the feature representations for all the samples belonging to the same class, while pushing apart the representations of samples across different classes.
We note that our proposed method HypStructure is not limited to the choice of euclidean classifica- tion losses as E and we report additional results with hyperbolic classification losses in Sections C.8 and C.9 respectively, demonstrating the wide applicability of our approach.
24

---

B.4 Implementation Details
B.4.1 Software and Hardware
We implement our method in PyTorch 2.2.2 and run all experiments single NVIDIA GeForce RTX-A6000 GPU. The code for our methodology will be open sourced for a wider audience upon acceptance, in the spirit of reproducible research.
B.4.2 Architecture, Hyperparameters and Training
We use the ResNet-18 28 network as the backbone for CIFARIO, and ResNet-34 as the backbone for CIFARI00 and ImageNet100 datasets. We use a ReLU activated multi layer perceptron with one hidden layer as the projection head h(..) where its hidden layer dimension is the same as input dimension size and the output dimension is 128. We follow the original hyperparameter settings from (3 for training the CIFAR10 and CIFARTI 100 models from scratch with temperature r 0.1, feature dimension 512, and training for 500 epochs with an initial learning rate of 0.5 with cosine annealing, optimizing using SGD with momentum 0.9 and weight decay 10-1 and batch size of 512 for all the experiments. For ImageNet100, we finetune the ResNet-34 for 20 epochs following 61 with an initial learning rate of 0.01 and update the weights of the last residual block and the nonlinear projection head, while freezing the parameters in the first three residual blocks. We use the same values as the regularization parameters for the CPCC loss in Equation (2) (f2-CPCC) and in Equation (8) (our proposed method HypStructure) for a fair comparison and find that the default regularization hyperparameter for the CPCC loss 1.0 for both b -PCC and HypStructur' performs well for the experiments on the CIFARIO and CIFAR 100 datasets. We observe that the 0.5. Additionally,
experiments the IMAGENETIO 100 dataset benefit from lower hyperparameter for the centering loss in our methodology as B = 0.01 for all the experiments. We use the default curvature value of 1.0 for the mapping and distance computations in the Poincaré ball.
set the
B.4.3 Datasets and Hierarchy
We use the following three datasets for our primary experimentation and training this work
1. CIFAR10 45 It consists of 50,000 training images and 10,000 test images from 10
different classes.
2. (55))) It also consists of 50,000 training images and 10,000 test images, however the images belong to 100 classes. Note that the classes are not identical to the CIFARIO dataset.
3. ImageNet100 (2)). This dataset is created as a subset of the large-scale ImageNet dataset following 559. The original ImageNet dataset consists of 1,000 classes and 1.2 million training images and 50,000 validation images. We construct the ImageNet100 dataset from this original dataset by sampling 100 classes, which results in 128,241 training images and 5000 validation images. We mention the specific classes used for sampling below.
Following [59].
the below 100 class id's for creating the ImageNet100 subset: n03877845,
n03000684, n031 16669, 103710721, n02825657, n021 13186, n018 17553, n04239074, n02002556, n04356056, n03187595, n03355925, m03125729, m02058221, n01580077, n03016953, n02843684,
n04371430, n01944390, m03887697, n04037443, n02493793, n01518878, n03840681, n041799113
n01871265, n03866082, 101110077, 1033888877 n03388549, n03908714, 209908874..085550222. n01855032, n02134084,
n03400231, 1004483377..33321334,.222220 m0177550 n02808304, n028840,,,000 m001000000000000101160169 n01601694,
n04136333, n03272562, n03895866,  n06785554 no2111889, n03447721, m02111899...344722.2 n0336659I.
n04376876, n03929855, m02128757, m0237,,2222222333 n07614500 n016950000 n024455 m02104551.
n04090263, n03127925, n04550184, n04600251, m04400055,,,,,4400215,,,, n02488702, n02447002,,,,,20070002
1024887022,0334444551 m03404251. (0, n03633091, no20011111 n02091635.
n03457902, n02233338, n02483362, n04461696 mo2871525, n01689811, m01498881 n01498041, m01498044, n02107312.
n01632458, n03394916, n04147183, n04418357, n03218198, n01917289, n02102318, n02088364,
n09835506, m02095570, n03982430, n04041544, n04562935, n03933933, n01843065. n02128925.
n02480495, nO3425413, n03935335, n02971356, m02124075, n07714571, n03133878, n02097130.
13799, n09399592, n03594945.
25

---

In addition to the training and validation images, we also require the label hierarchy for each of these datasets for the CPCC computation in C-CCCC and HypStructure approaches. For CIFARI00, we use the three-level hierarchy provided with the dataset release) We show this hierarchy in Table 3 where the top-level is the root of the tree.
Table 3: Class Hierarchy of the CIFARIO0 Dataset
Coarse Classes
Fine Classes
aquatic mammals
beaver, dolphin, olter, seal, whale aquarium fish, flatfish, ray, shark, trout orchids, poppies, roses, sunflowers, tulips bottles, buwls, cans, cups, plates apples, mushrooms, oranges, pears, sweet peppers clock, computer keyhoard, lamp, telephone, television bed, chair, couch, table, wardrobe bee, bectle, butterfly, caterpillar, cockroach hear, lenpard, lion, tiger, wolf bridge, castle, house, road, skyscraper cloud, forest, mountain, plain, sea camel, cattle, chimpanzce, elephant, kangaroo fox, porcupine, possum, raccoon, skunk crab, lobster, snail, spider, worm baby, boy, girl, man, woman crocodile, dinosaur, lizard, snake, turtle hamster, mouse, rabbit, shrew, squirrel maple, oak, palm, pine, willow bicycle. bus, motorcycle, pickup truck, train lawn-mower, rocket. streetcar, tank, tractor
fish
flowers food containers
fruit and vegetables
household electrical devices household furniture
insects
large carnivores
large man-made outdoor things
large natural outdoor scenes large omnivores and herbivores medium-sized mammals
non-inseet invertebrates
people
reptiles
small mammals trees vehicles
vehicles 2
Since no hierarchy is available for the CIFARIO and ImageNet100 datasets, construct a hierarchy for CIFAR10 10 manually, as seen in Figure For ImageNet 00, we create a subtree from the WordNet /hirarccyy, given the 100 aforementioned classes as leaves. We consider the classes which are one
level above the leaf nodes in the hierarchy as the coarse classes, following [104].
For the task of OOD detection, we use the following five diverse OOD datasets for CIFARIO and
CIFAR 100 as ID datasets, following the literature [83]: SVHN (65], Textures [9], Places365 [109]
LSUN 102 and iSUN 199] When ImageNet100 is used as the ID dataset, we use 4 diverse 0OD
datasets as the ones in (37]. namely subsets iNaturalist [90]. SUN [96]. Places [109] and
Textures [9]. These datasets have been processed that there is overlap with the ImageNet classes.
B,5 Delta Hyperbolicity Metrics
We Gromov's 6ret to evaluate the tree- tree-likeness of the data in Section 4.1 following [40]. For an arbitrary metric space X' with metric d, for any three points æ, y, z E X, we can define the Gromov product
(y. 2)= (d)()))(((
3) + d(xx x) d(y z))
Then, 9-hyperbolicity can be defined as the minimum value of such that for any four points x, y, z, X. the following condition holds:
(x, 2) 2 min((x, y) (w.2)w)-3
It can be shown that equivalently, there exists geometric definition of ô-hyperbolicity. A geodesic triangle in is d-slim if each of side contained in the 5-neighbourhood of the union of the other two sides. We define d-hyperbolicity as the minimum 5 that guarantees any triangle in A' is 8-slim. From Figure I0 when the curvature of the surface increases, the geodesic triangle converges to tree/star graph, and 8 gradually reduces to O.
Following [40] we use the scale-invariant metric 6 = normalized in [0, 1). and the diam(-) is the set diameter the maximal pairwise distance.
diam for evaluation, so that the 8 is
25
https://ww.cs.toomm//
www.::/wwww.ooooooeeeoomo edu/ kriz/cifar.html
https//wwww.aiikkoo//oootooooddd
26

---

A'
B
B
Figure 10: Example of 8-slim triangle, where each side of AABC is the geodesic distance two points in the metric space.
Table 4: Linear classification accuracy using SupCon 39
as E
Dataset
Method (SupCon)
Fine Accuracy (t)
Flat
94.53
CIFARIO
CPCC
95.08
HypStructure (Ours)
95.18
Flat
75.11
CIFAR100
CPCC
75.66
HypStructure (Ours)
77.66
C Additional Experimental Results
C.1 Results using Linear Evaluation
We also perform evaluation of the fine classification accuracy following the common linear evaluation protocol [39] where a linear classifier is trained on top of the normalized penultimate layer features. We report these accuracies for the models trained the CIFARIO and CIFARIO0 datasets in Table 4 for the Icaf-only variants of the models. We observe that the relative trend of accuracies is identical the reported using the kNN evaluation in Table and proposed method HypStructure outperforms the flat and CPCC methods on both the datasets.
C.2 Component-wise Ablation Study of HypStructure
To understand the role of each component in proposed methodology HypStructure, perform a detailed ablation study with the different components and measure the fine and the coarse accuracies on the CIFAR100 dataset. Specifically, we examine
1. the role of embedding all internal nodes in the label hierarchy (eq. (8 and line 10 in Algorithm D. as opposed to only using leaf nodes as in [14]. We refer to the inclusion of internal nodes Tin-
2. the role of hyperbolic class centroids computation using hyperbolic averaging (eq. (6) and line in Algorithm I as opposed to the Euclidean computation of class prototypes as in [104]. We refer to the hyperbolic class centroid computation as Whyp-
3. the role of the hyperbolic centering loss in our proposed methodology (eq. (8) and line 11 in Algorithm D. as opposed to not using centering loss. We refer to the inclusion of the centering loss as Leenter-
We ablate over the aforementioned settings, where a denotes the inclusion of that setting, and report the results on the CIFAR 100 dataset in Table Firstly, we observe that while the centering loss feemer improves the coarse accuracy only by a small increment, it leads to significant improvement in the fine accuracy (rows 1 + 2 and 4 - 5), indicating that the centering of the root in the poincare disk allows for a better relative positioning of the fine classes within the coarse class groups. Secondly, we observe that both the inclusion of internal nodes Tim and the hyperbolic computation of the class centroids whyp critical for accurately embedding the hierarchy, and removing either of these components (i.e. rows 5 - 3 for Time and rows 5 + 2 for Whyp). leads to degradation in both the fine as well as the coarse accuracies. The best overall performance is observed when all three of the components are included (row 5).
27

---

Table 5: Ablation study on the components of HypStructure, report the Classification accuracies based on the CIFARI00 model trained with ResNet-34.
HypStructure Components
Classification Acc.
Internal Nodes (Tmm)
Hyp. Class Centroids (Whyp) Hyp. Centering (Conter)
Fine
Coarse
75.03
84.77
75.61
84.81
76.22
85.70
76.59
86.23
76.91
86.22
C.3
OOD detection
C.3.1 Related Work and Methods
The goal of prior works in the OOD literature is the supervised setting of learning an accurate classifier for ID data, along with an ID-OOD detection methodology and this task has been explored in the generative model setting 42
63
97, and more extensively in the supervised discriminative The methods in this setting can categorized into four primarily:
77
model setting 2 30 36 37
sub-categories following [06
Post-Hoc Inference These methods design seatprooessinn/ccoring mechanisms on base classifiers such MSP 30). ODIN [51], ReAct 82), SSD+ (76), KNN+ 83 and RankFeat [80].
Training without outlier data These methods involve training-time regularization or different objective functions for improving OOD detection capabilities such as G-ODIN 36) CSI 85
LogitNorm [92] and CIDER 61].
Training with outlier data OE 31 and MixOE [005].
These methods assume access to auxiliary OOD training samples such
Data Augmentation These methods improve the generalization ability of image classifiers such as StyleAugment [22]. AugMix 332 and RegMixup [69].
Our proposed work be considered primarily in the Training without outlier data category, and we note that none of the prior works use any additional structural regularization term in the objective functions.
C.3.2 Dataset-wise OOD Detection Results
Table 6: Results on CIFARIO. OOD detection performance for ResNet-18 trained on CIFARIO. Training with HypStructure achieves strong OOD detection performance.
OOD Dataset AUROC (f)
Method
Avg. (f)
SVHN
Textures
Places365
LSUN
iSUN
ProxyAnchor
94.55
93.16
92.06
97.02
96.56
94.67
CE SimCLR
99.22
96.56
86.70
85.60
86.78
90.97
94.69
94.87
93.04
97.93
98.01
95.71
CIDER
99.72
96.85
94.09
99.01
96.64
97.26
SSD+
99.51
98.35
95.57
97.83
95.67
97.38
KNN+
99.61
97.43
94.88
98.01
96.21
97.22
CPCC
93.27
94.76
60.15
75.29
59.87
76.67
HypStructure (Ours)
99.75
98.89
94.80
99.67
95.64
97.75
We report the dataset-wise OOD detection results in Tables 7a and 7 for CIFAR100, CIFARI 10 and ImageNet100 respectively. We compare with several other state-of-the- the--ar baseline OOD detection
28

---

Table 7: Results ImageNer100. OOD detection performance for ResNet-34 trained on Ima-
geNet100. Training with HypStructure achieves strong OOD detection performance.
OOD Dataset AUROC (1)
Method
Avg. (1)
SUN
Places365
Textures
iNaturalist
CIDER 91.63
89.29
97.98
96.35
93.81
SSD+
88.97
85.98
98.49
96.42
92.46
KNN+
89.48
86.64
98.38
96.46
92.74
CPCCCC
90.95
86.87
97.41
90.08
91.33
HypStructure (Ours)
92.21
90.12
97.33
95.61
93,83
methods for CIFARIO and CIFAR100, namely ProxyAnchor 41 SimCLR CSI 85], and CIDER [6]] respectively. Results for these methods are taken from CIDER [6] where contrastive learning based OOD detection methods typically outperforms non-contrastive learning ones. For ImageNet 00, in the absence of the available class ids used to train the original models in CIDER [6], we finetune the ResNet34 models on the created ImageNetIO0 dataset. For CIDER and SupCon, we use the official implementations and hyperparameters provided by the authors.
We observe that our proposed method leads an improvement in the average OOD detection AUROC over all the ID datasets. In practice, we find that the Euclidean-centroid computational variant (first compute the Euclidean centroids and then apply the exponential map) of our proposed method performs slightly better than the hyperbolic-centroid computational variant (first apply the exponential map and then compute the hyperbolic average), for the specific task of OOD detection, while having equivalent performance on the ID classification task. Hence, we report the OOD detection accuracy corresponding to the first version.
C.4 Visualization of Learned Features
We provide additional visualizations of the learnt features from our proposed method HypStructure on the CIFARIO, CIFARIO0 and ImageNet10k datasets in Figures and 13 respectively.
(a)
(b)
Figure 11: Euclidean t-SNE Visualizations on CIFAR1O.
EROOEE Hadineter
(a)
(b)
Figure 12: Hyperbolic UMAP Visualizations on CIFAR100 and ImageNet100.
29

---

(a)
(b)
Figure 13: Hyperbolic UMAP Visualizations of ID-OOD separation on CIFARIO and ImageNer 100.
C.5 Effect of Centering Loss and Embedding Internal Node
Embedding the internal tree nodes HypStructure Tim (as compared to only leaf nodes in prior work CPCC) and placing the root node at the center of the Poincaré disk with feenter loss, helps in
embedding the hierarchy more accurately. To understand the impact of these components, we first visualize the learnt representations from HypStructure, with and without these components i.e. embedding internal nodes and centering loss vs leaf only nodes, via UMAP in [ 100) and Figure 15 (ImageNet100). We also provide a performance comparison (fine accuracy) in Table g
(CIFAR
Method
CIFARIO
CIFARIO0
ImageNet1(0)
HypStructure (leaf only)
94.54
76.22
89.85
HypStructure (with internal nodes and centering)
94.79
76.68
90.12
Table Fine accuracy comparison of HypStructure with vs. without internal nodes and centering on CIFARIO, CIFAR100, and ImageNet 100 datasets.
First, based on Figures Figure 14 and Figure 15] one can note that in the leaf-only setting without embedding internal nodes and centering loss (figures on the left), the samples belonging to the fine classes which share the same parent (same color) are in close proximity reflecting the hierarchy accurately, however the samples are not spread evenly. With the embedding of internal nodes and a centering loss (right), we note that the representations are spread between the center (root) to the boundary as well as across the Poincaré disk, which is more representative of the original hierarchy. This also leads to performance improvements as can be seen in Table S
Hypdtmecturn
(b)
Figure 14: Hyperbolic UMAP Visualizations on CIFAR100 using HypStructure without embedding the internal nodes and a hyperbolic centering loss (left), and with embedding the internal nodes along with centering loss (right).
30

---

(a)
(b)
Figure 15: Hyperbolic UMAP Visualizations ImageNet100 using HypStructure without embed- ding the internal nodes and hyperbolic centering loss (left), and with embedding the internal nodes along with centering loss (right).
C.6 Effect of Label Hierarchy Weights
Compared to ranking-based hyperbolic losses 666. our HypCPCC factors in absolute values of the node-to-node distances. The learned hierarchy with HypCPCC will not only implicitly encode the correct parent-child relations, but can also learn more complex and weighted hierarchical relationships more accurately. To demonstrate this, we modify the CIFARI0 tree hierarchy, and gradually increase the weight for the left transportation branch to 2x and 4x and use new weighted trees for the CPCC tree distance computation. We visualize the learnt representations in Figure 16] and we can observe that in the learned representations from left to right, the distance between the transportation classes (blue) are larger as compared to other classes, expected.
(a)
(b)
(c)
Figure 16: HypStructure can learn more nuanced representations with weighted hierarchy trees. Hyperbolic UMAP visualizations CIFAR10 using HypStructure with differently weighted left- subtrees.
C.7 Effect of Klein Averaging
We experiment with the two HypCPCC variants, using Klein Averaging or Euclidean mean for centroid computation, as mentioned in Section 3.2 and report the results in Table We empirically observe that using the Klein averaging leads to performance improvements across datasets.
Method
CIFARIO
CIFARIO0
ImageNet100
Euclidean
94.56
75.64
90.08
Klein
94.79
76.68
90.12
Table 9: Fine accuracy comparison between Euclidean and Klein centroid computation methods in HypCPCC on CIFAR10. CIFAR100, and ImageNet100 datasets.
31

---

C.8 Experiments with the Hyperbolic Supervised Contrastive Loss
We experiment with the Hyperbolic Supervised Contrastive Loss as proposed in 21 as the choice of the 4aa loss and refer to this loss as HoypSupCon- We follow the original setup as described by the authors for the measurement of the HyypSupCoa- where the representations from the encoders are not normalized directly, instead an exponential map is used to project these features from the Fuclidean space to the Poincaré ball first. Then, the inner product measurement in the &SupCon is replaced with the negative hyperbolic distances in the Poincaré ball to compute the EitySSupCon loss. We also experiment with our proposed methodology HypStructure along with the HyypSupoon loss and report the classification accuracies and hierarchy embedding metrics for both these settings in Table I0 We further report the OOD detection performance on CIFARIO I0, CIFARI00 and ImageNet100 as in-distribution datasets for both of these settings in Tables I 12 and [3 respectively. We observe that using HypStructure with a hyperbolic loss such as HyypSupCon as the Flat loss leads to improvements in accuracy across classification and OOD detection tasks while also improving the quality of embedding the hierarchy. This demonstrates the wide applicability of our proposed method HypStructure which can be used in conjunction with both euclidean and non-euclidean classification losses.
Table 10: Evaluation of hicrarchical information distortion and classification accuracy using HypSupCon [21] as Flat. All metrics are reported as mean (standard deviation) over seeds.
Dataset (Backbone)
Distortion Hierarehy
Classification Accuracy
Method
S (
CPCC (1)
Fine (1)
Coarse (1)
CIFARIO
Flat
0.128 (0.007)
0.745 (0.017)
94.58 (0.04)
98.96 (0.01)
(ResNet-18)
HypScructure
0.017 (0.001)
0.989 (0.001)
95.04 (0.02)
99.36 (0.02)
CIFAR100
0.168 (0.002)
0.664 (0.012)
75.81 (0.06)
85.26 (0.07)
(ResNet-34)
RypStructure
0.112 (0.005)
0.773 (0.008)
76.22 (0.14)
85.83 (0.06)
ImageNet100
Flat
0.157 (0.004)
0.473 (0.004)
89.87 (0.01)
90.41 (0.01)
(ResNet-34)
HypStructure
0.126 (0.002)
0.714 (0.003)
90.26 (0.01)
90.95 (0.01)
Table 11: Results on CIFARIO when using the HypSupCon(2 21
Flat using ResNet-18 18 as the
backbone. Training with HypStructure achieves improvements in OOD detection performance.
OOD Dataset AUROC (f)
Method
Avg. (t)
SVHN
Textures
Places365
LSUN
iSUN
HyypSupCon
89.45
93.39
90.18
98.18
91.31
92.51
PlypSSppon + HypStructure (Ours)
91.11
94.45
93.52
99.05
95,24
94.68
Table 12: Results on CIFARI00 when using the HypSupCon 21 as Eaa using ResNet-34 as the backbone. Training with HypStructure achieves improvements in OOD detection performance.
OOD Dataset AUROC (f)
Method
Avg. (t)
SVHN
Textures
Places365
LSUN
iSUN
PlypSSpponn
80.
79.61
74.02
70.22
82.35
77.27
HypSuppCon + HypStructure (Ours)
82.28
83,51
77.95
86.64
69.86
80.05
C.9 Experiments with Hyperbolic Backbone
We experiment with Clipped Hyperbolic Neural Networks (HNNs) 26
hyperbolic backbone and use our proposed methodology HypStructure in conjunction with the hyperbolic Multinomial Logistic Regression (MLR) loss. We report the classification accuracies and hierarchy embedding metrics on the CIFAR10 10 and CIFAR 100 datasets in Table [4, and the OOD detection performances using CIFARIO and CIFAR 100 as in-distribution datasets in Tables and [jrespecciii We observe that using HypStructure along with a hyperbolic backbone leads to improvements in classification accuracies, reduced distortion in embedding the hierarchy, and improved OOD detection performance overall, demonstrating the wide applicability of HypStructure with hyperbolic networks.
32

---

Table 13: Results ImageNer100 when using the HypSupCon 21] as Fla using ResNet-34 the backbone. Training with HypStructure achieves improvements in OOD detection performance.
OOD Dataset AUROC (t)
Method
Avg. (1)
SUN
Places365
Textures
iNaturalist
CoypSopCon
91.96
90.74
97.42
94.04
93.54
SaspSppCon HypStructure (Ours)
93.87
91.56
97.04
95.16
94.41
Table 14: Evaluation of hierarchical information distortion and classification accuracy using Clipped Hyperbolic Neural Networks (26] as the backbone. All metries are reported as mean (standard deviation) over 3 seeds.
Dataset (Backbone)
Method
Distortion of Hierarchy
Classification Accuracy
6. (})
CPCC (t)
Fine (t)
Coarse (?)
CIFAR10
(0.008)
0.604 (0.004)
94.81 (0.23)
89.71 (2.04)
(Clipped HNN
20
HypStructure
0.013 (0.002)
0,988 (0.001)
94.97 (0.12)
98.35 (0.22)
CIFARI00
Flat
0.098 (0.001)
0.528 (0.009)
76.46 (0.26)
49.26 (0.73)
(Clipped
HypStructure
(0.016)
(0.005)
(0.14)
(0.61)
Table 15: Results on CIFAR10 when using the Clipped Hyperbolic Neural Networks 26 as the
backbone. Training with HypStructure achieves improvements in OOD detection performance.
OOD Dataset AUROC (f)
Method
Avg. (1)
SVHN
Textures
Places365
LSUN
iSUN
Clipped HNN 26
92.63
90.74
88.46
95.66
92.41
91.98
Clipped HNN 26
HypStructure (Ours)
95.41
93,91
92.31
96.87
94.92
94.68
Table 16: Results on CIFAR100 when using the Clipped Hyperbolic Neural Networks 26 as the backbone. Training with HypStructure achieves improvements in OOD detection performance.
OOD Dataset AUROC (t)
Method
Avg. (1)
SVHN
Textures
Places365
LSUN
iSUN
Clipped HNN
89.94
83.77
77.26
82.87
82.35
83.23
Clipped HNN 26
HypStructure (Ours)
91.56
84.31
78.45
87.53
83.44
85.06
33

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and the introduction both clearly state the claims made by the paper, along with a clear description of the contributions, assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, the limitations of the current work as well the avenues for future improvements the current work can be found in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
34

---

Answer: [Yes]
Justification: Yes, for all the main results of the paper in Section] S a full set of assumptions and complete proof is provided in Section A in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversell, any informal proof provided in the core of the paper should be complemented by formal proof' provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, the paper discloses all the necessary details about the implemented architectures used, hyperparameters for each setting, algorithm pscudocode and other experimental details to reproduce all the experiments in the paper. These details can be found in Section B in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
35

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The paper includes the specific instructions to access the datasets used for all the experimentation. These can be found in Section B in the Appendix. The code for this project is released to the public.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /twpps.cc/puuli..e wwtpcccppppccccceecoec
wwweeeeee/ooeemmmeeceeee commisssions aessonoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: Yes, the paper uses standard data splits from publicly available benchmark sources and provides details regarding the choice of hyperparameters, optimizer and other decisions necessary for understanding the results. These details can found Scction B in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Yes, the paper reports the error bars and other information about the statistical significance of the results in the Section in the main paper.
Guidelines:
The answer NA means that the paper does not include experiments.
36

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eπror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, the paper provides details about the compute resources, hardware and software needed to reproduce the experiments in Section of the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp///bburrrp...cc//////aaaaa
Answer: [Yes]
Justification: Yes, the research conducted in this paper conforms in every respect, with the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
37

---

We discuss the potential positive and negative societal impacts of our work in Section of the Appendix.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper uses publicly available datasets, and does not release any data or code that have a risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have appropriately cited the original owners of data and code which is used in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
38

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperssi thoode com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release any new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing, nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
39

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
40


Reviews:
[{"review_id": "G4Uq1hU6fl", "reviewer": "Reviewer_dBDc", "summary": "The paper introduces HypStructure, a novel approach for learning structured representations using hyperbolic embeddings, which are well-suited for modeling hierarchical relationships due to their tree-like structure. The method incorporates a hyperbolic tree-based representation loss and a centering loss to embed label hierarchies into feature spaces with minimal distortion.\n\nExperiments demonstrate HypStructure's effectiveness in improving classification accuracy, especially in low-dimensional scenarios, and enhancing Out-of-Distribution (OOD) detection performance without compromising in-distribution accuracy.", "questions": "1. What is the rationale behind choosing the Klein weighted average for the first alternative in HypCPCC, considering the extra computation it incurs?\n2. Can the authors provide a comparison to [r2], which is (a special case of) their second alternative of HypCPCC?", "limitations": "The authors did not discuss the limitations in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Although it is already theoretically proved in the related work of [70, 72] that it is not possible to embed a tree in Euclidean space without loss, it is still informative to see that In section 2.2, example 1 and example 2 give good counter-examples to show this property. \n\n2. The paper is well-written and easy to follow, and the proposed model is simple yet effective. \n\n3. The paper provides a formal eigenvalue analysis that links the geometry of the learned representations to their performance", "weaknesses": "1. Sections 2.1, 2.2, and 3.1 are all from existing literature, which limited the contribution of the paper. Although the operations described in Section 3.1 are common hyperbolic operations, this section still lacks proper reference to the related papers.\n\n2. In HypCPCC, the authors proposed two alternatives of the loss,\n    * Map Euclidean vectors to Poincaré space then average.\n    * Averaging the Euclidean vectors then map to Poincaré space.\nIn the 1st alternative, The use of Klein weighted average incurs extra computation, Is it worth doing so?, In the 2nd alternative is exactly the same as [r2], which also calculates the prototypes for each class in hyperbolic space and then map to Poincaré space, [r2] also deploys supervised constructive learning, but the reference is missing and comparison is not stated.\n\n3. The statement in Theorem 5.1 is incorrect, an entry of $K$, denoted as $r^h$ should be a vector, but the theorem stated that $\\lambda_0 = 1 - r^1$, which does not make sense.\n\n3. Incorrect (but fixable) definition in line 708, the proof used $\\| u \\| = \\| v \\| = 1-\\epsilon $ but in the proof the authors used the fact that $\\| u \\| = \\| v \\| = 1-\\epsilon^2 $\n\n4. Incorrect proof in Corollary A.1, the last row of the proof does not hold, Poincaré distance cannot be the same as Euclidean distance, \"growing in the same trend\" does not mean \"proportional to\".\n\n[r2] Long, Teng, et al. \"Searching for actions on the hyperbole.\" CVPR2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for detailed comments, and address each of the questions and weaknesses below. \n\n## Contribution\n\nWhile our work builds on prior research, this does not diminish the contribution of our work and we request the reviewer to kindly refer to our note on novelty in the global response, we clearly outline the key differences between our work and prior works. \n\n---\n\n## Reference to Hyperbolic ops.\n\nThanks for pointing this out, we will include references to common Hyperbolic operations in the revised manuscript [1]\n\n[1] Hyperbolic trigonometry and its applications in the Poincare ball model of Hyperbolic geometry, Ungar A., Computers & Mathematics with Applications, 2001\n\n---\n\n## Comparison to [r2]\n\nThanks for pointing out this work, indeed [r2] is a related work, we will cite it in our revised manuscript and have attempted to provide a comparison below. However, first we discuss the key differences between [r2] and our work.\n\n1. **The usage of hierarchy and “prototypes” differ significantly.** [r2] is a two-step method, where first an action *word* Hyperbolic embedding is created based on action hierarchies (each word is a single tree node), and second a video embedding is learnt using positions of action words (called prototypes in [r2]) in the Hyperbolic space. In contrast, our method learns the hierarchical image embeddings end-to-end, does not depend on word embeddings, and performs a centroid computation on the label hierarchy tree nodes (many images belong to a single label node, hence class prototypes are computed by averaging).\n\n2. **The nature of components to learn hierarchical embeddings are different**. In [r2], both losses $L_H$ (Eq. 2 in [r2], as in [2]) and $L_2$ (Eq. 6 in [r2], as in [3]) are ranking losses, whereas HypCPCC factors in absolute values of the node-to-node distances. The learned hierarchy with HypCPCC will not only (implicitly) encode the correct parent-child relation like [2, 3], but also learn more complex and weighted hierarchical relationships more accurately (Fig. 16 in the rebuttal PDF).\n\nWe attempt to compare HypStructure and [r2] now. Since losses $L_H$ and $L_2$ have similar motivation as HypStructure, we add them as regularizers to the Hyperbolic classification backbone Clipped HNN and compare with our method:\n\n|Method|CIFAR10|CIFAR100|\n|------|:-----:|:------:|\n|Clipped HNN|95.08|77.42|\n|Clipped HNN + $L_H$|94.78|77.44|\n|Clipped HNN + HypStructure|**95.19**|**78.05**|\n\nWe see HypStructure achieves better performance than $L_H$. We face NaN issues while training with L_2 and other losses, and discuss the setup details in the global response.\n\n[2] Poincaré embeddings for learning hierarchical representations, Nickel et. al, NIPS ‘17\n\n[3] Hyperbolic entailment cones for learning hierarchical embeddings, Ganea et. al, ICML ‘18\n\n---\n\n## Rationale behind Klein avg.\n\nWe experiment with both HypCPCC variants mentioned in line 177-185. The 2nd variant, however, given our understanding, is not a special case of [r2], since it still requires a Euclidean averaging step for class prototype computation. We empirically observe performance improvements in accuracy using the 1st variant across datasets below (fine acc.).\n\n|Method|CIFAR10|CIFAR100|ImageNet100|\n|------|-------|--------|-----------|\n|Euc. (2)|94.56|75.64|90.08|\n|Klein (1)|**94.79**|**76.68**|**90.12**|\n\n---\n\n## Theorem 5.1 entry of K\n\nSince $K$ is $\\mathbb{R}^{n\\times n}$ (line 289), an *entry* of $K$ is -  $K_{i,j}$, the $i$-th row and $j$-th column scalar value of K. Each entry of $K$ can be denoted as $r^h$, where $r^h$ is the lowest common ancestor (LCA) of two samples represented by the $i$-th row and $j$-th column respectively so it is indeed a scalar. We also refer the reviewer to the matrix in lines 714-715, where each entry of $K$, denoted as $r^h_{i,j}$, is a scalar value.  (i,j subscript can be ignored, see 718-719, since every two leaves sharing the LCA of the same height have the same tree distance). We will make this more explicit in the revised version to avoid confusion.\n\n---\n\n## “The proof used $|𝑢|=|𝑣|=1−𝜖$…” \nThanks for pointing out this typo due to our cluttered notation! We intended to mean $|u|^2 = 1 - \\text{(a small number)}$, however the conclusion of the proof still holds with this change, as the reviewer noted. Let us rewrite the proof: in practice with $clip^1$ projection, to be consistent with the notation in the main body, we set $|u|=|v| = 1 - \\epsilon$ where $\\epsilon$ is a very small number. Then $|u|^2 = (1 - \\epsilon)^2 = 1 - 2\\epsilon + \\epsilon^2$. Since $\\epsilon$ is very small, we define $2\\epsilon - \\epsilon^2$ as $\\xi$, making $|u|^2 := 1 - \\xi$ where $\\xi$ is still a small number. Then replacing all $\\epsilon$ with $\\xi$ in line 708-709, we get the same conclusion. We will correct this error in the revised version.\n\n---\n\n## “Incorrect proof in Corollary A.1...”\n\nApologies for the confusion. With the statement “growing in the same trend”, we meant that Poincaré(u,v) is monotonically increasing with Euclidean(u,v), and the proof after Corollary A.1 will not depend on the metric, as can be seen from comments in 710. This is true since with $\\text{clip}^1$ transformation, Poincaré is approximately the log-scalar transform of Euclidean distance. The monotonically increasing property ensures the relative order of any two entries for Euclidean CPCC and Poincare CPCC matrices in K to be the same. Then, we can argue about the structure of K, either Euclidean or Poincare, to have the hierarchical diagonalized structure as in Fig 8a. But indeed technically, due to the logarithm, this is not “proportional to”. We will remove this notation and make it more clear.\n\n---\n\n## Limitations\n\nAnother potential limitation - our method relies on the availability (or construction) of an external hierarchy for computing the HypCPCC objective, which might be challenging if the hierarchy is unavailable or noisy. We will include this in the next iteration of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HT67UEyRMa", "reviewer": "Reviewer_kf6R", "summary": "This work introduces a regularization scheme based on Cophenetic Correlation Coefficient to more appropriately embed semantic label hierarchical structure in the representation. The method exploits the hierarchical benefits of hyperbolic space reformulating the CPCC regularization term to operate on the Poincare ball. The proposed method sees improvement in empirical performance demonstrating the effectiveness of the approach to learn a more separable embedding space for classification.", "questions": "See weaknesses.", "limitations": "Broader impact statement and societal impact are seemingly missing from this work. These limitations including those limitations of the analysis should be highlighted in more detail, you mention the Euclidean loss as a limitation. If I have missed them however in the text, please correct me.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "⁃\tThe authors present the work clearly with effective use of visual and writing structure. All figures/diagrams are useful in supporting the narrative and findings.\n\n⁃\tThe method is simple, highly generalizable, and leads to improved performance on benchmark tasks. It can therefore, be seen as an advantageous tool in hyperbolic learning that could possibly lead to impact and adaptation by practitioners in the field.\n\n⁃\tThe theoretical and analysis are generally good, with eigenspectrum analysis supporting your claims of hierarchical structure for the most part. This is a useful analysis that provides confidence in the findings supported by appropriate proofs.\n\n⁃\tExtensive details to support replication are provided.", "weaknesses": "⁃\tFrom the visualizations presented of the embedding space, notably the UMAP, your embeddings seem to have collapsed to the boundary in many places limiting the inherent hierarchy of the embeddings, this results in a limited hierarchy being represented. This in turn, leads me to question the extent of hierarchies learnt, when discussing the core intention of the work, and the claims made. One would expect that greater performance could be achieved if this had been addressed. I am aware that boundary collapse is still an unsolved problem, but careful tuning can limit its effects.\n\n⁃\tThe approach is simple but arguably not significantly novel given it is a hyperbolic reformulation of CPCC with minimal changes. With that being said, these simple methods do work somewhat well in practice and are useful to progressing the field. \n\n⁃\tThe use of a Euclidean linear evaluation is a confusing direction. You are aiming to learn a hyperbolic embedding space that preserves hierarchy, yet for downstream tasks you employ a Euclidean classifier, why? You will lose the desirable properties you are aiming to capture.\n\n⁃\tFurther experimentation on different hyperbolic models and downstream tasks would have helped demonstrate the generalization of the regularization to all of hyperbolic learning. Although, this cannot be expected in the rebuttal, it would have helped support the findings to present the work a more generalized approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments, and address each of the questions and weaknesses below. \n\n## Boundary collapse\n\nPart of the design of our HypStructure methodology, **including the centering loss and embedding of internal nodes** mitigates boundary collapse.  Embedding the internal tree nodes in HypStructure - $T_{\\text{int}}$ (as compared to only leaf nodes in prior work CPCC) and placing the root node at the center of the poincare disk with $l_{\\text{center}}$ loss (discussed in lines 100-104, 186-194 in the main paper, lines 931-946 in the Appendix) embeds the hierarchy more accurately and the mitigate aforementioned issues. To demonstrate this intuition and evaluate its impact on the learnt hierarchy and representations, we first visualize the learnt representations from HypStructure, with and without these components - i.e. embedding internal nodes and a centering loss vs leaf only nodes, via UMAP in Figures 14 (CIFAR100) and 15 (ImageNet100) of the rebuttal PDF. We also provide a performance comparison (fine acc.) in the Table below\n\n|Method|CIFAR10|CIFAR100|ImageNet100|\n|-------|:-------------:|:-----:|:-----:|\n| HypStructure (leaf only) | 94.54 | 76.22  | 89.85  |\n| HypStructure (with internal nodes and centering) | **94.79**  | **76.68**  | **90.12** |  \n\n\nFirst, based on Figures 14 and 15, one can note that in the leaf-only setting without embedding internal nodes + centering loss (figures on the left), the samples belonging to the fine classes which share the same parent (same color) are in close proximity, reflecting the hierarchy, however they are close to the boundary. With the embedding of internal nodes and a centering loss (right), we note that the representations are spread between the center (root) to the boundary as well as across the poincare disk, which is more representative of the original hierarchy. This also leads to performance improvements as can be seen in the table above. \n\nNote that we used the same $\\beta = 0.01$ penalty parameter across datasets for convenience and easy reproducibility, and with a more careful tuning of $\\beta$, the boundary collapse can be mitigated to a further extent. However, forcing the representations to be too close to the origin can lead to inter-class confusion and performance degradation as discussed in prior work [1] (Figure 4, page 7 of the paper). \n\n[1] Hyperbolic Busemann Learning with Ideal Prototype, Atigh et. al, NeurIPS ‘21\n\n\n---\n\n##  Novelty\n\nThanks for your comments in support of the simplicity of our work. We would also like to refer the reviewer to our response regarding technical novelty in the global response above. \n\n---\n\n## Euclidean linear evaluation\n\nWe provide **Euclidean linear evaluation on the Euclidean features**, as an added evaluation in the Appendix. Besides, it allows us to perform a fair comparison between important baselines, particularly l2-CPCC vs. HypStructure. To clarify, let us revisit our HypStructure pipeline: we first train an Euclidean loss SupCon which gives us an Euclidean feature (Fig. 6c), and then apply the exponential map to get the Poincare feature for CPCC computation. Because SupCon takes Euclidean inputs, it is reasonable to use Euclidean space for classification evaluation. From the visualizations in Figure 6b vs 6c, we note that the Hyperbolic regularizer empirically affects the geometry of the features even in the Euclidean space to be more structured and leads to performance improvements (Table 1, Fig. 7a, c) \n\nOn the other hand, Hyperbolic classifiers are applicable if we change the backbone model. We experiment with a Hyperbolic model, Clipped HNN [1] and present the fine acc. results below (more details about the setup in the global response above), evaluated with Hyperbolic Multinomial Logistic Regression Layer in [2]: \n\n| Method | CIFAR10 | CIFAR100 |\n|---|:---:|:---:|\n| Clipped HNN | 95.08 | 77.42 |\n| Clipped HNN + HypStructure | **95.19** | **78.05** |\n\nWe observe that HypStructure provides performance improvements with Hyperbolic backbones as well.  \n\n[1] Clipped hyperbolic classifiers are super-hyperbolic classifiers, Guo. et al, CVPR ‘22 \n\n[2] Hyperbolic neural networks, Ganea et. al, NeurIPS ‘18 \n\n---\n\n## Further Experimentation\n\nWe would also like to point the reviewer to additional experiments that we have performed with non-euclidean setups and we discuss more details about them in the global response. \n\n---\n\n## Broader impact and limitations\n\nThanks for pointing this out, we would like to clarify that we experiment with the Euclidean loss primarily to provide a fair comparison with the prior works, and it is not a requirement (or limitation) for our HypStructure methodology. It can be combined with any other loss function (such as HypSupCon) or backbone (Clipped HNN) as we have demonstrated above, highlighting the wide applicability of our method.\n\nIn addition to the discussion in Section 7, we enumerate other limitations which we foresee for future research and building on our methodology - our proposed method relies on the availability (or construction) of an external hierarchy for computing the HypCPCC objective, which might be challenging if the hierarchy is unavailable or noisy. In terms of Broader Impact, one potential area of impact that we recognize is the AI for science domain, where HypStructure can be helpful in learning more interpretable representations which are more reflective of the underlying relationships in the domain space. We include this in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AW5TjjNDqW", "reviewer": "Reviewer_gWWe", "summary": "The paper presents a novel approach, HypStructure, for learning structured representations. Comparing with the existing method, the proposed method adds an regularizer calculated from hyperbolic geometry. This approach aims to reduce distortion and improve generalization performance, particularly in low-dimensional scenarios. Extensive experiments are conducted on both the classification task and the out-of-distribution detection task.", "questions": "see my comments in the Weaknesses section.", "limitations": "There are two limitations mentioned in this paper. The first being that the L2-CPCC can only approximate the structured information since it is impossible to “embed a tree $T$ into $L2$ without distortion,” and as this work extends the L2-CPCC, it would similarly have this limitation. However, this is never explicitly stated in Section 7, where it mentions that HypStructure is only limited to Euclidean classification losses. There could have been more said about the limitations, but it does suffice.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is well organized. The paper extends and studies the existing L2-CPCC to the hyperbolic space, which effectively reduces distortion and enhances the representation of hierarchical relationships in the data. The paper also conducted comprehensive experiments as well as detailed theoretical analysis of the eigenspectrum of structured representations.", "weaknesses": "If my understanding of the proposed loss term is correct, $L_flat$ is not calculated in the hyperbolic geometry. Have you tried the $L_flat$ with hyperbolic network or hyperbolic geometry. I hope the authors could provide more explanation of the combined loss in different geometries. \n\nIn Section 2.1, it mentions that $D_i$ is the subset of data points with a specific class label, and $d(D_i, D_j)$ is the distance between the feature vectors of $D_i$ and $D_j$. However, it is not mentioned how the vectors for $D_i$ and $D_j$ are calculated. Is it simply the average of all the feature vectors in the subset? \n\nFor Example 1 in Section 2.2, tree $T$ and nodes $G, C, D, E$ are referenced in a way that implies there should be a figure accompanied by the example. While Figure 1 is referenced shortly before this, it is meant to be accompanied by Example 2. Is there a figure that has been omitted here?\n\nAlso, I would recommend proofreading the paper to correct all grammatical errors. For example, in the paragraph of Section 2.1, the first sentence “Using a composite objective as defined in Equation (2), we can enforce the distance relationship between a pair of representations in the feature space, to behave similarly as the tree metric between the same vertices” should be corrected to “Using a composite objective as defined in Equation (2), we can enforce the distance relationship between a pair of representations in the feature space to behave similarly to the tree metric between the same vertices.” This version removes the unnecessary comma and corrects “behave similarly as…” to “behave similarly to…”.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments, and address each of the questions and weaknesses below. \n\n## Hyperbolic $L_{flat}$\n\nWe choose the Supervised Contrastive Loss (SupCon) loss as the $L_{flat}$ loss in our experiments primarily to provide a fair comparison with the prior works, whereas our HypStructure methodology is flexible and can be combined with any other loss function.\n\nTo demonstrate the wider applicability of our method based on the reviewers suggestion, we evaluate the performance using the Hyperbolic Supervised Contrastive loss $L_{\\text{hyp}}$ proposed in [1] instead of the SupCon loss, and provide a comparison with using HypStructure in conjunction with this Hyperbolic loss in the Table below (fine acc.) \n\n\n| Method | CIFAR10 | CIFAR100 | ImageNet100| \n|---|:---:|:---:|:---:|\n| L_hyp (HypSupCon)  | 94.64 | 75.77| 90.02| \n| L_hyp (HypSupCon) + HypStructure (Ours)| **95.06**| **76.08** | **90.31** | \n\nIn addition to the aforementioned loss, we vary the backbone network - the Clipped HNN model, a type of Hyperbolic Neural Network, instead of using the Euclidean ResNet style architectures and provide the results for that setting as below. \n\n| Method | CIFAR10 | CIFAR100 |\n|---|:---:|:---:|\n| Clipped HNN | 95.08 | 77.42 |\n| Clipped HNN + HypStructure | **95.19** | **78.05** |\n\nWe observe that even with a classification loss function that optimizes for separability in the Hyperbolic space, or a Hyperbolic Neural Network as the backbone, our proposed methodology HypStructure is complementary and provides performance gains for better embedding the hierarchical relationship in the Hyperbolic space. (More details regarding the setup are in the global response above.)\n\nWe will include the results with the experiments on hyperbolic losses and backbone in the revised version of the paper as well. \n\n[1] Hyperbolic Contrastive Learning for Visual Representations Beyond Objects, Ge et. al, CVPR ‘23\n\n---\n\n\n## Definition of dataset distance $d(D_i, D_j)$\n\nAs we mention in line 75-76, the dataset distance $d(D_i, D_j)$ can vary depending on the choice of the distance metric and the design setup, hence we avoid writing an explicit definition in Sec. 2.1 and provide a general expression here instead. We follow this up with an example of the specific dataset distance for the l2-CPCC case in lines 97-98, where we see that $d(D_i, D_j) = \\rho_{l2} =$ _the Euclidean distance of two averages of all feature vectors_ in each subset. On the other hand, for our proposed methodology HypStructure, you can either compute the metric $d(D_i, D_j)$ on Hyperbolic space, which first uses exponential map (Eq. 4) and then Klein average (Eq. 6) followed by the Poincare distance computation (line 180-181), or compute the means of the Euclidean features as in $\\rho_{l2}$ and project the centroids to the Hyperbolic space using the exponential map, followed by the Poincare distance computation.\n\n---\n\n\n## Figure for Example 1 and 2\n\nThe accompanying figures for both Example 1 and Example 2 are provided in Figure 1. More specifically, Example 1 only uses the tree nodes on the left part of Figure 1, which is what we refer to as the tree $\\mathcal{T}$. \n\nThe Example 2 uses both the left part of the Figure to describe nodes in the tree, C, D and E, and the right part of the Figure to visualize this corresponding setup in an attempt to embed the tree in the Euclidean space, based on the arguments described in the text in Example 2. We will make this more explicit in the paper description and the caption in the figure. \n\n---\n\n## Grammatical errors\n\nThanks for the suggestions, we will proofread to improve readability and remove any grammatical errors in the camera ready version.\n\n---\n\n\n## Clarification for limitations\n\nThe aforementioned comments misunderstand the limitations of our proposed method and we would like to provide a few clarifications regarding the limitations. First, the $l_2$-CPCC methodology suffers from the challenges of embedding a tree **exactly**, owing to the representational capacity of the Euclidean space which has zero curvature. This leads to high distortion when embedding a tree-like hierarchy using the $l_2$-CPCC, as also shown in the distortion metrics in Table 1. In contrast, Hyperbolic spaces allow for embedding tree-like data in finite dimensions with minimal distortion owing to the negative curvature of the spaces, and hence our proposed methodology - HypStructure does not suffer from the same limitation as the $l_2$-CPCC, and therefore reduces the distortion in embedding the hierarchy (Table 1).\n\nAdditionally, while our submitted work experimented with primarily a Euclidean classification loss, (hence the corresponding statement in the limitations), over the rebuttal period based on the reviewers’ suggestions, we have experimented with other Hyperbolic losses and backbones, including the Hyperbolic SupCon loss and Clipped Hyperbolic Neural Network, which also shows improvements when combined with our proposed method, and hence our work is not limited to Euclidean classification losses. We will rephrase this statement to avoid further confusion.  We foresee a potential limitation - since our proposed method relies on the availability (or construction) of an external hierarchy for computing the HypCPCC objective, it can be challenging if the hierarchy is unavailable or noisy. We will include this in the limitations section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "U5bXQIsVBo", "reviewer": "Reviewer_3c64", "summary": "The paper introduces a novel regularization method, HypStructure, which utilizes hyperbolic geometry to improve the embedding of hierarchical relationships within feature representations. This approach enhances the learning of structured representations, reducing distortion and boosting generalization in low-dimensional scenarios. It also demonstrates superior performance in Out-of-Distribution (OOD) detection across various datasets through extensive empirical evaluation. Additionally, the paper includes an eigenvalue analysis that provides deeper insights into the structured representations, correlating positively with improved OOD detection performance. This advancement extends structured representation learning to hyperbolic spaces, achieving more discriminative and interpretable features that effectively capture the inherent hierarchies in complex datasets.", "questions": "See weaknesses.", "limitations": "As I said in the previous section, the main concern is the novelty. Potential improvements include extending the application scenarios of hyperbolic structured representations on more vision or language tasks that have not been explored before. Further theoretical investigation into establishing the error bounds of CPCC-style structured regularization objectives is of interest as well. The writing needs to be improved as well.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is the first to formally characterize properties of hierarchy-informed features via an eigenvalue analysis, and also relate it to the OOD detection task.\n2. The paper is easy to read and follow, making complex concepts accessible. The use of clear definitions, structured methodology sections, and detailed discussions helps in understanding both the theoretical underpinnings and practical implications of HypStructure. Visual aids and empirical results are presented in a manner that clearly supports the claims made.\n3. The significance of this work lies in its potential impact on a range of applications that require understanding and leveraging hierarchical relationships in data, such as image recognition and OOD detection.", "weaknesses": "1. The main concern of this paper is the novelty. I believe the method proposed by the author in this work has been explored in many previous works. For instance, in \"Hyperbolic Image Embeddings,\" \"Hyperbolic Contrastive Learning for Visual Representations beyond Objects\", etc. Although the paper characterizes properties of hierarchy-informed features via an eigenvalue analysis, the contribution is not significant enough to be accepted. \n2. The writing is also not good enough for me. For instance, two examples starting from line 107 are not necessarily included in the formal paper; it is better to be placed in the supplementarity materials. There are also some repetitive expressions, for instance, in line 351 and line 353 (While the current work).\n3. In summary, I believe the technical contribution of this paper is not significant enough to be accepted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed comments, and address each of the questions and weaknesses below. \n\n## Concerns about the novelty \nWe humbly disagree with the reviewers’ statement since our learning setting is **different from the existing works** in the hyperbolic geometry literature, where our focus is on using hyperbolic geometry to **embed a given class hierarchy.** We believe our work would be a suitable contribution to the space of structured representation learning and Hyperbolic geometry, and we will make key points to argue the novelty of our work, which will hopefully highlight our technical contributions to the field. \n\n\n1. **Implicit vs explicit incorporation of hierarchy in Hyperbolic geometry and complementary nature of HypStructure**:  Most previous works in Hyperbolic geometry assume a latent **implicit** hierarchy in distributions/parameters, while our work focuses on an **explicit** knowledge injection of external hierarchy into the representation space. So while the methods and motivations may _seem_ similar, there are important differences in the nature of the problem, and the statement that the “method proposed by the author has been explored in many previous works” is an overly simplified generalization. \n\nSpecifically, the “Hyperbolic Image Embeddings” paper replaces Euclidean layers with Hyperbolic classifiers which constrained the layer parameters in the Hyperbolic space, and has no similarities with HypStructure which instead utilizes external hierarchy for embedding regularization. Furthermore, the “Hyperbolic Contrastive Learning for Visual Representations Beyond Objects” proposes a Hyperbolic supervised contrastive loss implicitly assuming the existence of an unknown scene-object hierarchy. \n\nHowever, we want to emphasize that our approach offers a flexible framework that can be complementary and used in conjunction with other Hyperbolic losses/backbones assuming implicit hierarchies for performance gains. For instance, with a more recent variant of Hyperbolic classifier-based backbone (Clipped HNN), as well the Hyperbolic SupCon loss for further performance gains - please refer to the detailed results in the global response above, however we share these performance numbers here for reference. \n\n| Method | CIFAR10 | CIFAR100 | ImageNet100| \n|---|:---:|:---:|:---:|\n| L_hyp (HypSupCon)  | 94.64 | 75.77| 90.02| \n| L_hyp (HypSupCon) + HypStructure (Ours)| **95.06**| **76.08** | **90.31** | \n\n| Method | CIFAR10 | CIFAR100 |\n|---|:---:|:---:|\n| Clipped HNN | 95.08 | 77.42 |\n| Clipped HNN + HypStructure | **95.19** | **78.05** |\n\n\n\n2. **Address limitations of the l2-CPCC with strong empirical performance**: The central theme of our paper is **accurately** embedding **external knowledge** available in the form of hierarchical information, in the representation space. While our work draws inspiration from recent contributions in Hyperbolic machine learning, we propose a novel, practical and effective framework that addresses the severe limitations of the CPCC objective in embedding this hierarchical information. This is clearly demonstrated with the strong empirical performance on large-scale datasets and visualization of learned representations using HypStructure.\n\n3. **Theoretical understanding of structured representation learning and OOD detection** - to the best of our knowledge, our work is one of the first to draw theoretical connections between the paradigm of hierarchy-informed representation learning and tasks such as OOD detection, and we provide a provable hierarchical embedding that fills a theoretical gap in the broader field. These insights can serve to be useful in designing theoretically grounded, efficient and accurate representation learning methods that can learn general representations useful across tasks in practice as we have demonstrated for classification, OOD detection, and maintaining interpretable representations. \n\nBased on these grounds, we feel our work makes many novel contributions in terms of explicit modeling of structured hierarchy, improving and preserving representations with other Hyperbolic objectives and backbones, empirical effectiveness across tasks, and drawing theoretical connections to the OOD detection task (as you have rightly noted). We also foresee our contribution as motivating more works in the practical domain of embedding structured information explicitly using Hyperbolic geometry. We sincerely hope this provides a satisfactory justification to the reviewer regarding our contribution. \n\n---\n\n## Concerns about writing and examples \n\n\nWe have included these examples in the motivation section of our paper to emphasize the practical concerns and severe limitations of l2-CPCC, which serves as our motivation to develop better methods. These examples related to our label tree setting, are very informative for understanding the problem setup and the challenges  (as has also been noted by Reviewer dBDC as a strength of our paper). Regarding writing, thanks for your suggestions, we will proofread to improve readability and remove repetitive expressions in the camera ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for your valuable reviews and constructive feedback that has helped us improve our work. We first share the results of additional experiments we conducted based on these suggestions which demonstrate the wide applicability of our method, and then summarize our technical contributions and its novelty. 

## Additional experiments

1. **Hyperbolic Backbones and Losses**

According to the suggestions from reviewers dBDC, kf6R, and gWWe, we experimented with a Hyperbolic backbone, Clipped HNN [1], which is developed on the Hyperbolic MLR layer in the work Hyperbolic Neural Networks [2]. We also experiment with a composite objective combining this backbone and our HypStructure regularizer. 

For comparison, we also experiment with the objective function in the seminal work Poincaré Embedding [3] as a regularizer, since the methodology also aims to learn Hyperbolic (word) embeddings (as suggested by reviewer dBDc for learning action embeddings in [r2]). Following the setup in our paper, we use the first alternative for both HypStructure and PoincaréEmb in this comparison, i.e., we first ensure the features are in Poincaré ball, followed by the Klein averaging for each class node, and apply the regularizer accordingly. The fine accuracy results are as follows:

|Method|CIFAR10|CIFAR100|
|---|:---:|:---:|
|Clipped HNN|95.08|77.42|
|Clipped HNN+PoincareEmb|94.78|77.44|
|Clipped HNN+HypStructure|**95.19**|**78.05**|

We observe that HypStructure is a flexible regularizer compatible with Hyperbolic backbones and it improves the fine-level classification accuracy from the Clipped HNN baseline, and performs better than Clipped HNN + Poincare Embedding [3] as well. We also experimented with the Poincare Embedding [3] regularizer with the Supervised Contrastive (SupCon) loss, however the training runs into NaN issues due to numerical instability in the PoincareEmb computation. Note that for this experiment, we could not provide results on ImageNet100, since in our main experiments in Table 1 for ImageNet100, we rely on fine-tuning a pre-trained ResNet backbone on ImageNet-1K, whereas we were unable to find a comparable pretrained Hyperbolic model for a fair comparison to results in Table 1.

We also replaced the Euclidean Supervised Contrastive Loss, with the **Hyperbolic Supervised Contrastive Loss** $L_{\text{hyp}}$ as proposed in [4], and experiment with HypStructure in combination with this Hyperbolic loss. We report the results on different datasets below (fine acc.)

|Method|CIFAR10|CIFAR100|ImageNet100|
|---|:---:|:---:|:---:|
|L_hyp (HypSupCon)|94.64|75.77|90.02|
|L_hyp (HypSupCon) + HypStructure (Ours)|**95.06**|**76.08**|**90.31**|

We observe that HypStructure works well with Hyperbolic contrastive losses as well, and in fact leads to improved absolute accuracies on CIFAR10 and ImageNet100 than our previous results using the supervised contrastive losses.

[1] Clipped hyperbolic classifiers are super-hyperbolic classifiers, Guo. et al, CVPR ‘22

[2] Hyperbolic neural networks, Ganea et. al, NeurIPS ‘18

[3] Poincaré embeddings for learning hierarchical representations, Nickel et. al, NeurIPS ‘17

[4] Hyperbolic Contrastive Learning for Visual Representations Beyond Objects, Ge et. al, CVPR ‘23

[r2] Searching for actions on the hyperbole, Long et. al, CVPR ‘20



2. **Understanding the components of HypStructure**

Based on the suggestion from reviewers dBDC, kf6R, we perform additional experiments and visualizations to understand the working of HypStructure, specifically:  


- 2a. **The role of centering loss for the root and embedding internal nodes** - see Table below and comparison of visualizations in Figures 14 and 15 in the PDF

 |Method|CIFAR10|CIFAR100|ImageNet100|
 |---|:---:|:---:|:---:|
|HypStructure (leaf only)|94.54|76.22|89.85|
|HypStructure (with internal nodes and centering)|**94.79**|**76.68**|**90.12**|

- 2b. **The role of Klein averaging vs Euclidean averaging** for improved empirical performance 

|Method|CIFAR10|CIFAR100|ImageNet100|
|------|-------|--------|-----------|
|Euc. (2)|94.56|75.64|90.08|
|Klein (1)|**94.79**|**76.68**|**90.12**|

- 2c. **The capacity of HypStructure to learn complex hierarchical relationships** with differently weighted trees - Figure 16 in the PDF. 

---

## Novelty & Contribution

1. **Complementary nature to prior Hyperbolic works and address limitations of l2-CPCC**: While our work draws inspiration from recent works in Hyperbolic machine learning, we propose a novel, practical and effective framework that addresses the limitations of the l2-CPCC,as we clearly demonstrate with the strong empirical performance on large-scale datasets and visualization of learned embeddings. Furthermore, our approach offers a flexible framework that can be complementary and used in conjunction with other Hyperbolic losses/backbones for performance gains. We want to highlight that our work is very different from the majority of prior works in Hyperbolic learning that assume a latent implicit hierarchy in the learning process, while our work solves the problem of leveraging explicit external hierarchical knowledge in representation learning.

2. **Provable structured representation learning**: To the best of our knowledge, our work is one of the first to draw theoretical connections between the paradigm of hierarchy-informed representation learning and OOD detection. These insights can be very useful in designing theoretically grounded, efficient and accurate representation learning methods that can learn general representations useful across tasks in practice as we have demonstrated for classification and OOD detection, while maintaining interpretable representations.

We believe our work makes several novel contributions, and we anticipate that our work will inspire further research on embedding structured information and usage of Hyperbolic geometry. We hope this adequately justifies our contributions to the reviewers.

Meta-review: This paper has received active consideration and discussion by the reviewers, which in turn changed the scoring of the paper towards a consensus. The reviewers appreciate the theoretical nature of the paper and find the idea simple yet clear and effective. There were a number of limitations that required a rebuttal. The AC considers the questions about additional baselines, analyses, and issues with the theory solved. The reviewers have also reached a consensus towards accept, albeit close the borderline. The AC recommends acceptance since the theory and the methodology of the paper are of interest to the community and the additional experiments and clarifications of the rebuttal strengthen the paper. A number of initial reviewer concerns are a result of a lack of referencing; despite citing >100 papers, many papers on hyperbolic learning for computer vision specifically were missed. The AC urges the authors to includes the outcomes of the rebuttal in the paper and to include both the references mentioned by the reviewers and other highly related works (especially visual learning with hyperbolic hierarchical embeddings).
Decision: Accept (poster)