Learning Structured Representations with Hyperbolic Embeddings

Authors: aditya  sinha
siqi  zeng
makoto  yamada
han  zhao
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:29:10

Abstract: Most real-world datasets consist of a natural hierarchy between classes or an inherent label structure that is either already available or can be constructed cheaply. However, most existing representation learning methods ignore this hierarchy, treating labels as permutation invariant. Recent work [Zeng et al., 2022] proposes using this structured information explicitly, but the use of Euclidean distance may distort the underlying semantic context [Chen et al., 2013]. In this work, motivated by the advantage of hyperbolic spaces in modeling hierarchical relationships, we propose a novel approach HypStructure: a Hyperbolic Structured regularization approach to accurately embed the label hierarchy into the learned representations. HypStructure is a simple-yet-effective regularizer that consists of a hyperbolic tree-based representation loss along with a centering loss, and can be combined with any standard task loss to learn hierarchy-informed features. Extensive experiments on several large-scale vision benchmarks demonstrate the efficacy of HypStructure in reducing distortion and boosting generalization performance especially under low dimensional scenarios. For a better understanding of structured representation, we perform eigenvalue analysis that links the representation geometry to improved Out-of-Distribution (OOD) detection performance seen empirically.

Main paper: 
 introduction: 
 Real-world datasets, such as ImageNet [72] and CIFAR [45] , often exhibit a natural hierarchy or an inherent label structure that describes a structured relationship between different classes in the data. In the absence of an existing hierarchy, it is often possible to cheaply construct or infer this hierarchy from the label space directly [64] . However, the majority of existing representation learning methods [43, 7, 95, 29, 87, 33, 27, 39] treat the labels as permutation invariant, ignoring this semantically-rich hierarchical label information. Recently, Zeng et al. [104] offer a promising approach to embed the tree-hierarchy explicitly in representation learning using a tree-metric-based regularizer, leading to improvements in generalization performance. The approach uses a computation of shortest paths between two classes in the tree hierarchy to enforce the same structure in the feature space, by means of a Cophenetic Correlation Coefficient (CPCC) [79] based regularizer. However, their approach uses the ℓ 2 distance in the Euclidean space, distorting the parent-child representations in the hierarchy [70, 50] owing to the bounded dimensionality of the Euclidean space [8] .Hyperbolic geometry has recently gained growing interest in the field of representation learning [66, 67] . Hyperbolic spaces can be viewed as the continuous analog of a tree, allowing for embedding tree-like data in finite dimensions with minimal distortion [44, 73, 75, 24] . Unlike Euclidean spaces with zero curvature and spherical spaces with positive curvature, the hyperbolic spaces have negative curvature enabling the length to grow exponentially with its radius. Owing to these advantages, hyperbolic geometry has been used for various applications such as natural language processing [52, 73, 16] , image classification [40, 103, 18] , object detection [46, 21] , action retrieval [55] , and hierarchical clustering [100] . However, the aim of using hyperbolic geometry in these approaches is often to implicitly leverage the hierarchical nature of the data.In this work, given a label hierarchy, we argue that accurately and explicitly embedding the hierarchical information into the representation space has several benefits, and for this purpose, we propose HypStructure, a hyperbolic label-structure based regularization approach that extends the proposed methodology in [104] for semantically structured learning in the hyperbolic space. HypStructure can be easily combined with any standard task loss for optimization, and enables the learning of discriminative and hierarchy-informed features. In summary, our contributions are as follows:• We propose HypStructure and demonstrate its effectiveness in the supervised hierarchical classification tasks on three real-world vision benchmark datasets, and show that our proposed approach is effective in both training from scratch, or fine-tuning if there are resource constraints.• We qualitatively and quantitatively assess the nature of the learned representations and demonstrate that along with the performance gains, using HypStructure as a regularizer leads to more interpretable as well as tree-like representations as a side benefit. The low-dimensional representative capacity of hyperbolic geometry is well-known [6] , and interestingly, we observe that training with HypStructure allows for learning extremely low-dimensional representations with distortion values lower than even their corresponding high-dimensional Euclidean counterparts.• We argue that representations learned with an underlying hierarchical structure are beneficial not only for the in-distribution (ID) classification tasks but also for Out-of-distribution (OOD) detection tasks. We empirically demonstrate that learning ID representations with HypStructure leads to improved OOD detection on 9 real-world OOD datasets without sacrificing ID accuracy [106] .• Inspired by the improvements in OOD detection, we provide a formal analysis of the eigenspectrum of the in-distribution hierarchy-informed features learned with CPCC-style structured regularization methods, thus leading to a better understanding of the behavior of structured representations in general. 
preliminaries: 
 In this section, we first provide a background of structured representation learning and then discuss the limited representation capacity of the Euclidean space for hierarchical information, which serves as the primary motivation for our work.Structured representation learning [104] breaks the permutation invariance of flat representation learning by incorporating a hierarchical regularization term with a standard classification loss. The regularization term is specifically designed to enforce class-conditioned grouping or partitioning in the feature space, based on a given hierarchy.More specifically, given a weighted tree T = (V, E, e) with vertices V , edges E and edge weights e, let us compute a tree metric d T for any pair of nodes v, v ′ ∈ V , as the weighted length of the shortest path in T between v and v ′ . For a real world dataset D = {(x i , y i )} N i=1 , we can specify a label tree T where a node v i ∈ V , v i corresponds to a subset of classes, and D i ⊆ D denote the subset of data points with class label v i . We denote dataset distance between D i and D j as ρ(v i , v j ) = d (D i , D j ), where d(•, •) is any distance metric in the feature space, varied by design.With a collection of tree metric d T and dataset distances ρ, we can use the Cophenetic Correlation Coefficient (CPCC) [79] , inherently a Pearson's correlation coefficient, to evaluate the correspondence between the nodes of the tree, and the features in the representation space. Let d T , ρ denote the mean of the collection of distances, then CPCC is defined asEQUATION)For the supervised classification task, we consider the training set D in tr = {(x i , y i )} N i=1 and we aim to learn the network parameter θ for a feature encoder f θ : X → Z, where Z ⊆ R d denotes the representation/feature space. For structured representation learning, the feature encoder is usually followed by a classifier g w , and the parameters θ, w are learnt by minimizing L along with a standard flat (non-hierarchical) classification loss, for instance, Cross-Entropy (CE) or Supervised Contrastive (SupCon) [39] loss, with the structured regularization term as:L(D) = (x,y)∈D ℓ Flat (x, y, θ, w) -α • CPCC(d T , ρ).(EQUATIONUsing a composite objective as defined in Equation ( 2), we can enforce the distance relationship between a pair of representations in the feature space, to behave similarly to the tree metric between the same vertices. For instance, consider a simple label tree with a root node, a coarse level, and a fine level, where subsets of fine classes share the same coarse parent. For this hierarchy, we would expect the fine classes of the same parents (e.g., apple and banana are fruits) to have closer representations in the feature space, whereas fine classes with different coarse parents (e.g., an apple is a fruit and a tulip is a flower) should be further apart. The learned structure-informed representations reflect these hierarchical relationships and lead to interpretable features with better generalization [104] .Equation (1) offers the flexibility of designing a metric to measure the similarity between two data subsets, and [104] define the Euclidean dataset distance as ρ ℓ2 (D i , D j ) := ∥ 1 |Di| x∈Di f (x) -1 |Dj | x ′ ∈Dj f (x ′ )∥ 2 .The distance between datasets is thus the ℓ 2 distance between two Euclidean centroids of their class-conditioned representations, which is unsuitable for modeling tree-like data [8] . Additionally, this regularization approach in [104] is applied only to the leaf nodes of T for efficiency. However, this leaf-only formulation of the CPCC offers an approximation of the structured information, since the distance between non-leaf nodes is not restricted explicitly by the regularization. This approximation, therefore, leads to a loss of information contained in the original hierarchy T . Actually, it is impossible to embed d T into ℓ 2 exactly. Or more formally, there exists no bijection φ such that d T (φ(z i ), φ(z j )) = ∥z i -z j ∥ 2 irrespective of how large the feature dimension d is. We provide two such examples for a toy label tree in Figure 1 , below. of class similarity. To verify our claims, we consider the optimization of 512-dimensional ℓ 2 -CPCC structured representations for CIFAR10 [45] . The CIFAR10 dataset consists of a small label hierarchy as shown in Figure 2 (left). The optimal CPCC is achieved when each tree metric value corresponds to a single ρ ℓ2 . However, in Figure 2 (right), even with an optimization of the ℓ 2 -CPCC loss for the entire tree, we observe a sub-optimal train CPCC less than 1, where the distance between two coarse nodes, transportation and animal, is far away from the desired solution. Furthermore, optimization of the CPCC loss for only the leaf nodes, leads to an even larger distortion of the tree metrics.A B C D E F G H C D E A B O). It also consists of 50,000 training images and 10,000 test images, however the images belong to 100 classes. Note that the classes are not identical to the CIFAR10 dataset.3. ImageNet100( [72] ). This dataset is created as a subset of the large-scale ImageNet dataset following [59] . The original ImageNet dataset consists of 1,000 classes and 1.2 million training images and 50,000 validation images. We construct the ImageNet100 dataset from this original dataset by sampling 100 classes, which results in 128,241 training images and 5000 validation images. We mention the specific classes used for sampling below.Following [59] , we use the below 100 class id's for creating the ImageNet100 subset: n03877845, n03000684, n03110669, n03710721, n02825657, n02113186, n01817953, n04239074, n02002556, n04356056, n03187595, n03355925, n03125729, n02058221, n01580077, n03016953, n02843684, n04371430, n01944390, n03887697, n04037443, n02493793, n01518878, n03840681, n04179913, n01871265, n03866082, n03180011, n01910747, n03388549, n03908714, n01855032, n02134084, n03400231, n04483307, n03721384, n02033041, n01775062, n02808304, n13052670, n01601694, n04136333, n03272562, n03895866, n03995372, n06785654, n02111889, n03447721, n03666591, n04376876, n03929855, n02128757, n02326432, n07614500, n01695060, n02484975, n02105412, n04090263, n03127925, n04550184, n04606251, n02488702, n03404251, n03633091, n02091635, n03457902, n02233338, n02483362, n04461696, n02871525, n01689811, n01498041, n02107312, n01632458, n03394916, n04147183, n04418357, n03218198, n01917289, n02102318, n02088364, n09835506, n02095570, n03982430, n04041544, n04562935, n03933933, n01843065, n02128925, n02480495, n03425413, n03935335, n02971356, n02124075, n07714571, n03133878, n02097130, n02113799, n09399592, n03594945.In addition to the training and validation images, we also require the label hierarchy for each of these datasets for the CPCC computation in ℓ 2 -CPCC and HypStructure approaches. For CIFAR100, we use the three-level hierarchy provided with the dataset release 3 . We show this hierarchy in Table 3 , where the top-level is the root of the tree. Since no hierarchy is available for the CIFAR10 and ImageNet100 datasets, we construct a hierarchy for CIFAR10 manually, as seen in Figure 2 . For ImageNet100, we create a subtree from the WordNet4 hierarchy, given the 100 aforementioned classes as leaves. We consider the classes which are one level above the leaf nodes in the hierarchy as the coarse classes, following [104] .For the task of OOD detection, we use the following five diverse OOD datasets for CIFAR10 and CIFAR100 as ID datasets, following the literature [83] : SVHN [65] , Textures [9] , Places365 [109] , LSUN [102] and iSUN [99] . When ImageNet100 is used as the ID dataset, we use 4 diverse OOD datasets as the ones in [37] , namely subsets of iNaturalist [90] , SUN [96] , Places [109] and Textures [9] . These datasets have been processed so that there is no overlap with the ImageNet classes.We use Gromov's δ rel to evaluate the tree-likeness of the data in Section 4.1, following [40] . For an arbitrary metric space X with metric d, for any three points x, y, z ∈ X , we can define the Gromov product as(y, z) x = 1 2 (d(x, y) + d(x, z) -d(y, z))Then, δ-hyperbolicity can be defined as the minimum value of δ such that for any four points x, y, z, w ∈ X , the following condition holds:(x, z) w ≥ min((x, y) w , (y, z) w ) -δIt can be shown that equivalently, there exists a geometric definition of δ-hyperbolicity. A geodesic triangle in X is δ-slim if each of its side is contained in the δ-neighbourhood of the union of the other two sides. We define δ-hyperbolicity as the minimum δ that guarantees any triangle in X is δ-slim.From Figure 10 , when the curvature of the surface increases, the geodesic triangle converges to a tree/star graph, and δ gradually reduces to 0.Following [40] , we use the scale-invariant metric δ rel = 2δ diam(X ) for evaluation, so that the δ rel is normalized in [0, 1], and the diam(•) is the set diameter or the maximal pairwise distance. We also perform an evaluation of the fine classification accuracy following the common linear evaluation protocol [39] where a linear classifier is trained on top of the normalized penultimate layer features. We report these accuracies for the models trained on the CIFAR10 and CIFAR100 datasets in Table 4 for the leaf-only variants of the models. We observe that the relative trend of accuracies is identical to the ones reported using the kNN evaluation in Table 1 and our proposed method HypStructure outperforms the flat and ℓ 2 -CPCC methods on both the datasets.To understand the role of each component in our proposed methodology HypStructure, we perform a detailed ablation study with the different components and measure the fine and the coarse accuracies on the CIFAR100 dataset. Specifically, we examine 1. the role of embedding all internal nodes in the label hierarchy (eq. ( 8) and line 10 in Algorithm 1), as opposed to only using leaf nodes as in [104] . We refer to the inclusion of internal nodes as T int .2. the role of hyperbolic class centroids computation using hyperbolic averaging (eq. ( 6) and line 8 in Algorithm 1), as opposed to the Euclidean computation of class prototypes as in [104] . We refer to the hyperbolic class centroid computation as ω hyp .3. the role of the hyperbolic centering loss in our proposed methodology (eq. ( 8) and line 11 in Algorithm 1), as opposed to not using a centering loss. We refer to the inclusion of the centering loss as ℓ center .We ablate over the aforementioned settings, where a ✓ denotes the inclusion of that setting, and report the results on the CIFAR100 dataset in Table 5 . Firstly, we observe that while the centering loss ℓ center improves the coarse accuracy only by a small increment, it leads to a significant improvement in the fine accuracy (rows 1 → 2 and 4 → 5), indicating that the centering of the root in the poincare disk allows for a better relative positioning of the fine classes within the coarse class groups. Secondly, we observe that both the inclusion of internal nodes T int , and the hyperbolic computation of the class centroids ω hyp is critical for accurately embedding the hierarchy, and removing either of these components (i.e. rows 5 → 3 for T int and rows 5 → 2 for ω hyp ), leads to a degradation in both the fine as well as the coarse accuracies. The best overall performance is observed when all three of the components are included (row 5). The goal of prior works in the OOD literature is the supervised setting of learning an accurate classifier for ID data, along with an ID-OOD detection methodology and this task has been explored in the generative model setting [42, 63, 71, 77, 97] , and more extensively in the supervised discriminative model setting [2, 30, 36, 37, 51, 53, 82, 60] . The methods in this setting can be categorized into four sub-categories following [106] , primarily:Post-Hoc Inference These methods design post-processing/scoring mechanisms on base classifiers such as MSP [30] , ODIN [51] , ReAct [82] , SSD+ [76] , KNN+ [83] and RankFeat [80] .Training without outlier data These methods involve training-time regularization or different objective functions for improving OOD detection capabilities such as G-ODIN [36] , CSI [85] , LogitNorm [92] and CIDER [61] .Training with outlier data These methods assume access to auxiliary OOD training samples such as OE [31] and MixOE [105] .Data Augmentation These methods improve the generalization ability of image classifiers such as StyleAugment [22] , AugMix [32] and RegMixup [69] .Our proposed work can be considered primarily in the Training without outlier data category, and we note that none of the prior works use any additional structural regularization term in the objective functions. We report the dataset-wise OOD detection results in Tables 7a, 6 and 7 for CIFAR100, CIFAR10 and ImageNet100 respectively. We compare with several other state-of-the-art baseline OOD detection methods for CIFAR10 and CIFAR100, namely ProxyAnchor [41] , SimCLR [7] CSI [85] , and CIDER [61] respectively. Results for these methods are taken from CIDER [61] where contrastive learning based OOD detection methods typically outperforms non-contrastive learning ones. For ImageNet100, in the absence of the available class ids used to train the original models in CIDER [61] , we finetune the ResNet34 models on the created ImageNet100 dataset. For CIDER and SupCon, we use the official implementations and hyperparameters provided by the authors.We observe that our proposed method leads to an improvement in the average OOD detection AUROC over all the ID datasets. In practice, we find that the Euclidean-centroid computational variant (first compute the Euclidean centroids and then apply the exponential map) of our proposed method performs slightly better than the hyperbolic-centroid computational variant (first apply the exponential map and then compute the hyperbolic average), for the specific task of OOD detection, while having equivalent performance on the ID classification task. Hence, we report the OOD detection accuracy corresponding to the first version.We provide additional visualizations of the learnt features from our proposed method HypStructure on the CIFAR10, CIFAR100 and ImageNet100 datasets in Figures 11, 12 Embedding the internal tree nodes in HypStructure T int (as compared to only leaf nodes in prior work CPCC) and placing the root node at the center of the Poincaré disk with ℓ center loss, helps in embedding the hierarchy more accurately. To understand the impact of these components, we first visualize the learnt representations from HypStructure, with and without these components -i.e. embedding internal nodes and a centering loss vs leaf only nodes, via UMAP in Figure 14 (CIFAR100) and Figure 15 (ImageNet100). We also provide a performance comparison (fine accuracy) in Table 8 . First, based on Figures Figure 14 and This also leads to performance improvements as can be seen in Table 8 . MethodCompared to ranking-based hyperbolic losses [66] , our HypCPCC factors in absolute values of the node-to-node distances. The learned hierarchy with HypCPCC will not only implicitly encode the correct parent-child relations, but can also learn more complex and weighted hierarchical relationships more accurately. To demonstrate this, we modify the CIFAR10 tree hierarchy, and gradually increase the weight for the left transportation branch to 2× and 4× and use new weighted trees for the CPCC tree distance computation. We visualize the learnt representations in Figure 16 and we can observe that in the learned representations from left to right, the distance between the transportation classes (blue) are larger as compared to other classes, as expected. We experiment with the two HypCPCC variants, using Klein Averaging or Euclidean mean for centroid computation, as mentioned in Section 3.2 and report the results in Table 9 . We empirically observe that using the Klein averaging leads to performance improvements across datasets. We experiment with the Hyperbolic Supervised Contrastive Loss as proposed in [21] as the choice of the ℓ Flat loss and refer to this loss as ℓ HypSupCon . We follow the original setup as described by the authors for the measurement of the ℓ HypSupCon , where the representations from the encoders are not normalized directly, instead an exponential map is used to project these features from the Euclidean space to the Poincaré ball first. Then, the inner product measurement in the ℓ SupCon is replaced with the negative hyperbolic distances in the Poincaré ball to compute the ℓ HypSupCon loss. We also experiment with our proposed methodology HypStructure along with the ℓ HypSupCon loss and report the classification accuracies and hierarchy embedding metrics for both these settings in Table 10 . We further report the OOD detection performance on CIFAR10, CIFAR100 and ImageNet100 as in-distribution datasets for both of these settings in Tables 11, 12 and 13 respectively. We observe that using HypStructure with a hyperbolic loss such as ℓ HypSupCon as the Flat loss leads to improvements in accuracy across classification and OOD detection tasks while also improving the quality of embedding the hierarchy. This demonstrates the wide applicability of our proposed method HypStructure which can be used in conjunction with both euclidean and non-euclidean classification losses. We experiment with Clipped Hyperbolic Neural Networks (HNNs) [26] as a hyperbolic backbone and use our proposed methodology HypStructure in conjunction with the hyperbolic Multinomial Logistic Regression (MLR) loss. We report the classification accuracies and hierarchy embedding metrics on the CIFAR10 and CIFAR100 datasets in Table 14 , and the OOD detection performances using CIFAR10 and CIFAR100 as in-distribution datasets in Tables 15 and 16 respectively. We observe that using HypStructure along with a hyperbolic backbone leads to improvements in classification accuracies, reduced distortion in embedding the hierarchy, and improved OOD detection performance overall, demonstrating the wide applicability of HypStructure with hyperbolic networks. Justification: The abstract and the introduction both clearly state the claims made by the paper, along with a clear description of the contributions, assumptions and limitations.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Yes, the limitations of the current work as well as the avenues for future improvements to the current work can be found in Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methodology: 
 Hyperbolic spaces are more suitable for embedding hierarchical relationships with low distortion [75] and low dimensions. Hence, motivated by the aforementioned challenges, we propose a Hyperbolic Structured regularizer for hierarchy-informed representation learning. We begin with the basics of hyperbolic geometry, followed by the detailed methodology of our proposed approach. EQUATIONFor c → 0, we can recover the properties of the Euclidean geometry sincelim c→0 d Bc (z 1 , z 2 ) = 2∥z 1 -z 2 ∥. Since T z B d c is isomorphic to R d ,we can connect vectors in Euclidean space and hyperbolic space with the bijection between T z B d c and B d c [88] . For z = 0, the exponential map exp c 0 :EQUATIONAlternatively, to guarantee the correctness of Poincaré distance computation, we can also process any Euclidean vector with a clipping module [66] clipEQUATIONso the clipped vector is within the Poincare disk. We set ϵ as a small positive number in practice.Definition 3.3 (Klein Model). Klein model (K d c , g K ) consists of an 1/ √ c-radius open ball K d c = {z ∈ R d : c∥z∥ 2 <1} and a metric tensor g K different from g B . Similar to the mean computation in Euclidean space, let γ i = 1/ 1 -c∥z i ∥ 2 , the Einstein midpoint of a group of Klein vectors z 1 , . . . z n ∈ K d c has a simple expression of a weighted averageEQUATIONWe illustrate the relationship between the different hyperbolic models in Figure 3 EQUATION)For example, in Figure 3 , O ′ is the HypAve K of A ′ , B ′ , C ′ , and can be mapped back to the Poincaré disk to get the Poincaré prototype (HypAve B ) O of points A, B, C by a change of coordinates.At a high level, HypStructure uses a combination of two losses: a Hyperbolic Cophenetic Correlation Coefficient Loss (HypCPCC)), and a Hyperbolic centering loss (HypCenter) for embedding the hierarchy in the representation space. Below we describe the two components of HypStructure.The pseudocode of HypStructure is shown in Algorithm 1 in Appendix B.2.We extend the ℓ 2 -CPCC methodology in [104] to the hyperbolic space in HypCPCC. Three major steps of HypCPCC are (i) map Euclidean vectors to Poincaré space (ii) compute class prototypes (iii) use Poincaré distance for CPCC. Specifically, we first project each z i ∈ R d to B d c , and compute the Poincaré centroid for each vertex of T using hyperbolic averaging as shown in Equation (6) and Equation (7) . Alternatively, we can also compute Euclidean centroids z = 1 |Di| z∈Di z for each vertex, and project each z ∈ R d to B d c either by exp c 0 or clip c . After the computation of hyperbolic centroids, we use the pairwise distances between all vertex pairs in T in the Poincaré ball, to compute the HypCPCC loss using Equation (1) by setting ρ = d Bc .HypCenter (Hyperbolic Centering): Inspired by Sarkar's low-distortion construction [75] that places the root node of a tree at the origin, we propose a centering loss for this positioning, that enforces the representation of the root node to be close to the center of the Poincaré disk, and the representations of its children to be closer to the border of Poincaré disk. We enforce this constraint by minimizing the norm of the hyperbolic representation of the root node as ℓ center = ∥HypAve B (exp c 0 (z 1 ), . . . , exp c 0 (z N ))∥. Alternatively, for centroids computed in the Euclidean space and mapped to the Poincaré disk, we minimize ℓ center = 1/N N i=1 f θ (x i ) directly due to the monotonicity of tanh(•) in the exponential map.Finally, for α, β > 0, we can learn the hierarchy-informed representations by minimizingEQUATIONwhere ℓ Flat is a standard classification loss, such as the CE loss or the SupCon loss.Time Complexity: In a batch computation setting with a batch size b and the number of classes (leaf nodes) as k, the computational complexity for a HypStructure computation to embed the full tree will still be O(d • min{b 2 , k 2 }), which is the same as the complexity of a Euclidean leaf-only CPCC.The additional knowledge gained from internal nodes allows us to reason about the relationship between higher-level concepts, and the hyperbolic representations help in achieving a low distortion of hierarchical information for better performance in downstream tasks.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: Yes, the research conducted in this paper conforms in every respect, with the NeurIPS code of ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
experiments: 
 We conduct extensive experiments on several large-scale image benchmark datasets to evaluate the performance of HypStructure as compared to the Flat and ℓ 2 -CPCC baselines for hierarchy embedding, classification, and OOD detection tasks.Datasets and Setup Following the common benchmarks in the literature, we consider three realworld vision datasets, namely CIFAR10, CIFAR100 [45] and ImageNet100 [59] for training, which vary in scale, number of classes, and number of images per class. We construct the ImageNet100 dataset by sampling 100 classes from the ImageNet-1k [72] dataset following [59] . For CIFAR100, a three-level hierarchy is available with the dataset release [45] . Since no hierarchy is available for the CIFAR10 and ImageNet100 datasets, we construct a hierarchy for CIFAR10 manually in Figure 2 . For ImageNet100, we create a subtree from the WordNet [19] given 100 classes as leaves. More details regarding the datasets, network, training and setup are provided in the Appendix B.4. First, to assess the tree-likeness of the learnt representations, we measure the Gromov's hyperbolicity δ rel [23, 1, 38, 40] of the features in Table 1 . Lower δ rel indicates higher tree-likeness and a perfect tree metric space has δ rel = 0 (more details in Appendix B.5). To also evaluate the correspondence of the feature distances with ground truth tree metrics, we compute CPCC on test sets. We observe that HypStructure reduces distortion of hierarchical information over Flat by upto 59.4% and over ℓ 2 -CPCC by upto 45.4%, while also consistently improving the test CPCC for most datasets.We also perform a qualitative analysis of the learnt representations from HypStructure on the CIFAR10 dataset, and visualize them in a Poincaré disk using UMAP [57] in Figure 5a . We can observe clearly that the samples for fine classes arrange themselves in the Poincaré disk based on the hierarchy tree as seen in Figure 2 , being closer to the classes which share a coarse class parent.To examine the impact of feature dimension on the representative capacity of the hyperbolic space, we vary the feature dimension for HypStructure and compute the δ rel for each learnt feature.Comparing the distortion of features with the Flat and ℓ 2 -CPCC settings in Figure 4 , we observe that δ rel decreases consistently with increasing dimensions, implying that high dimension features using HypStructure are more tree-like, and better than Flat and ℓ 2 -CPCCs' 512-dimension baselines.Following [104] , we treat leaf nodes in the hierarchy as fine classes and their parent nodes as coarse classes. To evaluate the quality of the learnt representations, we perform a classification task on the fine and coarse classes using a kNN-classifier following [27, 95, 5, 110] and report the performance on the three datasets in Table 1 . We observe that HypStructure leads to upto 2.2% improvements over Flat and upto 0.8% improvements over ℓ 2 -CPCC on both fine and coarse accuracy. We also visualize the learnt test features from Flat vs HypStructure on the CIFAR100 dataset using Euclidean t-SNE [89] and show the visualizations in Figure 5b and Figure 5c respectively. We observe that HypStructure leads to sharper and more discriminative representations in Euclidean space. Additionally, we see that the fine classes belonging to a coarse class (the same shades of colors) which are semantically closer in the label hierarchy, are grouped closer and more compactly in the feature space as well, as compared to Flat. We also perform evaluations using the linear evaluation protocol [39] and observe an identical trend in the accuracy, we report these results in Appendix C.1.In addition to leveraging the hierarchy explicitly for the purpose of learning tree-like ID representations, we argue that a structured separation of features in the hyperbolic space as enforced by HypStructure is helpful for the OOD detection task as well. To verify our claim, we perform an exhaustive evaluation on 9 real-world OOD datasets and demonstrate that HypStructure leads to improvements in the OOD detection AUROC. We share more details below. Out-of-distribution (OOD) data refers to samples that do not belong to the in-distribution (ID) and whose label set is disjoint from Y in and therefore should not be predicted by the model. Therefore the goal of the OOD detection task is to design a methodology that can solve a binary problem of whether an incoming sample x ∈ X is fromP X i.e. y ∈ Y in (ID) or y / ∈ Y in (OOD).OOD datasets We evaluate on 5 OOD image datasets when CIFAR10 and CIFAR100 are used as the ID datasets, namely SVHN [65] , Places365 [109] , Textures [9] , LSUN [102] , and iSUN [99] , and 4 large scale OOD test datasets, specifically SUN [102] , Places365 [109] , Textures [9] and iNaturalist [90] when ImageNet100 is used as the ID dataset. This subset of datasets is prepared by [59] and is created with overlapping classes from ImageNet-1k removed from these datasets to ensure there is no overlap in the distributions.OOD detection scores While several scores have been proposed for the task of OOD detection, we evaluate our proposed method using the Mahalanobis score [76] , computed by estimating the mean EQUATIONwhere µ, Σ are the mean and covariance of in-distribution training features. [76] present the Mahalanobis score (eq. ( 9)) in a generalized version for multiple feature clusters. However, since they empirically observe that the single-cluster version achieves the highest performance [76] , we will focus on this version. After computing the OOD detection scores, we measure the area under the receiver operating characteristic curve (AUROC) as the primary evaluation metric following [47, 76] .We report the AUROC averaged over all the OOD datasets (5 datasets for CIFAR10 and CIFAR100, 4 datasets for ImageNet100) in Figure 7a and Table 2 In addition to the Flat (SupCon) and ℓ 2 -CPCC baselines, we also compare our method with other state-of-the-art methods (see Appendix C.3.1 for more details about existing OOD detection methods). We observe that HypStructure leads to a consistent improvement in the OOD detection score, with upto 2% in average AUROC. We also report the dataset-wise OOD detection results for the CIFAR100 ID dataset in Table 7a along with Average AUROC. To remove the bias in the Average AUROC metric towards any single dataset, we also evaluate the Borda Count (B.C.) [58] and report the same, along with a detailed comparison with more OOD detection methods in Table 7a , and Tables 6 and 7 in the Appendix C.3. We observe that HypStructure ranks in the highest performing methods consistently, thereby demonstrating a higher Borda Count as well. We additionally visualize the CIFAR100 (ID) vs SVHN (OOD) features learnt from HypStructure, using a hyperbolic UMAP visualization in Figure 7b . We observe that training with HypStructure leads to an improvement in the separation of ID vs OOD features in the Poincaré disk.Additional Experiments, Ablations and Visualizations: More experiments using hyperbolic contrastive losses and hyperbolic networks, ablation studies on each component of HypStructure and additional visualizations can be found in Appendix C. As seen in Figure 7a , we observe a significant improvement in the OOD detection performance using HypStructure with the Mahalanobis score eq. ( 9). After training a composite loss with CPCC till convergence, let us denote the matrix of the normalized in-distribution trained feature as Z ∈ R n×d . Naturally, we inspect the eigenvalue properties of Σ (i.e, Z), and observe that K = Z ⊤ Z ∈ R n×n exhibits a hierarchical block structure (Figure 8a ) where the diagonal blocks have a significantly higher correlation than other off-diagonal values, leading us to the following theorem. Theorem 5.1 (Eigenspectrum of Structured Representation with Balanced Label Tree). Let T be a balanced tree with height H, such that each level has C h nodes, h ∈ [0, H]. Let us denote each entry of K as r h where h is the height of the lowest common ancestor of the row and the column sample. If r h ≥ 0, ∀h, then: (i) For h = 0, we have 
eigenspectrum analysis of structured representations: 
 C 0 -C 1 eigenvalues λ 0 = 1 -r 1 . (ii) For 0 < h ≤ H -1, we have C h -C h+1 eigenvalues λ h = λ h-1 + (r h -r h+1 ) C0 C h . (iii) The last eigenvalue is λ H = λ H-1 + C 0 r H .We defer the eigenspectrum analysis for an arbitrary label tree to Appendix A. Theorem 5.1 implies a phase transition pattern in the eigenspectrum. There always exists a significant gap in the eigenvalues representing each level of nodes in the hierarchy, and the eigenvalues corresponding to the coarsest level are the highest in magnitude. CIFAR100 has a balanced three-level label hierarchy where each coarse label has five fine labels as its children. In Figure 8b , we visualize the eigenspectrum of CIFAR100 for HypStructure, ℓ 2 -CPCC and the Flat objective. We observe a significant drop in the eigenvalues for features learnt from two hierarchical regularization approaches, ℓ 2 -CPCC and HypStructure, at approximately the 20 th largest eigenvector (which corresponds to the number of coarse classes), whereas these phase transitions do not appear for standard flat features. We also observe that the magnitude of coarse eigenvalues are approximately at the same scale.In summary, Theorem 5.1 helps us to formally characterize the difference between flat and structured representations. CPCC style (eq. ( 1)) regularization methods can also be treated as dimensionality reduction techniques, where the structured features can be explained mostly by the coarser level features. For the OOD detection setting, this property differentiates the ID and OOD samples at the coarse level itself using a lower number of dimensions, and makes the OOD detection task easier. We visualize the OOD detection AUROC on SVHN (OOD) corresponding to the CIFAR100 (ID) features with the top-k principal component for different methods, in Figure 8c . We observe that for features learnt using HypStructure, accurately embedding the hierarchical information leads to the top 20 eigenvectors (corresponding to the coarse classes) being the most informative for OOD detection. Recall that CIDER [61] is a state-of-the-art method proposed specifically for improving OOD detection by increasing inter-class dispersion and intra-class compactness. We note that CPCC style (eq. ( 1)) methods can be seen as a generalization of CIDER on higher-level concepts, where the same rules are applied for coarse labels as well, along with the fine classes. When the ID and OOD distributions are far enough, using coarse level feature might be sufficient for OOD detection. 
related work: 
 Learning with Label Hierarchy. Several recent works have explored how to leverage hierarchical information between classes for various purposes such as relational consistency [14] , designing specific hierarchical classification architectures [101, 25, 68] , hierarchical conditioning of the logits [13] , learning order preserving embeddings [15] , and improving classification accuracy [91, 86, 48, 49, 108, 34] . The proposed structural regularization framework in [104] offers an interesting approach to embed a tree metric to learn structured representations through an explicit objective term, although they rely on the ℓ 2 distance, which is less than ideal for learning hierarchies. Hyperbolic Geometry. [66] first proposed using the hyperbolic space to learn hierarchical representations of symbolic data such as text and graphs by embedding them into a Poincaré ball. Since then, the use of hyperbolic geometry has been explored in several different applications. [40] proposed a hyperbolic image embedding for few-shot learning and person re-identification. [20] proposed hyperbolic neural network layers, enabling the development of hybrid architectures such as hyperbolic convolutional neural networks [78] , graph convolutional networks [12] , hyperbolic variational autoencoders [56] and hyperbolic attention networks [24] . Additionally, these hybrid architectures have also been explored for different tasks such as deep metric learning [18, 100] , object detection [46] and natural language processing [16] . There have also been several investigations into the properties of hyperbolic spaces and models such as low distortion [75] , small generalization error [84] and representation capacity [62] . However, none of these works have leveraged hyperbolic geometry for explicitly embedding a hierarchy in the representation space via structured regularization, and usually attempt to leverage the underlying hierarchy implicitly using hyperbolic models. 
discussion and future work: 
 In this work, we introduce HypStructure, a simple-yet-effective structural regularization framework for incorporating the label hierarchy into the representation space using hyperbolic geometry. In particular, we demonstrate that accurately embedding the hierarchical relationships leads to empirical improvements in both classification as well as the OOD detection tasks, while also learning hierarchyinformed features that are more interpretable and exhibit less distortion with respect to the label hierarchy. We are also the first to formally characterize properties of hierarchy-informed features via an eigenvalue analysis, and also relate it to the OOD detection task, to the best of our knowledge. We acknowledge that our proposed method depends on the availability or construction of an external hierarchy for computing the HypCPCC objective. If the hierarchy is unavailable or contains noise, this could present challenges. Therefore, it is important to evaluate how injecting noisy hierarchies into CPCC-based methods impacts downstream tasks. While the current work uses the Poincaré ball model, exploring the representation trade-offs and empirical performances using other models of hyperbolic geometry in HypStructure, such as the Lorentz model [67] is an interesting future direction. Further theoretical investigation into establishing the error bounds of CPCC style structured regularization objectives is of interest as well.the scalar factor of the tree distance matrix. Since each entry of K is the dot product of two samples, the relative order in K is the opposite. ■ Corollary A.1. If we use the Poincaré distance (eq. ( 3) in CPCC and let the curvature constant c = 1, the statement of cosine distance in Lemma A.1 still holds.Proof. Since the Poincaré distance (eq. ( 3)) is only defined for vectors with magnitude less than 1, let us consider the case where before the clipping operation, both u and v are outside the unit ball. After applying clip 1 , ∥u∥ = ∥v∥ = 1 -ϵ, where ϵ is a small constant (10^-5). Then∥u∥ 2 = (1 -ϵ) 2 = 1 -2ϵ + ϵ 2 . Define 2ϵ -ϵ 2 as ξ, making ∥u∥ 2 := 1 -ξ where ξ is also a small constant such that O(ξ 2 ) is negligible. Poincaré(u, v) = 2 ln ∥u -v∥ + ∥u∥ 2 ∥v∥ 2 -2u • v + 1 (1 -∥u∥ 2 )(1 -∥v∥ 2 ) = 2 ln ∥u -v∥ + 2 -2u • v -2ξ + ξ 2 ξ ≈ 2 ln ∥u -v∥ + √ 2 -2u • v -2ξ ξ = 2 ln ∥u -v∥ + ∥u∥ 2 + ∥v∥ 2 -2u • v ξ = 2 ln 2 ∥u -v∥ ξWe can see that the Poincaré distance monotonically increases with Euclidean distances ∥u -v∥. This property ensures the relative order of any two entries for Euclidean CPCC and Poincare CPCC matrices in K to be the same. Then, we can argue about the structure of K, either Euclidean or Poincare, to have the hierarchical diagonalized structure as in Figure 8a . So any statement applied for a Poincaré version of CPCC will also hold for the Euclidean CPCC counterpart. ■For each level of the tree, due to the optimization of CPCC loss, the corresponding off diagonal entries of K, which represent the intra-level-class similarities, are much smaller than inter-level-class values. We thus have a symmetric similarity matrix that takes on the following structure, where the red regions are greater than orange regions, which are further greater than the blue regions. K =              1 r 1 11            Each non-diagonal entry is called r h ij where i, j are the index of the diagonal block, or the finest label id of one sample, and h is the height of the lowest common ancestor of the two samples in the row and the column. Since every two leaves sharing the lowest common ancestor of the same height have the same tree distance, each entry of K with the same superscript will be the same so we can drop the i, j subscript in the notation. The size of each block is defined by the number of samples within one label. Then, the shown submatrix of K corresponds to the following tree in Figure 9 . Next, we present several useful lemmas and theorems. p i = p). Let r ii be the correlation for any pair of different variables within group i. Let r ij be the common correlation between any variable in group i and j. Denote the mean of the i-th diagonal block of R byR i = (1/p i )(1 + (p i -1)r ii ). Then: 1. R has p i -1 eigenvalues 1 -r ii (i = 1 : k).2. The rest of the eigenvalues are those from k × k symmetric matrix A whose diagonal elements are a ii = p i R i and whose off-diagonal elements area ij = √ p i • p j r ij .Lemma A. Proof. Using the definition of eigenvalues, we want to compute the determinant of matrix EQUATION       Adding the second till the last row to the first row, we have EQUATION       Subtracting the second till the last row by p times the first row results in an upper triangular matrix (ii) The rest of C h eigenvalues come from a C h × C h matrix, whose diagonal entries are the mean of each p i × p i diagonal block from K, and the off diagonal entries are √ p i p j r ij where r ij is the correlation between node i and node j of height h. 1) , C 0 -C h eigenvalues are all smaller than C h eigenvalues.      1 1 . . . . . . 1 0 1 -λ -p 0 . . . 1 -λ -p . . . . . . . . . 0 0 . . . . . . 0 1 -λ -p        Thus, det(M -λI) = (1 -λ + (d -1)p)(1 -λ -p) d-1 . ■ Notice that(iii) If m ≤ M -2∆(pmax-1) pmax(C h -Proof. Part (i) and (ii) can be extended from the proof of Theorem A.1. Let G be the n × C h matrix where G ij = 1 if the i-th sample is in group j, otherwise G ij = 0. For any n -C h eigenvectors in the orthogonal complement of the column space of G, the eigenvector of K is also the eigenvector of    K 1 0 • • • 0 0 K 2 • • • 0 . . . . . . . . . . . . 0 0 • • • K k    where due to the hierarchical structure of block matrix, K i has the format of            1 -r h r 1 -r h • • • r j -r h 0 • • • 0 r 1 -r h 1 -r h • • • 0 0 • • • 0 . . . . . . . . . . . . . . . . . . r j -r h 0 • • • 1 -r h 0 • • • 0 0 0 • • • 0 1 -r h • • • 0 . . . . . . . . . . . . . . . . . . 0 0 • • • 0 0 • • • 1 -r h            The rest of C h eigenvectors come from the symmetricC h × C h matrix A = (G ⊤ G) -1/2 (G ⊤ KG)(G ⊤ G) -1/2, by some basica algebra, we knowa ij = 1 √ pi • (sum of all r ij entries in p i × p j block) • 1 √ pj .For more details, we refer the reader to Theorem 3.1 in [3] where C 1 = k using their notation.Since the largest absolute value of K's eigenvalues, is bounded above by the largest row sum of the absolute values of K [35] , first n -C h eigenvectors are bounded above byU = max i (1 -r h ) + (p i -1)(r 1 -r h ) = (1 -r h ) + (p max -1)∆.On the other hand, for the rest of C h eigenvalues, we analyze matrix A:A ≤ A 1 + A 2 :=      1 + (p 1 -1)r h 0 • • • 0 0 1 + (p 2 -1)r h • • • 0 . . . . . . . . . . . . 0 0 • • • 1 + (p k -1)r h      +      (p max -1)(r 1 -r h ) p max m • • • p max m p max m (p max -1)(r 1 -r h ) • • • p max m . . . . . . . . . . . . p max m p max m • • • (p max -1)(r 1 -r h )     The inequality comes from the effect of the maximization of CPCC thatr 1 ≥ • • • r h ≥ r h+1 ≥ • • • r H and r h ≤ m.The eigenvalues of A 1 , A 2 have the analytical form, where A 1 's eigenvalues have the form of 1 + (p i -1)r h and A 2 's eigenvalues can be derived by Lemma A.2. By Weyl's inequality [93] , the minimum of these C h eigenvalues is at leastL = (1 + (p min -1)r h ) -[(p max -1)∆ - p max m + kp max m] ≥ (1 + (1 -1)r h ) -[(p max -1)∆ -p max m + kp max m].To guarantee eigenvalues from Part (ii) are larger, we want L ≥ U . We solve this inequality with m, and we will get the desired range of m. ■When r h = r 1 in Theorem A.2, we have ∆ = 0. Therefore, for a three level basic tree with only r 1 , r 2 , if m ≤ M/(p max (C 1 -1)), C 0 -C 1 eigenvalues are all smaller than C 1 eigenvalues.In general, we have shown that when m, i.e., the across group similarity is sufficiently small, the eigenvalue gap always exists. When the label tree T is balanced, we can further specify the expression of each eigenvalue and the amount of eigenvalue gaps.We now formally restate the Theorem 4.1 from the main paper and give its proof.Theorem 5.1 (Eigenspectrum of Structured Representation with Balanced Label Tree). Let T be a balanced tree with height H, such that each level has C h nodes, h ∈ [0, H]. Let us denote each entry of K as r h where h is the height of the lowest common ancestor of the row and the column sample. If r h ≥ 0, ∀h, then: (i) For h = 0, we haveC 0 -C 1 eigenvalues λ 0 = 1 -r 1 . (ii) For 0 < h ≤ H -1, we have C h -C h+1 eigenvalues λ h = λ h-1 + (r h -r h+1 ) C0 C h . (iii) The last eigenvalue is λ H = λ H-1 + C 0 r H .Proof. From Corollary A.1, we know that K ∈ R C0×C0 has a block-wise structure.Since all statements are presented recursively, we prove the theorem by structural induction on the height of the tree.The base case is Lemma A.2 with a two level hierarchy tree where only (i) and (iii) are applicable, and p = r 1 , C 0 = d, C 1 = 1. By Lemma A.2, K has C 0 -1 eigenvalues as λ 0 = 1 -r 1 , and one eigenvalue asλ 1 = 1 + (C 0 -1)r 1 = (1 -r 1 ) + r 1 /C -10 . Let us now assume that the theorem is true for the balanced tree whose root node is at height H -1. Then if we have a tree with height H. We call the resulting matrix K H . By the first bullet point of Theorem A.1 we directly get λ 0 from (i). Then by the second bullet point of Theorem A.1, the rest of the eigenvalues are from the symmetric matrix A H-1 ∈ R C1×C1 whose diagonal elements are γ = 1 + (C 0 /C 1 -1)r 1 and whose off diagonal elements areC 0 /C 1 • r j for j ≥ 2.The key is to observe that A H-1 is still a block structured matrix. After A H-1 is scaled by γ, the resulting matrix can be also seen as a result of maximizing CPCC where the off diagonal blocks have smaller values.Applying the hypothesis induction, we then know the expression of eigenvalues for A H-1 as (i) we have C 1 -C 2 eigenvalues of the formλ 1 = γ(1 - r 2 C 0 /C 1 γ ) = γ - C 0 C 1 r 2 = 1 + ( C 0 C 1 -1)r 1 - C 0 C 1 r 2 = 1 -r 1 + C 0 C 1 (r 1 -r 2 ) = λ 0 + C 0 C 1 (r 1 -r 2 ) (ii) For 0 < h ≤ H -2, we have C h+1 -C h+2 eigenvalues of the form λ h+1 -λ h = γ C 0 C 1 r h+1 γ - r h+2 γ C 1 C h+1 λ h+1 = λ h + (r h+1 -r h+2 ) C 0 C h+1 (iii) The last eigenvalue is λ H -λ H-1 = C 1 r H /γ • C 0 C 1 γ = C 0 r HTherefore, we proved the theorem by showing the induction step from K H-1 to K H holds. ■Note that the true symmetric covariance matrix K ′ might not be having the exact format as K, but it can be seen as a perturbation of K where ∥K ′ -K∥ ≤ ϵ, ϵ is a small constant. By Weyl's inequality [93] , the approximation error of each eigenvalue is bounded by[λ i -ϵ, λ i + ϵ].In this section, we first provide an overview of our algorithm, followed by a discussion on the choice of the flat loss and additional experimental details about the training and evaluation metrics.Our work proposes HypStructure, a structured hyperbolic regularization approach to embed hierarchical information into the learnt representations. This provides significant advancements in understanding and utilizing hierarchical real-world data, particularly for tasks such as representation learning, classification and OOD detection, and we recognize both positive societal impacts and potential risks of this work. The ability to better model hierarchical data in a structured and interpretable fashion is particularly helpful for domains such as AI for science and healthcare, where the learnt representations will be more reflective of the underlying relationships in the domain space. Additionally, the low-dimensional capabilities of hyperbolic geometry can lead to gains in computational efficiency and reduce the carbon footprint in large scale machine learning. However, real-world hierarchical data often incorporates existing biases which may be amplified by structured representation learning, and hence it is important to incorporate fairness constraints to mitigate this risk.The training scheme for our HypStructure based structured regularization framework is provided in Algorithm 1. At a high level, in HypStructure, we optimize a combination of the following two losses: (1) a hyperbolic CPCC loss to encourage the representations in the hyperbolic space to correspond with the label hierarchy, (2) a hyperbolic centering loss to position the representation corresponding to the root of the node at the centre of the Poincaré ball and the children nodes around it. Compute the Task loss: ℓ Flat (g w (z i ), y i )Get hyperbolic representations using exp. map (eq. ( 4)): zi ← exp c 0 (z i )8:Calculate class prototypes using hyp. Averaging (eq. ( 6)): ω i ← HypAve K ( zv 1 , . . . zv j )9:Compute pairwise hyp. distances (eq. ( 3)) ∀v i , v j ∈ V : ρ(v i , v j ) ← d Bc (ω i , ω j ) 10:Get hyp. CPCC loss (eq. ( 1): HypCPCC(d T , ρ)11:Compute hyp. centering loss using (Equation ( 6)): ℓ center = ∥HypAve B ( z1 , . . . , zB ∥)12:Get total loss using Equation ( 8): L(D B )13:Compute Gradients for learnable parameters at time t: u t (θ, w) ← ∇ θ,w L(D B )14:Refresh the parameters: (θ, w) t+1 ← (θ, w) t -η B u t (θ, w) Output: (z 1 , . . . z N ); θ, wWe use the Supervised Contrastive [39] (SupCon) loss as the first choice for a flat loss in our experimentation. Let q y be the one-hot vector with the y-th index as 1. The Cross Entropy (CE) loss, defined between the predictions g • f θ (x) and the labels y, as ℓ CE (g • f (x), y) := -i∈[k] q i log(g(f (x)) i ) has been used quite extensively in large-scale classification problems in the literature [11, 10, 43, 98] . However, several shortcoming of the CE loss, such as lack of robustness [81, 107] and poor generalization [17, 54] have been discovered in recent research. Contrastive learning has emerged as a viable alternative to the CE loss, to address these shortcomings [7, 95, 29, 87, 33, 27] . The underlying principle for these methods is to pull together embeddings for positive pairs and push apart the embeddings for negative samples, in the feature space. In the absence of labels, positive samples are created by data augmentations of images and negative samples are randomly chosen from the minibatch. However, when the labels are available, the class information can be leveraged to extend this methodology as a Supervised Contrastive loss (SupCon) by pulling together embeddings from the same class, and pushing apart the embeddings from different classes. This offers a more stable solution for a variety of tasks [39, 76, 83] . Definition B.1 (SupCon Loss). Given a training sample x i , feature encoder f θ (•) and a projection head h(•), we denote the normalized feature representations from the projection head as:EQUATIONFor the N training samples {(x i , y i )} N i=1 , we denote the training batch of 2N (augmented) pairs as {( xi , ỹi )} 2N i=1 and define the SupCon loss as:ℓ SupCon = 1 2N 2N i=1-logEQUATIONwhere N yi refers to the number of images with label y i in the batch, τ is the temperature parameter, • refers to the inner product, and u i and u k are the normalized feature representations using Equation (10) for xi and xk respectively.While the numerator in the formulation in Equation ( 11) only considers the samples (and its augmentations) belonging to the same class, the denominator sums over all the negatives as well. Overall, this encourages the network to closely align the feature representations for all the samples belonging to the same class, while pushing apart the representations of samples across different classes.We note that our proposed method HypStructure is not limited to the choice of euclidean classification losses as ℓ Flat and we report additional results with hyperbolic classification losses in Sections C.8 and C.9 respectively, demonstrating the wide applicability of our approach.We implement our method in PyTorch 2.2.2 and run all experiments on a single NVIDIA GeForce RTX-A6000 GPU. The code for our methodology will be open sourced for a wider audience upon acceptance, in the spirit of reproducible research.We use the ResNet-18 [28] network as the backbone for CIFAR10, and ResNet-34 as the backbone for CIFAR100 and ImageNet100 datasets. We use a ReLU activated multi layer perceptron with one hidden layer as the projection head h(.) where its hidden layer dimension is the same as input dimension size and the output dimension is 128. We follow the original hyperparameter settings from [39] for training the CIFAR10 and CIFAR100 models from scratch with a temperature τ = 0.1, feature dimension 512, and training for 500 epochs with an initial learning rate of 0.5 with cosine annealing, optimizing using SGD with momentum 0.9 and weight decay 10 -4 , and a batch size of 512 for all the experiments. For ImageNet100, we finetune the ResNet-34 for 20 epochs following [61] with an initial learning rate of 0.01 and update the weights of the last residual block and the nonlinear projection head, while freezing the parameters in the first three residual blocks. We use the same α values as the regularization parameters for the CPCC loss in Equation ( 2) (ℓ 2 -CPCC) and in Equation ( 8) (our proposed method HypStructure) for a fair comparison and find that the default regularization hyperparameter for the CPCC loss α = 1.0 for both ℓ 2 -CPCC and HypStructure performs well for the experiments on the CIFAR10 and CIFAR100 datasets. We observe that the experiments on the IMAGENET100 dataset benefit from a lower α = 0.5. Additionally, we set the hyperparameter for the centering loss in our methodology as β = 0.01 for all the experiments. We use the default curvature value of c = 1.0 for the mapping and distance computations in the Poincaré ball.We use the following three datasets for our primary experimentation and training in this work 1. CIFAR10 ( [45] ). It consists of 50,000 training images and 10,000 test images from 10 different classes. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss the potential positive and negative societal impacts of our work in Section B.1 of the Appendix. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper uses publicly available datasets, and does not release any data or code that have a risk of misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We have appropriately cited the original owners of data and code which is used in the paper. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [NA]Justification: The paper does not release any new assets.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: The paper does not involve crowdsourcing, nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.https://www.cs.toronto.edu/ kriz/cifar.htmlhttps://www.nltk.org/howto/wordnet.html 
 
 Appendix:  


Decision: This paper has received active consideration and discussion by the reviewers, which in turn changed the scoring of the paper towards a consensus. The reviewers appreciate the theoretical nature of the paper and find the idea simple yet clear and effective. There were a number of limitations that required a rebuttal. The AC considers the questions about additional baselines, analyses, and issues with the theory solved. The reviewers have also reached a consensus towards accept, albeit close the borderline. The AC recommends acceptance since the theory and the methodology of the paper are of interest to the community and the additional experiments and clarifications of the rebuttal strengthen the paper. A number of initial reviewer concerns are a result of a lack of referencing; despite citing >100 papers, many papers on hyperbolic learning for computer vision specifically were missed. The AC urges the authors to includes the outcomes of the rebuttal in the paper and to include both the references mentioned by the reviewers and other highly related works (especially visual learning with hyperbolic hierarchical embeddings).

Reviews: review: paper_summary: paper_summary: The paper introduces HypStructure, a novel approach for learning structured representations using hyperbolic embeddings, which are well-suited for modeling hierarchical relationships due to their tree-like structure. The method incorporates a hyperbolic tree-based representation loss and a centering loss to embed label hierarchies into feature spaces with minimal distortion.

Experiments demonstrate HypStructure's effectiveness in improving classification accuracy, especially in low-dimensional scenarios, and enhancing Out-of-Distribution (OOD) detection performance without compromising in-distribution accuracy.
strength_weakness: strength_weakness: 1. Sections 2.1, 2.2, and 3.1 are all from existing literature, which limited the contribution of the paper. Although the operations described in Section 3.1 are common hyperbolic operations, this section still lacks proper reference to the related papers.

2. In HypCPCC, the authors proposed two alternatives of the loss,
    * Map Euclidean vectors to Poincaré space then average.
    * Averaging the Euclidean vectors then map to Poincaré space.
In the 1st alternative, The use of Klein weighted average incurs extra computation, Is it worth doing so?, In the 2nd alternative is exactly the same as [r2], which also calculates the prototypes for each class in hyperbolic space and then map to Poincaré space, [r2] also deploys supervised constructive learning, but the reference is missing and comparison is not stated.

3. The statement in Theorem 5.1 is incorrect, an entry of $K$, denoted as $r^h$ should be a vector, but the theorem stated that $\lambda_0 = 1 - r^1$, which does not make sense.

3. Incorrect (but fixable) definition in line 708, the proof used $\| u \| = \| v \| = 1-\epsilon $ but in the proof the authors used the fact that $\| u \| = \| v \| = 1-\epsilon^2 $

4. Incorrect proof in Corollary A.1, the last row of the proof does not hold, Poincaré distance cannot be the same as Euclidean distance, "growing in the same trend" does not mean "proportional to".

[r2] Long, Teng, et al. "Searching for actions on the hyperbole." CVPR2020
questions: questions: 1. What is the rationale behind choosing the Klein weighted average for the first alternative in HypCPCC, considering the extra computation it incurs?
2. Can the authors provide a comparison to [r2], which is (a special case of) their second alternative of HypCPCC?
limitations: limitations: The authors did not discuss the limitations in the paper.
paper_summary: paper_summary: This work introduces a regularization scheme based on Cophenetic Correlation Coefficient to more appropriately embed semantic label hierarchical structure in the representation. The method exploits the hierarchical benefits of hyperbolic space reformulating the CPCC regularization term to operate on the Poincare ball. The proposed method sees improvement in empirical performance demonstrating the effectiveness of the approach to learn a more separable embedding space for classification.
strength_weakness: strength_weakness: ⁃	From the visualizations presented of the embedding space, notably the UMAP, your embeddings seem to have collapsed to the boundary in many places limiting the inherent hierarchy of the embeddings, this results in a limited hierarchy being represented. This in turn, leads me to question the extent of hierarchies learnt, when discussing the core intention of the work, and the claims made. One would expect that greater performance could be achieved if this had been addressed. I am aware that boundary collapse is still an unsolved problem, but careful tuning can limit its effects.

⁃	The approach is simple but arguably not significantly novel given it is a hyperbolic reformulation of CPCC with minimal changes. With that being said, these simple methods do work somewhat well in practice and are useful to progressing the field. 

⁃	The use of a Euclidean linear evaluation is a confusing direction. You are aiming to learn a hyperbolic embedding space that preserves hierarchy, yet for downstream tasks you employ a Euclidean classifier, why? You will lose the desirable properties you are aiming to capture.

⁃	Further experimentation on different hyperbolic models and downstream tasks would have helped demonstrate the generalization of the regularization to all of hyperbolic learning. Although, this cannot be expected in the rebuttal, it would have helped support the findings to present the work a more generalized approach.
questions: questions: See weaknesses.
limitations: limitations: Broader impact statement and societal impact are seemingly missing from this work. These limitations including those limitations of the analysis should be highlighted in more detail, you mention the Euclidean loss as a limitation. If I have missed them however in the text, please correct me.
paper_summary: paper_summary: The paper presents a novel approach, HypStructure, for learning structured representations. Comparing with the existing method, the proposed method adds an regularizer calculated from hyperbolic geometry. This approach aims to reduce distortion and improve generalization performance, particularly in low-dimensional scenarios. Extensive experiments are conducted on both the classification task and the out-of-distribution detection task.
strength_weakness: strength_weakness: If my understanding of the proposed loss term is correct, $L_flat$ is not calculated in the hyperbolic geometry. Have you tried the $L_flat$ with hyperbolic network or hyperbolic geometry. I hope the authors could provide more explanation of the combined loss in different geometries. 

In Section 2.1, it mentions that $D_i$ is the subset of data points with a specific class label, and $d(D_i, D_j)$ is the distance between the feature vectors of $D_i$ and $D_j$. However, it is not mentioned how the vectors for $D_i$ and $D_j$ are calculated. Is it simply the average of all the feature vectors in the subset? 

For Example 1 in Section 2.2, tree $T$ and nodes $G, C, D, E$ are referenced in a way that implies there should be a figure accompanied by the example. While Figure 1 is referenced shortly before this, it is meant to be accompanied by Example 2. Is there a figure that has been omitted here?

Also, I would recommend proofreading the paper to correct all grammatical errors. For example, in the paragraph of Section 2.1, the first sentence “Using a composite objective as defined in Equation (2), we can enforce the distance relationship between a pair of representations in the feature space, to behave similarly as the tree metric between the same vertices” should be corrected to “Using a composite objective as defined in Equation (2), we can enforce the distance relationship between a pair of representations in the feature space to behave similarly to the tree metric between the same vertices.” This version removes the unnecessary comma and corrects “behave similarly as…” to “behave similarly to…”.
questions: questions: see my comments in the Weaknesses section.
limitations: limitations: There are two limitations mentioned in this paper. The first being that the L2-CPCC can only approximate the structured information since it is impossible to “embed a tree $T$ into $L2$ without distortion,” and as this work extends the L2-CPCC, it would similarly have this limitation. However, this is never explicitly stated in Section 7, where it mentions that HypStructure is only limited to Euclidean classification losses. There could have been more said about the limitations, but it does suffice.
paper_summary: paper_summary: The paper introduces a novel regularization method, HypStructure, which utilizes hyperbolic geometry to improve the embedding of hierarchical relationships within feature representations. This approach enhances the learning of structured representations, reducing distortion and boosting generalization in low-dimensional scenarios. It also demonstrates superior performance in Out-of-Distribution (OOD) detection across various datasets through extensive empirical evaluation. Additionally, the paper includes an eigenvalue analysis that provides deeper insights into the structured representations, correlating positively with improved OOD detection performance. This advancement extends structured representation learning to hyperbolic spaces, achieving more discriminative and interpretable features that effectively capture the inherent hierarchies in complex datasets.
strength_weakness: strength_weakness: 1. The main concern of this paper is the novelty. I believe the method proposed by the author in this work has been explored in many previous works. For instance, in "Hyperbolic Image Embeddings," "Hyperbolic Contrastive Learning for Visual Representations beyond Objects", etc. Although the paper characterizes properties of hierarchy-informed features via an eigenvalue analysis, the contribution is not significant enough to be accepted. 
2. The writing is also not good enough for me. For instance, two examples starting from line 107 are not necessarily included in the formal paper; it is better to be placed in the supplementarity materials. There are also some repetitive expressions, for instance, in line 351 and line 353 (While the current work).
3. In summary, I believe the technical contribution of this paper is not significant enough to be accepted.
questions: questions: See weaknesses.
limitations: limitations: As I said in the previous section, the main concern is the novelty. Potential improvements include extending the application scenarios of hyperbolic structured representations on more vision or language tasks that have not been explored before. Further theoretical investigation into establishing the error bounds of CPCC-style structured regularization objectives is of interest as well. The writing needs to be improved as well.

Comments: title: Regarding further discussion and any additional concerns
Response to Reviewer dBDc
Response to Reviewer 3c64
Additional Results for the experiments with Hyperbolic Settings
Response to Reviewer kf6R
Response to concern about the statistical significance of the new results (Part 2)
Response to concern about the statistical significance of the new results (Part 1)
Response to Rebuttal
comment: Dear Reviewer gWWe,

We express gratitude for your time spent on reviewing our work and your valuable comments, and hope that
- the additional and exhaustive comparative experiments shared above help address your concerns about the statistical significance of our method's performance 
- the positive comments and feedback from all three other reviewers (kf6R, 3c64 and dBDc) on our theoretical contributions, overall experimentation and insights may help clarify the novelty and impact of our work to the community 

With the rebuttal discussion window closing today, we look forward to engaging in further discussion to confirm if your concerns have been addressed and if there are any other questions regarding our work, that we can answer.
Dear Reviewer dBDc, 

Thanks for your valuable feedback in the support of our work and for raising your rating! We are happy to see that our responses could address your concerns. We will include the clarifications and corrections to the proofs based on your suggestions, in the revised manuscript, and if you have any further questions for our work, we would be happy to continue the discussion.
Thank you for your detailed response and the clarifications provided. I appreciate the effort you put into addressing my concerns. After reviewing your explanation, most of my concerns have been resolved.

Although I acknowledge that the novelty and technical contribution might be somewhat limited, as pointed out by Reviewer 3c64, I believe that your proofs and the extensive experiments you conducted will still provide value to the community. In light of this, I will be raising my rating.
Thanks for raising your score and for the support of our work and it's contributions! We greatly appreciate the time you spent reviewing our paper, and going through the rebuttal and the other reviews, as well as your valuable suggestions for improving our work. We hope that the improvements across datasets, tasks and settings in terms of accuracies, distortion metrics and OOD scores are clearer in our updated set of results (as shared in response to reviewer gWWe, as well as the global comment with new results), and are grateful that our rebuttal could address your concerns. We would be happy to discuss further if you have any more questions about our work.
Thanks for the authors' rebuttal. The rebuttal strengthens my understanding of this paper. Combined with other reviews and rebuttals. I still believe the plug-in approach is useful, but the improvements on CIFAR 10 and 100 are not obvious. However, the theoretical understanding part and mathematical proof are valuable to the community. I would like to raise the score to 5. Good luck!
We are now sharing a more exhaustive set of results with the hyperbolic settings with more evaluation metrics used in our paper, with the results below (averaged over 3 seeds). **We also perform the t-test for experiments with vs. without HypStructure, and highlighted the higher performance number only if the p-value is smaller than 0.05, for statistical significance**

## Experiments with the Hyperbolic Supervised Contrastive Loss 

Fine accuracies, coarse accuracies, and distortion measurements with the L_hyp loss . 
 
| Method | Dataset | $\delta_{rel}$ (lower) | CPCC (higher) | Fine Accuracy (higher)| Coarse Accuracy (higher)|
|---|:---:|:---:|:---:|:---:|:---:|
| L_hyp (HypSupCon)  | CIFAR10| 0.128 (0.007) | 0.745 (0.017) | 94.58 (0.04) | 98.96 (0.01)|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|CIFAR10 |**0.017 (0.001)**| **0.989 (0.001)** | **95.04 (0.02)** | **99.36 (0.02)**|
| | | | | | 
| L_hyp (HypSupCon)  |CIFAR100|0.168 (0.002)|0.664 (0.012)|75.81 (0.06)|85.26 (0.07)|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|CIFAR100|**0.112 (0.005)**|**0.773 (0.008)**|**76.22 (0.14)**|**85.83 (0.06)**|
| | | | | | 
| L_hyp (HypSupCon)  |ImageNet100|0.157 (0.004)|0.473 (0.004)|89.87 (0.01)|90.41 (0.01)|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|ImageNet100|**0.126 (0.002)** |**0.714 (0.003)**|**90.26 (0.01)**|**90.95 (0.01)**|

Comparative Evaluation of OOD detection performance on a suite of OOD datasets with different ID datasets. 
OOD Detection AUROC Results on CIFAR10 (ID) 
| Method | SVHN | Textures| Places365 | LSUN | iSUN | Mean | 
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| L_hyp (HypSupCon)  |89.45 (0.18)|93.39 (0.24)|90.18 (0.09)|98.18 (0.19)	|91.31 (0.31)|92.502|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|**91.11 (0.21)**|**94.45 (0.13)**|**93.52 (0.33)**|**99.05 (0.17)**|**95.24 (0.42)**|**94.674**|

OOD Detection AUROC Results on CIFAR100 (ID) 
| Method | SVHN | Textures| Places365 | LSUN | iSUN | Mean | 
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| L_hyp (HypSupCon)  |80.16 (0.08)|79.61 (0.26)|74.02 (0.45)|70.22 (0.18)|**82.35 (0.19)**|77.272|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|**82.28 (0.19)**|**83.51 (0.29)**|**77.95 (0.31)**|**86.64 (0.54)**|69.86 (0.87)|**80.048**|

OOD Detection AUROC Results on ImageNet100 (ID) 

| Method | SUN | Places365| Textures | iNaturalist |Mean|
|---|:---:|:---:|:---:|:---:|:---:|
| L_hyp (HypSupCon)  |91.96 (0.18)|90.74 (0.26)|**97.42 (0.21)**	|94.04 (0.19)|93.54|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|**93.87 (0.05)**|**91.56 (0.13)**|97.04 (0.16)|**95.16 (0.24)**|**94.41**|



## Experiments using the Clipped Hyperbolic Neural Network Backbone 

Fine accuracies, coarse accuracies, and distortion measurements with the Clipped HNN backbone

| Method | Dataset | $\delta_{rel}$ (lower) | CPCC (higher) | Fine Accuracy (higher)| Coarse Accuracy (higher)|
|---|:---:|:---:|:---:|:---:|:---:|
| Clipped HNN  | CIFAR10|0.084 (0.008)|0.604 (0.004)|94.81 (0.23)|89.71 (2.04)|
| **Clipped HNN + HypStructure (Ours)**|CIFAR10 |**0.013 (0.002)**|**0.988 (0.001)**|94.97 (0.12)|**98.35 (0.22)**|
| | | | | | 
| Clipped HNN   |CIFAR100|0.098 (0.001)|0.528 (0.009)|76.46 (0.26)|	49.26 (0.73)|
| **Clipped HNN + HypStructure (Ours)**|CIFAR100|**0.064 (0.006)**|**0.624 (0.005)**|**77.96 (0.14)**|**55.46 (0.61)**|
| | | | | | 



Comparative Evaluation of OOD detection performance on a suite of OOD datasets with different ID datasets. 
OOD Detection AUROC Results on CIFAR10 (ID) 

| Method | SVHN | Textures| Places365 | LSUN | iSUN | Mean | 
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| Clipped HNN   |92.63 (0.24)|90.74 (0.18)|88.46 (0.19)|95.66 (0.11)|92.41 (0.25)|91.98|
| **Clipped HNN + HypStructure (Ours)**|**95.41 (0.44)**|**93.91 (0.32)**|**92.31 (0.41)**|**96.87 (0.21)**|**94.92 (0.31)**|**94.68**|

OOD Detection AUROC Results on CIFAR100 (ID) 

| Method | SVHN | Textures| Places365 | LSUN | iSUN | Mean | 
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| Clipped HNN  |89.94 (0.16)|83.77 (0.23)|77.26 (0.33)|82.87 (0.29)|	82.35 (0.15)|83.23|
|**Clipped HNN + HypStructure (Ours)**|**91.56 (0.21)**|**84.31 (0.09)**|**78.45 (0.28)**|**87.53 (0.52)**|**83.44 (0.37)**|**85.06**|

**Summary**: Based on the above results over multiple seeds, we can clearly observe that our proposed method leads to a **statistically significant** and consistent improvement in performance in terms of classification accuracies with a significantly more evident reduction in other metrics, including distortion of the representations, and a higher improvement on the mean AUROC in the OOD detection tasks, with improvements upto 3% across a suite of OOD datasets.
Thanks for the suggestion and for the support of our work! We will include the visualizations (Fig. 14 and 15)  and the new results on hyperbolic settings provided in the rebuttal in the camera ready version, along with a section outlining the limitations, and broader impact as well.
... (continued from Part 1 of this response) 

**Summary**: Based on the above results over multiple seeds, we can clearly observe that our proposed method leads to a **statistically significant** and consistent improvement in performance in terms of classification accuracies with a significantly more evident reduction in other metrics, including distortion of the representations, and a higher improvement on the mean AUROC in the OOD detection tasks, with improvements upto 3% across a suite of OOD datasets. 	


>I think that the new results in the rebuttal have marginal improvements, and I am not sure if such improvements are statistically important, which decides the novelty of the proposed method


We humbly disagree with the reviewers’ comment here and based on the above results, argue that the results would be a strong addition in demonstrating the wide-applicability of our proposed method **across tasks**, as has also been noted by Reviewer kf6R. Note that it has been discussed in multiple prior works [1] that improving accuracy on both classification as well as OOD detection tasks is fairly challenging. 

Furthermore, we would also like to point the reviewer to the global response where we clearly outline the novelty of our contribution which is not limited to the performance with hyperbolic losses/backbones, but is a lot more general, where **our work addresses the challenges of embedding explicit hierarchies and shows strong performance across tasks, learns interpretable representations and also draws important theoretical connections to tasks such as OOD detection, which are all very different from the prior works in the literature**. 

We hope that this clarifies the questions raised by the reviewer about the statistical significance and the novelty of the results. We would be happy to discuss any further questions about the work, and would really appreciate an increase in score if reviewers’ concerns are addressed.


[1] OpenOOD: Benchmarking Generalized Out-of-Distribution Detection,Yang et. al,  NeurIPS ‘22, Datasets and Benchmarks Track

[2] Learning Structured Representations by Embedding Class Hierarchy, Zeng et. al, ICLR ‘23
Thanks for taking the time to go through the rebuttal, owing to the rebuttal timeline and limited space we were only able to report results corresponding to a single seed in our initial response, however we are now sharing a more exhaustive set of results with the hyperbolic settings with more evaluation metrics used in our paper, and you can find the results below (averaged over 3 seeds). **Additionally, to address your concerns about statistical significance, we performed the t-test for experiments with vs. without HypStructure, and highlighted the higher performance number only if the p-value is smaller than 0.05.**

## Experiments with the Hyperbolic Supervised Contrastive Loss 

Fine accuracies, coarse accuracies, and distortion measurements with the L_hyp loss . 
 
| Method | Dataset | $\delta_{rel}$ (lower) | CPCC (higher) | Fine Accuracy (higher)| Coarse Accuracy (higher)|
|---|:---:|:---:|:---:|:---:|:---:|
| L_hyp (HypSupCon)  | CIFAR10| 0.128 (0.007) | 0.745 (0.017) | 94.58 (0.04) | 98.96 (0.01)|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|CIFAR10 |**0.017 (0.001)**| **0.989 (0.001)** | **95.04 (0.02)** | **99.36 (0.02)**|
| | | | | | 
| L_hyp (HypSupCon)  |CIFAR100|0.168 (0.002)|0.664 (0.012)|75.81 (0.06)|85.26 (0.07)|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|CIFAR100|**0.112 (0.005)**|**0.773 (0.008)**|**76.22 (0.14)**|**85.83 (0.06)**|
| | | | | | 
| L_hyp (HypSupCon)  |ImageNet100|0.157 (0.004)|0.473 (0.004)|89.87 (0.01)|90.41 (0.01)|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|ImageNet100|**0.126 (0.002)** |**0.714 (0.003)**|**90.26 (0.01)**|**90.95 (0.01)**|

Comparative Evaluation of OOD detection performance on a suite of OOD datasets with different ID datasets. 

OOD Detection AUROC Results on CIFAR10 (ID) 
| Method | SVHN | Textures| Places365 | LSUN | iSUN | Mean | 
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| L_hyp (HypSupCon)  |89.45 (0.18)|93.39 (0.24)|90.18 (0.09)|98.18 (0.19)	|91.31 (0.31)|92.502|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|**91.11 (0.21)**|**94.45 (0.13)**|**93.52 (0.33)**|**99.05 (0.17)**|**95.24 (0.42)**|**94.674**|

OOD Detection AUROC Results on CIFAR100 (ID) 
| Method | SVHN | Textures| Places365 | LSUN | iSUN | Mean | 
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| L_hyp (HypSupCon)  |80.16 (0.08)|79.61 (0.26)|74.02 (0.45)|70.22 (0.18)|**82.35 (0.19)**|77.272|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|**82.28 (0.19)**|**83.51 (0.29)**|**77.95 (0.31)**|**86.64 (0.54)**|69.86 (0.87)|**80.048**|

OOD Detection AUROC Results on ImageNet100 (ID) 

| Method | SUN | Places365| Textures | iNaturalist |Mean|
|---|:---:|:---:|:---:|:---:|:---:|
| L_hyp (HypSupCon)  |91.96 (0.18)|90.74 (0.26)|**97.42 (0.21)**	|94.04 (0.19)|93.54|
| **L_hyp (HypSupCon) + HypStructure (Ours)**|**93.87 (0.05)**|**91.56 (0.13)**|97.04 (0.16)|**95.16 (0.24)**|**94.41**|



## Experiments using the Clipped Hyperbolic Neural Network Backbone 

Fine accuracies, coarse accuracies, and distortion measurements with the Clipped HNN backbone

| Method | Dataset | $\delta_{rel}$ (lower) | CPCC (higher) | Fine Accuracy (higher)| Coarse Accuracy (higher)|
|---|:---:|:---:|:---:|:---:|:---:|
| Clipped HNN  | CIFAR10|0.084 (0.008)|0.604 (0.004)|94.81 (0.23)|89.71 (2.04)|
| **Clipped HNN + HypStructure (Ours)**|CIFAR10 |**0.013 (0.002)**|**0.988 (0.001)**|94.97 (0.12)|**98.35 (0.22)**|
| | | | | | 
| Clipped HNN   |CIFAR100|0.098 (0.001)|0.528 (0.009)|76.46 (0.26)|	49.26 (0.73)|
| **Clipped HNN + HypStructure (Ours)**|CIFAR100|**0.064 (0.006)**|**0.624 (0.005)**|**77.96 (0.14)**|**55.46 (0.61)**|
| | | | | | 



Comparative Evaluation of OOD detection performance on a suite of OOD datasets with different ID datasets. 

OOD Detection AUROC Results on CIFAR10 (ID) 

| Method | SVHN | Textures| Places365 | LSUN | iSUN | Mean | 
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| Clipped HNN   |92.63 (0.24)|90.74 (0.18)|88.46 (0.19)|95.66 (0.11)|92.41 (0.25)|91.98|
| **Clipped HNN + HypStructure (Ours)**|**95.41 (0.44)**|**93.91 (0.32)**|**92.31 (0.41)**|**96.87 (0.21)**|**94.92 (0.31)**|**94.68**|

OOD Detection AUROC Results on CIFAR100 (ID) 

| Method | SVHN | Textures| Places365 | LSUN | iSUN | Mean | 
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| Clipped HNN  |89.94 (0.16)|83.77 (0.23)|77.26 (0.33)|82.87 (0.29)|	82.35 (0.15)|83.23|
|**Clipped HNN + HypStructure (Ours)**|**91.56 (0.21)**|**84.31 (0.09)**|**78.45 (0.28)**|**87.53 (0.52)**|**83.44 (0.37)**|**85.06**|

... (continued in the next response)
Thanks for the authors' rebuttal. Personally, I think that the new results in the rebuttal have marginal improvements, and I am not sure if such improvements are statistically important, which decides the novelty of the proposed method. Therefore, I decide to keep my rate unchanged.
Thank you for your insightful rebuttal, and clarification on points raised during the review.

I'm grateful for your empirical evaluations and clarification on the boundary collapse and agree with the points made, however, I would still argue it would be beneficial to empirically demonstrate that collapse is not occurring given your strong claims made for hierarchy. Arguably, the small improvements when applying your centring method and accuracy metrics alone cannot refute the collapse claim or argue that the hierarchies you are aiming to capture are indeed captured. If I have misunderstood, or missed any further results that counter my points please do correct me.

Thanks for the additional evaluation on hyperbolic networks this is a strong addition that addresses one of the identified weaknesses.

I would emphasise the importance for a revised section outlining limitations, broader impact and societal for any revised manuscript, however, since you have identified these points in the rebuttal I have confidence the authors will provide this.

Given my original review is positive, and that I still have a positive outlook for this work, I have maintained my score for now, and will revisit after further discussion with other reviewers.