Title: Learning Structured Representations with Hyperbolic Embeddings
Venue: NeurIPS 2024
Accepted: yes
Keywords: hierarchical representation

representation learning

hyperbolic geometry
Abstract: Most real-world datasets consist of a natural hierarchy between classes or an inherent label structure that is either already available or can be constructed cheaply. However, most existing representation learning methods ignore this hierarchy, treating labels as permutation invariant. Recent work [Zeng et al., 2022] proposes using this structured information explicitly, but the use of Euclidean distance may distort the underlying semantic context [Chen et al., 2013]. In this work, motivated by the advantage of hyperbolic spaces in modeling hierarchical relationships, we propose a novel approach HypStructure: a Hyperbolic Structured regularization approach to accurately embed the label hierarchy into the learned representations. HypStructure is a simple-yet-effective regularizer that consists of a hyperbolic tree-based representation loss along with a centering loss, and can be combined with any standard task loss to learn hierarchy-informed features. Extensive experiments on several large-scale vision benchmarks demonstrate the efficacy of HypStructure in reducing distortion and boosting generalization performance especially under low dimensional scenarios. For a better understanding of structured representation, we perform eigenvalue analysis that links the representation geometry to improved Out-of-Distribution (OOD) detection performance seen empirically.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel regularization method, HypStructure, which utilizes hyperbolic geometry to improve the embedding of hierarchical relationships within feature representations. This approach enhances the learning of structured representations, reducing distortion and boosting generalization in low-dimensional scenarios. It also demonstrates superior performance in Out-of-Distribution (OOD) detection across various datasets through extensive empirical evaluation. Additionally, the paper includes an eigenvalue analysis that provides deeper insights into the structured representations, correlating positively with improved OOD detection performance. This advancement extends structured representation learning to hyperbolic spaces, achieving more discriminative and interpretable features that effectively capture the inherent hierarchies in complex datasets.\n\nSTRENGTHS:\n1. The paper is the first to formally characterize properties of hierarchy-informed features via an eigenvalue analysis, and also relate it to the OOD detection task.\n2. The paper is easy to read and follow, making complex concepts accessible. The use of clear definitions, structured methodology sections, and detailed discussions helps in understanding both the theoretical underpinnings and practical implications of HypStructure. Visual aids and empirical results are presented in a manner that clearly supports the claims made.\n3. The significance of this work lies in its potential impact on a range of applications that require understanding and leveraging hierarchical relationships in data, such as image recognition and OOD detection.\n\nWEAKNESSES:\n1. The main concern of this paper is the novelty. I believe the method proposed by the author in this work has been explored in many previous works. For instance, in \"Hyperbolic Image Embeddings,\" \"Hyperbolic Contrastive Learning for Visual Representations beyond Objects\", etc. Although the paper characterizes properties of hierarchy-informed features via an eigenvalue analysis, the contribution is not significant enough to be accepted. \n2. The writing is also not good enough for me. For instance, two examples starting from line 107 are not necessarily included in the formal paper; it is better to be placed in the supplementarity materials. There are also some repetitive expressions, for instance, in line 351 and line 353 (While the current work).\n3. In summary, I believe the technical contribution of this paper is not significant enough to be accepted.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nAs I said in the previous section, the main concern is the novelty. Potential improvements include extending the application scenarios of hyperbolic structured representations on more vision or language tasks that have not been explored before. Further theoretical investigation into establishing the error bounds of CPCC-style structured regularization objectives is of interest as well. The writing needs to be improved as well.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach, HypStructure, for learning structured representations. Comparing with the existing method, the proposed method adds an regularizer calculated from hyperbolic geometry. This approach aims to reduce distortion and improve generalization performance, particularly in low-dimensional scenarios. Extensive experiments are conducted on both the classification task and the out-of-distribution detection task.\n\nSTRENGTHS:\nThe paper is well organized. The paper extends and studies the existing L2-CPCC to the hyperbolic space, which effectively reduces distortion and enhances the representation of hierarchical relationships in the data. The paper also conducted comprehensive experiments as well as detailed theoretical analysis of the eigenspectrum of structured representations.\n\nWEAKNESSES:\nIf my understanding of the proposed loss term is correct, $L_flat$ is not calculated in the hyperbolic geometry. Have you tried the $L_flat$ with hyperbolic network or hyperbolic geometry. I hope the authors could provide more explanation of the combined loss in different geometries. \n\nIn Section 2.1, it mentions that $D_i$ is the subset of data points with a specific class label, and $d(D_i, D_j)$ is the distance between the feature vectors of $D_i$ and $D_j$. However, it is not mentioned how the vectors for $D_i$ and $D_j$ are calculated. Is it simply the average of all the feature vectors in the subset? \n\nFor Example 1 in Section 2.2, tree $T$ and nodes $G, C, D, E$ are referenced in a way that implies there should be a figure accompanied by the example. While Figure 1 is referenced shortly before this, it is meant to be accompanied by Example 2. Is there a figure that has been omitted here?\n\nAlso, I would recommend proofreading the paper to correct all grammatical errors. For example, in the paragraph of Section 2.1, the first sentence “Using a composite objective as defined in Equation (2), we can enforce the distance relationship between a pair of representations in the feature space, to behave similarly as the tree metric between the same vertices” should be corrected to “Using a composite objective as defined in Equation (2), we can enforce the distance relationship between a pair of representations in the feature space to behave similarly to the tree metric between the same vertices.” This version removes the unnecessary comma and corrects “behave similarly as…” to “behave similarly to…”.\n\nQUESTIONS:\nsee my comments in the Weaknesses section.\n\nLIMITATIONS:\nThere are two limitations mentioned in this paper. The first being that the L2-CPCC can only approximate the structured information since it is impossible to “embed a tree $T$ into $L2$ without distortion,” and as this work extends the L2-CPCC, it would similarly have this limitation. However, this is never explicitly stated in Section 7, where it mentions that HypStructure is only limited to Euclidean classification losses. There could have been more said about the limitations, but it does suffice.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces a regularization scheme based on Cophenetic Correlation Coefficient to more appropriately embed semantic label hierarchical structure in the representation. The method exploits the hierarchical benefits of hyperbolic space reformulating the CPCC regularization term to operate on the Poincare ball. The proposed method sees improvement in empirical performance demonstrating the effectiveness of the approach to learn a more separable embedding space for classification.\n\nSTRENGTHS:\n⁃\tThe authors present the work clearly with effective use of visual and writing structure. All figures/diagrams are useful in supporting the narrative and findings.\n\n⁃\tThe method is simple, highly generalizable, and leads to improved performance on benchmark tasks. It can therefore, be seen as an advantageous tool in hyperbolic learning that could possibly lead to impact and adaptation by practitioners in the field.\n\n⁃\tThe theoretical and analysis are generally good, with eigenspectrum analysis supporting your claims of hierarchical structure for the most part. This is a useful analysis that provides confidence in the findings supported by appropriate proofs.\n\n⁃\tExtensive details to support replication are provided.\n\nWEAKNESSES:\n⁃\tFrom the visualizations presented of the embedding space, notably the UMAP, your embeddings seem to have collapsed to the boundary in many places limiting the inherent hierarchy of the embeddings, this results in a limited hierarchy being represented. This in turn, leads me to question the extent of hierarchies learnt, when discussing the core intention of the work, and the claims made. One would expect that greater performance could be achieved if this had been addressed. I am aware that boundary collapse is still an unsolved problem, but careful tuning can limit its effects.\n\n⁃\tThe approach is simple but arguably not significantly novel given it is a hyperbolic reformulation of CPCC with minimal changes. With that being said, these simple methods do work somewhat well in practice and are useful to progressing the field. \n\n⁃\tThe use of a Euclidean linear evaluation is a confusing direction. You are aiming to learn a hyperbolic embedding space that preserves hierarchy, yet for downstream tasks you employ a Euclidean classifier, why? You will lose the desirable properties you are aiming to capture.\n\n⁃\tFurther experimentation on different hyperbolic models and downstream tasks would have helped demonstrate the generalization of the regularization to all of hyperbolic learning. Although, this cannot be expected in the rebuttal, it would have helped support the findings to present the work a more generalized approach.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nBroader impact statement and societal impact are seemingly missing from this work. These limitations including those limitations of the analysis should be highlighted in more detail, you mention the Euclidean loss as a limitation. If I have missed them however in the text, please correct me.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces HypStructure, a novel approach for learning structured representations using hyperbolic embeddings, which are well-suited for modeling hierarchical relationships due to their tree-like structure. The method incorporates a hyperbolic tree-based representation loss and a centering loss to embed label hierarchies into feature spaces with minimal distortion.\n\nExperiments demonstrate HypStructure's effectiveness in improving classification accuracy, especially in low-dimensional scenarios, and enhancing Out-of-Distribution (OOD) detection performance without compromising in-distribution accuracy.\n\nSTRENGTHS:\n1. Although it is already theoretically proved in the related work of [70, 72] that it is not possible to embed a tree in Euclidean space without loss, it is still informative to see that In section 2.2, example 1 and example 2 give good counter-examples to show this property. \n\n2. The paper is well-written and easy to follow, and the proposed model is simple yet effective. \n\n3. The paper provides a formal eigenvalue analysis that links the geometry of the learned representations to their performance\n\nWEAKNESSES:\n1. Sections 2.1, 2.2, and 3.1 are all from existing literature, which limited the contribution of the paper. Although the operations described in Section 3.1 are common hyperbolic operations, this section still lacks proper reference to the related papers.\n\n2. In HypCPCC, the authors proposed two alternatives of the loss,\n    * Map Euclidean vectors to Poincaré space then average.\n    * Averaging the Euclidean vectors then map to Poincaré space.\nIn the 1st alternative, The use of Klein weighted average incurs extra computation, Is it worth doing so?, In the 2nd alternative is exactly the same as [r2], which also calculates the prototypes for each class in hyperbolic space and then map to Poincaré space, [r2] also deploys supervised constructive learning, but the reference is missing and comparison is not stated.\n\n3. The statement in Theorem 5.1 is incorrect, an entry of $K$, denoted as $r^h$ should be a vector, but the theorem stated that $\\lambda_0 = 1 - r^1$, which does not make sense.\n\n3. Incorrect (but fixable) definition in line 708, the proof used $\\| u \\| = \\| v \\| = 1-\\epsilon $ but in the proof the authors used the fact that $\\| u \\| = \\| v \\| = 1-\\epsilon^2 $\n\n4. Incorrect proof in Corollary A.1, the last row of the proof does not hold, Poincaré distance cannot be the same as Euclidean distance, \"growing in the same trend\" does not mean \"proportional to\".\n\n[r2] Long, Teng, et al. \"Searching for actions on the hyperbole.\" CVPR2020\n\nQUESTIONS:\n1. What is the rationale behind choosing the Klein weighted average for the first alternative in HypCPCC, considering the extra computation it incurs?\n2. Can the authors provide a comparison to [r2], which is (a special case of) their second alternative of HypCPCC?\n\nLIMITATIONS:\nThe authors did not discuss the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}