Title: HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Efficient Training

Multi-modal Large Language Models

Granularity Levels

Hyperbolic Space
Abstract: Multi-modal large language models (MLLMs) have emerged as a transformative approach for aligning visual and textual understanding. They typically require extremely high computational resources (e.g., thousands of GPUs) for training to achieve cross-modal alignment at multi-granularity levels. We argue that a key source of this inefficiency lies in the vision encoders they widely equip with, e.g., CLIP and SAM, which lack the alignment with language at multi-granularity levels. To address this issue, in this paper, we leverage hyperbolic space, which inherently models hierarchical levels and thus provides a principled framework for bridging the granularity gap between visual and textual modalities at an arbitrary granularity level. Concretely, we propose an efficient training paradigm for MLLMs, dubbed as \blg, which can optimize visual representations to align with their textual counterparts at an arbitrary granularity level through dynamic hyperbolic radius adjustment in hyperbolic space. \alg employs learnable matrices with M\"{o}bius multiplication operations, implemented via three effective configurations: diagonal scaling matrices, block-diagonal matrices, and banded matrices, providing a flexible yet efficient parametrization strategy. Comprehensive experiments across multiple MLLM benchmarks demonstrate that \alg consistently improves both existing pre-training and fine-tuning MLLMs clearly with less than 1\% additional parameters. Code is available at \url{https://github.com/godlin-sjtu/HyperET}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes HyperET, a training paradigm for multi-modal large language models that operates in hyperbolic space to address purported granularity mismatches between visual and textual modalities. The authors claim that existing vision encoders like CLIP and SAM are inefficient because they align with language at single granularity levels, and propose using hyperbolic radius adjustment through Möbius multiplication operations to enable alignment at arbitrary granularity levels. The method demonstrates improvements across MLLM benchmarks with less than 1% additional parameters.\n\nSTRENGTHS AND WEAKNESSES:\npros\n\n- The core idea of leveraging hyperbolic geometry to explicitly model and adjust the semantic granularity seems interesting\n- The method is efficient (with 1% additional parameters), a practical consideration.\n- The paper is well written.\n\ncons\n\n- see the questions below\n\nQUESTIONS:\n- How do you control for the possibility that improvements stem simply from additional learnable parameters rather than the specific hyperbolic operations?\n- How does the method perform when the \"optimal\" granularity level varies within a single input (eg images with both fine-grained and coarse-grained elements)?\n- How sensitive is the method to the choice of hyperbolic curvature parameter c?\n- What happens to the hyperbolic structure during backpropagation, and how do gradient flows behave in the constrained hyperbolic manifold?\n- What is the theoretical capacity of the hyperbolic space compared to Euclidean representations. Could similar benefits be achieved through other geometric approaches?\n- Can you offer a more detailed explanation for the observed radius changes in fig2?\n- The learnable matrices seem to be applied in the attention layer. Is this choice based on empirical testing? How would applying the HyperET module to other components (such as the MLP projector) affect performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper identifies a \"granularity gap\" between vision and language modalities as a key source of inefficiency in training Multi-modal Large Language Models (MLLMs). It proposes HyperET, a novel training paradigm that uses hyperbolic space to model and align these different levels of granularity. The core idea is to use the \"hyperbolic radius\" as a proxy for visual granularity, which is then dynamically adjusted using learnable, parameter-efficient scaling matrices and Möbius multiplication operations. By adding less than 1% of new parameters, HyperET is shown to consistently improve the performance of existing MLLMs in both fine-tuning and pre-training scenarios across a wide range of benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. The paper's framing of MLLM inefficiency as a \"granularity gap\" is insightful, and using hyperbolic geometry as a solution is a principled and elegant approach.\n2. The method is rigorously tested across multiple models and tasks. Crucially, strong ablation studies confirm that the performance gains stem directly from the proposed hyperbolic mechanisms, not just from adding parameters.\n3. HyperET achieves significant performance improvements while adding negligible computational overhead (<1% additional parameters), making it a highly practical and accessible method.\n\n## Weaknesses\n1. The paper does not provide a qualitative analysis of why the model adjusts the hyperbolic radius for different tasks, missing an opportunity to offer deeper intuition behind the \"granularity adjustment\".\n2. The reliance on advanced concepts from hyperbolic geometry adds significant complexity that may not always be justified by the margin of performance improvement.\n3. The theory suggests that granularity is adjusted dynamically per input, a powerful feature that is not explicitly analyzed or discussed in the experimental results.\n\nQUESTIONS:\n1. Can you provide a qualitative analysis to make the concept of \"granularity adjustment\" more concrete? For instance, visualizing attention maps before and after applying HyperET could show if the model’s focus shifts from whole objects to finer details as the hyperbolic radius changes.\n\n2. To better establish the generality of your method, could you more directly compare HyperET's performance boost on a standard CLIP encoder versus an encoder that is already considered more granular, like DINOv2? This would clarify if the benefit is primarily in \"fixing\" CLIP or if it's a more universal alignment tool.\n\n3. The hyperbolic curvature -c is a key parameter in the methodology. Could authors discuss how this was chosen and how sensitive the model's performance is to this choice? This would improve the paper's practical guidance and reproducibility\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an efficient training paradigm called HyperET to address the inefficiency of cross-modal alignment at multi-granularity levels in MLLMs. Existing MLLMs rely on vision encoders like CLIP and SAM, which only align with language at a single granularity level, leading to heavy computational demands. HyperET leverages the properties of hyperbolic space to dynamically adjust the hyperbolic radius of visual representations, enabling alignment between visual and textual modalities at arbitrary granularity levels. Specifically, it employs learnable matrices with Möbius multiplication operations (including diagonal, block-diagonal, and banded matrices), significantly improving the performance of existing MLLMs in both pre-training and fine-tuning tasks with less than 1% additional parameters\n\nSTRENGTHS AND WEAKNESSES:\nStrength：\n1.\tBy adjusting the hyperbolic radius in hyperbolic space, it directly quantifies and aligns granular differences between vision and text, avoiding training inefficiencies caused by mismatched granularity in traditional methods\n2.\tThree parameter-efficient learnable matrix designs (diagonal, block-diagonal, banded) achieve significant performance gains with less than 1% additional parameters, reducing computational costs\n3.\tBased on hyperbolic geometry and Möbius operations, its effectiveness in granularity alignment is validated through theorems and derivations\n\nWeakness：\n1.\tAll experiments rely on CLIP as the vision encoder , leaving unanswered whether HyperET generalizes to other encoders or how it interacts with non-CLIP-based MLLMs.\n2.\tWhile three matrix configurations (diagonal, block-diagonal, banded) are proposed, the paper provides limited insight into when each is most effective. For example, it does not clarify why banded matrices with larger bandwidths do not consistently outperform simpler designs\n\nQUESTIONS:\n1.\tThe ablation study shows that hyperbolic space training outperforms Euclidean space with the same number of parameters, but the mechanism behind this superiority is not fully explained. Why is hyperbolic space uniquely effective for granularity alignment, and are there cases where Euclidean space could be more suitable?\n2.\tThe three matrix configurations (diagonal, block-diagonal, banded) are presented as flexible options, but their practical selection criteria are unclear. For specific tasks (e.g., high-resolution image understanding vs. low-level feature matching), how should researchers choose the optimal matrix type, and what is the theoretical basis for this choice?\n3.\tThe paper claims that HyperET achieves significant improvements with less than 1% additional parameters, but it does not compare the computational overhead of hyperbolic space operations with standard Euclidean operations. Could you provide quantitative analysis on the training/inference speed and memory usage of HyperET compared to baseline methods?\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces HyperET, a novel and efficient training paradigm for multi-modal large language models (MLLMs) designed to address the \"granularity mismatch\" between visual and textual modalities. The core idea is to leverage the natural hierarchical structure of hyperbolic space to represent and align features at varying levels of granularity. By dynamically adjusting the hyperbolic radius of visual representations via a parameter-efficient learnable matrix and Möbius multiplication, HyperET enables visual features to align with their textual counterparts at an arbitrary granularity level. The authors conduct comprehensive experiments on both fine-tuning and pre-training scenarios, demonstrating that HyperET consistently improves the performance of state-of-the-art MLLMs (e.g., LLaVA, MemVP) with less than 1% additional parameters, notably reducing object hallucination. The method is well-motivated, theoretically sound, and empirically validated across multiple standard benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n1. Strong and Well-Defined Motivation: The paper effectively identifies a critical and practical problem in MLLM training: the granularity mismatch between vision encoders and language models. The argument that this mismatch leads to inefficiency and performance issues like hallucination is compelling and provides a solid foundation for the work.\n2. Novel and Theoretically-Grounded Method: The proposed solution of using hyperbolic space to model and adjust semantic granularity is highly innovative and elegant. The connection between hyperbolic radius and granularity level is intuitive and provides a quantifiable framework for a previously abstract problem. The method is well-supported by clear mathematical proofs (Theorems 1 & 2), which adds significant credibility.\n3. Comprehensive Experimental Validation: The empirical evaluation is a major strength.\n\n### Weaknesses\n1. Indirect Justification of the Core Assumption: The fundamental assumption—that hyperbolic radius directly corresponds to semantic granularity—is not empirically verified. The paper lacks qualitative or quantitative analysis to visually or otherwise demonstrate that features at different radii actually capture different levels of semantic detail (e.g., from pixels/parts to global concepts).\n2. Insufficient Analysis of Computational Overhead: While the paper emphasizes \"parameter efficiency,\" it does not discuss the computational efficiency in terms of training time or FLOPs. Operations in hyperbolic space are generally more complex than their Euclidean counterparts, and an analysis of the wall-clock time or computational overhead would provide a more complete picture of the method's \"efficiency.\"\n3. Lack of Hyperparameter Details: For example, the choice and sensitivity of the curvature hyperparameter c, which is crucial for the geometry of the hyperbolic space, are not discussed.\n\nQUESTIONS:\nPlease see the weaknesses. Does the proposed method's effectiveness extend to MLLMs with non-Transformer architectures, such as those based on Mamba?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}