HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models
Authors: Zelin Peng
Zhengqin Xu
Qingyang Liu
Xiaokang Yang
Wei Shen
Venue: NeurIPS 2025 oral ()
Keywords: Efficient Training
Multi-modal Large Language Models
Granularity Levels
Hyperbolic Space
Primary area: applications

Abstract: Multi-modal large language models (MLLMs) have emerged as a transformative approach for aligning visual and textual understanding. They typically require extremely high computational resources (e.g., thousands of GPUs) for training to achieve cross-modal alignment at multi-granularity levels. We argue that a key source of this inefficiency lies in the vision encoders they widely equip with, e.g., CLIP and SAM, which lack the alignment with language at multi-granularity levels. To address this issue, in this paper, we leverage hyperbolic space, which inherently models hierarchical levels and thus provides a principled framework for bridging the granularity gap between visual and textual modalities at an arbitrary granularity level. Concretely, we propose an efficient training paradigm for MLLMs, dubbed as \blg, which can optimize visual representations to align with their textual counterparts at an arbitrary granularity level through dynamic hyperbolic radius adjustment in hyperbolic space. \alg employs learnable matrices with M\"{o}bius multiplication operations, implemented via three effective configurations: diagonal scaling matrices, block-diagonal matrices, and banded matrices, providing a flexible yet efficient parametrization strategy. Comprehensive experiments across multiple MLLM benchmarks demonstrate that \alg consistently improves both existing pre-training and fine-tuning MLLMs clearly with less than 1\% additional parameters. Code is available at \url{https://github.com/godlin-sjtu/HyperET}.

HyperET: Efficient Training in Hyperbolic Space for Multi-modal Large Language Models
Zelin Peng', Zhengqin Xu', Qingyang Liu', Xiaokang Yang', Wei Shen (es) MoE Key Lab of Artificial Intelligence, Al Institute, School of Computer Science, SITU State Key Laboratory of Infrared Physics, Shanghai Institute of Technical Physics, CAS {zelin. peng, fateß11, narumimaria, xkyang, w asen) ju. edu.cn cn
Abstract
Multi-modal large language models (MLLMs) have emerged as a transformative approach for aligning visual and textual understanding. They typically require extremely high computational resources (e.g.. thousands of GPUs) for training to achieve cross-modal alignment multi-granularity levels. We argue that key source of this inefficiency lies in the vision encoders they widely equip with, e.g., CLIP and SAM, which lack the alignment with language multi-granularityy levels. To address this issue, in this paper, we leverage hyperbolic space, which inherently models hierarchical levels and thus provides a principled framework for bridging the granularity gap between visual and textual modalities at an arbitrary granularity level. Concretely, we propose an efficient training paradigm for MLLMs, dubbed as HyperET, which can optimize visual representations to align with their textual counterparts at an arbitrary granularity level through dynamic hyperbolic radius adjustment in hyperbolic space. HyperET employs learnable matrices with Möbius multiplication operations, implemented via three effective configurations: diagonal scaling matrices, block-diagonal matrices, and banded matrices, providing a flexible yet efficient parametrization strategy. Comprehensive experiments across multiple MLLM benchmarks demonstrate that HyperET consistently improves both existing pre-training and fine-tuning MLLMs clearly with less than 1 % additional parame- ters. Code is available at https::/ github..... com/ hom/godlii in- hetnsjjttHyyperteeee
Introduction
Thanks advancements in pre-trained foundation models in both computer vision and natural language processing [60, 61, 58, 1, 29, 50, 14, 18, 50, 15], researchers have been inspired to explore the alignment of visual and language models, leading to the development of multi-modal large language models (MLLMs). This aligmment is often achieved through adapters, e.g., Q-former [16]. As result, MLLMs [2. 10, 16. 40. 49, 4, 65] rapidly develop in recent years, demonstrating strong performance in tasks that require both visual and textual understanding, c.g., image captioning and visual question answering (VQA).
Although modern multimodal large language models (MLLMs), e.g., Qwen-VL series [5, 64, 4] and InternVL series [13, 12, 11], achieve cross-modal alignment across wide range MLLM tasks that inherently involve multi-granularity levels, their success heavily depends on extensive data scaling and massive computational resources. For example, Intern VL [13] requires training on hundreds of millions of image-text pairs using up to 640 GPUs. Such a resource- resource-intensive training scheme raises serious concerns about efficiency, reproducibility, and long-term sustainability in the MLLM
Corresponding Author: wei. shenes] edu. en
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

community, especially for researchers and institutions constrained by limited computational resources. To address these concerns, is essential identify the underlying cause of this inefficiency. We argue that a key factor lies in the vision encoders that are commonly used, e.g. CLIP [54]. SAM [29], and DINOv2 [46]. These encoders are typically aligned with language at single granularity level, e.g.. either pixel-Ievel or object-level, and are thus insufficient to deal with tasks required for alignments at different granularity levels. This mismatch in granularity during training significantly impeees the optimization process, leading to inefficient cross-modal alignment and increased reliance on large-scale computational resources.
To solve this issue, we propose to directly quantify the granularity levels by leveraging hyperbolic space [7]. Empirical observations in prior works demonstrate that visual representations different hierarchical levels (e.g., image-level and object-level) naturally stratified in hyperbolic space [20, 52, 47]. This property enables the use of hyperbolic radius [57]-defined as the distance from a point to the origin in hyperbolic space-to to quantify granularity levels [47]. Specifically, points in hyperbolic space closer to the origin (smaller radius) encode low-level visual features (e-g., pixel-level information), while points near the boundary (larger radius) represent high-level visual semantics (e.g., image-level concepts). This hierarchical level suggests that adjusting the hyperbolic radius of visual representations can effectively align them with language models at arbitrary granularity levels.
Building on this insight, we propose an efficient training paradigm (HyperET) for MLLMs that is capable of optimizing visual representations to align with their textual counterparts at arbitrary granularity levels. This is achieved through learnable matrices equipped with Möbius multiplication operations [62], which enable direct and continuous adjustment of the hyperbolic radius of visual representations, thereby facilitating cross-modal alignment. practice, we introduce three parameter- efficient forms of the learnable matrices for adjusting the hyperbolic radius: (1) diagonal scaling matrices, (2) block-diagonal scaling matrices, and (3) banded scaling matrices. These designs significantly reduce the number of trainable parameters while retaining the capacity to address granularity mismatch. To further enhance parameter flexibility, the matrices can be extended to dense version with fully populated learnable elements. This expansion increases the expressive capacity of our proposed training paradigm, facilitating more effective alignment across diverse cross-modal scenarios.
To evaluate the generalization capability and effectiveness of HyperFT, we conduct extensive experi- ments across multiple pre-trained MILM benchmarks and various downstream MI.MM tasks, e.g., ScienceQA [42]. Results demonstrate that the proposed HyperET can be easily plugged-and-plas and consistently improve various MLLMs, including LLaVA-1.5 [39] and LLaVA-Ner [32] for pre-training, as well as Me [26] and LaVIN [43] for fine-tuning on downstream tasks. Notably, HyperET introduces less than 1 1% of the total trainable parameters, ensuring high parameter efficiency.
2 Related Work
2.1 Multi-modal Large Language Models
Multi-modal large language models (MLLMs) [31, 10, 33, 38, 2, 55, 36, 70] make significant breakthroughs recent advancements, aiming equip large language models [73, 60, 61, 3] with the capability to process and interpret visual information. Most MILMS achieve this goal by integrating CLIP's vision encoder [69, 54] into pre-trained large language models through adapters, e.g., MLPs [40, 39], Q-Former [16, 35], and attention mechanisms [2]. Despite its effectiveness, this straightforward connection between the visual and language modalities still fails to align tre-trained models from different modalities, thereby resulting in inferior performance, e.g., hallucinations, across various downstream tasks.
2.2 Towards Alignment in MLLM
This failure mainly stems from the fact that the CLIP's vision encoder [50] is designed for standard classification tasks and is not equipped to handle the more fine-graing visual understanding tasks required by the language modality [59, (1,,51,77) 27]. To better align the vision and language modalities in MILMs, recent works explore parameter-efficient fine-tuning methods [43, 56, 71, 26, 44]. For example, LAVIN [43] introduces adapters in both the vision encoder and LLaMA [60] to achieve
better alignment of the modalities. Another line of research [66, 59, 27] seeks to overcome this
2

---

bottleneck by incorporating additional vision models, such as DINOv2 [46], to construct a more powerful and capable vision branch. Despite the advancements the field, few studies focus on understanding the changes in the representation of vision encoders that enable MI.LMs to tackle more complex vision tasks. this work, we aim to bridge this gap by leveraging hyperbolic space to directly model the granularity levels and adapt the granularity of visual representations to an appropriate level.
2.3 Learning in Hyperbolic Space
Unlike Euclidean space, hyperbolic space can be viewed as the continuous analog of a tree [6]. making it inherently suitable for capturing hicrarchical levels among various data types. Since visual and textual concepts are inherently hierarchical, recent works [17, 30, 52, 47, 48] show that hyperbolic space serves as a promising manifold for preserving granularity levels in vision-language model representations, leading to strong performance across downstream tasks. Most existing methods directly reshape the original hierarchical structure to align different modalities. In contrast, our method preserves the original hierarchical structure and specifically leverages the intrinsic property of hyperbolic radius to enable visual representations to adapt their hierarchical level, aligning them with the language modality. This approach provides complementary perspective existing efforts in the MILLM community.
3 Preliminary Concepts
Hyperbolic Geometry. In contrast to Fuclidean or spherical geometries, hyperbolic geometry characterized by a constant negative curvature, which fundamentally distinguishes its geometric properties and computational behaviors. Following prior studies [8. 57]. we utilize the classical Poincaré ball model-one of five principal analytic models for constructing hyperbolic space due to its demonstrated efficacy in representing hierarchical levels [23, 19, 45]. The Poincaré ball model (D" )c), characterized by radius of defined as follows:
1/ L and constant negative eurvature -c (c 0) is formally
D gg
X
E c|||| <1}
Ex
where λe,x [- and g = I, denotes the Euclidean metric tensor, serving the foundation for the hyperbolic space construction.
Hyperbolicity. Building upon the theoretical framework of gyrovector spaces [62, 63], we incorporate Möbius operations into hyperbolic space, specifically the Möbius addition operation "#." and Möbius multiplication operation "8." In hyperbolic geometry, the tangent space TxD" at any point X E D. serves a first-order approximation of DD," representing
n-dimensional Euclidean space that locally approximates the hyperbolic structure. The tangent space TXID" and D are mapped to each other by exponential (T&Da D expx (-)) and logarithmic (D TSD" (-)) maps, respectively. Detailed mathematical definitions and derivations are provided in the supplementary material.
4 Methodology
This section first provides the necessary background on the conventional training paradigm from the perspective of parameter space tuming (Sec. 4.1) contextualize our approach, followed by the detailed presentation of HyperET in Sec. 4.2. Then, Sec. 4.3 provides theoretical analysis on adjusting the hyperbolic radius of visual representations.
4.1 Parameter Space Tuning
large language models (MLLMs), whether through full fine-tuning or parameter-efficient fine-tuning, aims to adapt pre-trained visual and language models to target multi-modal scenarios. However, these tuning methods, which rely solely on gradient updates, operate as constraint-free adjustments in Fuclidean space. They often implicitly assume that visual representations can sufficiently adapt the required granularity level (e.g.. transitioning
Parameter space tuning in
3

---

Hyperbolic Radius Adjustment Module
Scaling Matrix
HRA
Pretrained Weight Matrix
W
WB-D
W
W S
W W
Wo o
Diagonal
Block-Diagonal
Full
Efficient (1)
Number Trainable Params
Flexible (n²
Input
Euclidean Space: R
Hyperbolic Space:
Zero Elements
Tunable Elements
Exponential Map
Logarithmic Map
HRA: Hyperbolic Radius Adjustment
Multiplication
Figure I: The schematic representation of HyperET. In HyperET, we efficiently train MLLMs in hyperbolic space by adjusting the hyperbolic radius using a tunable scaling matrix W,, Here, W, can be configured into three sarameter-efficient variants, i.e., Diagonal, Block- Boock-Diggonal and Banded.
from image-level pixel-level), which may lead to inefficiency in alignment at certain granularity level. In contrast, the core technique of our proposed HyperET involves hyperbolic radius adjustment, which explicitly adjusts the granularity level of visual representations MLLMs. HyperET provides a simple yet effective solution to granularity mismatch challenges.
4.2 General Hyperbolic Radius Adjustment
As previously discussed, hyperbolic radius adjustment provides direct mechanism for optimizing the granularity level of visual representations, effectively bridging the granularity gap between visual and language modalities. In practice, introduce radius adjustment constraint into the weight update process, generally defined as follows:
Radw> /Radw3 =
Radwo s -Rad
(2)
where Radwo and Radw; represent the hyperbolic radii of W and W respectively, quantifying
their granularity levels. The hyperbolic weight matrices W and W are derived by projecting their Euclidean counterparts W and Wo into hyperbolic space through exponential mapping operations defined Sec. 3. The scaling coefficient s is task-adaptive, dynamically adjusting to optimize performance. However, this modification requires two-step procedure: (1) computing the hyperbolic radius of WD and (2) subsequently adjusting it. To streamline this process, we propose a more efficient approach that directly optimizes W without intermediate radius computations.
Hyperbolic Radius. Without loss of generality, for any point X E Tya in hyperbolic space, its hyperbolic radius is formally defined as follows:
Radx == d (X = Vc (vanh-"(v[IIi
where 0 denotes the origin point in hyperbolic space. In the subsequent analysis, demonstrate that Mobiüs multiplication operations 8e defined Sec. can enable precise control over the hyperbolic radius. This critical property is formally stated in the following theorems:
Theorem (Hyperbolic Radius Scaling) For a point E D hyperbolic space, the hyperbolic radius adjustment function is expanded as follows:
2 Ve
Radx
(s Radx)
Vc 2
tanh -1 (Velss 00 xx X|I)
Vc
Rad
where G. here is instantiated as a Mobitss scalar multiplication operation. Therefore, hyperbolic radius adjustment can precisely controlled through e between and X, with the scaling coefficient serving as a primary learnable parameter.

---

According to Theorem 1, &e can provide an equivalent mechanism during parameter space tuning. Consequently, hyperbolic radius adjustment in Eq. (2) can be easily achieved as follows:
W & W
(5)
Then, building upon the exponential mapping expo (-) and the logarithmic mapping logo (.g)) defined in Sec. 3, we achieve general hyperbolic radius adjustment via the following reformulation of Eq. (5):
w logo"' (s & expo
ex (W))),
(Wo)),
(6)
where is a learnable parameter for adjustment. Given that a constrained parameter set particicuyllrly when limited to a single learnable parameter-may ineffectively adjust hyperbolic radius during restricted training iterations, propose more flexible parameterization strategy through matrix- based formulations, termed flexible adjustment, which further enables precise control over hyperbolic radius optimization.
Flexible Adjustment for Hyperbolic Radius. replace the scaling coefficient which is satisfied: right-hand side of Eq. (2) can be reformulated as follows:
To achieve this, we adopt scaling matrix W, to
WW Consequently, the
Radwa Radw?
W. Will
Radww
In this framework, the learnable parameters transition from scalar values to matrix-based formulations, significantly enhancing flexibility within constrained training iterations. The following theorem demonstrates that hyperbolic radius can be dynamically scaled through Mobiüs matrix multiplication operations, generalized instantiation of &c.
Theorem (Hyperbolic Radius Flexibility Scaling) For a point X scaling matrix X, E Rax=, the flexible radius adjustment function is formally defined through Eq. (7) and Mobitls matrix multiplication operations as follows:
D hyperbolic space and a
XX, X
2
|X,X|| Ve
Radx
Radx)
Vc
|
2
2
tanh Tauh-((Golla (Vellx, 0. XII
vc
Radx,
(8)
Analogically, the scaling matrix W. is able to direct adjust the hyperbolic radius Radx through Mobiüs matrix multiplication operations, providing precise control over representation granularity.
Building upon Theorem 2, we introduce matrix-based formulation by replacing the scaling coeffi- cient with W., .. resulting the following reformulation Eq. (6);
W = logo (W, Be exPo D.C
(Wo)).
With up to O(n²) learnable elements, Wa in Eq. (9) offers significantly enhanced flexibility for hyperbolic radius adjustment. Notably, Eqs. (5) and (9) become equivalent when W, is constrained to diagonal matrix with uniform scaling factors, i.e., W, = diag(wi, (42, ... wn) E RRax and wi w i * j. Moreover, in alignment with the current paradigm of parameter-effiiient fine-tuning, we also introduce a parameter-efficie variant of W.. designed to maintain ficxibility while reducing computational overhead.
Efficient Adjustment for Hyperbolic Radius. To realize this strategy, we define diagonal scaling matrix WD diag(w;, (w1, w2..... .,.) E R where each wi is a learnable scalar and * wy when i * j. significantly reducing the number of learnable parameters. In this form, only the diagonal elements are learnable parameters. making the diagonal scaling matrix W the most parameter- efficient configuration for hyperbolic radius adjustment. Building upon this foundation, we extend WP into two more flexible variants by incrementally introducing learnable off off-diagonal elements, balancing enhanced adjustment capability with parameter efficiency: (1) Block-diagonal scaling matrix BBD diag(Ro R1, R2,..., R,),, where R. E IR x Here, r is the block size (we assume
5

---

is divisible by r). This formulation allows intra- block- interactions while maintaining sparsity
W11
Win
across blocks; (2) Banded scaling matrix WB
E R
where wij = 0 for all i.j
such that j ji d. Here, d denotes the bandwidth, indicating that nonzero elements allowed within d entries above and below the main diagonal. These two variants provide mechanisms to capture localized interactions while preserving parameter efficiency. Notably, both WBBD and W degenerate to W under specific configurations: when = n for the block-diagonal matrix and for the banded matrix. This highlights the inherent hierarchical flexibility enabled by our parametrization strategy and reflects insightful way of designing fine-tuning methods.
To summarize these fine-tuning strategies, Fig. provides unified illustration of different variants of the scaling matrix W Our introduced hyperbolic radius adjustment, i.e., HRA, adjusts visual representations by applying learnable scaling matrices the frozen pre-trained weights Wo in hyperbolic space, enabling precise adjustment over arbitrary granularity levels.
4.3 Theoretical Analysis
To demonstrate the impact of hyperbolic radius adjustment, we provide theoretical analysis of the Möbius multiplication operation. The following deduction shows that the proposed Möbius multiplication operation
directly adjust the granularity level of visual representations.
Deduction. For Yo WoX, where Ruxk is the input embedding, Wo is the pre-trained weight and Yo is the pre-trained visual representation, the forward pass of HyperET is as follows:
Yo WX logo &e exPo (xPo((W.))
(10)
Then, according to the definition of hyperbolic space, Yo is firstly projected into the hyperbolic space D and our HyperET is applied to adjust the hyperbolic radius, resulting in the new visual representation Y, expressed as:
Die(y Yd expo (Yo) W X
(11)
Y expo Dio (Y) W. &c W X
(12)
where W expo
D.c (Wo) and D =expo" (X). According to the Theorem 2, the hyperbolic
D.c
radius of D is obtained as:
Rady Rad
W. " x
W, W X
Rad
W "8 x
I X
W. W X
Rady.
| D
s. s-Rad...
(13)
where s is scaling coefficient. Therefore, our hyperbolic radius adjustment method can directly adjust the hyperbolic radius of visual representations, thus is able to bridge the granularity gap between visual and textual modalities at an arbitrary granularity level.
Experiments
Our experimental evaluation encompasses two MLLM scenarios, (1) MLLM's fine-tuning (Sec. 5.1), and (2) MLLM's pre-trininng (Sec. 5.2). A detailed ablation study is presented in Sec. 5.3 and 5.4.
5.1 MI.LM's Fine-tuning
Experimental Setting. We evaluate our method on ScienceQA [42], challenging large-scale VQA benchmark encompassing diverse scientific domains. Our comparative analysis includes Mem VP and other LLaMA-baser models with input-space visual prompting: LLaVA [40], and LaVIN [43]. We

---

Table 1: Comparision with SoTA fine-tuning methods ScienceQA test set [42]. Question
categories: NAT natural science, SOC social science, LAN = language science, TXT = w/ text context, IMG w/ image context, NO context, G1-6 grades 1 1-6, G7-12 = grades 7-12. "Ours": we here realize the extra learnable parameters as diagonal matrices, i.e., W Vision encoder: CLIP.
#Trainable Params
Language Model
Subject
Context Modality
Grade
Method
Average
NAT SOC LAN
TXT IMG NO G1-6 G7-12
Human
90.23 84.97 87.48 89.60 87.50 88. 19.10991.55 82.42
88.40
Fully Fine-Tuning
LLaVA
13B
Vicuna-
99.36 95.95 88.00 89.49 88.00 90.66/
90.90
90.92
Parameter-efficiett Tunnng
LaVIN
3.8M
LLaMA-7B
89.25 94,94 85.24 88.51 87,46 88.08 90.16 88.07
89.41
Va
3.85M (+-0.5M)
ILaMA-7B
89.35 96.06 86.54 88.29 88.01 89.33 91.36 87.65 90.03 (++0.62)
Me
3.9M
LLaMA-7B
94.45
95.05 88.64 93.99
92.36 90.94 93.10
93.01
93.07
MemVP+Ours 3,95M
(+0.05M)
LLaMA-7B
94.85 95.05 90.55
94.57
92.91 92.20
93.65
94.00
93.78 (+0.711)
LaVIN
5.4M
LLaMA-138
13B 90.32
94.38 87.73 89.44
87.65 90.31
99.19
89.26
90.50
LavIN+Ours
5.45M
(4-0005M)
LLaMA-13B
90.57 95.63 89.89 89.61 88.75 92.02
91.95 90.58 91.46 (4-0:96
Mc
5.5M
LLaMA-13B
95.07 95.15 90.00 94.43
92.86 92.47
93.61 94.07
93.78
MemVP+Ours 5.55M (+0.05M)
LLaMA-13B 96.19 95.78 90.86
95.51
94.25 93.18
94.88
94.44 94.72 (+-0.94)
follow the experiment setting in [43]. All models utilize a CLIP pre-trained ViT-L/14 visual encoder. The weights of HyperET in this task are implemented using the three parameter-efficient scaling matrices, i.e., W.,, WB-D and W and are adapted in the attention layer, consistent with most parameter-efficient tuning methods, e.g., LoRA [24]. The curvature is 0.01. All experiments are conducted using a maximum of 8 NVIDIA H800 GPUs.
Comparing to SOTA. Our experimental evaluation compares the proposed approach with statc-of- the-art parameter-efficient fine-tuning (PEFT) methods, including LaVIN [43] and Mem [26]. As demonstrated in Table 4, which presents both baseline and HyperET enhanced results, our method establishes new state-of-the-arr performance. HyperET achieves this breakthrough with minimal parameter overhead (fewer than 1%), delivering substantial improvements to both LaVIN and MemVV frameworks. Particularly noteworthy are the gains observed with LLaMA- as the backbone language model, where HyperET enhances average cross-domain accuracy by 0.96% for LaVIN and 0.94% for MemVP. Notably, when integrated with MemVP using LLaMA-7B, HyperET achieves performance par with the more computationally intensive Mem VP-LLMMA- 13B configuration, i.c., 93.78. This result demonstrates that HyperET enhances visual representation and achieves comparable performance improvements while utilizing 100, 000 x fewer parameters (0.05M vs 6B) than Mem using LLLMA- 13BB, highlighting its remarkable parameter efficiency.
5.2 MLLM's Pre-training
Experimental Setting. Our pre-training evaluation framework builds upon LLaVA-1.5 [39], employ- ing identical datasets to evaluate HyperET's effectiveness in MLLM pre-training. Our comparative analysis includes LL. [39] and LLaVA-Next [39] and train and fine-tune our HyperET with the same experiment setting. The weights of HyperET in this task are implemented using the highest flexible scaling matrices, i.e., W., and are adapted in the attention layer, consistent with most parameter-efficient training methods, e.g., LoRA [24].
Comparing to SOTA. Our experimental framework evaluates the proposed method against state- of-the-art pre-trained MLLMs across 12 standard visual language benchmarks. We implement HyperET using the most flexible matrix configuration, i.c., W. s. maximizing the model's 's adapt- ability. While this approach resembles full fine-tuning in structure, the actual parameter count remains remarkably efficient at approximately 50M-less than 1% of the language model's 13B parameters maintaining parameter efficiency. As shown in Table 2, our approach demonstrates significant performance improvements over -L- [39], particularly mitigating the limitations of CLIP-based encoders. Notably, on the POPE benchmark [37] for object hallucination detection, HyperFT substantially reduces visual hallucinations, providing empirical evidence that hyperbolic radius optimization effectively enhances the cross-mode alignment at an arbitrary level.
7

---

Table 2: Comparison with SoTA pre-trained methods 12 MLLM benchmarks, including
VQAv2 [21], GQA [25], VW: VisWiZ [22], SQA: ScienceQA-IMG [42]. TVQA: TextVQA [53], PE: POPE [37], ME: MME [67], MB: MMBench [41], MB MMBBnchh-Chine, [41], SD: SEED- (In-the- [40] and M-Vet [68]. Top-1 accuracy is reported (Best in bold, second best is underlined). Lan. Model: Language model. Benchmark names
Bench [34], LVA": :
abbreviated due to space limits. "Ours": we here realize the extra learnable parameters as full
matrices, i.c. W. Vision encoder: CLIP.
Method
Lan. Model VQAv2 GQA VW SQA TVQA
PE
ME
MB
MB CN
SD
LVA W M-
LLaVA-1.5
Vicuna-7B
78.5
62.0 50.0 66.8
58.2
85.9 1510.7 64,3 58.3
58.6
63.4
30.5
LLaVA-1.5+Ours
Vicuna-7B
80.3
63.7
51,9 69.1
60.8
87.7 1536.2 66.8
60.5
60.2
65.6
32.4
LLaVA-1.5
Vicuna-
80.0
63.3
53.6 71.6
61.3
85.9 1531.3 67.7
63.6
61.6
70.7
35.4
LLaVA-1.5+Our
Vicuna-133
82.3
65.7 55.2 73.7
63.9
88.7
1584.7 69.8
65.2
63.4
72.6
38.3
LLaVA-Next
Vicuna-7B
81.8
64.2
57.6 70.1
64.9
86.5
1519
67.4
60.6
70.2
81.6
43.9
LLaVA-Next+Oum
Vicuna-7B
82.9
65.4 58.9 70.8
65.1
88.9
1551 69.9 62.5
71.0
82.9
44.8
Table 3:
Comparative analysis of fine-tuning spaces and flexibility levels ScienceQA test
Table 4: Ablation studies of HyperET across vision encoders with varying granularity levels on ScienceQA test set.
set [42]. All experiments utilize Mem [26] with LLaMA-13B
the backbone language model. The notation is defined as follows: W
Method |ang. Model Vision Encoder]
Average
represents diagonal scaling matrices, WB-D de- notes block-diagonal scaling matrices. w in- dicates banded scaling matrices, and W corre- sponds to Euclidean space fine-tuning matrices. Key parameters include d for banded size and 2 for block size. &e: Möbius matrix multiplication.
McmVP L.LMMA-3BB MemVP LLaMA-13B
DINOV2
91.47
SAM
91.16
Efffciintt training
+W
LLaMA-13B
DINOV2
91.98 (+0.511)
+WB
LLaMA-13B
SAM
92.05 (+0.89)
se
Efficient training in hyperbollc space
+W
LLaMA-13B
DINOV2
93.38 (+1191)
#Trainable Params (M)
Method
d
&
Average
+W!
LLaMA-13B
SAM
93.74 (+22.58)
MemVP
5.5 Efficient training
93.78
Table 5: Ablation study the key components of HyperET on selected five MILM benchmarks. We here realize the extra learnable parameters as full matrices, i.e., W,, & Möbius matrix multiplication. W se corresponds to Euclidean space fine-tuning matrices with the same number of parameters.
W
5.55
(400055
93.81 (4-0.03)
W WB-D
5.64
(40.14)
2
93.70 (-0.08)
5.71
(40.21)
93.65 n.133)
Efficient training hyperbolic
space
+W!
5.55 (+0005)
0
X
93.91
+W
5.55 (+0.05)
94.72 (+-0.94)
Method
VQAv2 GQA VW
SQA
TVQA
5.64 (+0.14)
94.79
(+1.01)
+
5.78 (+-0.28)
4
94.84
Baseline
80.0
63.3
53.6
71.6
61.3
Efficient training
6.08 (+0.58)
8
V
94.82
+W sc
80.8
63.8 53.8 71.7
61.8
5.71 (+0021)
94.89 (+LII))
Efficient training hyperbolic Space
+W
5.86 (+0036)
94.8
+W.
82.3
65.7
55.2
73.7
63.9
6.15 (+00665
94.83
81.1
64.0
53.9
71.9
62.1
5.3 Ablation Study on Fine-tuning
Here, we do an ablation study on the ScienceQA [42], employing MemVP [26] as the backbone.
Introducing Diffferent Flexibility into HyperET. We systematically investigate our proposed three distinct parameterization strategies for HyperET 's learnable components. These strategies include diagonal scaling matrices D and their two extended variants: banded scaling matrices F and dlock-diagonll scaling matrices WB-D, which systematically increase parameter flexibility through progressively relaxed structural constraints. As evidenced in Table 3, enhanced flexibility achieved through ! fine-tuning pproidds only marginal performance improvements (0.17% accuracy gain over WP) while introducing over-fitting risks when further increasing learnable parameters via expanded banded sizes. These findings demonstrate that our fine-tuning method effectively accomplishes two key objectives: (1) efficient adaptation to optimal hyperbolic radii and (2) robust cross-modal alignment for downstream tasks, all while maintaining parameter efficiency.
8

---

Effectiveness of Training in Hyperbolic Space. To isolate the performance gains attributable to hyperbolic space rather than parameter increases. we conduct controlled experiments by adding an equivalent number of parameters through matrix multiplication in Euclidean space, shown in Table 3. The results demonstrate that such parameter augmentation either yields no significant performance improvement or even degrades model performance. This empirical evidence strongly validates the necessity of employing hyperbolic space for adjusting vision encoder in MLLMs, as the observed improvements in Table 3 cannot be explained by mere parameter increases.
Necessity of Mobiüs matrix multiplication. We conduct ablation studies to evaluate the impact of the Möbius matrix multiplication operation. As shown in Table 3, a comparison between rows 6 and 7 reveals that standard matrix multiplication (denoted by" x") underperforms 0.81% compared the Möbius matrix multiplication, (represented by "V"). This performance gap underscores the essential role of Möbius operations in precisely adjusting the hyperbolic radius of visual representations.
Discussion of Different Vision Encoder. To further clearly demonstrate the primary contributing factor behind the performance improvements, we present additional experimental results designed to isolate and eliminate the influence of merely increasing the number of trainable parameters. As shown in Table 4, fine-tuning SAM [29] and DINOv2 [46] in Euclidean space with the same number of additional parameters as HyperET yields only marginal gains. In contrast, fine-tuning with HyperET results substantial performance improvements. These findings clearly illustrate that our main contribution arises specifically from the proposed hyperbolic radius adjustment mechanism, rather than simply from introducing more learnable parameters.
5.4 Ablation Study on Pre-training
To further assess the contribution of each key component in the proposed HyperET, we iso- late each component in separate experiments, and the results are presented in Table 5. Com- paring row 2 and row 3, training in Euclidean space instead of hyperbolic space leads sig- nificantly lower performance, indicating that the granularity gap mismatch cannot be effectively addressed in Euclidean space. The observed improvement over the baseline is primarily at- tributable to the increased number of parameters rather than the alignment of granularity levels. Intriguingly, comparing row 2 and row 4, simply transitioning from Euclidean space to hyperbolic space yields slight improvement, which we at- tribute to the inherent ability of hyperbolic space to capture granularity levels. However, this strat- egy lacks explicit mechanisms for hyperbolic radius adjustment, limiting its effectiveness in addressing granularity mismatches. Finally, by adjusting the hyperbolic radius through Möbius matrix multiplication and integrating it with learnable matrices W.. the results show significant per- formance improvements, e.g., a 2. % gain on the TextVQA [53], demonstrating the effectiveness of HyperET. Additionally, we visualize the changes in the hyperbolic radius in Fig. 2 after training, and the observed change in the hyperbolic radius indicates the MLLM's requirement for multi- multi-granuularity levels, demonstrating the necessity of our proposed HyperET.
0.90
Hyperbolic Radius
0.88
0.86
0.84
0.82
0.80
Fine tuning
Visualization of hyperbolic radius changes in visual representation after training across different MLLM benchmarks. Normalizing the hyperbolic radius to a range of 0-1 facilitates comparison. A smaller hyperbolic radius corre- sponds to a more low granularity level of visual representation. "Zero-shot": maintaining the pre- trained weights of the vision encoder, i.c., CLIP, without additional training.
Figure 2:
6 Conclusion
This work proposes an efficient training paradigm (HyperET) for multi modal large language models in hyperbolic space. By dynamically adjusting the hyperbolic radius of visual representations through learnable matrices and Möbius multiplication operations, HyperET effectively bridges the gramularity gap between visual and textual modalities at an arbitrary granularity level. Our experiments across multiple MI.LM benchmarks demonstrate that HyperET consistently improve existing pre-training and fine-tuning basclines by large margins with less than 1 1% additional parameters.

---

Acknowledgment. and in part by the Shanghai Municipal Science and Technology Major Project under Grant 2021SHZDDZX012..
This work was supported by the NSFC under Grant 62322604, 62176159,
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint (xXiv:2303.08772 2023.
[2]
-aan-Baptsste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736 2022.
[3] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprini AxXiv:3305.14000 2023.
[4] Jinze Bai, Shuai Bai, Shusheng Yang. Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308 12966, 2023.
[5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jiangiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 12000.11322. 13923, 2025.
[6] Martin R Bridson and André Hacfliger. Metric spaces of non non-positive curvature, volume 319. Springer Science & Business Media, 2013.
[7] James W Cannon, William J Floyd, Richard Kenyon, Walter R Parry, et al. Hyperbolic geometry. Flavors of geometry, (195--115):2, 1997.
[8] Jiaxin Chen, Jie Qin, Yuming Shen, Li Liu, Fan Zhu, and Ling Shao. Learning attentive and hierarchical representations for 3d shape recognition. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 23,28, 2020, Proceedings, Part XV 16, pages 105-122. Springer, 2020.
[9] Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:230.115195, 2023.
[10] Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski. Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, al. Pali: A jointl-scalled multilingual language-image model. arXiv preprint arXiv: 06794, 2022.
[11] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao. Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:24/2.0522 2024.
[12] Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma. et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 1604. 2024.
[13] Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qin- glong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
10

---

[14] Wii-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
[15] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finttune language models. Journal of Machine Learning Research, 25(70): 2024.
[16] Wenliang Dai, Jumnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpor vision-language models with instruction tuning, 2023.
[17] Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ra- makrishna Vedantam. Hyperbolic image-tex text representations. In International Conference on Machine Learning, pages 7694-777 7731. PMLR. 2023.
[18] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2023.
[19] Octavian Ganea, Gary Bécigneul, and Thomas Hofmann. Hyperbolic neural networks. Advances in neural information processing systems, 31, 2018.
[20] songwei Ge, Shlok Mishra, Simon Kornblith, Chun-Liang Li, and David Jacobs. Hyperbolic contrastive learning for visual representations beyond objects. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6840-6849, 2023.
[21] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
[22] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608-3617, 2018
[23] Chengyang Hu, Ke-Yue Zhang, Taiping Yao, Shouhong Ding, and Lizhuang Ma. Rethinking generalizable face anti-spoofing via hierarchical sesootype-guided distribution refinement in hyperbolic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (032-1041) 2024.
[24] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 106.028006 2021.
[25] Drew A Hudson and Christopher Manning. Gqa: new dataset for real-world visual reasoming and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[26] Shibo Jie, Ychui Tang, Ning Ding, Zhi-Hong Deng, Kai Han, and Yunhe Wang. Memory-space visual prompting for efficient vision-languae fine-tuning. arXiv preprint 240.:::20000066155 2024.
[27] Oguzhan Fatih Kar, Alessio Tonioni, Petra Poklukar, Achin Kulshrestha, Amir Zamir, and Federico Tombari. Brave: Broadening the visual encoding of vision-languag models. In European Conference on Computer Vision, pages 113-132. Springer, 2024.
[28] Valentin Khrulkov, Leyla Mirvakhabova, Exgeniya Ustinova, Ivan Oseledets, and Victor Lempit- sky. Hyperbolic image embeddings. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 6418-6428, 2020.

---

[29] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander Berg, Wan- Wan Lo, et al. Segment anything. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 4015-4026, 2023.
[30] Fanjie Kong, Yanbei Chen, Jiarui Cai, and Davide Modolo. Hyperbolic learning with synthetic captions for open-world detection. In Proceedings of the IEEE/CVV Conference on Computer Vision and Puttern Recognition, pages 16762-16771. 2024.
[31] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9579-9589, 2024.
[32] Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild. 2024.
[33] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Fanyi Pu, Jingkang Yang. Chunyuan Li, and Ziwei Liu. Mimic-it: Multi-modal in-context instruction tuning. arXiv preprint arXiv:2306. 05425, 2023.
[34] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307. 16/25, 2023.
[35] Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: pre- pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
Bootstrapping language- image
[36] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06335, 2023.
[37] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen, Evaluating object hallucination in large vaniun-language models. arXiv preprint S.Ki::2305.13355 2023.
[38] Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large multi-modal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26763-26773, 2024.
[39] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296-26306, 2024.
[40] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2023.
[41] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprint arXiv:2307. 06281, 2023.
[42] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Long-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-25211 2022.
[43] Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. Cheap and quick: Efficient languael instruction tuning for large language models. Advances in Neural Information Processing Systems, 36, 2023.
[44] Feipeng Ma, Hongwei Xue, Yizhou Zhou, Guangting Wang, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, and Xiaoyan Sun. Visual perception by large language model's 's weights arXiv preprint 2rX5::200..23339 2024.
12

---

[45] Maximillian Nickel and Douwe Kiela. Learning continuous hierarchies in the lorentz model of hyperbolic geometry. In International conference on machine learning. pages 3779-3788.
PMLR, 2018.
[46] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/ 09.4,.77993.00 193, 2023.
[47] Avik Pal, Max van Spengler, Guido Maria D' Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional entailment learning for hyperbolic vision- vision-language models, arXiv preprint arXiv:2410.06911 12, 2024.
[48] Zelin Peng, Zhengqin Xu, Zhilin Zeng, Changsong Wen, Yu Huang, Menglin Yang, Feilong Tang, and Wei Shen. Understanding fine-tuning clip for open-vocabulary semantic segmentation in hyperbolic space. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 13562-4572 2025.
[49] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 9748-8763. PMLR, 2021.
[51] Pooyan Rahmanzaaehgervi. Logan Bolton, Mohammad Reza Taesiri, and Anh Totti Nguyen. Vision language models are blind. In Proceedings of the Asian Conference on Computer Vision, pages 18-34, 2024.
[52] Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, and Ajanthan Thalaiyasingam. Accept the modality gap: An exploration in the hyperbolic space. Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pages 27263-27272 2024.
[53] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8317-8326, 2019.
[54] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 230.:/333../.., 2023.
[55] Quan Sun, Qiying Yu. Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222, 2023.
[56] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer learning for lision-and-lanuuee tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2022.
[57] Dídac Surís, Ruoshi Liu, and Carl Vondrick. Learning the predictability of the future. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12607-12617, 2021.
[58] Yehui Tang, Fangcheng Liu, Yunsheng Ni, Yuchuan Tian, Zheyuan Bai, Yi-Qi Hu, Sichao Liu, Shangling Jui, Kai Han, and Yunhe Wang. Rethinking optimization and architecture for tiny language models. arXiv preprint S103::2402.0299999 2024.
[59] Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma. Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. Proceedings of the IEEF/VVF Conference Computer Vision and Pattern Recognition, pages 9568-9578 2024.
13

---

[60] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint 1.4i:22302.13977, 2023.
[61] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXi::2307.0922: 2023.
[62] Abraham A Ungar. Hyperbolic trigonometry and its application in the poincaré ball model of hyperbolic geometry. Computers & Mathematics with Applications, 41(1-2)11351447 2001.
[63] Abraham Albert Ungar. A gyrovector space approach to hyperbolic geometry. Synthesis Lectures on Mathematics and Statistics, 1(I):1-194, 2008.
[64] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[65] Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, et al. Visionllm: Large language model is also open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36, 2023.
[66] Wenxuan Wang, Quan Sun, Fan Zhang, Yepeng Tang, Jing Liu, and Xinlong Wang. Diffusion feedback helps clip see better. arXiv preprint arXiv:2407.20171, 2024.
[67] Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(12)nwae4033 2024.
[68] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinehao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02444 2023.
[69] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11975-11986, 2023.
[70] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tunee audio-visual language model for video understanding. arXiv preprint arXiv:2306.0288 2023.
[71] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
[72] Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba to modlil large language model for efficient inference. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 10421-10429, 2025.
[73] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-asaa-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatinn right after your answer (even for NA).
The checklist answers
integral part of your paper submission. They visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] provvided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). ). general, answering "[No] " or "[NA]" is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The abstract clearly states the paper's main contributions and scope. We propose an efficient training paradigm for MLLMs, dubbed as HyperET, which can optimize visual representations align with their textual counterparts an arbitrary granularity level through dynamic hyperbolic radius adjustment in hyperbolic space. Our method consistently improves both existing pre-training and fine-tuning MLLMs by large margins with less than 1% additional parameters.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals not attained by the paper.
15

---

2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [NA]
Justification: Those are not discussed in the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e-g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: All assumptions are proved clearly in the main paper or the supplemental material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
16

---

Answer: [Yes]
Justification: Detailed experimental settings and information are provided in Sec. 4 and Sec. which are sufficient to reproduce the results reported in this paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution
specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
The code will be made public after acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips cc/ Coo.eeSumissiiooool.......
public/guien /ttgg//dd//CodSuuie
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nips. wtippsccpppulii httpp//pcc.gpiuee///e/e/eeepeeee /eoiidde//eeebmiiiic. SolodeSubmissioooool.... ssioololicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
17

---

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: All the experimental settings are clarified in Sec. 5 and supplementary materi- als.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not report error bars statistical significance measures mainly because the experiments were conducted in stable and controlled environment, where repeated trials showed minimal variability. Additionally, due computational resources and time constraints, extensive repeated runs were not feasible. Moreover, omitting such measures aligns with common practice in this research area.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma emror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
18

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Computing resource requirements are detailed in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp://neett http:/::////u.....eeee wwtpwblic/pttiiieececccc www.thhiiicceieeeeeeeeeeeeee
Answer: [Yes]
Justification:
We ensure our research adheres to the guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
19

---

11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper properly credits all original papers, states their versions, and respects the copyright and terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
20

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
The core method development this research does not involve LLMs
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our L.LM policy (https: http///eerrpp..pp http///eriss.c//oofeeeeeeeeeeeee www./o/oeeeencee///000000 for what should or should not be described.

---

Appendix of HyperET
This appendix provides additional theoretical and empirical details support our work. Section introduces the formal definition of hyperbolicity. Section B presents the full derivation of the core theorem introduced the main manuscript. Section D offers an extended empirical comparison between HyperETT and representative training methods in hyperbolic space, i.e., MERU, further validating the effectiveness HyperET.
A The Definition of Hyperbolicity
This section introduces the concepts of hyperbolicity utilized in this work, including Möbius addition operation, Möbius multiplication operation, and the tangent space.
Möbius addition operation. For X, Y E De the Möbius addition operation is defined as:
X C Y = (1+ 2x Y) + cl Y ) + (1 c|x 2
(14)
1 + 2e X, Y) + X Y
Möbius scalar multiplication operation. For scalar E R and vector X E W the Möbius scalar multiplication operation defined as:
X X
X = (1/Vejtanh(r: tanh (We
(15)
Möbius matrix multiplication operation. Refer to the relation definition in [28]. for a matrix M E Taxx and and point X E D if MX + O, the Möbius matrix multiplication operation is defined as:
M 2 X = ( Itanh [MXX] X
MX
tanh tanh (vclpx)))
|MX
Tangent Space. The tangent space TDD" at point X E DD is the first order approximation of D which is an n-dimensional Euclidean space, The tangent space TxD2 Dr and DV are mapped to each
other by exponential (T + Dh expx (-)x and logarithmic (ID + Tx logx (-)) maps, respectively. For any X, Y W and V TxI the mapping functions are given for V + 0 and Y & X by:
1 2
V
expx (V)
X the tanh Ve
(16)
Vel
2 tanh
X
+ Y
DC logx
(Voll
X B. YII)
(17)
VeXax
X + Y||
B Derivation of the Theorem
Theorem (Hyperbolic Radius Scaling) For a point E Tra in hyperbolic space, the hyperbolic radius adjustment function is expanded as follows:
2 Vc
s Radx = (s Radx)
Vc 2
2
tanh ((olls /.
Vc
RRdd R...
(18)
where @. here is instantiated as a Mobiüs scalar multiplication operation. Therefore, hyperbolic radius adjustment be precisely controlled through @. between and X, with the scaling coefficient serving as a primary learnable parameter.
Proof. In hyperbolic space, considering point X E D with hyperbolic radius Radx (2/ /c) tanh- -1 [|), the detailed expansion of the hyperbolic radius adjustment function is

---

formulated as:
(s v
8 Radx
Radx)
Vc
2
Vc
tanh
(taan(( Radx))
Vc
2
VC
X
tanh
tanh(s
Radx
Vc
X
tanh(s
Le 3/ Radx
tanh Vc
(Ve
Vol
tanh(s tanh -1 '(vC
tanh -1 (VV
X
Ve
Vel|
2
tanh
(Vell (v. x()
Ve
Rad
(19)
Consequently, according to Eq. (19), the scaling scalar s can adjust the hyperbolic radius Radx via the Möbius scalar multiplication operation.
Theorem (Hyperbolic Radius Flexibility Sealing) For point X D hyperbolic space and a scaling matrix X, E R=*", the flexible radius adjustment funetion is formally defined through Eq. (7) and Mobiüs matrix multiplication operations as follows:
|X,X|| X
X Ve
Radx
Radx)
vc
|
2
Ve tanh -1 (VC X, 8.X]] XI)
Radx,00.
(20)
Proof. In hyperbolic space, considering a point X DD2 and a scaling matrix X, E R the detailed expansion of the hyperbolic radius flexibility adjustment function can be formulated as:
X.X
Radx
X
2
|X,X| Ve
XX,X|
tanh t
(tanh(
Radx
Vc
[
2
X
2
tanh(
|X,X|I |X||
Radxx) )
Ve tanh -1
X.X X
XI
2
tanh
-1
(Vo
(anh(
|X
X 2
ve
Radxx
X,X
Vc
Vol.
X
2
X,X
X,X
tanh (Vo
(1/ve) tanh
tanh (Vc|x(l)
Ve
X
X,X
2
tanh
(Veel
X, 8. X(I)
Ve
Radx,8.x
(21)
Consequently, according to Eq. (21), the scaling matrix X, can adjust the hyperbolic radius Radx via the Möbius matrix multiplication operation.

---

C Validating HyperET via Comparison with Hyperbolic Training Methods
To further validate the effectiveness of our approach, we also compare HyperET with representative training methods that utilize hyperbolic space, i.e., MERU [17]. MERU [17] and its variants establish an asymmetric visual-semmn semantcc hierarchy that emphasizes subordinate relationship wherein textual features dominate visual ones, inherently intensifying the granularity gap between text and visual modalites, leading to inferior performance. In contrast, HyperET attempts to align visual and textual modalities at arbitrary granularity levels. As shown in Table 6, our experimental results demonstrate that HyperET achieves superior performance compared to MERU, empirically validating the effectiveness and contribution of HyperET in the context of hyperbolic geometry.
Table 6: Comparison with MERU [17] on 12 MLLM benchmarks, including VQAv2 [21], GQA [25], VW: WisWiZ [22], SQA: ScienceQA-IMe [42], TVQA: TextVQA [53], PE: POPE [37], ME: MME [67], MB: MMBench [41], MB CN MMBeench-Chinese [41], SD: SEED-Bence [34], LVA W; LLaVA -enhhBh (In-the- Wild) [40] and M- Vet [68]. Lan. Model: Language model. Benchmark names
abbreviated to consistent with the main manuscript. "Ours": we here realize the extra learnable
parameters as full matrices, i.e., W.. Vision encoder: CLIP.
Method
Lan. Model VQAv2
Vodc/ GQA VW SQA TVQAJ PE ME MB SD LVA W M-Vet
LLaVA-1.5
Vicuna- | 80.0
63.3
53.6 71.6
61.3
185.9 1531.3 67.7
61.6
35.4
Vicuna- 13B
78.9
61.0
48.0
65.7
60.2
84.7
1456.8
62.3
59.1
56.6
68.4
32.5
LLaVA-1.5+Ours
Vicuna-
82.3
65.7
55.2
73.7
63.9
88.7 1584.7
69.8
65.2
63.4
72.6
38.3
D Validation on Non-Transformer MILMs
To demonstrating our method's ' effectiveness on different architectures, we experiment HyperET on a non- Ton-Trasffrmer architecture [72], i.e., Mamba. As shown in Table 7, method yields consistent gains across all MLLM benchmarks, further demonstrating the robustness and generalizability of our approach.
Table 7: Comparison with Cobra [72] on 5 MLLM benchmarks, including VQAv2 [21], GQA [25], VW: VisWiZ [22]. TVQA: TexIVQA [53]. PE: POPE [37]. Benchmark names are abbreviated to consistent with the main manuscript. "Ours": we here realize the extra learnable parameters full matrices, i.e., W..
Method
VQAv2
GQA
VW
TVQA
PE
Cobra
79.2
63.9
56.2
59.5
87.6
Cobra+Ours
80.9
65.2
56.9
60.6
88.8


Reviews:
[{"review_id": "sd04yUUvvj", "reviewer": "Reviewer_ijbW", "summary": "This paper introduces HyperET, a novel and efficient training paradigm for multi-modal large language models (MLLMs) designed to address the \"granularity mismatch\" between visual and textual modalities. The core idea is to leverage the natural hierarchical structure of hyperbolic space to represent and align features at varying levels of granularity. By dynamically adjusting the hyperbolic radius of visual representations via a parameter-efficient learnable matrix and Möbius multiplication, HyperET enables visual features to align with their textual counterparts at an arbitrary granularity level. The authors conduct comprehensive experiments on both fine-tuning and pre-training scenarios, demonstrating that HyperET consistently improves the performance of state-of-the-art MLLMs (e.g., LLaVA, MemVP) with less than 1% additional parameters, notably reducing object hallucination. The method is well-motivated, theoretically sound, and empirically validated across multiple standard benchmarks.", "questions": "Please see the weaknesses. Does the proposed method's effectiveness extend to MLLMs with non-Transformer architectures, such as those based on Mamba?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n1. Strong and Well-Defined Motivation: The paper effectively identifies a critical and practical problem in MLLM training: the granularity mismatch between vision encoders and language models. The argument that this mismatch leads to inefficiency and performance issues like hallucination is compelling and provides a solid foundation for the work.\n2. Novel and Theoretically-Grounded Method: The proposed solution of using hyperbolic space to model and adjust semantic granularity is highly innovative and elegant. The connection between hyperbolic radius and granularity level is intuitive and provides a quantifiable framework for a previously abstract problem. The method is well-supported by clear mathematical proofs (Theorems 1 & 2), which adds significant credibility.\n3. Comprehensive Experimental Validation: The empirical evaluation is a major strength.\n\n### Weaknesses\n1. Indirect Justification of the Core Assumption: The fundamental assumption—that hyperbolic radius directly corresponds to semantic granularity—is not empirically verified. The paper lacks qualitative or quantitative analysis to visually or otherwise demonstrate that features at different radii actually capture different levels of semantic detail (e.g., from pixels/parts to global concepts).\n2. Insufficient Analysis of Computational Overhead: While the paper emphasizes \"parameter efficiency,\" it does not discuss the computational efficiency in terms of training time or FLOPs. Operations in hyperbolic space are generally more complex than their Euclidean counterparts, and an analysis of the wall-clock time or computational overhead would provide a more complete picture of the method's \"efficiency.\"\n3. Lack of Hyperparameter Details: For example, the choice and sensitivity of the curvature hyperparameter c, which is crucial for the geometry of the hyperbolic space, are not discussed.", "was_revised": true, "final_justification": "Comprehensive experiments across different architectures are crucial to demonstrate the method's generalizability. The current version, lacking this, is not sufficient for a clear accept. Therefore, I will maintain my score of Borderline Accept.", "rebuttal": "**W1: Indirect Justification of the Core Assumption.**\n\n**A1**: Our core assumption that hyperbolic radius corresponds to semantic granularity is supported by both experimental results in prior works [a-c] and new empirical evidence. As shown in the Table R1 below, the hyperbolic radii of the features output from vanilla CLIP's text and image encoders are typically large (7.5 $\\pm$ 0.03 and 8.2 $\\pm$ 0.05, respectively), which are known to excel at capturing image-level semantics, but struggles with finer-grained understanding (e.g., only 6.6\\% mIoU on the PASCAL-Context dataset and 36.4 mAP on base class object detection on the COCO dataset). After optimizing the hyperbolic radii of embeddings (reducing to 6.1 $\\pm$ 0.03 and 7.2 $\\pm$ 0.05 for segmentation and 6.4 $\\pm$ 0.03 and 7.5 $\\pm$ 0.05 for detection), we observe dramatic performance improvements across all benchmarks (e.g., a 12.6\\% mIoU improvement on the PASCAL-Context dataset). These results demonstrate that features at different hyperbolic radii capture different levels of semantic detail. Moreover, the mean hyperbolic radii exhibit a progressive decrease from image-level (global concept, largest) to object-level (intermediate) and finally to pixel-level (smallest), demonstrating an intrinsic granularity hierarchy that adapts to different visual tasks. Another evidence is provided in Table R2 below. In this table, we calculate the hyperbolic radii of feature maps computed by three different visual encoders, i.e., CLIP, DINOv2 and SAM, on the ScienceQA dataset. Both DINOv2 and SAM are known as more fine-grained encoders than CLIP, where SAM is pre-trained on pixel-level tasks, endowing it with the finest granularity level. This granularity difference is precisely reflected in their hyperbolic radius values, where SAM, CLIP and DINOv2 exhibits the smallest (7.0$\\pm$0.01), the largest (8.2$\\pm$0.05) and the intermediate (7.4$\\pm$0.09) hyperbolic radius, respectively. Moreover, DINOv2's broader granularity scope naturally results in greater variation in its hyperbolic radius range, consistent with its more diverse feature extraction capabilities.\n\n| Granularity level | Hyperbolic radius | Dataset | Performance |\n|------------------|------------------|---------|-------------|\n| *Open-Vocabulary Semantic Segmentation* | | | |\n| Image-level | 7.5$\\pm$0.03 /8.2$\\pm$0.05 | PC-459 | 6.6 |\n| **Pixel-level** | 6.1$\\pm$0.03 /7.2$\\pm$0.05 | PC-459 | **19.2** |\n| *Open-Vocabulary Object Detection* | | | |\n| Image-level | 7.5$\\pm$0.03 /8.2$\\pm$0.05 | COCO | 36.4 / 21.6 |\n| **Object-level** | 6.4$\\pm$0.03 /7.5$\\pm$0.05 | COCO | **42.7 / 26.3** |\n\n*Table R1: Comparative performance across different granularity levels. For semantic segmentation: PASCAL-Context dataset (PC-459); for object detection: COCO dataset ($\\text{AP}^{\\text{Base}}_{50} / \\text{AP}^{\\text{Novel}}_{50}$).*\n\n| Visual encoder | Hyperbolic radius |\n|---------------|------------------|\n| CLIP | 8.2$\\pm$0.05 |\n| DINOv2 | 7.4$\\pm$0.09 |\n| SAM | 7.1$\\pm$0.01 |\n\n*Table R2: Comparison of hyperbolic radii across different visual encoders on the ScienceQA dataset.*\n\n**W2: Insufficient Analysis of Computational Overhead.**\n\n**A2**: Since we do not introduce any extra architectures, and the logarithmic and exponential maps can be completed within linear complexity [d], the training/inference overhead is negligible compared with the Euclidean counterpart. --See table below\n\n| Method | Training time (h) | Memory (GB) | FLOPs (T) | Latency (ms) |\n|--------|------------------|-------------|-----------|--------------|\n| LLaVA-1.5-7B | 13.5 | 34.5 | 8.55 | 113.0 |\n| LLaVA-1.5-7B+Ours | 13.6 (+0.1) | 35.2 (+0.7) | 8.61 (+0.06) | 114.1 (+1.1) |\n\n*Table R3: Efficiency comparison with baseline methods.*\n\n**W3: Lack of Hyperparameter Details.**\n\n**A3**:  The curvature hyperparameter c is set to 0.01 in our experiments. As shown in the table R4 below, our method exhibits negligible performance variations when c ranges from 0.001 to 1. This demonstrates our approach is not sensitive to the choice of the curvature hyperparameter c.\n\n| Curvature hyperparameter c | 1 | 0.1 | 0.01 | 0.001 |\n|---------------------------|------|------|------|-------|\n| Average (%) | 93.56 | 93.64 | 93.78 | 93.71 |\n\n*Table R4: Ablation study of curvature hyperparameter c on ScienceQA Test Set.*\n\n**Q1: Does the proposed method's effectiveness extend to MLLMs with non-Transformer architectures, such as those based on Mamba?**\n\n**A4**:  Thank you for your valuable suggestions. We have validated the robustness of our method using the mainstream architecture for MLLMs, i.e., Transformers. Although few recent studies have explored non-Transformer architectures, e.g., Mamba, they are proposed primarily for acceleration purposes, and most of their performance remains comparable to that of Transformer-based models. Since our approach directly adjusts the hyperbolic radius of features without relying on any specific architectural design, it is likely to be compatible with Mamba as well. Unfortunately, the time constraints during rebuttal limit this exploration. We will explore our method on other architectures in future work.\n\n[a] COMPOSITIONAL ENTAILMENT LEARNING FOR HYPERBOLIC VISION-LANGUAGE MODELS. ICLR2025.  \n[b] Hyperbolic Contrastive Learning for Visual Representations beyond Objects. CVPR2023.  \n[c] Rethinking the compositionality of point clouds through regularization in the hyperbolic space. Neurips 2022.  \n[d] Hyperbolic Fine-tuning for Large Language Models. ICML2024."}, {"review_id": "rKBOftM53d", "reviewer": "Reviewer_1nn1", "summary": "This paper proposes an efficient training paradigm called HyperET to address the inefficiency of cross-modal alignment at multi-granularity levels in MLLMs. Existing MLLMs rely on vision encoders like CLIP and SAM, which only align with language at a single granularity level, leading to heavy computational demands. HyperET leverages the properties of hyperbolic space to dynamically adjust the hyperbolic radius of visual representations, enabling alignment between visual and textual modalities at arbitrary granularity levels. Specifically, it employs learnable matrices with Möbius multiplication operations (including diagonal, block-diagonal, and banded matrices), significantly improving the performance of existing MLLMs in both pre-training and fine-tuning tasks with less than 1% additional parameters", "questions": "1.\tThe ablation study shows that hyperbolic space training outperforms Euclidean space with the same number of parameters, but the mechanism behind this superiority is not fully explained. Why is hyperbolic space uniquely effective for granularity alignment, and are there cases where Euclidean space could be more suitable?\n2.\tThe three matrix configurations (diagonal, block-diagonal, banded) are presented as flexible options, but their practical selection criteria are unclear. For specific tasks (e.g., high-resolution image understanding vs. low-level feature matching), how should researchers choose the optimal matrix type, and what is the theoretical basis for this choice?\n3.\tThe paper claims that HyperET achieves significant improvements with less than 1% additional parameters, but it does not compare the computational overhead of hyperbolic space operations with standard Euclidean operations. Could you provide quantitative analysis on the training/inference speed and memory usage of HyperET compared to baseline methods?", "limitations": "no", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength：\n1.\tBy adjusting the hyperbolic radius in hyperbolic space, it directly quantifies and aligns granular differences between vision and text, avoiding training inefficiencies caused by mismatched granularity in traditional methods\n2.\tThree parameter-efficient learnable matrix designs (diagonal, block-diagonal, banded) achieve significant performance gains with less than 1% additional parameters, reducing computational costs\n3.\tBased on hyperbolic geometry and Möbius operations, its effectiveness in granularity alignment is validated through theorems and derivations\n\nWeakness：\n1.\tAll experiments rely on CLIP as the vision encoder , leaving unanswered whether HyperET generalizes to other encoders or how it interacts with non-CLIP-based MLLMs.\n2.\tWhile three matrix configurations (diagonal, block-diagonal, banded) are proposed, the paper provides limited insight into when each is most effective. For example, it does not clarify why banded matrices with larger bandwidths do not consistently outperform simpler designs", "was_revised": true, "final_justification": "Thanks for your response. It resolves most of my concerns, and I will maintain the score.", "rebuttal": "**W1: All experiments rely on CLIP as the vision encoder, leaving unanswered whether HyperET generalizes to other encoders or how it interacts with non-CLIP-based MLLMs.**\n\n**A1**: This is a misunderstanding. We already provided comparisons with non-CLIP-based MLLMs. These results are presented in Tables 4 of the main manuscript, including employing DINOv2 and SAM as the vision encoder. Our HyperET also yields consistent performance improvements using these visual encoders. These results indicate that 1) the granularity mismatch also persists in non-CLIP visual encoders, as the required granularity levels vary among different tasks; 2) the proposed HyperET can explicitly address this issue, effectively unleashing the full potential of these vision encoders.\n\n**W2: While three matrix configurations (diagonal, block-diagonal, banded) are proposed, the paper provides limited insight into when each is most effective. For example, it does not clarify why banded matrices with larger bandwidths do not consistently outperform simpler designs.**\n\n**A2**: In general, the banded configuration is preferable and typically outperforms diagonal and block-diagonal structures. However, for downstream tasks with limited data (e.g., the ScienceQA dataset), where fewer parameters are sufficient for adaptation, the block-diagonal configuration can achieve performance comparable to that of the banded strategy.\n\n**Q1: The ablation study shows that hyperbolic space training outperforms Euclidean space with the same number of parameters, but the mechanism behind this superiority is not fully explained. Why is hyperbolic space uniquely effective for granularity alignment, and are there cases where Euclidean space could be more suitable?**\n\n**A3**: Many prior studies [a,b,c] have demonstrated, both theoretically and empirically, that hyperbolic space is particularly effective for granularity alignment tasks, owing to its hierarchical geometry that inherently preserves multiple granularity levels, which is a fundamental advantage over Euclidean space. Nonetheless, for tasks involving flatter structures or single-level granularity (e.g., transferring CLIP to classification tasks), Euclidean space may still offer competitive performance theoretically.\n\n**Q2: The three matrix configurations (diagonal, block-diagonal, banded) are presented as flexible options, but their practical selection criteria are unclear. For specific tasks (e.g., high-resolution image understanding vs. low-level feature matching), how should researchers choose the optimal matrix type, and what is the theoretical basis for this choice?**\n\n**A4**: The practical criteria for selecting among diagonal, block-diagonal, and banded matrix structures primarily consider task complexity and data scale. (1) Diagonal matrices require the fewest parameters, making them suitable primarily for simple tasks (e.g., edge detection). (2) Block-diagonal matrices, which leverage intra-block parameter modeling, offer an effective balance between parameter efficiency and modeling capacity, making them particularly competitive in low-data scenarios (e.g., fine-tuning MLLMs on the ScienceQA dataset). (3) Banded matrices, which encompass block-diagonal structures and additionally incorporate local connections, generally achieve optimal performance for complex tasks with abundant data (e.g., high-resolution image understanding). In summary, the banded configuration is generally preferable; however, diagonal and block-diagonal structures remain competitive choices for simpler tasks or data-constrained scenarios.\n\n**Q3: The paper claims that HyperET achieves significant improvements with less than 1% additional parameters, but it does not compare the computational overhead of hyperbolic space operations with standard Euclidean operations. Could you provide quantitative analysis on the training/inference speed and memory usage of HyperET compared to baseline methods?**\n\n**A5**: Since we do not introduce any extra architectures, and the logarithmic and exponential maps can be completed within linear complexity [d], the training/inference overhead is negligible. --See table below\n\n| Method               | Training time (h) | Memory (GB) | FLOPs (T) | Latency (ms) |\n|----------------------|------------------|-------------|-----------|--------------|\n| LLaVA-1.5-7B         | 13.5             | 34.5        | 8.55      | 113.0        |\n| LLaVA-1.5-7B+Ours    | 13.6 (+0.1)      | 35.2 (+0.7) | 8.61 (+0.06) | 114.1 (+1.1) |\n\n*Table R1: Efficiency comparison with baseline methods.*\n\n[a] Poincaré Embeddings for Learning Hierarchical Representations. NeurIPS2017.  \n[b] Hyperbolic Neural Networks. NeurIPS2018.  \n[c] Hyperbolic Image Embeddings. CVPR2020.  \n[d] Hyperbolic Fine-tuning for Large Language Models. ICML2024."}, {"review_id": "84FFNlHBwS", "reviewer": "Reviewer_hmUq", "summary": "This paper identifies a \"granularity gap\" between vision and language modalities as a key source of inefficiency in training Multi-modal Large Language Models (MLLMs). It proposes HyperET, a novel training paradigm that uses hyperbolic space to model and align these different levels of granularity. The core idea is to use the \"hyperbolic radius\" as a proxy for visual granularity, which is then dynamically adjusted using learnable, parameter-efficient scaling matrices and Möbius multiplication operations. By adding less than 1% of new parameters, HyperET is shown to consistently improve the performance of existing MLLMs in both fine-tuning and pre-training scenarios across a wide range of benchmarks.", "questions": "1. Can you provide a qualitative analysis to make the concept of \"granularity adjustment\" more concrete? For instance, visualizing attention maps before and after applying HyperET could show if the model’s focus shifts from whole objects to finer details as the hyperbolic radius changes.\n\n2. To better establish the generality of your method, could you more directly compare HyperET's performance boost on a standard CLIP encoder versus an encoder that is already considered more granular, like DINOv2? This would clarify if the benefit is primarily in \"fixing\" CLIP or if it's a more universal alignment tool.\n\n3. The hyperbolic curvature -c is a key parameter in the methodology. Could authors discuss how this was chosen and how sensitive the model's performance is to this choice? This would improve the paper's practical guidance and reproducibility", "limitations": "no", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The paper's framing of MLLM inefficiency as a \"granularity gap\" is insightful, and using hyperbolic geometry as a solution is a principled and elegant approach.\n2. The method is rigorously tested across multiple models and tasks. Crucially, strong ablation studies confirm that the performance gains stem directly from the proposed hyperbolic mechanisms, not just from adding parameters.\n3. HyperET achieves significant performance improvements while adding negligible computational overhead (<1% additional parameters), making it a highly practical and accessible method.\n\n## Weaknesses\n1. The paper does not provide a qualitative analysis of why the model adjusts the hyperbolic radius for different tasks, missing an opportunity to offer deeper intuition behind the \"granularity adjustment\".\n2. The reliance on advanced concepts from hyperbolic geometry adds significant complexity that may not always be justified by the margin of performance improvement.\n3. The theory suggests that granularity is adjusted dynamically per input, a powerful feature that is not explicitly analyzed or discussed in the experimental results.", "was_revised": true, "final_justification": "Based on the rebuttal and the discussion with other reviewers, my concerns have been largely addressed. The clarifications provided were helpful, and as a result, I'm raising my score from 3 to 4.", "rebuttal": "**W1: The paper does not provide a qualitative analysis of why the model adjusts the hyperbolic radius for different tasks, missing an opportunity to offer deeper intuition behind the \"granularity adjustment\".**\n\n**A1**: Different tasks typically require distinct granularity levels (pixel-, object-, or image-level, etc), and vanilla CLIP suffers from granularity mismatch on various tasks that degrades performance. By adaptively adjusting hyperbolic radii of CLIP's feature embeddings, we align model granularity with task requirements. Experiments on open-vocabulary semantic segmentation and object detection demonstrate consistent and significant improvements when matching granularity, e.g., 12.6\\% mIoU improvement on the PASCAL-Context dataset  (see the table below).  \n\n| **Task/Feature granularity** | **Hyperbolic radius** | **Dataset** | **Performance** |\n|-----------------------------|-----------------------|-------------|-----------------|\n| *Open-Vocabulary Semantic Segmentation* | | | |\n| Pixel-level/Image-level | 7.5$\\pm$ 0.03/8.2$\\pm$ 0.05 | PC-459| 6.6 |\n| **Pixel-level/Pixel-level** | 6.1$\\pm$ 0.03/7.2$\\pm$ 0.05 | PC-459 | **19.2** |\n| *Open-Vocabulary Object Detection* | | | |\n| Object-level/Image-level | 7.5$\\pm$ 0.03/8.2$\\pm$ 0.05 | COCO | 36.4 / 21.6 |\n| **Object-level/Object-level** | 6.4$\\pm$ 0.03/7.5$\\pm$ 0.05 | COCO | **42.7 / 26.3** |\n\n*Table R1: Comparative performance across different granularity levels. For semantic segmentation: PASCAL-Context dataset (PC-459); for object detection: COCO dataset ($\\text{AP}^{\\text{Base}}_{50} / \\text{\\AP}^{\\text{Novel}}_{50}$).*\n\n**W2: The reliance on advanced concepts from hyperbolic geometry adds significant complexity that may not always be justified by the margin of performance improvement.**\n\n**A2**: Our approach is not complex: it maps embeddings from Euclidean space into hyperbolic space through a mathematical transformation, adjusts their radii via scaling operations, and subsequently projects them back. Each step has been rigorously validated through ablation studies (see Tables 3 and 5 in the main manuscript), confirming the necessity and effectiveness of this lightweight geometric adaptation.\n\n**W3: The theory suggests that granularity is adjusted dynamically per input, a powerful feature that is not explicitly analyzed or discussed in the experimental results.** \n\n**A3**: Thank you for your suggestion. To demonstrate the capability of our method in dynamically adjusting granularity per input, we present a representative example from the 2nd iteration of the Visual Question Answering (VQAv2) dataset (Image ID: 53141, Question: \"What is in the sky?\"). In the image of this example, a flight is located at the center of the image against a clear sky background. The two below matrices show the hyperbolic radii of the features on the feature map extracted from the last layer of CLIP's image encoder, before and after the adjustment by our method, respectively. It can be observed that, our method adjusts the mean hyperbolic radius from 8.28 to 8.02,  shifting the granularity level from the original image-level to the fine-grained level, with smaller values concentrated on the object of interest. This effectively aligns feature granularity with the VQA task's requirements.   \n\n**Before**\n|       |       |       |       |       |       |       |\n|-------|-------|-------|-------|-------|-------|-------|\n| 8.39  | 8.38  | 8.26  | 8.31  | 8.25  | 8.33  | 8.42  |\n| 8.24  | 8.27  | 8.29  | 8.36  | **8.23**  | **8.21**  | 8.34  |\n| **8.22**  | 8.30  | 8.28  | **8.22**  | 8.32  | 8.37  | 8.31  |\n| 8.35  | **8.21**  | **8.13**  | **8.11**  | **8.05**  | 8.36  | 8.36  |\n| 8.32  | 8.33  | **8.13**  | **8.09**  | **8.19**  | **8.21**  | 8.39  |\n| 8.41  | 8.32  | **8.23**  | 8.32  | 8.25  | **8.24**  | 8.25  |\n| 8.34  | 8.36  | 8.33  | 8.30  | 8.38  | 8.35  | 8.37  |\n\n**After**\n\n|       |       |       |       |       |       |       |\n|-------|-------|-------|-------|-------|-------|-------|\n| 8.02  | 8.19  | 8.06  | 8.11  | 8.07  | 8.12  | 8.02  |\n| 8.13  | 8.07  | 7.99  | 7.95  | 7.99  | 7.95  | 8.11  |\n| 8.07  | 8.00  | 7.99  | 7.89  | 7.92  | 7.98  | 8.14  |\n| 8.05  | 8.01  | 7.86  | **7.81**  | **7.89**  | 8.03  | 8.12  |\n| 8.02  | 8.03  | 7.93  | **7.87**  | 7.89  | 8.01  | 8.09  |\n| 8.21  | 8.02  | 7.99  | 8.02  | 8.05  | 8.03  | 8.05  |\n| 8.17  | 8.12  | 8.03  | 8.00  | 8.08  | 8.05  | 8.02  |\n\n**Q1:** **Can you provide a qualitative analysis to make the concept of \"granularity adjustment\" more concrete? For instance, visualizing attention maps before and after applying HyperET could show if the model's focus shifts from whole objects to finer details as the hyperbolic radius changes.** \n\n**A4:** We use the same example as in W3. The two matrices below represent the attention maps extracted from the last layer of CLIP's image encoder, before and after hyperbolic radius adjustment. These maps clearly demonstrate a shift in focus from coarse object-level regions to finer, more discriminative details. (1) Before adjustment, attention is broadly distributed across regions, e.g., the sky and background, with moderate activation values (e.g., 0.34-0.66) even in irrelevant areas. (2) After adjustment, the model sharply focuses on the central object (a flight), with high attention values (e.g., 0.75-0.91) in key regions, while background areas exhibit minimal activation (mostly < 0.1).\n\n**Before**\n\n|       |       |       |       |       |       |       |\n|-------|-------|-------|-------|-------|-------|-------|\n| 0.12  | 0.14  | 0.16  | 0.15  | 0.17  | 0.21  | 0.06  |\n| 0.25  | 0.27  | 0.21  | 0.03  | 0.25  | **0.31**  | 0.14  |\n| **0.34**  | 0.05  | **0.46**  | **0.35**  | **0.33**  | 0.12  | 0.21  |\n| 0.23  | **0.45**  | **0.56**  | **0.59**  | **0.43**  | 0.24  | 0.15  |\n| 0.12  | 0.24  | **0.66**  | **0.54**  | **0.53**  | **0.34**  | 0.16  |\n| 0.08  | 0.23  | **0.45**  | 0.23  | **0.43**  | 0.23  | **0.34**  |\n| 0.04  | 0.23  | **0.33**  | 0.26  | 0.04  | 0.15  | 0.09  |\n\n**After**\n|       |       |       |       |       |       |       |\n|-------|-------|-------|-------|-------|-------|-------|\n| 0.07  | 0.04  | 0.08  | 0.05  | 0.12  | 0.11  | 0.01  |\n| 0.09  | 0.07  | 0.11  | 0.01  | 0.04  | 0.01  | 0.06  |\n| 0.04  | 0.05  | 0.16  | 0.75  | 0.23  | 0.02  | 0.03  |\n| 0.03  | 0.05  | 0.76  | **0.91**  | **0.83**  | 0.14  | 0.05  |\n| 0.02  | 0.14  | 0.16  | 0.74  | 0.13  | 0.04  | 0.06  |\n| 0.01  | 0.12  | 0.15  | 0.03  | 0.03  | 0.13  | 0.04  |\n| 0.05  | 0.03  | 0.03  | 0.06  | 0.02  | 0.05  | 0.01  |\n\n\n**Q2:** **To better establish the generality of your method, could you more directly compare HyperET's performance boost on a standard CLIP encoder versus an encoder that is already considered more granular, like DINOv2? This would clarify if the benefit is primarily in \"fixing\" CLIP or if it's a more universal alignment tool.**\n\n**A5:** The reviewer may overlook some experimental results in our paper. We already provided such comparisons with non-CLIP-based MLLMs, which are presented in Tables 4 of the main manuscript, including employing DINOv2 and SAM as the vision encoder. In Table 4, our HyperET also yields consistent performance improvements. These results indicate that 1) the granularity mismatch also persists in more granular visual encoders, e.g., DINOv2 and SAM, as the required granularity levels vary among different tasks; 2) HyperET can explicitly address this issue, serving as a more universal alignment strategy to effectively unleash the full potential of these vision encoders. \n\n**Q3:** **The hyperbolic curvature -c is a key parameter in the methodology. Could authors discuss how this was chosen and how sensitive the model's performance is to this choice? This would improve the paper's practical guidance and reproducibility.**\n\n**A6:** The curvature hyperparameter c is set to 0.01 in our experiments. As shown in the table below, our method exhibits negligible performance variations when c ranges from 0.001 to 1. This demonstrates the our approach is not sensitive to the choice of the curvature hyperparameter c.\n\n| Curvature hyperparameter $c$ | 1 | 0.1 | 0.01 | 0.001 | \n|------------------------------|---|---|---|----|\n| Average (%) | 93.56 | 93.64 | 93.78 | 93.71 |\n\n*Table R2: Ablation study of curvature hyperparameter $c$ on ScienceQA Test Set.*"}, {"review_id": "gA0m0y0aV7", "reviewer": "Reviewer_g5HU", "summary": "The paper proposes HyperET, a training paradigm for multi-modal large language models that operates in hyperbolic space to address purported granularity mismatches between visual and textual modalities. The authors claim that existing vision encoders like CLIP and SAM are inefficient because they align with language at single granularity levels, and propose using hyperbolic radius adjustment through Möbius multiplication operations to enable alignment at arbitrary granularity levels. The method demonstrates improvements across MLLM benchmarks with less than 1% additional parameters.", "questions": "- How do you control for the possibility that improvements stem simply from additional learnable parameters rather than the specific hyperbolic operations?\n- How does the method perform when the \"optimal\" granularity level varies within a single input (eg images with both fine-grained and coarse-grained elements)?\n- How sensitive is the method to the choice of hyperbolic curvature parameter c?\n- What happens to the hyperbolic structure during backpropagation, and how do gradient flows behave in the constrained hyperbolic manifold?\n- What is the theoretical capacity of the hyperbolic space compared to Euclidean representations. Could similar benefits be achieved through other geometric approaches?\n- Can you offer a more detailed explanation for the observed radius changes in fig2?\n- The learnable matrices seem to be applied in the attention layer. Is this choice based on empirical testing? How would applying the HyperET module to other components (such as the MLP projector) affect performance?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "pros\n\n- The core idea of leveraging hyperbolic geometry to explicitly model and adjust the semantic granularity seems interesting\n- The method is efficient (with 1% additional parameters), a practical consideration.\n- The paper is well written.\n\ncons\n\n- see the questions below", "was_revised": true, "final_justification": "Most of my questions have been resolved.", "rebuttal": "**Q1: How do you control for the possibility that improvements stem simply from additional learnable parameters rather than the specific hyperbolic operations?**\n\n**A1**: We have presented experimental results to isolate and eliminate the influence of merely increasing the number of trainable parameters. As shown in Table 5 in the main manuscript, fine-tuning MLLMs in Euclidean space with the same number of additional parameters as HyperET yields only marginal gains. In contrast, fine-tuning with HyperET results in substantial performance improvements. These findings clearly illustrate that our main contribution arises specifically from the proposed hyperbolic radius adjustment mechanism, rather than simply from introducing more learnable parameters.\n\n**Q2: How does the method perform when the \"optimal\" granularity level varies within a single input (eg images with both fine-grained and coarse-grained elements)?**\n\n**A2**: There might be a misunderstanding here. The term \"granularity'' refers to an intrinsic property of a task, e.g. semantic segmentation is a more granular task than image classification, rather than elements in a single image. A granularity level can correspond to a range of hyperbolic radii. Thus, we assume the reviewer was asking how our method performs when the ``optimal'' hyperbolic radius varies within a single input. This is straightforward, since our method performs hyperbolic feature learning on the dense feature maps. The features at different locations naturally have varied hyperbolic radii. But, note that the variance of these hyperbolic radii is small (typically 0.01 in our experiments), evidencing that they belong to one hyperbolic radius range, corresponding to one granularity level.\n\n**Q3: How sensitive is the method to the choice of hyperbolic curvature parameter c?**\n\n**A3**: The curvature hyperparameter c is set to 0.01 in our experiments, but as shown in the table below, our method exhibits only minor performance variations when c ranges from 0.001 to 1. This demonstrates the stability of our approach across different curvature settings.\n\n| Curvature hyperparameter c | 1    | 0.1  | 0.01 | 0.001 |\n|----------------------------|------|------|------|-------|\n| Average (%)                | 93.56| 93.64| 93.78| 93.71 |\n\n*Table R1: Ablation study of curvature hyperparameter $c$ on ScienceQA Test Set.*\n\n**Q4: What happens to the hyperbolic structure during backpropagation, and how do gradient flows behave in the constrained hyperbolic manifold?**\n\n**A4**: Our method preserves the intrinsic hyperbolic geometry by learning to adjust only the radii of points within hyperbolic space. Gradient flows $\\nabla_{\\mathbb{H}}$ follow standard Euclidean backpropagation $\\nabla_{\\mathbb{E}}$, with an additional radius-aware scaling:\n$\\nabla_{\\mathbb{H}} = (1-\\tanh^2(r)) \\cdot \\nabla_{\\mathbb{E}}$, where $r$ is the radius of a point. This technique guarantees stability near the boundary of the Poincaré ball, ensuring updated points remain within the manifold. Furthermore, it is compatible with standard optimizers, e.g., Adam.\n\n**Q5: What is the theoretical capacity of the hyperbolic space compared to Euclidean representations. Could similar benefits be achieved through other geometric approaches?**\n\n**A5**: Hyperbolic space preserves a larger theoretical capacity than Euclidean space due to its intrinsic exponential volume growth, where the circumference scales as $C(r) \\propto e^{r}$ compared to the linear $C(r) \\propto r$ scaling in Euclidean space, where $r$ is radius. Other geometric approaches, e.g., kernel-based methods, lack clear geometric interpretability and often require manual kernel design. Hyperspherical space offers geometric interpretability but is hindered by intrinsic limitations, particularly its saturated volume growth, which restricts hierarchical granularity representation.\n\n**Q6: Can you offer a more detailed explanation for the observed radius changes in fig2?**\n\n**A6**: Yes. Figure 2 illustrates the variation of representation norms (hyperbolic radii) among different tasks. More concretely, the VQAv2 dataset is a general visual question answering benchmark that typically requires representations with larger norms, whereas the GQA dataset emphasizes detailed spatial reasoning tasks, leading to representations characterized by comparatively smaller norms.\n\n**Q7: The learnable matrices seem to be applied in the attention layer. Is this choice based on empirical testing? How would applying the HyperET module to other components (such as the MLP projector) affect performance?**\n\n**A7**: The placement of learnable matrices within the attention layers is consistent with mainstream parameter-efficient methods (e.g., LoRA) and is empirically validated, as shown in the table below. The observed performance gains may result from a hypothesis that interactions in the attention layers more effectively influence the adjustment of the hyperbolic radius.\n\n| Module               | Average |\n|----------------------|---------|\n| Attention layer (Ours) | **93.78** |\n| MLP Projector        | 93.11   |\n\n*Table R2: Ablation study of placement of learnable matrices on ScienceQA Test Set.*"}]

Author rebuttal:

Meta-review: This paper provides a solution to the granularity mismatch problem between visual and text modalities in MLLMs by leveraging the natural structure of hyperbolic space to align features at various granularities, when training multimodal LLMs. Arbitrary granularity levels can be used by dynamically adjusting the hyperbolic radius using a learnable matrix with Möbius multiplication operations. The authors demonstrate strong performance benefits by conducting comprehensive experiments on both pre-training and fine-tuning scenarios, and particularly reduce object hallucination. 

Strengths
- All reviewers find the granularity mismatch problem is well motivated, and the hyperbolic space solution proposed in the paper to be novel, and elegantly grounded in theory, and well supported by clear mathematical proofs.
- All reviewers also find the empirical evaluation setup to be very strong.

Weaknesses 
- While many weaknesses have been stated, the authors have addressed all concerns via the rebuttal process and all reviewers have found the answers to be satisfactory. Some of these are 1) Empirical validation that the hyperbolic radius corresponds to different granularities of semantic features, 2) stating efficiency in terms of FLOPs, 3) Demonstrating improvement on MAMBA architectures and not just transformer models, 4) ablations on types of matrix configurations, 5) dynamic choice of granularity per input.

Overall, the paper has received accept scores from all reviewers. The paper is a strong new idea, that is well motivated, works well compared with very strong baselines, is theoretically grounded and is also highly interpretable in its mechanics. Recommend accepting this paper.
Decision: Accept (oral)