Modality Alignment across Trees on Heterogeneous Hyperbolic Manifolds
Authors: Wu Wei
Xiaomeng Fan
Yuwei Wu
Zhi Gao
Pengxiang Li
Yunde Jia
Mehrtash Harandi
Venue: ICLR 2026 Poster (2026)
Keywords: hyperbolic geometry
modality alignment
multimodal learning
Primary area: learning on graphs and other geometries & topologies

Abstract: Modality alignment is critical for vision-language models (VLMs) to effectively integrate information across modalities. However, existing methods extract hierarchical features from text while representing each image with a single feature, leading to asymmetric and suboptimal alignment. To address this, we propose Alignment across Trees, a method that constructs and aligns tree-like hierarchical features for both image and text modalities. Specifically, we introduce a semantic-aware visual feature extraction framework that applies a cross-attention mechanism to visual class tokens from intermediate Transformer layers, guided by textual cues to extract visual features with coarse-to-fine semantics. We then embed the feature trees of the two modalities into hyperbolic manifolds with distinct curvatures to effectively model their hierarchical structures. To align across the heterogeneous hyperbolic manifolds with different curvatures, we formulate a KL distance measure between distributions on heterogeneous manifolds, and learn an intermediate manifold for manifold alignment by minimizing the distance. We prove the existence and uniqueness of the optimal intermediate manifold. Experiments on taxonomic open-set classification tasks across multiple image datasets demonstrate that our method consistently outperforms strong baselines under few-shot and cross-domain settings.

Published as a conference paper at ICLR 2026
MODALITY ALIGNMENT ACROSS TREES ON HETERO- GENEOUS HYPERBOLIC MANIFOLDS
Wei Wu 1.2- Xiiaomeng Fan Yuwei Wu! Zhi Gao 1,2 Pengxiang Li
Yunde Jia Mehrtash Harandì
Beijing Key Laboratory of Intelligent Information Technology. School of Computer Science & Technology, Beijing Institute of Technology
2 Guangdong Laboratory Machine Perception and Intelligent Computing, Shenzhen MSU-BIT University
Department of Electrical and Computer System Engineering, Monash University
https: weriisssammmaiffoo -elddiinenniin qithub. io/ ht.Hyyyodaalligg
ABSTRACT
Modality alignment is critical for vision-language models (VLMs) to effectively integrate information across modalities. However, existing methods extract hier- archical features from text while representing each image with single feature, leading to asymmetric and suboptimal alignment. To address this, propose Alignment across Trees, a method that constructs and aligns tree-like hierarchi- cal features for both image and text modalities. Specifically, semantic-aware visual feature extraction framework that applies a cross-attention mechanism to visual class tokens from intermediate Transformer layers, guided by textual cues to extract visual features with coarse-to-fine semantics. We then embed the feature trees the two modalities into hyperbolic manifolds with dis- tinct curvatures to effectively model their hierarchical structures. To align across the heterogeneous hyperbolic manifolds with different curvatures, we formulate a KL distance measure between distributions on heterogeneous manifolds, and learn an intermediate manifold for manifold aligmment by minimizing the distance. We prove the existence and uniqueness of the optimal intermediate manifold. Exper- iments on taxonomic open-set classification tasks across multiple image datasets demonstrate that our method consistently outperforms strong baselines under few- shot and cross-domain settings.
introduce
INTRODUCTION
In vision-language models (VLMs), modality alignment aims to bridge the modality gap and en- able the effective integration of information across different modalities (Liang et al. 2022] Zhang let al.) 2025). In real-world scenarios, multimodal semantics are inherently hierarchical (Dhiloon
et al. 2002; Stevens et al.) 2024). For example, in biology, the semantics an organism follow a taxonomy kingdom, phylum, class, order, family, genus, and species. To align such hierarchical semantics across modalities, existing methods extract hierarchical features from textual labels while representing images using only a single feature (Khattak al., 2023b Li et 2024). single visual feature is inherently asymmetric to hicrarchical textual features, namely, it fails to capture the complete textual information, thereby causing suboptimal aligmment, as shown in Figure I
In this paper, we propose Alignment across Trees, a method that constructs hierarchical features from both images and texts, and aligns such tree-like features to enhance modality alignment. In- spired by the findings that intermediate Transformer layers encode coarse information (Chen et al., 2024), we exploit the class tokens from intermediate layers to construct hierarchical visual features. To this end, we need to overcome two challenges: (1) extracting hierarchical visual features that carry coarse-to-fime semantic information. (2) Textual features are relatively pure, whereas visual features encode more complex and diverse information such background (Pal et al. 2025), result- ing in distinct geometric structures that reside on heterogeneous manifolds. Alignment across such heterogeneous manifolds remains underexplored and challenging.
Eqqual contribution. Corresppnniigg authors.

---

Published as a conference paper at ICLR 2026
Avo
llngle ribual
Thenal
caode:
Rucoratioome
Acsscs
scymmutiiec alypnue Factldran
inferior profkatin
Impertads
Merrachica
hierarcical
Textral
Arce
Burectifomes
Arzzon
Eitt
syniaciiiic digssnos, on byperbelic cosiilid
v
Arims
sptinal prodiction
Figure 1: Comparison between previous methods and our method. Previous methods extract single visual feature to align with hierarchical textual features in Euclidean spaces. This asymmetric aligmment leads to inferior prediction. In contrast, our method achieves symmetric alignment by extracting hierarchical visual features on hyperbolic manifolds, leading to optimal prediction.
To address the first challenge, propose semantic-aware framework that leverages textual cues to construct hierarchical visual features with coarse-to-fine semantics. Intermediate-laye tokens are projected into the final layer to enhance discriminative power. Next, introduce cross-atts cross-attention module in which textual features at each hierarchy serve as queries and class tokens from different layers keys and values produce hierarchy-specific visual features. The obtained visual features, paired with their textual counterparts, form symmetric eextual-visure feature trees.
To address the second challenge, introduce a heterogeneous manifold aligmment algorithm. Hy- perbolic manifolds, with their negative curvature, naturally model data hierarchies (Nickel & Kicla, 2017 Gao al. 2022a). To capture geometric differences between textual and visual modalities, we embed each modality into an individual hyperbolic manifold with learnable curvature, allowing the feature trees to adapt to their intrinsic geometry. To align the two trees on heterogeneous hy- perbolic manifolds, we explore intermediate manifold close to both textual and visual manifolds. Specifically, we model data on each hyperbolic manifold as wrapped normal distribution (Nagano) al. 2019 Gao al., 2022b) and develop the KL divergence to measure manifold distance. By minimizing the derived distance between the intermediate manifold and textual and visual mani- folds, obtain the optimal intermediate manifold and prove its existence and uniqueness. We then use hyperbolic cones on the intermediate manifold to align cross- moddl tree features while imposing geometric constraints the visual and textual trees. Curvatures optimized efficiently using the implicit function theorem (Lorraine et al. 2020) for curvature gradients.
We evaluate our approach the taxonomic open-set (TOS) classification task (Wu al. 2024) using four image datasets under four settings: few-shot, base-to-base, base-to-novel, and base-to- whole. We use three evaluation metrics (Wu al., 2024): Leaf Accuracy (LA), Hierarchical Con- sistent Accuracy (HCA), and Mean Treecut Accuracy (MTA). Our method consistently surpasses all baselines across datasets and settings, showing clear advantages modality alignment. Specif- ically, for HCA, our method improves by up to 7.72% (1-shot) and 28.83% (16-shot), highlighting its effectiveness. Moreover, visualizations confirm that the extracted image-feature trees capture hi- erarchical semantics, while ablation studies validate the contribution of the heterogeneous manifold alignment algorithm. Our contributions can be summarized as:
We propose Alignment across Trees, a method that constructs hierarchical features from both images and texts, and aligns such tree-like features, thereby enhancing modality alignment.
We introduce a semantic- visual feature extraction framework, which builds hierarchical visual features with coarse-to-fine semantics.
We introduce a heterogeneous manifold alignment algorithm, which embeds image and text fea- ture trees into hyperbolic manifolds with distinct curvatures and aligns them via an intermediate manifold optimized through manifold distance minimization.
2

---

Published as a conference paper at ICLR 2026
2
RELATED WORKS
2.1 MODALITY ALIGNMENT
Existing modality alignment methods be categorized into pre-training and prompt learning. Pre- training methods, such as CLIP (Radford et al. 2021) and ALIGN (Jia et al., 202I). train vision- language models (VLMs) on large-scer scale image- text pairs to bridge the modality gap. Recent efforts focus on scaling datasets (Gadre et al. 2024; Schuhmann et al. 2022) or improving training strate- gies (Li et al. 2023) [Snn al.] 2023). Prompt learning introduces learnable prompt tokens the input for modality alignment with significantly fewer computational resources than pre-training (Gu al. 2023). Representative works include CoOp for continuous prompt optimization in the lan- guage branch (Zhou et al. 2022b), CoCoOp for conditional prompts based on visual features (Zhou al. (022a), and VPT for optimizing visual prompt tokens (Jia al.)
2022); subsequent studies 2025 distribution estima- explore multi- -modal prompt fusion (Khattak et al. [023a Zhang et al.
Khattak et al., 2023b Park et at 2024). Some works use relative representations to preserve relational geometry, enabling com- munication between embedding spaces without enforcing a shared space. For example, Moschella| ct al. (2023) demonstrate that pairwise relational structure enables information transfer across em- bedding spaces, and Cannistraci et al.] (2024) extend this principle through wwwriance-preseevving transforms. Different from existing methods that model data Euclidean space and use latent spaces, our method utilizes hyperbolic manifolds for modeling and constructs an intermediate man- ifold for communication.
tion (Fan et al. 2025a), and regularization techniques (Zhu et al., 2023
Taxonomy classification predicts labels at multiple hierarchy levels. Several works (Goo et al,, 2016 Kim & Frahm 2018) utilize CNN-based networks to learn hierarchical feature space. These methods rely on predefined hierarchical structures and fixed sets of categories for classification. To solve this issue. some works construct multi-modal alignment method for taxonomy classi- fication. Wu et al. (2024) first extract a single visual feature and compute contrastive losses with multi-level textual features using prompt learning, introducing metrics for hierarchical consistency. BioCLIP (Stevens et al., 2024). BioCLIP2 (Gu et al. (025), and Biotrove (Yang et al. (024) form prompts from coarse-to-tine annotations for pretraining. |Sastry et al. (2025) apply transitive entail- ment constraints to features of hierarchical textual labels. Moreover, some works implicitly try to learn an identifiable hierarchical structure. For example, Kivva et al. (2022) study identifiability in deep generative models, and Kong et al. (024) explore latent hierarchies by introducing structured discrete variables that capture hierarchical dependencies. However, these methods align hierarchi- cal textual features with single visual representation (i.e., asymmetric alignment) the Euclidean spaces, leading to suboptimal alignment and ignoring the geometric structure of hierarchical multi- modal data. In contrast, our method extracts hierarchical visual features align hierarchical textual features (i.e., symmetric aligmment), and we model them in hyperbolic manifolds for improved align- ment. The reason we choose hyperbolic manifolds is that they offer a more effective way to capture hierarchical geometric structures, as their volume grows exponentially with the radius, aligning with the exponential increase data size along the depth of hierarchical structures (Fan et al. 2025b).
2.2 LEARNING ON HYPERBOLIC MANIFOLDS
Modeling via hyperbolic manifolds has shown superior performance in many tasks due their pabilities in encoding data with hierarchical structures. Hyperbolic neural networks (Guo et al.) 2022
Shimizu et al. 1 He et al.) 2025b; Malik et al. 2025 Skopek et al.) 2020 Gao et al. Yu et al. 2025) incorporate several hyperbolic operations on top of a neural network to ob- tain hyperbolic embeddings. Recently, hyperbolic neural networks have been applied to diverse modalities such as graphs (Fu et al.] [023) 2024, Malik et al.] 2025), text (He et al.) 2025a), im- ages (Wagg et al. 2024b Franco et al. 2024; Li al.] 2025b Gao et al. 2023 Li et al. 2025a), videos (Long et al., 2020; Hong et al. (023a). and audio (Hong et al. (023b). Moreover, recent work (Ramasinghe et al.) 2024: Desai et al. 2023 Pal et al. 2025 Wang et al. 2024a) has focused on developing multimodal methods on hyperbolic manifolds by combining entailment learning with CLIP to learn embeddings in hyperbolic manifolds. Mandica et al. (2024) explore hyperbolic em- beddings in VLMs with billions of parameters. Existing methods assume that the curvatures of visual and textual modalities are the same, which cannot precisely model the geometric structures of each modality, further hindering the effectiveness. In contrast, we model visual and textual modal-
2021
3

---

Published as a conference paper at ICLR 2026
ities manifolds with different curvatures for precisely capturing their geometries, and we design a heterogeneous manifold alignment algorithm for better alignment.
PRELIMINARIES
Hyperbolic manifold. Unlike Euclidean spaces with zero curvature, a hyperbolic manifold is smooth Riemannian manifold with constant negative curvature -c (c 0) LLe 2006). We choose the Lorentz. model L (Canmon et al. 1997) for hyperbolic manifolds due to its com- putational efficiency and numerical stability. The Lorentz model is defined as L { Ra+1 x)c - The hyperbolic vector can written as z = [pspcee Ttimel, where Trime R corresponds the hyperboloid's axis of symmetry and &space R" represents the remaining spatial coordinates.
( denotes the Lorentzian inner product that is computed as (x. c (Fepace Uspace) Ztime Utsme. where (--)) the Euclidean inner product. The induced Lorentzian norm is |aolcc- = (xx cl. The following hyperbolic operations are used in our work.
Distance. The Lorentzian distance between æ, y E L is de y) = vI/c arccosh (-c(æ, y) c).
Tangent space. The tangent space L at tangent point æ, denoted as Ta consists of all langent vectors at that tangent point. Any vector in ambient space u E g=+11 can be projected to the tangent space C via proja (u)
ca u.
Exponential map. The exponential map expm_. (v) projects v from Ta C to L as
expm (v) cosh (Ve||/] a sinh (Vc|lollc) v.
Vellullc
Logarithmic map. The logarithmic map logm projects vector æ from L to Ty Cy as
logm (croosss(c(y,,))))) V(ely, proj,(xx)
Wrapped normal distributions. The wrapped normal distribution (Nagano et al. 2019) on _ is defined as
de(2,, u)
No- (a: u. 6)
(
exp
28
(3)
where u the Fréchet mean, normalization constant.
dispersion parameter, and Z(d) dispersion dependent
Hyperbolic entailment cones. Hyperbolic entailment cones w(x) are regions for every possible point in the hyperbolic manifold, such that all points w(x) are semantically linked to æ as its child concepts. As such, points w(x) are expected to contain specific information for the general concept æ. The cone is defined by half-aperture
(v/)
2k
w(x) = sin -1
(elliaapocce
(4)
where h = 0.1 is constant.
4 METHOD
We propose the method, Alignment across Trees, which contains a semantic-aware visual feature extraction framework to construct textual and visual feature trees and a heterogeneous manifold alignment algorithm to align the feature trees (See Figure 2)
4.1 SEMANTIC-AWARE VISUAL FEATURE EXTRACTION FRAMEWORK
Existing vision- languael models (e.g., CLIP and BLIP) and prompt learning methods (e.g., CoOp) typically align tokens from the last layer of the Vision Transformers (ViT) with the textual modality. Recent studies show that intermediate layers ViT encode coarse semantics, while the final layer encodes fine-grained information (Chen et al. 2024). Motivated by this observation, leverage
4

---

Published as a conference paper at ICLR 2026
Aves
Ducerotifrmes
Testual encoder
Cress- attvation
Visaal crender
dbd visual Teatures
toxtual features
Sentatti-AAvar Vasual
Acerus
Expenential
Heterggneeus manifold search
textual for textual entallment constraints
manifold Og for visual entaliment constraints
alignment
Intermediate manifold with for cross-modall alignment
Figure 2: Pipeline of our method.
class tokens from m intermediate layers {h, } and the final layer (denoted by h,,) to construct hierarchical visual features, illustrated Figure 3 We enhance the discriminative power of intermediate-layee tokens by mapping them to the final layer's representation space. Specifically, for hp.. from the P, layer onward, we disable cross-token self-attention by removing query and key computations that each token no longer attends to others. h2, is then forwarded to the final layer through only linear projection, residual connections, and MLP updates, preserving its original information for alignment. The mapped tokens are denoted by hip (hn doesn't 't need to be mapped).
To construct hierarchical visual features that align with hierarchical textual features (4: ( where И denotes the depth of the textual hierarchy, we design a cross-attention mechanism where textual features serve as queries and class tokens from different layers ((h 1m and h,,) serve as keys and ( values. The hierarchical visual features [vi; ; vn) are computed as:
QK
[M1: 12l mn) Softmax
Vain-
Vd
(5)
where Q [aa;...;:: ta We. K [ i.......: WK, Van [hppi-..; h.....::.JJW
Wo. WK. Wr E Rd dxd are learnable parameters. this way, we can model the hierarchical text feature tree T. [ and hierarchical image feature tree Ve {vr 1 with the symmetric semantic information.
4.2 HETEROGENBOLE MANIFOLD ALIGNMENT ALGORITHM
The hyperbolic manifolds are well-suited for mod- eling hierarchical features. Given the geometric dif- ferences between textual and visual feature trees, we embed them in separate hyperbolic manifolds with distinct, learnable curvatures C1 (text) and C2 (im- age). Formally,
ldandton
6,12
cocodal
Henrchica testral
Testeal mender
expm (t+), 0i expm (v.))
(6)
vinal saater
Avez
The curvatures and C2 are data-driven and treated as trainable parameters optimized together with the model via the loss functions. To align the image features and text features located in different hyper- bolic manifolds, the heterogeneous manifold align- ment algorithm constructs an intermediate hyper- bolic manifold, shown in Figure
Beccstifonnca
cluo teken pash wakiu
Figure Structure of semantic- aware visual feature extraction framework.
5

---

Published as a conference paper at ICLR 2026
4.2.1 INTERMEDIATE MANIFOLD CONSTRUCTION
To minimize geometric distortion and preserve the original structures, we introduce an intermediate manifold _ that bridges the textual and visual manifolds. Directly measuring the dissimilarity between hyperbolic manifolds is underexplored. We define distance function Dc(:,) quan- tify how dissimilar two hyperbolic manifolds are. Using this distance, the optimal curvature cj is obtained by optimizing the following objective,
arg min Je(cs)
We model the distributions on hyperbolic manifolds by wrapped normal distributions and use KL divergence between the distributions to define the manifold distance Dr(:.). Since the KL diver- gence on hyperbolic manifolds does not have an analytic expression (Cho et al. (023), we present approximate expression for the KL divergence and utilize it to define the manifold distance, as shown in T'eeorm
Theorem
Given two manifolds n and _ the distributions on the two manifolds are
1 Le (w.01)
1
Lg. (0.
Par, = Nc(x u1. 5)
262 Peeg,u Ne( u3 6)
262
Z(6)
Z(5)
where un and u3 are the Fréchel means, and >> is dispersion parameter. We define the distance between L and L as affine transformation of the Kullback-Leibler (KI.) divergence, which is
Pe(Cr,c% -V +
+ 2 cosh|l -v
2/
where is a constant.
Note that Dz(t, -) is not formal metric due to the asymmetry and violation of the triangle inequality inherited from the KL divergence. We present the minimizer Dr(C", in Proposition to show the soundness of D)(:,))
Proposition 1 The minimum of De(Ce, ((S) in Eq. is uniquely attained c3 C1-
Proposition I shows that in Theorem I _ and _ are optimally aligned when ca c₁ showing the soundness of Theorem
Based on the derived distance, Je(cs) in Eq. is formulated as [
Je(ca) = D2 (((((,))_)))))) + De(c")
-Ve + 253s cosh[([/s verr + -Va 2/23 coshi((/s Volr
(10)
2/ercs
2/020s
We demonstrate the existence and uniqueness of the minimizer of Jc(ss) by Proposition
Proposition 2 Ja(ca) has unique minimizer cg [miiner, o2). c2
Proofs of Theorem Proposition I and Proposition 2 are provided in Appendix A We apply the golden section search Kiefer 1953) to find the minimizer of Je(C3), which can effectively solve the one-dimensional unconstrained optimization problem.
Eystou crcal
U1
4.2.2 INTER-MODAL GEOMETRIC ALIGNMENT MECHANISM
After obtaining the curvature ca, we utilize the exponential map to project the textual and visual features to E
expmo (t+),
vf expm (v:).
(11)
Following Desai et al. 2023 we utilize the entailment to achieve inter-modal geometric alignment. Pal et al. (2025)
+ merrs Eyphosi fa
shows that the text generally provides a broader context than images. Thus, as to each hierarchy, we force the visual feature 0 to be entailed in the textual feature 1. Le., 8. located
4 da
Figure 4: Illustration of entailment.

---

Published as a conference paper at ICLR 2026
in w (t, that is the entailment cone of t, The loss function forces of8h be in w(t,), which is modeled as
Jent ( ) max(0, o(o," t.:) w(t))))
(12)
where (0, (. is the exterior angle,
% time
+ 4 time
( = T /0t, 0 cos
(13)
||
(cs ))
space
Overall, for all hierarchies, the loss function for cross-modal alignment is modeled as
H
Jent CV (
Σ (e .4,)
(14)
where = fff
H Ni=1 and T = {t 1H 11=1. The process illustrated Figure
4.3 IN-MODAL GEOMETRIC STRUCTURE CONSTRAINTS
Since text and image features hierarchical, we impose in-modal geometric structure constraints on each modality, requiring fine- rreeggrandd features to be entailed by coarse coarse-grained ones. We use the hyperbolic cones _ and _ model the textual entailment and visual entailment, respectively. For the i-th level (coarser) and the + 1-th level (finer) that are adjacent, we push t into cone w(tt)) and 1,41 into cone w ((x").. The losses on visual and textual modalities are formulated as 22
Jent (u 1411, v.+) = max(0, ( 0.0) -(x
(15)
Jent (1 ( = max(0, < ( ( w(('))
For all hicrarchies, the loss functions of hierarchical constraints visual and textual modalities
#-1
H-1
Jvene
Σ
Jena (uff1
IT = Σ
(t t()).
(16)
i=1
This process is also illustrated in Figure
4.4 OPTIMIZATION STRATEGY
We employ CLIP to extract the hierarchical textual and visual features Te and Ve. Following the prompt learning paradigm, introduce learnable tokens 0 to train our model. The overall loss function is formulated as
J(0, C1, 62) = Jpo Vo) + a (Te) Jvene + Jon (V's, T
(17)
s.t. Cg = arg min Jc(cs) De(C ( + )
where Spro is the loss in Wu 2024 and a weighting factor. We optimize the parameters using gradient descent,
d. c 4
d.J 02 c2 n
dJ de²
0 - n d0'
"
(18)
dei
where 17 denotes the learning rate, and the gradients with respect to and Co computed
d.J aJ OJ dcg. dJ aJ OJ dcg
(19)
dc1 dc
Ocg dc1²
dez
dcz
dcg do2
dej However, the terms dc, and dez non-differentiable nature of the golden section search. To address this challenge, we apply the
dej cannot be computed through standard backpropagation due to the
implicit function theorem to compute these derivatives,
0gg
8JJa 0eg
-1
J
O
8J.e Ocg
JJ. dezdcs
(20)
dc,
dct Ocs
doz
ca=

---

Published as a conference paper at ICLR 2026
5 EXPERIMENTS
5.1 SETTINGS
Taxonomic Open Set (TOS) Classification. TOS classification (Wu al., 2024 organizes labels as semantic tree, requiring the classifier to predict across multiple levels of semantics.
Datasets. We evaluate all methods on four datasets: Cifar100 (Krizhevsky et al. 2009). SUN (Xiao et al. 2010), ImageNet (Deng et al. 2009), and Rare Species (Stevens et al. 2024).
Tasks. We evaluate our method on two types of tasks. (1) Few-shot. We adopt a 1-shot and 16-shot training setting, where a fixed number of samples are randomly selected from each class. (2) Base- bebaas/has-toonove///seetete
generalization. We equally split each dataset into base and novel classes. The model is trained on base classes and evaluated on base classes (base-to-base), novel classes (base-to-novel), and whole classes (base-to-whole) across all 4 datasets.
Metrics. Following Wu et al. 2024), we evaluate performance using three metrics: Leaf Accuracy (LA), Hierarchical Consistent Accuracy (HCA), and Mean Treecut Accuracy (MTA).
Implementation Details. Following Wu al. (2024), conduct experiments based the prompt tuning method MaPLe (Khattak et al. 2023a). We also conduct experiments on the Prompt- SRC (Khattak et al., 2023b) method to further validate the effectiveness of our method. More details of datasets, tasks, metrics, and implementation are provided Appendix B
5.2
FEW-SHOT SETTING RESULTS
Table 1: TOS classification results on the 1-shot and 16-shot settings. We bold the best results.
K-
Base Method
Cifar100
SUN
ImageNet
Rare Species
Variant
Shot
LA
HCA
MTA
LA
HCA
MTA
LA
HCA
MTA
LA
HCA
MTA
Vanilla
68.75
4.65
50.60
63.98
25
50.31
68.91
2.97
48.16
41.55
5.09
44.75
MaPLe
Pro'eeCt
69.33
48.10
83.36
64,29
50.45
76.73
66.16
20.44
85.18
39.92
13.22
70.04
+Ours
71.37
53.19
85.29
67.57
57.92
80.55
66.33
25.56
85,98
46.77
20.94
76.83
Vanilla
72.48
14.36
70.58
42.14
57.19
82
4.46
54.10
6.72
44.72
PromptSRC
ProTeeCt
73.07
49.54
85.16
70.61
55.52
78.73
68.43
21.58
85.63
44,56
20.36
74.42
+Ours
73.54
51.91
85.76
70.64
57.79
79.94
68.86
25.13
86.45
46.98
23.03
77.32
Vanilla
75.01
17.54
52.21
71.86
33.25
54.29
70.70
4.15
48.16
50.94
5.30
40.41
MaPLe
+ProTeCt
75.34
61.15
88.04
72.17
59.71
82.27
69.52
31.24
87.87
14
24.82
78.79
+Ours
77.92
69.38
90.89
75.47
68.67
86.02
71.41
43.79
88.78
69.96
53.65
87.27
Vanilla
77.71
15.07
56.86
75.75
45.23
59.42
71.50
2.48
46.71
59,20
11.64
55.82
PRo +ProTeCt
78.76
66.74
75.54
66.01
32.89
88.31
33.92
82.47
+Ours
78.90)
68.47
91.12
76.54
69.18
86.20
71.67
42.26
89.64
67.38
50.77
87.60
We compare our method with MaPLe, PromptSRC. MaPLe + ProTeCt, and PromptSRC: + ProTeeC methods in Table [T] Results show that our method significantly improves the performance of the compared methods in both I-shot and 16-shot settings, demonstrating its effectiveness in modality alignment. Notably, in the 16-shot setting, our method achieves up to 19.02% improvement on LA, a 28.83% improvement on HCA, and a 8.48% improvement on MTA, indicating its ability to align the textual and visual modalities with the hierarchical semantic structures. See Appendix C.I for comparisons between our methods and other hyperbolic alignment methods.
WHOLE GENER CONERALIZATION
Sableepreseme the comparison between our method and existing methods. Our method consistently outperforms all baselines across all metrics and settings, demonstrating strong generalization to novel classes. On Cifar our method achieves improvements of 1.38%, 5.66%, and 4.90% in LA, HCA, and MTA, respectively, on novel classes, highlighting the effectiveness of our method.
5.4 ABLATION STUDY
Variants of our method. To evaluate the effectiveness of each component, we design three vari- ants: (i) Ours-Euc, which does not employ hyperbolic constraints; (ii) Ours-HypV1, where all
8

---

Published as a conference paper at ICLR 2026
Table 2:
generalization results across multiple datasets.
See Appendix C.2 for more results.
Dataset
Base Method
Variant
LA
HCA
MTA
Base
Novel
HM
Whole
Base
Novel
HM
Whole
Buse
Novel
HM
Whole
Vanilla
80.77
76.85
78.76
69.45
38.51
37.62
38.06
33.31
65.23
61.61
63.37
55.82
MaPLe
ProTTeCt
81.77
76.67
79.14
55.43
53.74
85,30
83.15
76.37
SUN
+Ours
82.79
77.11
79.85
69.82
73.38
56.23
63.67
$7.09
88.85
81.24
84.87
78.61
Vanilla
82.30
78.68
80.58
71.50
51.77
48.25
49.94
47.05
68.89
63.67
67.24
58.93
PromptSRC
+ProTecC
82.36
80.33
86.67
82.76
84.67
79.12
+Ours
83.40
78.72
80.99
72.04
73.11
59.10
65.36
58.42
89.02
82.86
85.83
79.74
Vanilla
82.60
75.80
79.05
71.45
9.94
6.94
8.17
7,00
61.46
50.66
55.54
54.14
MaPLe
+ProTeCt
82.66
74.56
78.40
70.03
35.86
45.40
89.65
77.15
82.93
77.34
Cifar100
+Ours
82.76
75.94
79.20
72.52
66.42
41.52
51.10
45.79
91.01
82.05
86.30
81.84
Vanilla
85.43
80,28
82.77
74.86
14.06
14.74
14.39
11.95
63.56
55. 60
59.31
55.41
PromptSRC
+ProTeCt
85.36
78.72
81.91
73.82
64.76 76
40.02
42.36
91.19
79.38
84.88
80.53
+Ours
85.58
80.28
82.85
74.89
67.66
40.12
50.37
42.87
91.74
79.82
85.37
81.72
Table 3: Ablation results on C00, SUN, and Rare Species using MaPLe under different k-shot Additional ablation results and analyses are provided Appendix C
settings.
Cifar100
SUN
Rure Species
K-Shot
Variant
LA
ICA
MTA
LA
HCA
MTA
LA
HCA
MTA
+ProTeCt
69.33
48.10
83.36
64.29
50.45
76.73
39.92
13.22
70.04
+Ours-Euc
69.79
49.77
84.54
67.19
56.25
79.80
46.56
20.28
74.25
Ours-HypVV
69.80
51.86
85.17
66.78
57.56
80.22
45.51
20.86
76.62
+Ours-HypV2
70.98
51.67
85.23
67.17
57.87
80.44
45.81
20.82
76.54
+Ours
71.37
53.19
85.29
67.57
57.92
80.55
46.77
20.94
76.83
+Pro'leCt
75.34
61.15
88.04
72.17
59.71
82.27
48.14
24.82
78.79
+Ours-Eue
76.99
68.01
90.55
74.07
66.81
85.36
68.96
51.81
87.15
+Ours-HypVI
77.62
69.05
90.82
75.10
68.26
85,99
67.41
52.85
87.18
+Ours-HypV2
77.69
69.33
90.71
75.19
68.65
85,92
69.67
52.73
87.01
+Ours
77.92
69.38
90.89
75.47
68.67
86.02
69.96
53.65
87.27
modalities share a common learnable curvature; and (iii) Ours-HypV2 in which each modality is assigned an independently learnable curvature. We conduct ablation in the few-shot setting, with results summarized Table
Effectiveness of Semantie-Aware Visual Feature Extraction Framework. Ours-Euc consistently outperforms ProTeCt. This gain results from our framework's ability to extract coarse-to-fin vi- sual features and construct symmetric feature trees, facilitating symmetric alignment and enhancing performance.
Effectiveness of Alignment on Hyperbolic Manifolds. The comparison of Ours-Eue with the hyperbolic variants (Ours-HypV1, Ours-HypV2, and Ours) demonstrates that incorporating our hy- perbolic alignment constraint helps better preserve hierarchical relationships.
Effectiveness of Intermediate Manifold Search. The comparisons among the hyperbolic vari- ants highlight the importance of intermediate manifold search. By aligning visual and textual fea- tures through an optimized intermediate manifold, our approach achieves more effective hierarchical alignment across modalities.
Efficiency Analysis. We further measure the computational overhead of our method and that of learning a single shared curvature. The time cost for the single-curvature method and our method is 74 S and 74.5 per batch, and the memory cost is 10,400 MB and 10,400 MB, respectively. The results demonstrate that the additional time and memory overhead introduced by learning multiple curvatures is negligible. More details are provided in Appendix CC16
5.5 VISUALIZATION
Visualization of learned representations. Figure TigureBsshore shows that our semantic-aware visual feature extraction framework produces more separable hierarchical representations across all taxonomic

---

Published as a conference paper at ICLR 2026
Phylum-loyeklt
Ch
Onlesleved Bi)
Family-ooootli
Spocielloee(B
Phylum-lovcc(O
Clavllveiit
OnlevievdtOy
Family-lovetO)
Gena--evekk
Spocie-love(O0
Figure 5: T-SNE visualization of learned image representations from the baseline ProTeCt (B) and our method (O), colored by laxonomic labels. Our method demonstrates improved feature separa- bility across taxonomic categories. Additional quantitative results are provided in Appendix C.7
levels compared ProTeCt. The clearer inter-class boundaries and compact intra-class distributions demonstrate the effectiveness of our method in extracting coarse-to-fin visual features that align with the hierarchical text structure, validating our tree-based alignment approach.
Visualization of attention maps. We use GradCAM (Selvaraju et al.) 2017) visualize the attention maps generated by model to analyze its behavior across different taxonomic levels. As shown in Figure when aligned with text prompts at different granularities, our model attends to distinct visual regions for the same image. For instance, when distinguishing at the class level (e.g., mammal), the model focuses on features like fur, while the genus level (e.g., ailuropoda), it shifts attention to facial characteristics. This confirms that our semantic-aware feature extraction framework adaptively captures the most relevant visual cues for each taxonomic level, generating appropriate hierarchi- cal features for alignment. See Appendices C.I3 and for more details.
Origiial
Figure 6: Visualization of attention maps of our model and the baseline ProTeeC across taxonomic levels.
CONCLUSION
In this work, we have presented an Alignment across Trees method address asymmetric modal- ity misalignment in lanionnl models. The proposed method consists of semantic-aware visual feature extraction framework and a heterogeneous manifold alignment algorithm. The frame- work leverages class tokens from intermediate Transformer layers and text-guided cross-attention module to produce visual features with coarse-to-fine semantics. The tree-like visual and textual fea- are then embedded into distinct hyperbolic manifolds with various curvatures. The proposed heterogeneous manifold alignment algorithm constructs an intermediate manifold by formulating and minimizing the manifold distance, and aligns textual and visual features on this intermediate manifold. Extensive experiments on taxonomic open-set classification tasks demonstrate that our method extracts symmetric cross-modal features, captures and aligns their geometric structures on hyperbolic manifolds, leading to consistent improvements various datasets and settings.
Limitations and future work. Considering the complexity the underlying geometric structures, single-curvature hyperbolic manifold is inherently limited in capturing such rich semantics. More- over, our method does not explicitly account for the fact that hierarchical data real-world scenarios often exhibit domain shifts. In future work, plan to leverage spaces to separately model visual and textual features, and we plan to develop a robust method that can handle cross-domain variations, improving the generalization in the open world.
expressive mixed-curvatuee
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the National Natural Science Foundation of China (NSFC; Grant No. 62406009), the Natural Science Foundation of Shenzhen (Grant No. (CYI2022380011220000000 the Shenzhen Science and Technology Program (Grant No. [CYJ202412022 10024058666)) the Key Research Platforms and Projects of the Guangdong Provincial Department Education (Grant No. 2023ZDZX1034) and the Opening Project of the State Key Laboratory of General Artificial Intelli- gence, BIGAI/Peking University, Beijing, China (Grant No. SKLAGI20250P07).
REPRODUCIBILITY STATEMENT.
To ensure the reproducibility of our work, we have made significant efforts to provide comprehen- sive implementation details and resources. We provide detailed training procedures and hyperpa- rameters required to reproduce our experiments in Appendix B Complete proofs for all theoretical contributions, including clear statements of assumptions and mathematical derivations, are provided in Appendix A For the Rare Species dataset, which has not been previously used for TOS classifi- cation tasks, provide data preprocessing steps and adaptation procedures in Appendix [B.] For our newly proposed base-to-base, base-to-novel, and base-to-whole evaluation settings, we provide detailed descriptions of the base/novel tree partitioning methodology and experimental configura- tions in Appendix [B.]] We are committed to open-sourcing our code and trained models to facilitate future research. We believe these materials will enable researchers reproduce our findings and further advance our work.
REFERENCES
Irene Cannistraci, Luca Moschella, Marco Fumero, Valentino Maiorea, and Emanuele Rodolà. From bricks bridges: Product of invariances to enhance latent space communication. In International Conference on Learning Representations, 2024.
James W Cannon, William J Floyd, Richard Kenyon, Walter Parry, et al. Hyperbolic geometry. Flavors of geometry, 31(59-115)22, 1997.
Haozhe Chen, Junfeng Yang, Carl Vondrick, and Chengzhi Mao. Invite: Interpret and control vision- language models with text explanations. In The Twelfth International Conference on Learning Representations, 2024.
Seunghyuk Cho, Juyong Lee, and Dongwoo Kim. Hyperbolic vae via latent gaussian distributions. Advances in Neural Information Processing Systems, 36:569-588 2023.
Yean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. In International Conference on Learning Representations, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 248-255, IEEE, 2009.
Karan Desai, Maximilian Nickel, Tanmay Rajpurohit, Justin Johnson, and Shanmukha Ramakr- ishna Vedantam. Hyperbolic image-text representations. In International Conference on Machine Learning, pp. 7694-7731. PMLR, 2023.
Inderjit S Dhillon, Subramanyam Mallela, and Rahul Kumar. Enhanced word clustering for hierar- chical text classification. Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 191-200, 2002.
Xiaomeng Fan, Yuchuan Mao, Zhi Gao, Yuwei Wu, Jin Chen, and Yunde Jia. Beyond the seen: Bounded distribution estimation for open vocaaulcyyylary learning. arXiv preprint arXiv:2510.0470) 2025a.
11

---

Published as a conference paper at ICLR 2026
Xiaomeng Fan, Yuwei Wu, Zhi Gao. Mehrtash Harandi, and Yunde Jia. Curvature learning for generalization of hyperbolic neural networks: X. fan et al. International Journal of Computer Vision, pp. 1-37, 2025b.
Christiane Fellbaum. Wordnet. WordNet Electronic Lexical Database, pp. 69, 1998.
Luca Franco, Paolo Mandica, Konstantinos Kallidromitis, Devin Guillory, Yu-Teng Li, Trevor Dar- rell, and Fabio Galasso. Hyperbolic active learning for semantic segmentation under domain shift. In International Conference on Machine Learning. PMLR, 2024.
Xingcheng Fu, Yuecen Wei, Qingyun Sun, Haonan Yuan, Jia Wu, Hao Peng, and Jianxin Li. Hy- perbolic geometric graph representation learning for hierarchy-imbalarre node classification. In Proceedings of the ACM Web Conference 2023, pp. 460-4644400 468, 2023.
Xingcheng Fu, Yisen Gao, Yuecen Wei, Qingyun Sun, Hao Peng, Jianxin Li, and Xianxian Li. Hyperbolic geometric latent diffusion model for graph generation. In International Conference Machine Learning (ICML), 2024.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances Neural Information Processing Systems, 36, 2024.
Zhi Gao, Yuwei Wu, Yunde Jia, and Mehrtash Harandi. Curvature generation in curved spaces for few-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8691-8700, 2021.
Zhi Gao, Yuwei Wu, Mehrtash Harandi, and Yunde Jia. Curvature-adaptive meta-learning for fast adaptation to manifold data. IEEE Transactions on Pattern Analysis and Machine Intelligence, 4552):154--1562, 2022a.
Zhi Gao, Yuwei Wu, Yunde Jia, and Mehrtash Harandi. Hyperbolic feature augmentation via distri- bution estimation and infinite sampling on manifolds. Advances in neural information processing systems, 35:34421-34435. 2022b.
Zhi Gao, Chen Xu, Feng Li, Yunde Jia, Mehrtash Harandi, and Yuwei Wu. Exploring data geometry for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24325-24334, 2023.
Wonjoon Goo, Juyong Kim, Gunhee Kim, and Sung Ju Hwang. rrxuuliey semantic deep convolutional neural networks. In European Conference on Computer Vision, pp. 86-101. Springer, 2016.
Jianyang Gu, Samuel Stevens, Elizabeth G Campolongo, Matthew Thompson, Net Zhang, Jiaman Wu, Andrei Kopanev, Zheda Mai, Alexander E White, James Balhoff, al. Bioclip 2: Emer- gent properties from scaling hicrarchical contrastive learning. Advances in Neural Information Processing Systems, 2025.
Jindong Gu. Zhen Han, Shuo Chen, Ahmad Beirami, Bailan He, Gengyuan Zhang, Ruotong Liao, Yao Qin, Volker Tresp, and Philip Torr. A systematic survey of prompt engineering on vision- language foundation models. arXiv preprint 1.XXv:2307./29800 2023.
Yunhui Guo, Xudong Wang. Yubei Chen, and Stella X Yu. Clipped hyperbolic classifiers are super- hyperbolic classifiers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11 11-20, 2022.
Neil He, Rishabh Anand, Hiren Madhu, Ali Maatouk, Smita Krishnaswamy, Leandros Tassiulas, Menglin Yang. and Rex Ying. Helm: Hyperbolic large language models via mixture-of-currature experts. arXiv preprint ar 2505v 24722, 2025a.
Neil He, Menglin Yang, and Rex Ying. Lorentzian residual neural networks. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining 1, pp. 436 447,477 2025b.
12

---

Published as a conference paper at ICLR 2026
Jie Hong, Pengfei Fang, Weihao Li, Junlin Han, Lars Petersson, and Mehrtash Harandi. Curved geometric networks for visual anomaly recognition. IEEE Transactions on Neural Networks and Learning Systems, 2023a.
Jie Hong, Zeeshan Hayder. Junlin Han, Pengfei Fang, Mehrtash Harandi, and Lars Petersson. Hyper- bolic audio-visual zero-shot learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7873-7883, 2023b.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hicu Pham, Quoc Lc, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International conference on machine learning, pp. 4904-4916. PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-moda prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113-19122, 2023a.
Muhammad Uzair Khattak, Syed Talal Wasim, Muzammal Naseer, Salman Khan, Ming-Hsura Yang, and Fahad Shahbaz Khan. Self-regulating prompts: Foundational model adaptation without forgetting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 15190- 15190-15200, 2023b.
Jack Kiefer. Sequential minimax search for a maximum. Proceedings of the American mathematical society, 403):502-506, 1953.
Hyo Jin Kim and Jan-Michael Frahm. Hierarchy of alternating specialists for scene recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 451 467, 2018.
Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35:15687-15701, 2022.
Lingjing Kong, Guangyi Chen, Biwei Huang, Eric Xing, Yuejie Chi, and Kun Zhang. Learning dis- crete concepts in latent hierarchical models. Advances in Neural Information Processing Systems, 2024.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 and cifar- 100 datasets. cifar siflr 2009.
https:
// www. cs. pooronto.edd/ooa..
John M Lee. Riemannian manifolds: an introduction to curvature, volume 176. Springer Science & Business Media, 2006.
Pengxiang Li, Yuwei Wu, Zhi Gao, Xiaomeng Fan, Wei Wu, Zhipeng Lu, Yunde Jia, and Mehrtash Harandi. Geometry-awe distance measure for diverse hierarchical structures in hyperbolic spaces. arXiv preprint arXiv:2506. 18533, 2025a.
Wenrui Li, Zhe Yang, Wei Han, Hengyu Man, Xingtao Wang, and Xiaopeng Fan. Hyperbolic- constraint point cloud reconstruction from single rgb-d images. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 39, pp. 4959-4967, 2025b.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Patlern Recognition, pp. 23390-23400, 2023.
Zheng Li. Xiang Li. Xinyi Fu, Xin Zhang, Weigiang Wang, Shuo Chen, and Jian Yang. Promptkd: Unsupervised prompt distillation for vision- language models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26617-26626, 2024.
13

---

Published as a conference paper at ICLR 2026
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. Ad- vances in Neural Information Processing Systems, 35:17612-17622 2022.
Teng Long, Pascal Mettes, Heng Tao Shen, and Cees GM Snoek. Searching for actions on the hyper- bole. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1141-1150, 2020.
Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International Conference on Artificial Intelligence and Statistics, pp- 1540-1552. PMLR, 2020.
Nikita Malik, Rahul Gupta, and Sandeep Kumar. Hyperdefender: A robust framework for hyperbolic gnns. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 19396- 19404, 2025.
Paolo Mandica, Luca Franco, Konstantinos Kallidromitis, Suzanne Petryk, and Fabio Galasso. Hy- perbolic learning with multimodal large language models. European Conference on Computer Vision, pp. 382-398. Springer, 2024.
Luca Moschella, Valentino Maiorea, Marco Fumero, Antonio Norelli, Francesco Locatello, and Emanuele Rodolà. Relative representations enable zero-shot latent space communication. In International Conference on Learning Representations, 2023.
Yoshihiro Nagano, Shoichiro Yamaguchi, Yasuhiro Fujita, and Masanori Koyama. A wrapped nor- mal distribution on hyperbolic space for gradient- based learning. In International conference on machine learning, pp. 4693 4702. PMLR, 2019.
Maximillian Nickel and Douwe Kicla. Poincaré embeddings for learning hierarchical representa- tions. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017.
Avik Pal, Max van Spengler, Guido Maria D' D'Amely di Melendugno, Alessandro Flaborea, Fabio Galasso, and Pascal Mettes. Compositional entailment learning for hyperbolic vision-language models. In The Thirteenth International Conference on Learning Representations, 2025.
Jinyoung Park, Juyeon Ko, and Hyunwoo J Kim. Prompt learning via meta-regularrizaton. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26940- 26950, 2024.
Namuk Park and Songkuk Kim. How do vision transformers work? In International Conference on Learning Representations, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Sameera Ramasinghe, Violetta Shevchenko, Gil Avraham, and Ajanthan Thalaiyasingam. Accept the modality gap: An exploration in the hyperbolic space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27263-27272, 2024.
Srikumar Sastry, Aayush Dhakal, Eric Xing, Subash Khanal, and Nathan Jacobs. Global and local entailment learning for natural world imagery. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15770-15780 2025.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25288 2022.
14

---

Published as a conference paper at ICLR 2026
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based local- ization. In Proceedings of the IEEE international conference computer vision, pp. 618-626 2017.
Ryohei Shimizu, YUSUKE Mukuta, and Tatsuya Harada. Hyperbolic neural networks++. In Inter- national Conference on Learning Representations, 2021.
Ondrej Skopek, Octavian- Ganea, and Gary Bécigneul. Mixed-curvature variational autoen- coders. In International Conference on Learning Representations, 2020. 2020..
Samuel Stevens, Jiaman Wu, Matthew J Thompson, Elizabeth G Campolongo. Chan Hee Song, David Edward Carlyn, Li Dong, Wasila Dahdul, Charles Stewart, Tanya Berger-War Wolf, et al. Bioclip: A vision foundation model for the tree of life. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 19412-19424, 2024.
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389, 2023.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEEE, 2015.
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. Transformer dissection: unified understanding of transformer's 's attention via the lens of kernel. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing (EMNLP), 2019.
Yiping Wang. Yifang Chen, Wendan Yan, Alex Fang, Wenjing Zhou, Kevin Jamicson, and Simon S Du. Cliploss and norm-based data selection methods for multimodal contrastive learning. Ad- vances in Neural Information Processing Systems, 37:15028 15069, 2024a.
Yuan Wang, Yali Li, and Shengjin Wang. G'3-Iq:
Marrying hyperbolic alignment with explicit remantic-geomest modeling for 3d visual grounding. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 13917-13926, June 2024b.
Xin Wen, Bingchen Zhao, Yilun Chen, Jiangmiao Pang, and Xiaojuan Qi. What makes clip more robust to long-tailed pre-training data? a controlled study for transferable insights. Advances in Neural Information Processing Systems, 57:36567-36601. 2024.
Tz- King Wu, Chih-Hui Ho, and Nuno Vasconcelos. Protect:
Prompt tuning for taxonomic open classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16531-16544 2024.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3485-3492 2010.
Chih-Hsuan Yang, Benjamin Feuer, Talukder Jubery, Zi Deng, Andre Nakkab, Md Zahid Hasan, Shivani Chiranjeevi, Kelly Marshall, Nirmal Baishnab, Asheesh Singh, al. Biotrove: A large curated image dataset enabling ai for biodiversity. Advances in Neural Information Processing Systems, 17:102101-102122 2024.
Peilin Yu, Yuwei Wu, Zhi Gao, Xiaomeng Fan, Shuo Yang, and Yunde Jia. Hyperbolic dual feature augmentation for open-environment, arXiv preprint arXiv:2506 08906, 2025.
Fei Zhang, Tianfei Zhou, Jiangchao Yao, Ya Zhang, Ivor W Tsang. and Yanfeng Wang. Decouple before align: Visual disentanglement enhances prompt tuning. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- Langual models. In Proceedings of the IEEE/CVF conference on computer vision and paftern recognition, pp. 1681 168255,, 2022a.
15

---

Published as a conference paper at ICLR 2026
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9))2337-2344 2022b.
Beier Zhu, Yulei Niu, Yucheng Han, Yue Wu, and Hanwang Zhang. Prompt-aligme gradient for prompt tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15659-15669, 2023.
A PROOFS
A.1 PROOF OF THEOREM 1
We present the proof of theorem follows. We first restate the main setting for clarity. Let La and _ be two manifolds. For clarity, 2 can be understood as the curvature of a manifold representing the distribution of textual and visual features, while f corresponds to the curvature of an intermediate manifold. The feature distributions on the two manifolds are given by
d2 (x, 2b)
Peruu = No U1, 0) = 2(8) exp
28
(21)
d2 (x, Ut3
Per,ug Ne(x uz, Z(8)
exp
282
where C1, represent the negative curvatures of the two hyperbolic manifolds, and u1, Us 8 are parameters of the Gaussian distributions. We assume the parameters of _ and Per, (i.e, Ci u1 and 8) are constant. The distance on hyperbolic manifolds can be computed as
(x u) arccosh" u)
(22)
For simplicity, we approximate the distance function in Eq. (2 using Taylor expansion around the point y.
1 arccosh (-rcoss?2-e((,,.. c)
arccosh? (v) + [arccosh?]] (v) (- c(c ui y)
(23)
arccosh? (y) 2 2arccosh(y)y
V
2 arrcosh (v)
(xr, u)
V
We define the distance between _ and _ based on the Kullback-Leibler (KL) divergence:
1 Ep [d (x, un)] + Ep [d (z, us)]
KL (Poy,ui Pey.us)=
28²
282
(24)
R((r, 8) + 28 del de(Lor L
where R(c1, 8) 25" Ep [db (x, u1)] is constant and de (L'1, (2)) Ep [de (x, us)]
Thus, de(L9, (es) affine transformation of the Kullhack-Leibler (KI.) divergence.
16

---

Published as a conference paper at ICLR 2026
We observe that dg((") _ is a function of cz. and we approximate it using Taylor expansions (as shown in Eq. (23
de dc 2
v1)
arccosh (y1) 2 arcrnss((112g
Vu
arccosh (1/1)
Ep
(x,, uz)
c3
arccosh² (v1)
2arccosh(2n In
v vi
2 arccosh (311)
+ Ep
(xx, us)
Vii -
(25)
arccosh2 (B1) 2 ar Iu v
arccosh (11) (- (EE [2c], us)
C3
Vii -
arccosh2 (v1)
2 arccosh((pp (y Jun
Vvi
2 arccosh (11) (- (m,, u3)
C3
V -1
Since in and uz are mapped from midpoints in the tangent space (refer Eq. (T for the exponential map, where we use the tangent space at the origin), we can further expand the Lorentzian inner product:
(u1, us) c = Mitimme Migime
U Ispace: M.ssucee?
+ 1
(Mpsce 143spacee
C1
C3
1 + sinhl Veill
2
sinh (
/03l 0
sinh(Ve1
[ sinh /c3 (0)
Vallol
sinh (nnn"(/Villlll))
sinh ( [|] +1 sinh(Veilla sinh( (c33 b|))
C
C3
Vcic3
V (sinh (Val| + (ssinn?
b11) sinh( sinh ( a|]
cosh V cosh / /
sinh / sinh
cosh ((Vei Vc3) |+])
Vcccs
(26)
where ||o||| is the Euclidean norm of the midpoint of features in the tangent space, which depends on the choice of er and C3. We define r = ||²|| as constant value.
Thus, we have
de(Ca, (*) de(Cs), L v1)
A(y1) + B(y1)
cosh Vca)r)
(27)
C3
VC103
where A(y1) and B(y1) are computed as
A( = arccooh?(y)) aarccosh(v)) B(11)
arccosh(y1)
(28)
Vei -1
v
Notice that (33;))))))))) y1) is function of Cz. which we denote as f(03 V1, (r, de(L%, (1)
(29)
17

---

Published as a conference paper at ICLR 2026
Next, we focus selecting the Taylor expansion point /1- To ensure that dz(C") 2 th, (1,7) is good approximation the distance function, we need to find a yf that satisfies (f (1, = 0, i.e,
d
A(yi)
B(yi)
f(e3: c1.) de3
G7 2c
1
[Avi)) + B(vt)
(30)
c7
This equation has numerical solution v sa 3.016 and the corresponding B(vi) > 0
Thus, we have
;n)
[llz;yi,,,,
(31)
B(vi) -vei + 2 2/3ccosh va)r
2 2/003
Let (e(L,,00) -V V Varm then we have
2/0res
De(C (0)) de(Ca, ( (y))
KL (Par,us Pes.us) R(C1, u1
(32)
28 26 B(y
which indicates that our D2( L is an affine transformation of the Kullback-Leibler (KL) divergence and the coefficient B8*)(y]) is positive.
The proposed distance derived from the KL divergence, which does not satisfy symmetry or the triangle inequality. Consequently, the proposed distance does not satisfy these properties either. In the future, we consider leveraging JS divergence derive the formal distance metric between manifolds.
A.2 PROOF OF PROPOSITION 1
We just need to prove that (c c1,7) is monotonically decreasing on (0, ej and monotonically increasing on [c1, oo).
To analyze the monotonicity of f(cs: vi r), we examine its second derivative:
0² de3
+3 cosh ir(va, voi)
3cgr sinh [r(ves - (1) 4
(33)
The denominator is always positive for > 0. Defining the numerator as:
N(ca)) = (3 22))) cosh [r(ves- /ci)
3cgr sinh [r(va- - voi) 4//1,
(34)
require (c3) 0 for all C3 2 Cmin O. where cmin is positive lower bound for curvatures C12 C2 Cz.
POSITIVITY ANALYSIS OF (c3)
Let 03 - va and L = /ci- - VCmin > 0. For large r:
Case 1: c3 ci (d > The dominant term is 2
(3 + 12cx) - 3ear] erd Its coefficient positive since:
+3 30yr = 3/33 + 2 3/2 3car
Vr > O, C3 > 0.
(35)
The minimum occurs at C3 = ci:
((a)) = (1 -1) 2
when r 2
(36)
v ei
18

---

Published as a conference paper at ICLR 2026
Case 2: Cmin I C3 C1 (d 0)
The dominant term is [v + r + +cgr]] e rd with -d > 0. Its coefficient always positive. At Cg Ciinn
[(mmi)) VCmin
Cmin) coshoorro m0mi sinh(r AVc
(37)
For rL 2, have cosh(rL) 2 e²/, /3 and sinh(r 2 e /3, leading to:
/ N (Cmin) 2 3 Cmin (3 + 2² cmin) + 3Cminr] 6" - 4
(38)
3/2 22 e 4/01. 2 min
(39)
This is positive when:
12
3/2 c 4/e1 rL> In mina
(40)
3/2 I?2 "min
SUFFICIENT CONDITION FOR r
Define L = /ei - VCmin > O. For r > Tmin with:
1 2
12 ci
Tmin max
In
(41)
Voi L'
Cauin I
3/2
we have N(cs)) > 0 for all c3 2 Cmin- Thus:
/e3: (1r) >0 Veg 0ey
Cmin-
(42)
MONOTONICITY CONCLUSION
The function f is strictly convex for C3 2 Cmin- Combined with the first derivative analysis:
0f
At C3 = Ci+ Brg
For 63 < C1 (C3 2 Cmin);
01 Beg
For 03 > C1. 8es
81
Thus f(es; yT c1,7)
is monotonically decreasing on [Cmin: c1] and monotonically increasing on (c1, Thus, Proposition holds.
A.3 PROOF OF PROPOSITION
62, without loss of generality assume Ci < C2- When C₁ = C2. proposition holds trivially. For Ci C2, We prove the existence and uniqueness of the minimizer G3 E [cr, 02] by analyzing the first derivative of LD(c)):
dJe C
D
(43)
dcg
2/22
2/
where C = 00 Veg coshl((/s [o car Vorr and D Verrr.
Vai
/C3 cosh (eah(((/33-) - Vei rr] sirhsin(((
Define M 002 Va > 0. At the endpoint = cT:
dLn
00 /e cosh cosh(Mr) cir sinh(Mr)
de3
2
For r > max
Vog M . the numerator is strictly negative. At C3 = ez:
3
4
dLp
c
v 02 coss((M))) + car sinh(Mr)
de3 ley=02
2/7ich
19

---

Published as a conference paper at ICLR 2026
which is strictly positive when Veg
By continuity of the derivative and the intermediate value theorem, there exists cg E (c1, 62) where the derivative vanishes.
To establish uniqueness, extend the curvature bound from appendix radius threshold:
A.2
Define the enhanced
1
12 12
T min max
Tmin:
V02 Vo2
Vmmin
Mmin
In
3/2
M min
(44)
Cuin
where Mmin 02 Cmin- For 2 max
mina
M
4 the second derivative
yeu
3
d'Lp deg
throughout (e1, c2] (proof methodology identical to appendix a unique minimizer ej E [c1.
A.2
This strict convexity guarantees
Thus proposition holld for all cases.
B EXPERIMENTAL SETTINGS
B.1 DATASETS
Our experiments conducted four datasets: Cifar100 (Krizhevsky et al., 2009), SUN (Xiao jet al., 2010). ImageNet (Deng et al., (009). and Rare Species (Stevens et al., (024). Cifar is a dataset containing 100 classes of images, each with 600 samples, designed for fine-grained classi- fication tasks. SUN is a scene recognition dataset that includes 397 scene categories, representing wide variety of indoor and outdoor environments. ImageNet is large-scale dataset with millions of labeled images across 1,000 categories, primarily used for large-scale image classification tasks. Rare Species, on the other hand, focuses on rare species classification and provides hierarchical annotations spanning multiple taxonomic levels, including kingdom, phylum, class, order, family, genus, and species.
It is worth noting that Cofar100, SUN, and ImageNet do not natively include hierarchical labels. However, these datasets were extended into hierarchical versions by WW (2024) using generic public taxonomy (e.g.. WordNet (Fellbaum) 1998)) or a specialized taxonomy related to the appli- cation, such as scientific taxonomies. In contrast, Rare Species offers a more rigorous hicrarchical taxonomy. Each sample is annotated at multiple levels, from kingdom to species, with well-defined and consistent hierarchical relationships. Unlike the other datasets, where hierarchical labels are inferred or constructed from external sources, Rare Species ensures that every leaf node is at the same depth, and each sample has a corresponding label at every level. This consistency makes Rare Species particularly suited for TOS classification.
To prepare the Rare Species dataset for TOS classification, we construct a hierarchical tree struc- ture based the taxonomic annotations. The Rare Species dataset provides annotations at seven taxonomic levels for each image: kingdom, phylum, class, order, family, genus, and species. A key challenge in constructing this hierarchy is that identical species names may refer to different organ- isms across different taxonomic lineages. Specifically, there are 28 out of 400 species names that refer to multiple organisms. To address this ambiguity, we create unique identifiers by concatenat- ing all seven taxonomic levels, ensuring each leaf node represents distinct biological entity. This also aligns with the BIOCLIP's text style during pretraining. Additionally, since all samples in the Rare Species dataset belong to the animal kingdom (Animalia), we skip the grouping process at the kingdom level during tree construction. The complete preprocessing procedure detailed in Algo- rithm] I We also consider alternative approach where we use the common name accompanied by the species name to handle ambiguity at the species level, as discussed in Appendix C.11
B.2 METRICS
We evaluate TOS classification performance using three metrics proposed by Wu al. (004 Leaf Accuracy (LA), Hierarchical Consistent Accuracy (HCA), and Mean Treecut Accuracy (MTA). We include here for completeness the definitions introduced in Wu et al. (2024).
Problem Formulation. A class taxonomy Vrax organizes classes into a tree where classes of sim- ilar semanties recursively assembled into superclasses each graph node. For image æ.,
20

---

Published as a conference paper at ICLR 2026
Algorithm Hierarchical Tree Construction for Rare Species Dataset
Require: Dataset D with taxonomic annotations for each image
Ensure: Hierarchical tree structure
1: Initialize empty tree T and empty dictionary nodes
2: Step 1: Create unique identifiers for leaf nodes
3: for each image i in D do
4: Extract taxonomic labels: {ki. cs. 0s, Ji, Si,
8,)
5: Create unique identifier: uid; +- concat(k;, ki, Ps, Ci. Ot: fs: Ia,
6: Create leaf node with uid, and add to nodes [udd;]
7: end for
8: Step 2: Recursively build internal nodes do
9: for level from
10: {From genus to phylum (skip kingdom)
Group nodes by taxonomic prefix level &
12:
for each unique prefix prefix, at level & do children + all nodes with matching prefix, Create parent node with identifier prefix; Connect parent to all nodes children 16: Add parent node to nodes/prefix,)
13:
14:
15:
17:
end for
18:
end for
19: Return Tree structure T with hierarchical taxonomy
classifier predicts label given the label y and model parameters 0:
i( a v.
arg maxp(ty)
x;; V,
(45)
Kyy
Leaf Accuracy (LA). It is defined as
N 11 (a,;; Meat)) byd..
LA
(46)
where by, denotes the leaf node corresponding to groundtruth label yi of the i-th image It mea- sures the classification accuracy at the leaves the taxonomic tree. This enables comparison of hierarchical classifiers to standard, or flat, classifiers which only consider the leaf classes.
Hierarchical Consistent Accuracy (HCA). It is defined as
1 A N
HCA
Lij(z;; Σ
t..]
II 1[0(x;)h) A(ty.) (tu Hw.ll mEA((
(47)
where A(n) denotes all the ancestors node n. While LA considers successful any correct clas- sification the leaf level of the tree, the HCA is stricter. declares a success only when all the ancestors of the leaf node are correctly classified. In other words, each sample needs to be classified correctly at each tree level to be viewed as correctly classified under the HCA. LA is an upper bound for the HCA.
Mean Treecut Accuracy (MTA). estimates the expected accuracy under the TOS classification settings. It computes the average accuracy over set of treecuts To E Ω:
N
MTA
1 [0(0;
VT.)
(48)
Ω
T.ER
Note that the number treecuts of a tree is very large. It is impossible to evaluate on all treecuts of given tree. Following Wu et al. (2024), randomly sample [Ω] 25 treecuts from T in all experiments. The treecuts are generated once and used in the evaluation all methods, thus ensuring fairness.
21

---

Published as a conference paper at ICLR 2026
B.3 TRAINING PROCEDURES
Algorithm 2 Training Process of Heterogeneous Manifold Alignment (One Iteration)
Require: soft prompts 0, learnable curvatures c1 C2, learning rate 11. weight of entailment loss
Ensure: Updated parameters 0, C1. C2
1: Step 1: Construct intermediate manifold
2: cg < GoldenSSecioonSearchh (n(;; ,2, (Eq. 10
3: Step 2: Compute implicit gradients
4: Beg 0og +- +Implicttrradient/ (1, (1,02)) oz) Eq. 20
0cs acg
5: Step 3: Extract and map features
6: {v,) [t+ <- ExtractFeaturesO) {Sec. 4.1
7: for = to do
8:
t.t expmo (t;), 0.0h expmm (v:)
9: t. < expm (t,), ca < expm 2s (v;)
end for
11: Step 4: Compute losses
12: ((,,c1,0)) = (po V.) + (JTem( + Jvem (Voo) Jem( TrS)) {Eq. 17
13: Step 5: Update parameters
14: 0 - 0 llrr. J
15: dJ + aJ + as 0ca
(Total derivative)
dei 8c,
dej dc.
deg
+
Bez
0J
+ deg 002
dcg
(Total derivative)
C1 c
lr dai
18:
02
lr dJ
19: return 0, C1,
B.4 TOS CLASSIFICATION
For TOS classification experiments on Cifar100, SUN, and ImageNet, our training settings largely aligned with Wu et al. (2024) to ensure a fair comparison with baseline methods. For Rare Species, we use consistent settings across all baseline reproductions and our method experiments.
Training epochs. On Cifar 100 and SUN, both MaPLe and PromptSRC are trained for 200 epochs with batch sizes of 128 and 32, respectively. On ImageNet and Rare Species, both models are trained for 10 epochs with batch sizes of and 8, respectively.
Model initialization. We use pretrained ViT-B/16 CLIP model for initialization. For Cifar100, SUN, and ImageNet, we use OpenAl's CLIP checkpoint. For Rare Species, we use BioCLIP Stevens et al. 2024) checkpoint as the OpenAl pretrained CLIP performs poorly on this dataset.
Choice of intermediate layers. Across all experiments, we choose the 4hh layer, 7 layer (counting from 0) as the intermediate layers for our Semantic Aware Feature Extraction framework introduced in Section 4.1
Optimization. All experiments use the SGD optimizer with a cosine learning rate scheduler. For MaPLe experiments Cifar100, SUN, and ImageNet, adopt learning rate of 0.02 ensure fair comparison with ProTeCt (Wu et al. 2024). For other experiments, the learning rate chosen from {0.01, 0.02, 0.03) depending the prompt learning method and the dataset.
Other hyperparameters. Across all experiments, we use a fixed weight a 0.5 for the en- tailment loss. The initial values of curvatures for visual and textual manifolds are chosen from (0.5, 0.25, 0.05, 0.025) depending on the dataset.
Data preprocessing. For Cifar100, SUN, and ImageNet, Wu et al. (0024). For Rare Species, the preprocessing procedure is depicted in Appendix B.
the preprocessed data provided by
For all experiments, report LA, HCA, and MTA, averaged three independent runs.
22

---

Published as a conference paper at ICLR 2026
B.5 BASE-TO-NOVEL GENERALIZATION OF TOS CLASSIFICATION
To construct base and novel tree pairs, we first partition the leaf nodes of the semantic tree equally into two disjoint subsets: base leaves Vn and novel leaves V'v. For each subset V/s (where S e {B, N), we define the corresponding base/novel tree as the subgraph formed by the union of all root-to-leaf paths for every leaf E V/s. Detailed preprocessing procedure illustrated in Algorithm This construction yields rooted subtree satisfying:
(a) The leaf set equals V/s exactly.
(b) Internal nodes comprise all ancestors of 3
(c) Edges preserve the original ancestor-descendaan relationships.
Algorithm Construction of Base and Novel Trees
Require: Semantic tree T (V, E) with root and leaf set y
Ensure: Base tree TB and novel tree TN
1: Step 1: Partition leaf nodes
2: Partition y into two equal disjoint subsets: VB and V'N
3: where D/l = VNvl = / and VB n J/v = 0
4: Step 2: Construct base tree Ta
5: Initialize V/8 + 0, EB <- 0
6: for each leaf v Y' do
7: Trace path P. from root r to leaf v
8: VB +BB (all nodes in P.}
9: EB EB U (all edges in P.}
end for
11: TB + -VB,EB))
12: Step 3: Construct novel tree TN
13: Initialize VN +- 0, EN 0
14: for each leaf c E V/y do
15: Trace path P. from root leaf v
16: VN + +V U fall nodes P.)
17: EN+L EN U (all edges in P.}
18: end for
TN EN),
20:
return TB, TN
For base-to-novel generalization experiments, we adopt the same settings as standard TOS classifi- cation experiments but modify only the hyperparameters. Specifically, we train all methods for 10 epochs across all datasets to maintain generalizability, as we observe that training for more epochs leads dramatic decrease performance on the novel trees. For each of the metries (i.e., LA, HCA, MTA), we report results on the base tree, novel tree, and their harmonic mean, defined as:
2 Metricbase Metricnovel Metricbase Metricuovel
Metricharmony
(49)
where Metric E {LA, HCA, MTAJ.
C MORE EXPERIMENTAL RESULTS
C.1 COMPARISONS WITH HYPERBOLIC ALIGNMENT METHODS
We compare our method with hyperbolic alignment methods, MERU (Desai et al. 2023) and Hy- CoCLIP (Pal et al. 2025). Results are presented in Table Compared to the Mate-of-the-ar HyCo- CLIP method, our method achieves up to 25.89% improvement on LA, a 41.17% improvement on HCA, and 50.79% improvement MTA. The results demonstrate the limited ability of hyperbolic methods aligning modalities with hierarchical semantic structures, highlighting the necessity of method.
23

---

Published as a conference paper at ICLR 2026
Table TOS classification results across three datasets compared with hyperbolic alignment meth-
ods.
Cifar
SUN
ImageNet
Method
HCA
MTA
LA
HCA
MTTA
1.A
HCA
MTA
MERU HYCOCLIP
48.57
8.42
37.72
53.67
20.60)
45.88
39.35
1.17
40.03
60.21
9.31
42.96
58.40
26.52
48.98
45.74
0.98
38.87 38.82
Ours
78.9
68.47
91.12
76.54
69.18
86.20
71.63
42.15
89.61
Table 5: Base-to-barn
generalization results across four datasets.
Base Method
LA
HCA
MTA
Dataset
Variant
Base
Novel
HM
Whole
Base
Novel
HM
Whole
Base
Novel
HM
Whole
Vanilla
80.77
76.85
78.76
69.45
38.51
37.62
38.06
33.31
65.23
61.61
63.37
55.82
MaPLe
+ProTeCt
81.77
76.67
79.14
69.80
64.27
55.43
59.52
53.74
85.30
81.10
83.15
76.37
+Ours
82.79
77.11
79.85
69.82
73.38
56.23
63.67
57.09
88.85
81,24
84.87
78.61
Vanilla
82.30
78.68
80.58
71.50
51.77
48.25
49.94
47.05
68.89 89.89
65. 67
67
58.93
PromptSCC
+ProTeCt
82.36
78.40
80.33
71.94
66.86
58.80
62.57
57.06
86.67
82.76
84.67
79.12
+Ours
83.40
78.72
80,99
72.04
73.11
59,10
65.36
58.42
89.02
82.86
85.83
79.74
Vanilla
82.60
75.80
79.05
71 .44
9.94
6.94
8.17
7.00
61.46
50. 66
55
54.14
MaPLe
+ProTeCC
74.56
77.34
Cifar100
+Ours
82.76
75.94
79.20
72.52
66.42
41,52
51.10
45.79
91.01
82.05
86.34)
81.84
Vanilla
85.43
80.28
82.77
74.86
14.06
14.74
14.39
11.95
63.56
55. 60
59.31
55.41
PromptSC
+ProTeCt
85.36
78.72
81.91
73.82
64.76
49.47
42.36
79.38
84.88
80.53
+Ours
85.58
80.28
82.85
74.89
67.66
40.12
50.37
42.87
91.74
79.82
85.37
81.72
Vanilla
78.59
75.78
77.16
67.50
1.84
1,96
1.90
1.64
48.57
45 85
47.17
45.10
MaPLe
+ProTeCt
78.23
75.42
76.80
67.48
33.92
29.28 28
31.43
90.86
89.42
86.61
ImageNet
+Ours
78.74
75.88
77.28
67.53
50.81
31.70
39.04
33.60
92.09
88.11
90.06
86.73
Vanilla
79.67
77.12
78.37
68.70
5.26
4.22
3.78
53.57
51.71
52.62
52.33
PromptSRC
+ProTeCt
79.52
77.02
78.25
68.67
40.01
29.57
34.01
29.90
91.10
87. $7.90
89.47
86.81
+Ours
79.80
77.15
78.45
68.74
49.59
30.32
37.63
33.31
92.03
88.08
90.01
86.95
Vanilla
52.95
47.83
50.26
43.93
9.25
7.78
8.45
5.72
50.45
52.94
51.67
45.52
MaPLe
+ProTeCt
52.61
47.32
49.82
44.18
29.94
16.14
20.97
17.15
79.72
72.32
75.84
72.81
Rare Species
+Ours
68.54
48.01
56.47
53.02
55.00
17.22
26.23
29.79
88.04
74,18
80.52
78.62
Vanilla
52.40
13.45
57.58
55.90
PrompeSRC
PPoTTeCt
57.51
53,30
55.33
50.44
40.69
21.01
27.71
24.57
82.75
77. 19
79.87
76.98
+Ours
64.93
53.72
58,80
53.32
51.89
21.11
30.01
29.29
87.70
77.27
82.16
78.79
C.2 COMPLETE RESULTS OF BASE-TO-BASEEE TEA.---BBSSSSAASSSS-OOVLLVEEEEEE GENERALIZATION
Table presents the complete results of results across multiple datasets. between our method and existing methods. Our method consistently outperforms all baselines across all metrics and settings, demonstrating strong generalization to novel classes.
generalization
C.3 MORE ABLATION RESULTS ON TOS CLASSIFICATION.
Table summarizes our ablation results on Cifar100, SUN, and Rare Species using MaPLe and PromptSRC under 1-shot and 16-shot settings. Ours-Fuc consistently outperforms ProTeCt, demon- strating the effectiveness of our semantic-aware visual feature extraction framework. Our method consistently outperforms two hyperbolic variants as well as Ours-Euc, demonstrating the effective- ness of our heterogeneous manifold alignment algorithm.
We also conduct ablation experiments under rine-to-basebbasee baass/bsseto-noove//haee poovel/base-o-wwiie settings on the SUN dataset. As shown in Table we observe similar performance patterns, where method consistently achieves superior results across different evaluation protocols. These results validate the generalization capability of proposed approach.
24

---

Published as a conference paper at ICLR 2026
Table 6: Ablation results on Cifar100, SUN, and Rare Species under different k-shot settings (Ex- tended).
K- Shot
Base Method
C0
SUN
Rare Species
Variant
LA
HCA
MTA
LA
HCA
MTA
LA
HCA
MTA
+ProTeCt
69.33
48.10
83.36
64.29
50.45
76.73
39.92
13.22
70.04
+Ours-Bue
69.79
49.77
84.54
67.19
56.25
79.80
46,56
20.28
74.25
MaPI.e
-uurs-ypVI
69.80
51.86
85.17
66.78
57.56
80.22
45.51
20.86
76.62
+urs-HypV2
70.98
51.67
85.23
67.17
57.87
80.44
45.81
20.82
76.54
+Ours
71.37
53.19
85.29
67.57
57.92
80.55
46.77
20.94
76.83
+ProTeCt
73.07
49.54
85.16
70.61
55.52
78.73
44.56
20.36
74.42
+Ours-Eur Euc
73.16
50.05
85.17
70.09
56,84
79.80
46.96
22.60
77.17
PromptSRC
+Ours-HyPVI
72.21
51.03
85.36
70.39
57.76
79.84
46.22
22.74
77.22
+Ours-HypV2
72.43
51.59
85.25
70.56
57.75
79.86
46.59
22,65
77.20
+Ours
73.54
51.91
85.76
70.64
57.79
79.94
46.98
23.03
77.32
+ProTTeCt
75.34
61.15
88.04
72.17
59.71
82.27
48.14
24.82
78.79
Ours-Euc
76.99
68.01
90.55
74.07
66.81
85.36
68.96
51.81
87.15
MaPLe
Ours-HypVV
77.62
69.05
90.82
75.10
68.26
85.99
67.41
52.85
87.18
+uur--HypV2
77.69
69.33
90.71
75.19
68.65
85.92
69.67
52.73
87.01
+Ours
77.92
69.38
90.89
75.47
68.67
86.02
69.96
53.65
87.27
16
Pero
78.76
66.74
90.79
75.54
66.01
84.75
56.40
33.92
82.47
+Ours-Eo
78.34
68.05
90.85
75.81
68.81
86.17
66.50
48.73
87.26
PromptSRC
+Ours-HyPVI
78.82
68.24
91.00
76.50
69.10
86.02
67.25
49.98
87.31
+Ours-Hip D2
78.55
68.18
91.06
76.47
69.17
86.09
66.92
49.48
87.13
+Ours
78.90
68.47
91.12
76.54
69.18
86.20
67.38
50.77
87.60
C.4 ABLATION RESULTS ON MAB--TO-NNS/D--S---ONOOVVE/EEEE GENERALIZATION.
We also conduct an ablation study on tings. The results in Table demonstrate the effectiveness and robustness of our semantic-aware visual feature extraction and intermediate manifold search module.
to-novel/base-too generalization set-
Table Ablation results for haac-tl--acchaaeetoonee//seeeees generalization experiments on SUN.
Base
LA
HCA
MTA
Variant
Method
Base
Novel
HM
Whole
Base
Novel
HM
Whole
Base
Novel
HM
Whole
+ProTeCt
81.77
76.67
79.14
69.80
64.27
57.43
59.52
53.74
85.30
81.10
83.15
76.37
+Ours-Bue
81.83
76.74
79.20
69.79
71.74
53.75
61.46
54.39
88.22
80.80
84.35
77.81
MaPLe
+Ours-HypV
82.22
76.90
79.47
69.68
73.00
55.02
62.75
56.64
88.42
80.80
84.44
78.16
+ Ours-IlypV2
82.15
76,93
79.45
69 69.66
73.28
55.75
63.32
56. 16
88.65
80.90
84.60
78.08
+Ouss
82.79
77.11
79.85
69.82
73.38
56.23
63.67
57.09
88.85
81.24
84.87
78.61
+ProTeCt
82.36
78.40
80.33
71.94
66.86
62.57
57.06
86.67
82.76
84.67
79.12
+Ours- Fue
82.85
78.83
80.79
71.93
72.28
57.69
64.17
$7.51
88.74
82.65
85.59
78.93
PromptSRC
-Ours-HypVi
83.10
78.53
80.75
72.02
72.96
58.93
65.20
$7.99
88.78
82.74
85.65
79.23
V2
83.02
78.30
80.59
71.89
72.72
57.84
64.43
58.41
88.88
82.72
85.69
79.58
+Ours
83.40
78.72
80.99
72.04
73.11
59.10
65.36
58.42
89.02
82.86
85.83
79.74
C.5 ABLATION RESULTS ON MANIFOLD DISTANCE DEFINITION.
We also conduct ablation experiments proposed distance measure with three baselines: squared difference (a c3) squared log dis- tance (Incy Inc3) 2 and squared reciprocal distance (ei cs 1 Our method consistently achieves the best performance. These results validate that our distance measure better captures the intrinsic relationships between manifolds with different curvatures compared to simple mathematical trans- formations.
manifold distance definition. Table 8 compares
25

---

Published as a conference paper at ICLR 2026
Table 8: Comparison of different distance metries for hierarchical learning. Our proposed hyperbolic distance achieves the best performance across all evaluation metrics.
Method
De(ca,
LA
HCA
MTA
Squared distance
(c1
ca)3 (Incl ln
71.03
50.76
84.58
Squared log distance
71.10
52.42
84.95
Squared reciprocal distance
71.05
52.11
84.71
VCi cosllll(/( veirrl
Ours
71.37
53.19
85.29
2/0103
C.6 ABLATION RESULTS ON THE CROSS-ATIE ATTENTION MODULE.
Table 9: Performance of our method without the cross attention module.
Metrics
Base Method
Variant
LA
HCA
MTA
Vanilla
68.75
4.65
50.60
+ProTecC
69.33
48.10
83.36
MaPLe
+Ours
71.37
53.19
85.29
+Ours w/o cross attention
67.43
48.42
84.04
Vanilla
72.48
14,36
51.91
+ProTeCt
73.07
49.54
85.16
PromptSRC
+Ours
73.54
51.91
85.76
+Ours w/o cross attention
73.12
50.11
85.08
We also conduct an ablation analysis on the cross-attention mechanism by removing it. Specifically, we manually design set of weighting coefficients for the class tokens different layers construct hierarchical visual features. For example, the weight for level 1 is [0.2397, 0.3235, 0.4368), and the weight for level is [O. 1628, 0.2966. 0.54051] for training. During inference, given an image and set of candidate textual labels, to all textual labels, and the label with the highest similarity is the final prediction. Results are shown in Table 9 highlighting the importance of this mechanism.
compute visual features at all levels and evaluate their similarities
C.7 ADDITIONAL QUANTITATIVE RESULTS FOR LEARNED IMAGE REPRESENTATIONS
Table 10: Clustering performance of our learned representations across taxonomic levels. Our method consistently surpasses the baseline in all three metrics: Silhouette Score and Calin- ski-Harabasz Index, and Davies-Bouldin Index.
Silhouette Score (f)
Calinski- Index (t)
Davies-Bouldin Index (L)
Taxonomic Level
Baseline
Ours
Baseline
Ours
Baseline
Ours
Phylum
0.0106
0.0112
105.7
128,0
4,48
3.78
Class
0.0191
0.0250
113.1
136.44
3,92
3.50
Order
0.0301
0.0381
48.5
56.8
3,33
3.04
Family
0.0563
0.0671
33.8
39.5
3.24
2.99
Genus
0.0680
0.0796
27.0
31.6
3.16
2.94
Species
0.0621
0.0727
23.4
27.3
3.38
3.19
To comprehensively demonstrate that our learned features outperform the baseline across all hier- archical levels, employ three clustering quality metries for quantitative comparison: Silhouette Score,
Index, and Davies-Bouldis Index. The Silhouette Score measures how similar sample is to its cluster compared to other clusters, with values ranging from -1 to
26

---

Published as a conference paper at ICLR 2026
1 where higher scores indicate better-defined clusters. The Calinski-Harbbass Index evaluates the ratio of between-cluser dispersion to within-cluster dispersion, where higher values signify more distinct and compact clusters. The Davies-Bouldis Index quantifies the average similarity between each cluster and most similar cluster, with lower values indicating better cluster separation.
Table TO presents the quantitative comparison between method and the baseline. The results consistently demonstrate that our hierarchical feature extraction framework produces more discrim- inative and well-separated representations at each semantic level, indicating that semantic-aware approach effectively captures the inherent structure of visual concepts at multiple granularities.
C.8 SENSITIVITY ANALYSIS
Table 11: Performance of our method with different choice of intermediate layers {pj}グ=1 for cur- vatures of the visual and textual manifolds.
Intermediate Layers
HCA
MTA
[0,5,11]
71.08
52.52
84.65
[2.6,11]
71.20
52.69
84.89
[4,7,11] (ours)
71.37
53.19
85,29
[6,8,11]
71.25
52.63
84.85
[8,9,11]
71.18
52.46
84.67
Table 12: Sensitivity analysis. (a) Performance with different curvature initialization values. (b) Performance with different weight factors (a).
(a)
(b)
Image
Text
LA
HCA
MTA
LA
HCA
MTA
0.25
0.5
70.62
52.70
84.95
0.3
70.87
52.88
84.76
0.5
0.25
70.88
52.95
85.02
0.4
71.02
52.90
84.90
0.5
71.37
53.19
85.29
0.5 (ours)
71.37
53.19
85.29
0.6
0.6
70.87
53.15
85.20
0.6
70.94
53.12
85.17
0.7
0.7
71.18
52.72
85.04
0.7
70.74
52.94
84.99
We conduct experiments analyze the sensitivity of hyperparameters, including the choice of in- termediate transformer layers {py}7: the initial values for the learnable curvatures, and the loss weighting factor. Specifically, we conduct experiments on the Cifar100 dataset by adjusting these hyperparameters. The results shown in Table II and 12 indicating that the method is only minimally sensitive hyperparameter variation.
Table 13: Performance comparison with different backbone architectures. (a) Results using MaPLe with ViT-L/14 backbone. (b) Results using PromptSRC with ViT-L/14 backbone.
(a)
(b)
Method
LA
ICA
MTA
Method
LA
HCA
MTA
MaPLe
75.79
10.13
58.94
PromptSRC
79.45
12.23
58.34
+ProTeCt
75.54
55.90
85.89
+ProTeCt
79.59
57.29
86.53
+Ours
77.04
56.54
86.71
+Ours
79.99
59.01
87.39
To further investigate the robustness of our method, additionally utilize another architecture on Cifar100 to conduct experiments on few-shot setting.
with deeper depths (i.e.. Results in Table 13 demonstrate the robustness of our method.
27

---

Published as a conference paper at ICLR 2026
500000
400000
Prequency
300000
200000
100000
0.0
0.2
1.0
Error Value
Figure 7: Histogram of the error ratio of Taylor approximation.
C.9 DISTRIBUTION OF TAYLOR APPROXIMATION ERROR FOR HYPERBOLIC DISTANCE COMPUTATION, SHOWING MEAN RELATIVE ERROR OF 5.3% ACROSS 1M VECTOR PAIRS.
The Taylor approximation used to compute the KL--based manifold distance is applied the hy- perbolic distance. We conduct an empirical analysis to evaluate this approximation error in Taylor. Specifically, we randomly select 1,000,000 pairs of hyperbolic vectors generated by our method, compute both the exact hyperbolic distance and its maylor-approximate value, and measure the rel- ative error defined as error Ideneet - dapproal The histogram of this ratio is illustrated in Figure [deanet Results show that the mean relative error 5.3%, with maximum of 32.4% 4% and minimum less than 0.1% 16, demonstrating that the approximation error is acceptable and controllable in practice.
C.10 ANALYSIS ON THE ESTIMATION OF
The approximation constant r is the norm of feature midpoints on tangent spaces. In our imple- mentation, estimated per batch by computing the mean feature vector the tangent space and taking its norm. This estimation makes it naturally consistent with the training dynamics-ijg value evolves smoothly with feature distribution of the model. We compute the relative error defined as |eroet-ri] where T Pezact is computed on full-batch data. We further visualize the trajectory error, = |rsecal of error, during training, as shown in Figure
Table 14: Analysis on hyperparameter r in our hyperbolic distance formulation.
LAA
HCA
MTA
70.58
52.41
84.48
70.52
52.46
84.50
Ours
71.37
53.19
85.29
Finally, set fixed constant, i.e., 1 and = 2, where results shown in Table 14 Results demonstrate that our method is robust to the variation of r. In conclusion, our method is robust the approximation of r.
28

---

Published as a conference paper at ICLR 2026
0.8
0.6
EEror
25
50
75
100
125
Epoch
Figure 8: Trajectory of the relative error in estimating r.
C.I1
AN ALTERNATIVE ADDRESSING SPECTES-LEVEL. AMBIGUITY RARE SPECTES
Table 15: TOS classification results Rare Species using alternative labeling strategy without hierarchical concatenation, under 1-shot and 16-shot settings.
Metrics
Shot
Baseline Method
Variant
LA
HCA
MTA
Vanilla
38.09
1.84
35.19
MaPLe
+ProTeCt
35.67
7.84
61.69
+Ours
38.34
10.55
69.47
Vanilla
39.13
6.09
43.68
PromptSRC
+rroTeCt
37.92
9.93
64.45
+Ours
39.97
12.47
71.93
Vanilla
50.69
2.01
35.19
MaPI.e
+ProTeCt
44.68
16.19
61.69
+Ours
66.71
44.01
77.27
16
Vanilla 59.99
4.55
40.54
PRomptSRC
+ProTeCt
55.53
28.66
74.76
+Ours
65.16
44.43
79.74
Table 16: to- mo-nove//base-oor to wo-hoe generalization results on Rare Species using an alternative labeling strategy without hierarchical concatenation.
Base
LA
HCA
MTA
Variant
Method
Base
Novel
Whole
Base
Novel
HM
Base
Novel
Whole
Vanilla
50.38
39.29
44.15
41.09
3.20
2.90
3.04
1.84
8.51
8.15
8.33
7.42
MaPLe
+ProTeCt
45.50
38.80
37.92
21.78
8.44
12.17
10.97
18.69
17.45
18.05
18.14
+Ours
66.86
40.40
50.37
47.77
50.97
10.10
16.86
24.57
19.15
20.01
19.57
20.07
Vanilla
60.13
42.20
49.59
46.43
6.06
6.04
6.05
4.01
7.64
8.00
7.82
6.25
PromptSRC
+ProTeCt
54.33
42.02
47.39
44.93
33.22
11.75
17.36
17.36
18.08
16.12
17.04
17.88
+Ours
64.93
44.21
52.60
48.85
48.36
12.09
19.34
24.74
18.92
16.72
17.75
18.42
We additionally design an alternative approach that combines the common name with the species name resolve ambiguity the species level. Specifically, replace non-unique labels with
29

---

Published as a conference paper at ICLR 2026
combination of the scientific name and the common name to guarantee uniqueness without relying on the concatenated hierarchy. We re-conduct experiments without using the concatenation strategy. Results in Table [5] and [6 show that method significantly improves the performance of the compared methods, demonstrating its superiority. Notably, in the 16-shot setting of the few-shot task, our method achieves up to a 22.03% improvement LA, 27.8 27.82% improvement on HCA, and 15.58% improvement on MTA, indicating its ability to align the textual and visual modalities with the hierarchical semantic structures.
C.12 POTENTIAL DATA IMBALANCE.
Table 17: Analysis on class imbalance using the MaPLe baseline.
Method
LA
HCA
MTA
MaPLe
68.75
4.65
50.60
+ ProTect
69.33
48.10
83.36
Ours
71.37
53.19
85,29
+ Ours-balanced
71.26
53.54
85.70
Class imbalance is a common issue. This issue is particularly prevalent in taxonomic open-set clas- sification tasks. At the leaf level, sample sizes per category are typically balanced, whereas at higher taxonomic levels, the distribution the capability to handle such an imbalance issue (Wen et al. (0024). Therefore, we did not explic- itly consider the imbalance issue in current design. In future work, plan develop more robust algorithms to address this issue. As a preliminary attempt, we have implemented a sample re- weighting strategy mitigate potential bias. As shown in 17] this balanced approach brings consistent improvements in hierarchical metrics (HCA and MTA). We attribute these improvements the fact that the re- re-weighting strategy effectively addresses the imbalance at non-leaf nodes, al- lowing the model to better learn the hierarchical structure of the data.
be highly imbalanced. We note that CLIP has already shown
C. MORE VISUALIZATIONS ON RARE SPECIES
We visualize the attention maps generated by the baseline ProTeCt (Wu al.) 2024) and our model using GradCAM (Selvaraju et al. (2017) to analyze their behaviors across different taxonomic levels. The results are shown in Figures 9 and 10 When aligned with text prompts at different granularities, our model exhibits hierarchical attention patterns, focusing on distinct visual regions that are most discriminative for each taxonomic level. Specifically, we observe the following attention patterns:
Giant Panda (Ailuropoda melanoleuca): At the class level (Mammalia), the model pri- marily attends the distinctive fur texture and body shape, which are key features distin- guishing mammals from other vertebrate classes (e.g., Chondrichthyes), At the order level (Carnivora), attention shifts to the limbs and paws, capturing the characteristic plantigrade locomotion that differentiates carnivorans from other mammalian orders (e.g.. Primates). At the genus level (Ailuropoda), the model focuses on the facial features and distinctive black-and-whir coloration pattern, which uniquely identifies giant pandas from other bear genera (e.g.. Tremarctos).
Forty-Spotted Pardalote (Pardalotus quadragintus): At the class level (Aves), the model attends to the head plumage and feather structure, fundamental avian characteristics that distinguish birds from other vertebrates. At the order level (Passeriformes), attention concentrates on the bill morphology, particularly its short and stubby shape that charac- terizes pardalotes and differentiates them from other passerines with elongated bills (e.g., honeyeaters with long, slender bills) or from non- pan-passeenne orders with distinct bill struc- tures (e.g., Psittaciformes with curved beaks). At the family level (Pardalotidae), the model highlights the distinctive wing patterns, specifically the characteristic spotted markings that distinguish pardalotes from other passerine families within the same order.
Grey Reef Shark (Carcharhinus amblyrhynchos): At the class level (Chondrichthyes), the model focuses on the caudal fin structure, defining feature of cartilaginous fishes
30

---

Published as a conference paper at ICLR 2026
Panda
original
class-level
order-level
genus-level
Forty-Spotted Pardalote
original
class-level
order-level
family-level
Grey Reef
Shark
original
class-level
order-level
species-level
Figure 9: Visualization of attention maps generated by the baseline ProTeCt (Wu al. 2024 Rare Species.
Panda
original
class-level
order-level
genus-level
Forty-Spotted
Pardalote
original
class-level
order-level
family-level
Grey Reef Shark
original
class-lcvel
order-level
specics-level
Figure 10: Visualization of attention maps generated by our model on Rare Species.
31

---

Published as a conference paper at ICLR 2026
that distinguishes them from bony fishes (Osteichthyes). At the order level (Carcharhini- formes), attention is directed towards the gill region, specifically the presence of five gill slits a diagnostic feature of ground sharks that differentiates them from other shark orders (e.g., Hexanchiformes with six or seven gill slits). At the species level (Amblyrhynchos), the model identifies the snout morphology and mouth position, which species-specific characteristics distinguishing the grey reef shark from other Carcharhinus species.
These visualization results demonstrate that model learns biologically meaningful features each taxonomic level, aligning with domain knowledge in taxonomy and supporting its strong per- formance on hierarchical classification tasks.
C.14 VISUALIZATION FOR ARTIFACT CLASSES.
original
artifact
mechanism
carousel
original
artifact
wheeled vehicle
taxicab
original
artifact
wheeled vehicle
police van
Figure 11: Visualization attention maps generated by the baseline ProTeCt (Wu et al. 2024 on ImageNet.
We also visualize the attention maps generated by our model and baseline when encountering artifact classes. The results are presented in Figures [1 and 12 When processing text descriptions at different granularities, our model demonstrates adaptive attention semantically relevant regions. For instance, when determining whether images of a taxicab or a police van belong to the wheeled vehicle category, our model correctly attends the wheel region, whereas the baseline model fails to focus on this discriminative feature. This further demonstrates that our semantic-aware visual feature extraction framework effectively leverages textual cues guide attention towards hierarchically- appropriate visual regions, enabling more precise alignment between coarse-to-fin semantics across modalities.
C.15 LEARNED CURVATURES ON DIFFERENT DATASETS.
We additionally provide the learned and intermediate curvatures across different datasets, with re- sults shown in Table 18 We observe that the curvatures of the text, image, and intermediate mani- folds differ substantially for each dataset, indicating that data-driven curvature learning is necessary to match the varying geometric structures. The large divergence among the three learned curvatures further confirms that different modalities indeed require different curvature adaptations, validating the core motivation of our method.
32

---

Published as a conference paper at ICLR 2026
original
artifact
mcchanism
carousel
original
artifact
wheeled vehicle
taxicab
original
artifact
wheeled vehiele
police van
Figure 12: Visualization of attention maps generated by our model on ImageNet.
Table 18: Learned curvatures and the corresponding optimal curvatures for intermediate manifolds of our method on different datasets.
Dataset
Images
Text
Intermediate
Cifar100
0.11
0.37
0.26
SUN
0.019
0.029
0.026
ImageNet
0.022
0.028
0.026
Rare Species
0.15
0.40
0.30
C.16 EFFICIRNCY ANALYSIS
We evaluate the efficiency of our method on the Rare Species dataset, using batch sizes of for training and 1024 for inference. We measure the per-batch time cost and memory usage, reporting results averaged over 1,000 batches as shown Figure I3 The results demonstrate that our method introduces only modest overhead approximately 30% more time and 2% more GPU memory dur- ing training, and 10% more time and 4% more GPU memory during inference. Moreover, learning multiple curvatures introduces almost additional cost.
D MORE DISCUSSIONS
D. THEORETICAL JUSTIFICATION FOR THE SEMANTIC-AWARE FEATURE EXTRACTION MODULE
In this section, provide detailed thcoretical justification for the design of our semantic-aware feature extraction module, specifically addressing the rationale behind layer selection and the sup- pression of cross-token attention. We further discuss the robustness and novelty of this module compared to existing approaches.
Our strategy of selecting specific intermediate transformer layers is grounded in the Information Bottleneck (IB) theory applied to deep networks (Tishby & Zaslavsky 2013). As network depth increases, representations undergo progressive compression process that reduces the mutual infor-
33

---

Published as a conference paper at ICLR 2026
1.27
1.30
L.22
140
180
1/12
Relatve Value Tve. Burelano
Rdabive Barelioo
0.8
0.6
02-
Chevifip
Dan-thpp2
Chta
Prefec
Gus-HbpVi
Omnelty122
Oan
12
1.77
1.10
Relaivee Value (Va. Burdan))
Relaske Bardinc)
0.8
066
0.6
0.4
0.1
02
00
00
Protect
Eas
Das4lspVi
0awel1912
Pronei
GandityVi
Oar
Obeo
Figure 13: Efficiency analysis of different methods. (a) Training time consumption; (b) Training
memory consumption; (c) Inference time consumption; (d) Inference memory consumption.
mation between the features and the input. This compression induces a natural hierarchy of semantic granularity:
Early layers capture coarse and low-level visual cues.
Intermediate layers model mid-level semantic representations.
Late layers are highly compressed and encode fine-grained semantics.
This theoretical progression motivates our coarse-to-fine feature extraction strategy. While select layers to align with these distinct granularities, we empirically demonstrate that the specific choice of layers within these ranges does not significantly affect performance (see Table in Appendix C..).
From a theoretical standpoint, the self-attentio mechanism operates smoothing operator (Cordonnier al., 2020 Tsai al. 2019). This mechanism inevitably mixes information across tokens and induces semantic smoothing at each scale (Park & Kim (022). Such cross-token mixing hinders the preservation of specific granularity semantics because the feature of each token becomes entangled with information from other spatial regions. Consequently, the visual evidence aggregated into the class token may no longer align with the intended semantic granularity of that layer. Motivated by these findings, disable cross-token attention during feature extraction to ensure that the class token summarizes disentangled, semantically coherent features at each granularity, which is crucial for robust multi-modal alignment.
pnessage-passsngg or kernel-
34

---

Published as a conference paper at ICLR 2026
To further verify that the effectiveness of our component is not sensitive to specific architectures (e.g.. varying depths of ViT), we conducted additional experiments using larger backbone, CLIP- ViT-L/14, on the C0 few-shot setting. As shown in Table B in Appendix C.8 method achieves consistent performance gains over baselines (e.g.. MaPLe and ProTecC) on the deeper backbone. These results demonstrate the generalizability and robustness of our core component across different model capacities.
While utilizing intermediate features is well-established technique, our semantic-aware module differs significantly from existing methods that typically operate in a purely feed-forward manner (e.g., weighted fusion concatenation). Our approach introduces two key innovations:
We introduce cross-token attention suppression construct features that strictly preserve coarse-to-fine semantics, strategy that has not been explored in existing feature extraction methods.
We design semantic-aware module that utilizes textual semantics to actively guide the coarse-to-fie visual feature construction.
Moreover, our contribution extends beyond feature extraction. We align these hierarchical, tree-like semantic features across heterogeneous manifolds, ensuring that the similarity in using intermediate layers does not diminish the novelty of our overall framework.
D.2 HOW DOES OUR MODEL AVOID DEGENERATE SOLUTIONS?
Our model avoids the degeneration of fine-grained features (i.e., preventing fine-grained features from collapsing into coarser representations) through three synergistic mechanisms:
Feature construction preserves fine-level detail. We explicitly include the final-layer class token to construct visual representations. Specifically, utilize the finest-grained textual feature as the query to compute the attention weights on candidate tokens. Empir- ically, the average attention weights on layers 4, and 11 are 0.06, 0.08, and 0.86, spectively. We observe that the attention weights on this final-layer token are significantly larger than those intermediate layers. This indicates that the finest-grained features suc- cessfully capture fine- fine-grained details and do not collapse into coarser representations.
Fine-grained alignment loss directly prevents collapse. Our training objective (Tee, Ve) in Eq. (7) explicitly includes distance aligmment loss between the finest- level textual and visual features. This objective forces the model to maintain a discrimi- native fine-grained structure, any degeneration would inevitably increase this alignment loss.
-odality-speeificc geometric constraints enforce boundary separation. The constraints defined in E5. encourage fine-grained features to reside near the boundary of the hy- perbolic manifolds. Because points near the boundary in hyperbolic space have larger radii and correspond to more specific semantics, our method naturally prevents fine-level fea- tures from collapsing to coarser levels.
E THE USE OF LARGE LANGUAGE MODELS (LLMs)
In this work, we employed Large Language Models (LLMs) as assistive tools in the following ca- pacities:
Code Development: We utilized LLMS to help understand existing codebases and assist in writing new code implementations. Specifically, LLMs were used for:
Explaining complex code segments and algorithmic implementations
Generating code snippets and suggesting optimizations
Debugging and identifying potential issues in implementations
Writing Assistance: LLMs were employed to improve the clarity and readability of our manuscript through:
35

---

Published as a conference paper at ICLR 2026
Refining technical descriptions and explanations
Improving grammar and language flow
Suggesting alternative phrasings for better clarity
Limitations LLM Usage: We emphasize that:
All research ideas, experimental design, and core contributions original work by the authors
All LLM L.M-gennrrted content was carefully reviewed, verified, and modified by the authors
The experimental results and analysis are entirely based on our own implementations and observations
The specific LLMs used include ChatGPT, Claude, and Deepseek-RI.
36


Reviews:
[{"review_id": "ZgoHPeE8lW", "reviewer": "Reviewer_L9Aq", "summary": "This paper proposes a hyperbolic representation approach for vision–language alignment in a hierarchical manner. \n\nThe approach assumes we have textual labels at multiple levels (e.g. not just the original animal species but also the biological family, order, and class). It hypothesizes that different layers of a deep model similarly discern representations at different levels of granularity that can be aligned to the different textual hierarchy levels.\n\nBased on this idea, the paper investigates hyberbolic representations. First, hierarchical representations are induced for the two modalities in a deep model by eliminating cross-token self-attention, adding cross-modality attention. The two modality-specific manifolds are bridged by constructing an intermediate hyperbolic space, to which the original ones are mapped subject to hierarchical entailment constraints.", "questions": "- Understanding how the model avoids degenerate solutions is important. Can you explain to readers of the paper which parts of the model ensure that the fine-grained features do not become overly coarse?\n\n- What do we observe on truly diverse classes, e.g. WordNet's \"artifact\" class?\n\nMinor issues:\n\n- The explanation of Eq. 3 mistakenly refers to $\\delta$ and $u$ that do not appear in the formula.\n\n- bad formatting in L. 197 \"fine-grained information(Chen et al., 2024).\"\n\n- typo \"can not\" \n\n- The paper suffers from extensive negative vspace hacking. There are various orphan headings such as \"4.2 HETEROGENEOUS MANIFOLD ALIGNMENT ALGORITHM\" and \"4.2.2 INTER-MODAL GEOMETRIC ALIGNMENT MECHANISM\". There are also many headings that have insufficient spacing, e.g. \"4.2.1 INTERMEDIATE MANIFOLD CONSTRUCTION\" and \"4.4 OPTIMIZATION STRATEGY\".\n\n- There are a number of poorly typeset mathematical expressions, e.g. Eq. (49).", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Interesting idea worth exploring\n- Some promising experimental results\n- Generalization to novel classes investigated", "weaknesses": "- Most of the examples focus on hierarchical relationships from biological taxonomy. While there are experiments on more diverse datasets such as ImageNet, the paper does not provide much analysis and insight into how the model behaves on other kinds of hierarchies, especially when the data is more diverse.\n\n- On the Rare Species Dataset, the authors do not stick with the original labels but instead use hierarchical label names created by concatenating the various hierarchy levels in Algorithm 1 (Hierarchical Tree Construction). This seems to massively alter the nature of the task, as now the parent hierarchy is already given in each textual label. If the motivation for this step, as claimed by the authors, is just to deal with ambiguous labels, then why not add a small extra term just to ambiguous labels?\n\n- The paper currently lacks a larger discussion of methods for hierarchy/taxonomy alignment. The paper presents a single method but doesn't explain the design space and design choices well enough. What alternatives could have been chosen? The relationship to the broader field of hierarchical/taxonomy alignment could be explored further.\n\n- The alignment occurs in multiple phases rather than a single one, in particular it occurs via the Heterogeneous Manifold Alignment Algorithm but also earlier via cross-attention in the Semantic-Aware Visual Feature Extraction Framework. This cross-attention between image and text in the Semantic-Aware Visual Feature Extraction Framework (Eq. 5) appears to allow information to be copied over between modalities. It is not obvious how the approach ensures that each modality indeed yields a hierarchy for that particular modality rather than just copying over information from the other modality. How well does the model work without this cross-attention?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oVJcqdmVOv", "reviewer": "Reviewer_jhgh", "summary": "This paper addresses a key limitation in current vision-language models (VLMs): the asymmetric and potentially suboptimal alignment of modalities, where images are represented by single features while text uses multi-level features. The proposed method, Alignment across Trees, constructs hierarchical (tree-structured) features for both modalities and embeds them in separate hyperbolic manifolds with learnable curvatures. It further introduces a semantic-aware visual feature extraction framework based on cross-attention guided by textual prompts, and a heterogeneous manifold alignment technique that uses KL divergence between distributions on different manifolds via an intermediate curvature. Experiments on multiple taxonomic open-set classification benchmarks show that this approach outperforms existing methods in aligning hierarchical multimodal data.", "questions": "Could visualizations be supported by metrics like inter/intra-class variance or cluster separation?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The paper identifies and tackles a issue in VLMs—the mismatch in representational hierarchy between vision and language. Figure 1 clearly illustrates this contrast, highlighting how the proposed tree-based structure achieves more symmetric alignment.\n\nS2.This paper has comprehensive ablation and visualization.", "weaknesses": "W1. The Taylor approximation used to compute the KL-based manifold distance (Appendix A) lacks empirical evaluation. No sensitivity analysis is provided for the approximation constant 𝑟, raising concerns about robustness and stability.\n\nW2. The paper does not analyze potential failure modes, computational overhead from learning multiple curvatures.\n\nW3. The approach to class imbalance across different tree levels is not explained, leaving open questions about potential bias in curvature learning or alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0LTk1tAlzJ", "reviewer": "Reviewer_1GsE", "summary": "This paper tackles the modality gap in hierarchical vision-language learning. The key idea is to construct symmetric tree-like features for both images and texts, and then align them on hyperbolic manifolds. Specifically, the method extracts hierarchical visual features from intermediate transformer layers using a text-guided attention module. To handle geometric differences between modalities, it embeds them into separate hyperbolic spaces with learnable curvatures and aligns them through an optimally constructed intermediate manifold. The approach demonstrates strong performance on taxonomic open-set classification across several datasets.", "questions": "Please kindly refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses an important and interesting problem of achieving efficient and symmetric modality alignment for hierarchical semantic structures.\n- The work presents a novel approach by leveraging hyperbolic spaces with learnable curvatures and an intermediate manifold for modality alignment between images and text.\n- Comprehensive experiments demonstrate the method's effectiveness, showing consistent improvements across multiple datasets and settings for taxonomic classification.", "weaknesses": "- The design of the semantic-aware feature extraction module appears somewhat heuristic and lacks strong theoretical justification. The critical choices of which specific intermediate transformer layers to use and the decision to disable cross-token attention are based on empirical observations rather than derived from first principles. This raises concerns about the generalizability and robustness of this core component, as its effectiveness might be sensitive to the specific architecture (e.g., different ViT depths) or pre-training dataset. Furthermore, the utilization of intermediate features is a well-established technique, diminishing the perceived novelty of this part of the proposed framework.\n- Concern about sensitivity to hyperparameters. The proposed method introduces a non-trivial number of hyperparameters, and its performance is likely sensitive to their careful tuning. Key hyperparameters include the choice of intermediate transformer layers, the initial values for the learnable curvatures, and the loss weighting factor α. The paper would be strengthened by a sensitivity analysis demonstrating the robustness of the results to variations in these critical settings. Without such an analysis, it is difficult to assess the true practicality and generalizability of the approach, as the reported strong performance might be contingent on a specific, finely-tuned configuration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ihyVqqxe8o", "reviewer": "Reviewer_8Seg", "summary": "This paper proposes \"Alignment across Trees,\" a novel method for aligning vision and language modalities by constructing symmetric, hierarchical feature representations for both. The core idea is to address the asymmetry in prior work where hierarchical text features are aligned with a single, flat image feature. The method introduces two main contributions: 1) a semantic-aware visual feature extraction framework that uses cross-attention with textual cues to build a hierarchical tree of visual features from intermediate Transformer layers, and 2) a heterogeneous manifold alignment algorithm that embeds the image and text feature trees into separate hyperbolic manifolds with distinct, learnable curvatures. To align these, the paper defines a novel distance measure between heterogeneous hyperbolic manifolds and learns an optimal intermediate manifold. The approach is evaluated on taxonomic open-set classification, where it shows consistent and significant improvements over strong baselines.", "questions": "1. The manuscript introduces a new function D to measure the distance between hyperbolic manifolds. Is this function a formal distance metric (i.e., does it satisfy non-negativity, identity of indiscernibles, symmetry, and the triangle inequality)? A brief discussion on this would be helpful.\n\n2. The learnable curvatures are an interesting component. Could the authors provide some analysis on the learned values? For instance, do textual and visual manifolds for semantically similar/distinct data end up with correspondingly similar/distinct curvatures? This could provide valuable insight.\n\n3.  Figure 5: The claim that the method achieves \"improved feature separability\" is hard to verify by visual inspection alone, especially for the \"Order\" and \"Class\" levels. Could it be supplemented with a quantitative clustering metric (e.g., Silhouette Score or Calinski-Harabasz Index) to more formally support this claim?\n\n4. Figure 6: These attention maps are very compelling. However, to demonstrate that the method is responsible for this hierarchical attention, it is essential to show the corresponding attention maps from the baseline. Does the regularization and alignment strategy actually cause the model to shift its attention in this structured way compared to the baseline?\n\n5. The abstract mentions that existing methods represent each image with a \"single feature.\" Models like ViT produce patch-level features. What is the final, single-vector representation (e.g., the [CLS] token) used for alignment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper clearly identifies a key limitation in existing VLMs, asymmetric feature representation, and proposes a well-motivated solution.\n\n2. The use of heterogeneous hyperbolic manifolds with learnable curvatures to capture the intrinsic geometry of each modality is technically sophisticated. The formulation of an intermediate manifold for alignment, supported by proofs of existence and uniqueness, provides a strong theoretical foundation. Even though I do not have enough experience in this line of work to confirm with certainty the correctness of the proofs.\n\n3. The method demonstrates substantial and consistent performance gains over strong baselines across multiple datasets, metrics, and experimental settings (few-shot and generalization).", "weaknesses": "1. The paper’s literature review, while good, overlooks several relevant lines of work. It would be stronger if it discussed the concept of emergent alignment in models, and positioned the proposed \"intermediate manifold\" with respect to related concepts such as:\n    - The idea of a universal or shared latent space, as explored in work on relative representations [1,2]. The proposed \"intermediate manifold\" seems to be a generalization of this concept to heterogeneous hyperbolic spaces.\n    - The general problem of identifiability in representation learning (e.g., [3]), as the proposed work is implicitly trying to learn an identifiable hierarchical structure\n\n\n2.  Some of the visual results used to support the paper's claims are difficult to interpret without further evidence. For instance, the qualitative improvement in clustering in Figure 5 is not immediately obvious, and the attention map visualizations in Figure 6 lack a crucial comparison to the baseline.\n\n3. While the paper includes ablations, further analysis would strengthen the claims. For example, an ablation on the novel manifold distance definition and an analysis of the learned curvatures would provide deeper insight into the method's behavior.\n\n--- \n\n[1] Moschella, et al, \"Relative representations enable zero-shot latent space communication,\" in ICLR, 2023.\n\n[2] Cannistraci, et al, \"From Bricks to Bridges: Product of Invariances to Enhance Latent Space Communication,\" in ICLR, 2024.\n\n[3] Kivva, et al, \"Identifiability of deep generative models without auxiliary information,\" NeuriIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)